why curve fitting doesn't work? | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Optimization, statistics, and avoidance of curve fitting 21 replies

Curve Fitting: The Ultimate Conundrum 17 replies

EA Backtesting and Curve Fitting Concern 2 replies

(curve)-fitting and overfitting, good examples wanted 0 replies

"curve fitting" or just improving?? 5 replies

Attachments: why curve fitting doesn't work?
Exit Attachments

why curve fitting doesn't work?

  • #1
  • First Post: Sep 12, 2007 8:04am Sep 12, 2007 8:04am
during my search for a profitable system I've repeatedly met with a strange phenomenon - a system optimised for a particular period more often than not don't work for the present. I've encountered this often enough to the point that I no longer trust any optimised results of a system. yet I still don't understand why this would happen. how could a working system, when optimised, stop working right away?

when we are given a system, we build our confidence by backtesting it, we want to see how it has worked in the past. we decide whether or not to trade it based on its past performance. optimization gives us the best track record of the system. this performance record, if we were not told that it was optimised, is no different than any other track record to us. yet if we believed in what we saw, it could lead us to financial ruin. what's so wrong with optimisation that made it do this?

no matter how much we optimse a system, it is still based on historical data. and if we could go back in time with this optimised system, we would almost certain make the fantastic profits shown by the system. yet if we trade it in the now and future, last year's 200% profitable system could easily wipe out your account. how could this happen?
  • #2
  • Sep 12, 2007 3:37pm Sep 12, 2007 3:37pm
Each year, the "shape" of the chart looks different. Compare one year to another.... they will be wildly different. You've got to step back and see what's really happening.... a 2007 EURUSD market is not a 2006 EURUSD market. Etc.

If your EA can stand the test of time.... say 6 years of backtesting..... then you may have something. I suppose even 2 years in a row of good results is worth a forward test.

Quoting bluefox
during my search for a profitable system I've repeatedly met with a strange phenomenon - a system optimised for a particular period more often than not don't work for the present. I've encountered this often enough to the point that I no longer trust any optimised results of a system. yet I still don't understand why this would happen. how could a working system, when optimised, stop working right away?

when we are given a system, we build our confidence by backtesting it, we want to see how it has worked in the past. we decide whether or not to trade it based on its past performance. optimization gives us the best track record of the system. this performance record, if we were not told that it was optimised, is no different than any other track record to us. yet if we believed in what we saw, it could lead us to financial ruin. what's so wrong with optimisation that made it do this?

no matter how much we optimse a system, it is still based on historical data. and if we could go back in time with this optimised system, we would almost certain make the fantastic profits shown by the system. yet if we trade it in the now and future, last year's 200% profitable system could easily wipe out your account. how could this happen?
  • #3
  • Sep 12, 2007 4:57pm Sep 12, 2007 4:57pm
Quoting bluefox
during my search for a profitable system I've repeatedly met with a strange phenomenon - a system optimised for a particular period more often than not don't work for the present. I've encountered this often enough to the point that I no longer trust any optimised results of a system. yet I still don't understand why this would happen. how could a working system, when optimised, stop working right away?

when we are given a system, we build our confidence by backtesting it, we want to see how it has worked in the past. we decide whether or not to trade it based on its past performance. optimization gives us the best track record of the system. this performance record, if we were not told that it was optimised, is no different than any other track record to us. yet if we believed in what we saw, it could lead us to financial ruin. what's so wrong with optimisation that made it do this?

no matter how much we optimse a system, it is still based on historical data. and if we could go back in time with this optimised system, we would almost certain make the fantastic profits shown by the system. yet if we trade it in the now and future, last year's 200% profitable system could easily wipe out your account. how could this happen?
Optimization of a trading system is very tricky. It takes a lot of experience and knowledge to be able to optimize a trading system without ruining the system. The price data on which you build your system is mostly random (as far as your system is concerned) except for the inefficiencies that your system is attempting to exploit. When you build your system there are different variables involved like the periods of your indicators SL and TP levels etc. This variables are your degrees of freedom. During the optimization different values for your degrees of freedom are taken and tested to find the best once. The problem is that if you for instance have 2 variables that you want to optimize and each of them have a range of say 1-100 with a 1 step you have 10000 different tests. if out of those tests only a couple are profitable your results are just random. The chances that the market will behave exactly as it did in the past are non-existent. That's why simply optimizing a system and using the values that showed the best profit will never work. You need to build and optimise your system on sample data and do the final testing on out of sample data. You also need to analize your optimization rezults. The best way i found to do that is to use excel. you would import the most relevant info for each test. I usually analize the following for each test:
Net profit;
%Profitable;
Profit factor;
Max. DD;
Return on Account;

Then i take those results and create a pivot table. Next i create a 3D graph of the PT See Picture. Once i have my graph i pick the range or numbers that are profitable and in the same time have a good portion of the chart covered. In this case its the purple which has a range between 10000-20000. Now i go back to my Pivot table and use excel's conditional formatting to only highlight the numbers in that range. Then i pick a number that is in the center of a highlighted area. Now the optimized values that have generated that number are the once i would use for my trading the system. See picture.

You could do the same for other values of the optimization like the DD. Anyway i hope this will help you get started and make certain things a bit more clear.
Attached Image(s) (click to enlarge)
Click to Enlarge Name: 3d.gif Size: 81 KB
Click to Enlarge Name: PT.gif Size: 40 KB
Stubbornly persistent
  • #4
  • Sep 12, 2007 7:30pm Sep 12, 2007 7:30pm
Too many variables. When one word from a guy can move mountains in the market, backtesting will never work because you never know what moved the market during your backtest. Fundamentals don't care about technicals.

I read about the big traders and they all trade on fundamentals. They all trade on the probability of a certain currency being in trouble, etc. All EA's and systems are purely based on technicals which is a fraction of what actually moves the market.

The other problem is widening spreads. Backtesting is very flawed because you never know what the spread was at the time of the candle, especially the large ones.

I am no expert and trade purely as a hobby and these are my humble observations. The more I have studied technicals, the more I am convinced candlesticks are probably the most powerful indicator for short term trading. If the system doesn't include resistance points at round numbers either it probably won't work.
"Its not where your at, but where your coming from"
  • #5
  • Sep 12, 2007 7:51pm Sep 12, 2007 7:51pm
Quoting athalon7

I read about the big traders and they all trade on fundamentals. They all trade on the probability of a certain currency being in trouble, etc. All EA's and systems are purely based on technicals which is a fraction of what actually moves the market.
they have also the source of information that you haven't
  • #6
  • Sep 12, 2007 11:04pm Sep 12, 2007 11:04pm
One of the things that you can use in your systems development is "expectancy".

Expectancy = (averagewin * win %) - (averageloss * loss%)

Your expectancy value on forward based (out of sample) testing should be higher than the value on your in sample data.

If you have a high expectancy, and it continues on forward testing, then that's a pretty reliable system.

Expectancy, in theory, is harder to "curve fit" than the actual curve itself.

Most new systems developers rely on that curve. And that's not what a person should really be focusing on (it's important, but there are other values that are more important.)
Mr. Trend
  • #7
  • Sep 13, 2007 12:37am Sep 13, 2007 12:37am
Hello Mr.Trend, thanks for the reply. the backtest is done on 5 minutes data from December 2002 to December 2006, my EA has generated about 815 trades during this period. I've used price only to determine entry and exit. no complex indicators invloved.

I've calculated the expectancy and it is about 94. according to Dr.Van Tharp, this means that for every one lot trade I make, I could expect a $94 profit. however my system has generated 111 trades since December 2006, and the result is a net drawdown of $8690, instead of a handsome profit.

here I have a system that has been optimised over 4 yrs' history of 5 minutes data. the optimisation result showed many clusters of profitable parameters. I didn't pick the best one, instead I chose what I thought was the most stable one. I checked it trade by trade using chart generated by MT4's visual mode. apart from a few trades that didn't really happen because of weird spread problems. every trade was there. and the missing trades did not make a dent in the system's profits.

yet despite all this, it lost out big this year. thank God I didn't trade it live because deep down I suspected it was too good to be ture. so I run it on simulation instead. but still, the result really took me by surprise. how could it lose so much?! how could it go so wrong?!

this kind of strengthened my belief that there is no such set-and-forget systems. after all, any other performance fields, such as chess or olympic games or nuclear physics, the performers must study and practice so many yrs in order to become competent in their fields, and longer to become experts. how could I expect trading to be any different?

now I'm at a loss as to how to begin my journey. in order to trade I must have a sytem. in order to have confidence I must do backtesting of the system. but how do I know the parameters of the system are not optimised? especially when I know that a system could show large profits with hundreds of trades over many yrs and still not trustworthy? even if I picked the parameters randomly, there is still the possibility that they happen to be among the groups of the optimised parameteres.

as I'm writing this I began to realise that backtesting can not work. the only way to profit from the market is to become immersed in it and constantly adapt to it. there is no fixed system that I could pick up and follow and become rich. I'll have to do it myself.
  • #8
  • Edited 3:26am Sep 13, 2007 12:50am | Edited 3:26am
I am a 100% mechanical set-and-forget trader. 10 years from now I will come back to this thread and say the same thing. Here is some wisdom I've picked up along the way:

http://www.forexfactory.com/showthread.php?t=44686

Aparsai has posted a 100% gain w/ Pipboxer's live account since it opened 3 months ago. This is another nondiscretionary set-and-forget system.

http://www.forexfactory.com/showthread.php?t=13186

Quoting bluefox
as I'm writing this I began to realise that backtesting can not work. the only way to profit from the market is to become immersed in it and constantly adapt to it. there is no fixed system that I could pick up and follow and become rich. I'll have to do it myself.
  • #9
  • Sep 13, 2007 1:27am Sep 13, 2007 1:27am
Curve fitting doesn't really work. Because anyone can pick any number of moving averages, and I guarantee you will find one that works. I know -- I did it when I first started developing.

Backtesting is really just a snapshot of system assurance.

I always recommend that a person backtest the sample (2002-2007), then do random tests of 2002-2004, 2004-2005, 2004-2007, 2006-2007. See how that works.

If that still works well, including expectancy, then start a live test, on a micro account, risking ten cents a pip. Get about 300 trades under your belt, and then re-evaluate. If it still looks good, then I'd trade it.

That's pretty much an outline of how I develop systems. For every twenty tries, I get about two right.
Mr. Trend
  • #10
  • Sep 13, 2007 7:39am Sep 13, 2007 7:39am
bluefox, this is a great post and thread. Thanks for starting it. I have a number of replies to various posters so please bear with me.

First, to my knowledge, no one every said curve fitting does work. Despite that, I see it a lot, especially here on Forex Factory. Someone picks up a copy of MT4, slaps on a couple EMAs, does an optimization run, posts an awesome chart, everyone downloads it, skips the demo account part, and loses money.

Whether you embrace curve fitting or more accurately, optimization or not, you are an unwilling participant. The fact that you selected forex over other markets, you chose a particular system, you chose your indicators, you chose specific parameters, you chose the time frame to trade and, you chose the currency pair(s) to trade are all fine examples of curve fitting. Entire books have been written on backtesting and optimization. I wonder how many of those you studied before reaching your conclusions. You mentioned nuclear physics (by the way, I have a nuclear engineering degree so thanks for the industry mention) and I can tell you that we studied long and hard before taking the controls of a reactor. There's no nuclear forum for people who say it won't work simply because they haven't studied it.

The science of selection, backtesting, out of sample data, evaluation of results, genetic algorithms, and walk forward analysis are all important parts. As I said, entire books have been written on the subject and my posting commnets on each would not do them justice.

Optimization is like playing with matches. If you know what you're doing you can cook your dinner and heat your house. If you don't, you're going to burn your house down. I've stated elsewhere that there is a huge reality gap between automation and results. In that gap lie all the subjects that I mentioned above.

tdion, I disagree with your comment that a strategy should withstand 6 years of backtesting. You did point out that markets change and that's true. So, why would you optimize on a market that is no longer applicable to present day trading? The more data you expose to your strategy the more you have to dumb it down to accomodate those markets. Do you want a mediocre strategy that wanders through the market or do you want it to perfrom well right now? Do racecar owners set up their car for each track or do they build one slow car that will do OK on each track in the circuit?

Sergiu, awesome post! One of the best I've seen this year but probably a bit advanced. Degrees of freedom is absolutely critical. I've read, and agree, that for every degree of freedom you should have at least 30 trades available for testing. Those numbers can add up quickly. The other great part of your post is that the absolute highest peak isn't necessarily the best. If performance drops off rapidly around the peak then it's not a good combination. It's better to find a high, round top where you will continue to profit as the market shifts. One way to accomplish this is by the use of gentic algorithms for testing rather than the brute force optimiation methods that come with charting software. The other way is to actually analyze your results and plot them as you have.

athalon, a variable spread can be a problem but you can set a conservative number in your testing program and do quite well. If your strategy only trades at news releases then yes, a variable spread would be an issue. The other alternative is to use a fixed spread broker. You made some other uneducated comments but to each his own. I'm not here to convince you to autotrade.
  • #11
  • Sep 13, 2007 9:38am Sep 13, 2007 9:38am
Man Phil. No reply for me. ...sniff...

Mr. Trend
  • #12
  • Sep 13, 2007 9:42am Sep 13, 2007 9:42am
That's funny. It's clear from your post that you know what you're talking about. I have had about the same success rate as you: 1 out of 10 systems warrants further study.
  • #13
  • Sep 13, 2007 10:13am Sep 13, 2007 10:13am
Quoting philmcgrew
You made some other uneducated comments but to each his own. I'm not here to convince you to autotrade.
I base my comments on my observations and am always willing to learn. Please enlighten.
"Its not where your at, but where your coming from"
  • #14
  • Sep 14, 2007 12:21am Sep 14, 2007 12:21am
Hello philmcgrew! "Optimization is like playing with matches. If you know what you're doing you can cook your dinner and heat your house. If you don't, you're going to burn your house down." I can't agree with you more. yet I've also found it very difficult to resist optimising a system. I just couldn't help but wondering how well it could have performed in the past. I started this thread out of frustration and the need to find out why. I still don't understand how could a system that produced such a realistic equity curve (attached) over more than a couple of yrs go all wrong. I agree with you that optimisation is a double-edged sword. how to wield such a sword is what I want to find out. sergiu's post give me ideas, now I wish I had more excel lessons when I was at school.
Attached Image (click to enlarge)
Click to Enlarge Name: StrategyTester.gif Size: 6 KB
  • #15
  • Sep 14, 2007 1:32am Sep 14, 2007 1:32am
Well you're question is a good one. The only reason why you forward test a system is to see if the market conditions at the present are different from the market conditions in the past that had an impact on your edge.

You also forward test to see what issues you will experience that will reflect losses on your account, such as untimely server resets, geopolitical and geographical hazards, general slippage (in the case of futures), slippage during extreme volatility, etc.
Mr. Trend
  • #16
  • Sep 14, 2007 3:16am Sep 14, 2007 3:16am
Quoting Mr Trend
Well you're question is a good one. The only reason why you forward test a system is to see if the market conditions at the present are different from the market conditions in the past that had an impact on your edge.

You also forward test to see what issues you will experience that will reflect losses on your account, such as untimely server resets, geopolitical and geographical hazards, general slippage (in the case of futures), slippage during extreme volatility, etc.
I also believe that better conviction can only be obtained by forward testing. however even if a system has been proved by forward testing, there is still the nagging suspicion of "how long will it continue working?". in a sense, forward testing is just like backtesting. by the time the forward testing is done, it becomes history. and if I trade a system based on this, I'm still effectively trading history.

I'm more and more inclined to the belief that successful trading is more about pattern recognition than following a system. no system can stand the test of time, but a method might.

for example, as long as price moves, there will always be congestions and trends. however over time the characteristics of these congestions and trends change. a system can not adapt to this, and by the time I sense sth wrong it's usually too late.

but if I could learn to recognise the congestions and trends with my own eyes, then everytime I apply this ability I also refine my skills at doing this. in other words, if I could use my brain to process the market then I will constantly adapt.

every trade will be important, it teaches me about subtle changes in the market. every trade will be unimportant once it's done, because when the next one comes, I'll be facing a new market.

I think if I could acquire this skill, I won't need any system at all, and I won't fear changes in the market. I don't kow how to approach this dream yet. I'm not even sure it's possible. but I'll certainly explore.
  • #17
  • Edited 4:17am Sep 14, 2007 4:01am | Edited 4:17am
Quote
but if I could learn to recognise the congestions and trends with my own eyes, then everytime I apply this ability I also refine my skills at doing this. in other words, if I could use my brain to process the market then I will constantly adapt.

Good way to look at it. I once read a good observation that the market is a blend of fundamentals and technicals. If it were all technicals the market would move in such a tight range nobody could make money. If it were all fundamentals there would be so much movement that prediction would be virtually impossible.

So the question is are you going to try and predict the market or trade reactionary. Many hedge strategies are based on reactionary trading. Most technical strategies are based on prediction. Are you going to get a "bias" based on the strength of the overall currency against the other or strictly look at lagging technicals? Many good strategies mix both. Look at the EUR/USD over the last 6 years. If you had the bankroll all you had to do was buy on the dips with huge stop losses and you would have made a killing as many did. The overall bias was the strength of the EUR against the dollar. No huge systems or technical discussions. Just buy on the dips.

The other question is mixing your targets and stops with your methods correctly. I sometimes see strategies where these nice "head an shoulder" patterns are drawn on daily charts. A 50 pip target in this case does not fit the analysis and you better be using a large stop.

Debate can take place all day and education level is based on experience. My experience so far is a true mechanical system just will not work. They can aid in analysis and save time, but trading ultimately is based on the dynamics of the market at the exact time you place the trade and not prior. The market is nothing more than people on a small scale and governments and countries on a large scale. These are dynamic entities with free will and by all means unpredicatable.
"Its not where your at, but where your coming from"
  • #18
  • Sep 14, 2007 5:02am Sep 14, 2007 5:02am
Quoting bluefox
Hello philmcgrew! "Optimization is like playing with matches. If you know what you're doing you can cook your dinner and heat your house. If you don't, you're going to burn your house down." I can't agree with you more. yet I've also found it very difficult to resist optimising a system. I just couldn't help but wondering how well it could have performed in the past. I started this thread out of frustration and the need to find out why. I still don't understand how could a system that produced such a realistic equity curve (attached) over more than a couple of yrs go all wrong. I agree with you that optimisation is a double-edged sword. how to wield such a sword is what I want to find out. sergiu's post give me ideas, now I wish I had more excel lessons when I was at school.
The test you have posted doesn't mean anything (openprice only)
  • #19
  • Sep 14, 2007 6:12am Sep 14, 2007 6:12am
Quoting melpheos
The test you have posted doesn't mean anything (openprice only)
oh, why is that? I've designed my EA to make decisions only on completed bars, and my entry is always a stop order well above or below market price. that's why I could use open prices only to backtest. as far as I know, this is the most reliable way to do backtesting. of course I coule be wrong about this.
  • #20
  • Sep 14, 2007 7:18am Sep 14, 2007 7:18am
Quoting bluefox
I also believe that better conviction can only be obtained by forward testing. however even if a system has been proved by forward testing, there is still the nagging suspicion of "how long will it continue working?". in a sense, forward testing is just like backtesting. by the time the forward testing is done, it becomes history. and if I trade a system based on this, I'm still effectively trading history.
Good point.

I have a H4 strategy that i have backtested and optimised over the last 6 months (approximately 100 trades on average) on various pairs. The results are good once i have optimised the 2 indicator inputs and the maximum SL size.

When i backtest it on the previous 6 months data, these "optimum" settings can then make an overall loss.

But then markets evolve and is what happened 12 months ago still relevant on H4 charts?

My plan is to backtest and optimise the past 6 months H4 data, trade these inputs live for the following month, and then backtest/optimise & possibly change the inputs at the end of each month. How does this plan sound?

Cheers.
0 traders viewing now