Thoughts on backtesting and optimizing

I have recently started backtesting and optimizing various strategies and to my biggest surprise they all produced smooth and upward sloping equity curve after having optimized the variables for the different indicators. First I thought I was super smart because what ever I came up with was profitable but then – luckily before I would have gone live – I came across Merlin’s article “The System Behind The System” see abstract: http://www.forexfactory.com/forexfor...ead.php?t=4446 – thanks mate for dashing my hopes for putting my arse into an Enzo within 3 months. Anyhow, Merlin suggests in his article that when we optimize strategies we should avoid “optimizing the past”. This is what he suggested on how to divide up the historical price series in order to testing of the robustness of a strategy. “How a trader divides their data is more art than science. General guidelines I use are 5% Build Data, 40% Test Data, 40% Walkforward Data, and 15% Unseen Data. I also like to spread a specific symbol (i.e. EURUSD) among at least three data categories.” <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o></o>

<o> </o>

After having changed my approach according to Merlin’s suggestion it turned out that most of my earlier strategies were just ok and it’s no point trading them. And there is a question which I’d like to discuss here and it would be great if we could get some consensus on the issue as it - I believe - seems to be overlooked by many when it comes to backtesting. So, what is the proper selection of the time frame we should pick to backtest a system within? As far as I am concerned the price of a currency pair is moving in a very different fashion from time to time based on the particular market cycle it is traveling through. I, in the future, try to use fundamental analysis to understand how market cycles are changing but when it comes to interpreting history I am not sure how one can assess when the particular market cycle changed and for what interval. I am trying historical volatility, for example to detect some changes in the way the nature of prices have been changing but I am not sure if that’s the right way. To make it short, my question is what tells me the right time frame within I should run my historical price data series when backtesting the robustness of a strategy. I tend not to go with the general suggestion according to which I should backtest a strategy for as long as it is possible back in time because if the strategy has performed during all possible past market conditions it should be doing so in the future too. <o></o>

<o> </o>

Thanks guys, hope I hear back from you on this. <o></o>

im glad to see my article is actually getting some use

hazi, fist let me say you seem to have a very good handle on this backtesting game. many fail to even make it to the point you are at now, where you understand that a process must be used or otherwise you will be fooled.

as for your questions, they are very deep and hard to answer. i dont think there is any fact to give you, just my opinion (btw, you will find that mechanical trading is VERY much discretionary because of all the decisions you have to make when building the system).

as for what type of market to backtest on, it is best, of course, if your system works on any kind of market. and that, of course, is the hardest to find system: much more common is the system that works during certain types of markets, like trending or non-trending, high volatility vs. low volatility. for determine trends, i like to use the ADX. you can use it in all time frames and it generally does a good job of telling you what kind of market you are in. for volatility, you can use stddev volatility as you mentioned. these two tool can be used in combination to tell you what kind of market you are in. you should use the tools as blunt objects rather than surgical knives if you know what i mean.

once you know what kind of market your strategy works on, i see no reason you cant cherrypick walkfoward data that matches your build and test data. as i said over and over in my article, you have to be aware of hindsight, and if you become aware of how hindsight creeps into your system design, you can make certain moves without contaminating your system.

good luck with your development and feel free to ask any further questions...

Great article and post, Merlin!
I'm really impressed with your web site...these forums must be the only forums where flame wars never start. I come back here everyday.

I hope I'm not asking a rather stupid question here but is there any mq4 indicators that show stddev volatility, or I should just figure that out manually?

Thanks for your response Merlin. I agree with you on ADX, I also use it a lot, it’s a great indicator. Would you be able to let <?xml:namespace prefix = st1 ns = "urn:schemas-microsoft-com:office:smarttags" /><st1ersonName>me</st1ersonName>/us know how many data points (just approximately, of course) do you usually use for the procedure and was the ti<st1ersonName>me</st1ersonName> period that you picked in your article arbitrary or there was a specific reason for using it? I am asking this because in the article you used two different ti<st1ersonName>me</st1ersonName> periods for the EURUSD and AUDUSD pairs, for example. Thanks for your ti<st1ersonName>me</st1ersonName>, mate. <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o></o>

Merlin, how long do you usually keep one strategy running "unchecked"? I mean how do you know when your current strategy loses its strength/profitability? Of course if it stops delivering the stats you picked it based on, that can be a sign, obviously but do you rerun the tests with walk forward and unseen data from time to time as well? Thanks mate.

when you have backtested your strategy you already now the maximum drawdown, consecutive losses etc.. when the real numbers differ from the backtested result then you can throw it out.

if you want to see some sort of backtesting reliability just compare yout forward test results with backtested result for the same period.

and when you do optimization stuff you need to test it with out of sample data. for example if you have optimized it for 2004.07.01 - 2005.07.01 then test it on 2005.07.01 - 2006.07.01 and if the results match then you have not curve fitted your system and it shows some promise.

Fair enough too and thanks for your thoughts Piccolo. Is "forward testing" the same as testing the model with "unseen data" i.e. the approximate last 15% of the price series? Thanks.