Do you not use data that is X years old when backtesting/constructing a system?

I'm going through another build process and I'm wondering if I should be considering using data from 2006 let alone even older. I feel like the market is entirely different now.

Does anyone else feel this? Should I perhaps only be using data that is maybe 2 or 3 years old?

I am building a system that trades daily bars. I don't think I would expect someone trading a 1 minute chart to care about what happened in 2006. There are enough bars of data in one year to probably build a system off the 1 minute.

Is that the question then? How many bars are relevant to construct something to work now?

It really depends on what you are trying to achieve, if for instance you are looking at a seasonal effect, you probably want to go back as far as possible. If the system is more short term and perhaps volatility dependent then back-testing over different time intervals may give you an idea of how you system functions in different market regimes. If on the other hand your system depends on something which has only recently entered the market, then going back to 2006 may make no sense.

There is no right answer, it depends on what you are trying to capture with your model.

I'm building a simple quantitative breakout model built on daily candles meant to capture the daily trend based off the long term trend. ie - Is open price above a specific MA? Then only look for a breakout going long and then base the TP/SL/trigger off of probabilities based on analysis of previous candles and ATR.

How far back would you look in that instance? It's very easy for me to analyze all years of data, but I'm wondering if I'm shooting the system in the foot by thinking data from 2006 or older is relevant anymore.

If you are going to optimize parameters (like MA length) you need to partition your data so you can optimize on one data set then test these parameters on unseen data. I have never had any success with parameter optimization, with even 2 or 3 degrees of freedom in a bad model, it will fall apart on unseen data.

No. The MA length or period isn't really the key factor in the system. The key factors are how far from the open do I place the trigger to avoid false signals. Then from the trigger what are my TP and SL? It's a balance of putting the trigger far enough away to avoid taking a long trade when it's really likely going to be a down day. Then, what do I make the TP/SL such that it's not trading in the noise, but still not large enough that I have unrealistic goals.

For example, on eur/usd I wouldn't place a trigger of 60 pips and then have a tp/sl of 200/200 for a single day bar. There are days that the movement would complete it, but the ATR of eur/usd is only 130-140 right now. Most of the days would end in an uncompleted trade.

The more specific question in my instance is what periods should I backtest and build the system depending on ATR? Please see the image.

http://i.imgur.com/GNvJd.jpg

That is a daily chart of eur/usd. Do I want to use data from Aug 2008 - Aug 2009? To me that seemed like an extreme period of trading as things were getting shook up in the market. Thus, perhaps it's data that would negatively affect the true conditions of the market as they are now. It seems things have settled somewhat since then.

Conversely do I want to use data from before Aug 2008? The ATR of all periods before then had never reached where they are now. The ATR has settled into a steadily increasing uptrend, but I would probably only want to concern myself with the most recent characteristics. I worry that this is curve fitting.

What do you think?

I would probably start by concentrating on the period you have highlighted.
But I would still check out what happens in the other periods.
Google 'walk forward testing',it's what you need to do, and it's a pain in the arse in MT4.

Yea I figured that is what I was going to do going forward. I was hoping to better understand how large of a period I need to consider with each walk forward test. Perhaps one year is good enough? Effectively several hundred bars of data still. Perform the walk forward test once a month and adjust accordingly.

Doing it in MT4 is a pain in the ass. Doing it in excel, at least for this system, isn't too difficult.

Why not do it like this: optimize using 2006-2007 and see how your system would perform in 2008. Keep rolling window stats on your system, and see when they start to diverge from expected averages. The trick is to stop trading and re-optimize before they diverge so much that you start losing.

You should also do at least a simple cross-validation like 10 fold (http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29). You split the data in 10 and optimize 10 times on all but a chunk, then test on that chunk. The systems stats should not change a lot. If the stats are all over the place on each different test, you are curve-fitting.

You should not aim for a particular testing period, but on a target number of trades. Testing on 10 year data, but having only 20 trades in the whole period likely won't work very well in the future. But if you do 100 trades a day, one week of data is enough. If you couple it with rolling system stats like I said above.

I think it begs the question of when curve fitting is good and bad then. It's one thing to build a system, curve fit it over the data from the last year and then expect it to run perfectly for the next decade. It's a completely different thing to build a system, curve fit it over the data from the last year and then readdress the parameters every month to insure the system is adjusting to the fluctuating market behavior.

At least in my systems instance the parameters I would set in 2006 would have been completely different then what I should have set in 2010. The ATR between the years is completely different. The principle of how the system is traded is still the same, but it's difficult to backtest the two periods equally. I feel using the K-fold method is just as inadaquate.

I hope this begins to highlight why I'm having such difficulty really laying down the framework of the system.