Walk Forward Analysis - the process of testing

One thing I am struggling to understand about Walk Forward Analysis is how to select the parameters to use, and possibly when to change them?

So if you take the stepped WFA approach where the in sample (IS) data is, for example, a 3mth period and the out of sample (OOS) is 1mth.... (my strategy is based on an hour tf)

I am doing this in MT4 manually, based on this approach… http://articles.mql4.com/1012

Optimization/IS WFA/OOS
June to Sep ----------> Oct
Jul to Oct -------------->Nov
AUg to Nov ------------>Dec
Sep to Dec ------------> Jan
Oct to Jan ------------->Feb

So if i take the best passes from each optimization - let's say there are 150 for each. I run these through the respective OOS. So this may then produce 20 passes/results that are profitable.
Do I then take these 20 passes for each OOS data, Oct, Nov, Dec, Jan, Feb, combine them = 100 passes and then see if there are passes with similar settings?
Then hopefully take the settings that match closest across all these OOS sets - is that how it should work?
(so in excel I create a key based on the 4 parameters, and then put this in a pivot table to see if there are matches across all the sets)

So then the final settings would to a degree be based on data that is far from the current date. I get that this hopefully makes it more robust, but possibly less optimal? (So if I take those setting that match closest, and run them over the whole period June to Feb, the results will not be that good, as this window is larger than the original IS optimization windows which is only 3 months.
So if this is the correct approach, do you then run with these settings live over March and then repeat and have different settings for April and so on?

Or is that just for testing the robustness, and for the actual parameter selection it would be better off just running a single WFA
eg
Optimization/IS WFA/OOS
June to Sep ----------> Oct

and then run live with the best parameter set from the Oct OOS test for a month and then repeat?

How are others managing their WFA testing?


Thanks

Simon

You should always trade your strategy in the same way you tested it.

So if you always optimised your parametersets on the last 3 months of data within your WFA, you should do the exact same thing when you want to trade it live.
So today, you would want to optimise on Dez, Jan and Feb, and then trade your strategy until 1.4.2014, and then you would re-optimise your parameterset on Jan, Feb, Mar and so on.

Quote

Disliked

So if i take the best passes from each optimization - let's say there are 150 for each. I run these through the respective OOS. So this may then produce 20 passes/results that are profitable.

I am not sure if I understand you correctly, but are you first choosing the best parameters from optimisation and then you pick the parameters that were also good in OOS?
Thats very dangerous, dont do this! Because when you want to trade your strategy live, you wont have the possibility to incorporate OOS data into your decision, because well, you can not look into the future. So what you do is making your OOS date In-Sample actually.

You may only use In-Sample Data within your parameterset decision routines, because that way you can simulate how these decision routines would have behaved in the past, and hopefully get a compareable result when you trade it live.


Hope I could help.

-Darwin

PS: for reference, WFA is not a solid and trustworthy analysis approach: http://www.forexfactory.com/showthread.php?p=7312800

Thanks Darwin,

Yes essentially that is what i have been doing


the reason for this is that i have read in several articles that you combine the OOS sets and take the best parameter sets from these.
for example, taken from wikipedia... http://en.wikipedia.org/wiki/Walk_forward_optimization


so i have essentially been taking the best results from each OOS set for Oct, Nov, Dec, Jan and Feb - put these in one table and then find the best parameter set. My criteria for choosing the best parameter set might not actually be the the most profitable set, but ideally where the parameter set is profitable in the most months, with greater leaning to profitability in the later months. (I don't have a formula for this, just manual observation)

I can also see your point though, that to a degree this makes the OOS data effectively IS, but then it also does test the robustness of the parameter set at the same time.

Now with regard to the re-optimisation period, I have found some interesting statistics whilst testing on my EA.

I wanted to see the results past the 1 month OOS period.

So for example I run the optimisation for
June to Sep

I then also run the best parameter sets over OOS data for Oct, Nov, Dec, Jan, Feb individually, so that I could try to see when the parameters sets might degrade over time, and give some indication of how soon they needed to be reoptimised.

What i noticed is that the first month of the OOS data, is never the most profitable. It is usually the 2nd and third month that are better.
I have repeated this on different pairs and different optimisation periods and found it to the same.

Therefore that would suggest that possibly i would need to run the parameter set live for 3 months and then reoptimise? So that then suggests a 3 month optimisation period followed by a 3 month live period potentially.

Have you witnessed anything similar?

That means that you line up all the OOS results to one equity/test-report and then view the performance of the system based on this combined result.





There is no real reason why this approach should not work, and also I can not provide any empirical experiments or something. But well, imo its the best approach if you trade your systems in the exact same ways you tested them, because only then you can see your test results as somehow predictive.
So if you want to stick with this method, test it! That means, in your first IS test, just take the best parameterset for OOS.
In your second IS test, combine the IS test 1+2 and take the best parameterset from them for OOS.
For 3rd IS test, choose from IS test 1+2+3 etc, just in the way you want to use for forward trading ("put these in one table and then find the best parameter set.")






I did, yes.

But from theroy, the smaller your IS timespan, the better. So in my oppinion the characteristics described by you are random fluctuations, and should vanish if you analyse your system on a huger dataset.

Tough, I am not sure about this, perhaps some systems behave better with higher oos timespan?!
One should dig into this


-Darwin

i thought it might mean that after i posted this. But then if i did follow that, then my results wouldn't be as good - as i mentioned i have been finding the 1st month OOS test isn't the most profitable.

I will experiment using different combinations of IS and OOS timeframe lengths, and see if i can improve this.

sorry i didn't quite follow this. - would it not be the same as just increasing the IS length, as you also suggested?

Yes, you should experiment with differnet combinations, but keep in mind: from a theoretical point of view, OOS should be as small as possible. So perhaps you look at the randomness of markets if you see that the first oos month is not the best.


What you could do in order to get more samples, to reduce randomness in outcomes, take a smaller roll forward window.
So, do IS on 1.1.2012-1.1.2013, OOS on 1.1.2013-1.2.2013, then next WF window do 7.1.2012-7.1.2013 and OOS accordingly (for example! I used 1 year IS just because its easier to read in dates)



No, because you would do independent IS tests and then combine their OOS tests. Thats most likely not the same as doing a huge IS test with 1 OOS test


However, its not about me thinking this is a better approach (I do not know if it is..), but if you want to do this in live trading, then test your strategy according to the same rules and methods.


-Darwin

As a side question: Do you guys do backtesting with whole of the data and then do walk forward analysis (wfa) ? Or just skip the backtest, with the logic that WFA will filter bad strategies better than backtesting and optimizing?