Why "Walk Forward Analysis" is still unreliable and useless!

.
.
.
ATTENTION: THIS THREAD IS OUTDATED
Please go to http://datfra-expert-advisor.com/ in order to find the final result of all the research!


























Hello, I am Darwin and today I want to talk about the limitations that walk forward analysis suffers from. This is my 3rd article, so if you do not know how WFA works, please read the other 2 (you can find them here on the forum).

The target audience of this article is everybody that deals with ExpertAdvisors and Backtesting / Walk Forward Analysis

Some of you might already have seen a few posts of me where I talk about some research I do in the fields of Trading System Analysis (in the course of writing a meta-algorithm that can build, analyse and trade strategies on its own). The goal is to write an algortithm that is so powerful that it can take every EA and, due to in-depth analysis, tell you how and when to trade it in order to make profit, no matter how good or bad the underlying EA is

So here is a new article in which I would like to lay down some insights that I could get in the process of writing this algorithm (DATFRA - Darwins Algorithmic Trading Framework)


Well, lets begin. My first concern is that the design of Walk Forward Analysis is, in its nature, unrewarding and not the kind of analysis a trader wants.

Also, I claim that the results of a WFA are more or less random, and if a system works well after a successful WFA, then not because the test was successful, but because the trader designing the system did a good job.

In this article I do not yet want to show how this problems can be solved, I just want to demonstrate that they exist. In my next article I will explain how I think this all can be solved in an elegant way.



The fundamental design problem

Walk Forward Analysis is designed to evaluate a trading construct you give to it.
This construct consists of:

* Trading System (eg an Expert Advisor)
* Market/Timeframe (eg "EURUSD / H4")
* System's Parameter ranges (eg "Moving Average Period from 50-150")
* Optimisation (In Sample) Timespan (eg "Optimise on 2 years of data")
* Forward Trading (Out of Sample) Timespan (eg "Forward Trade for one month")
* Preferred characteristic (eg "forward trade the candidate with highest profit").

So all this has to be pre-determined by the trader, out of intuition, and not based on true facts and data. But god, these are the most important decisions, how should one "guess" them?!

And then, WFA will only be able to tell you if this construct would have worked in the past or not, thats it.

So in order to find the best trading construct, you have to use trial&error and repeat WFA step multiple times. This would then, step by step, even lead to the worst case, your "unseen" out-of-sample tests would slowly become "known" in-sample data and the whole advantage of WFA over backtesting would fade away completely.


This design related problems are already showing that WFA can not be the end of the road in terms of system analysis.

In a perfect world, you should give the analysis algorithms only the trading system and the market/timeframe, no other parameters. And then, the algorithm should tell you the best choices for all the other parts of the trading construct, based on data and facts, not the other way round.

Side Note: it should NOT just tell you how to trade your systems, it should give you the possibilities to look into the system's characteristics on your own. You should never be forced to trust any algorithms without the possibilities to check it's findings!

This is very, very important. It is not very much of value to evaluate a single trading construct, but it is a gamechanger if you can look into your strategies in a way that would allow you to just "see" how they work and what trading construct will work best (More on this in my next article)



Even worse: Unreliable results because of lacking data

Ok, so even if a trader could come up with a good trading construct out of intuition/knowledge, WFA would still be a more or less random thing. But first, let's make a rough calculation:


An example trading system and a small estimation of its parameterspace-size

So, a system that enters trades based on a Moving Average Crossover and RSI Indicator, and exits them using a different Moving Average Crossover has at least 5 Parameters (2x2 for MA-Periods + RSI Threshold). It's 6 if you take into account the StopLoss.

Let's say the "fast" Moving Average Periods can be 10-50 and the "slow" ones 50-250, the RSI threshold can be 1-100 and the StopLoss 50-150 pips (this is no real system, just an example!)

So this system can already be traded in 40*200*40*200*100*100 different ways. That is 640 billion (640.000.000.000), which is quite a huge number.

One might question my exact example strategy, but can not question the millions or billions of possible parameter combinations, even for small systems.

But thankfully, if we take into account that a lot of these parameter-combinations would behave very similar, we do not need to evaluate them all, but we need at least a meaningfull sample of it, like a few hundred thousand or a few million.

So, keep this huge amount in mind, even for small systems, because with every new dimension for our optimisation problem's solution space (every new parameter) the amount of possible parameter-combinations grows exponentially.



Walk Forward Analysis - missed data during optimisation

Ok, now lets look at the first step of WFA, and the first problem: Missed data because of inefficient algorithm design and computing time concerns.

During optimisation step of WFA, the algorithm should, in a perfect world, evaluate all 640 billion combinations in order to determine which of them work best. Of course this is not possible, but a "meaningfull" sample (let's say 500.000) would be feasible and _needed_ if we want to look at the "real" picture.

The problem is, due to limitations of WFA algorithms, optimisation has to be done in every single Walk Forward Window.

Let's say we do a WFA on 10 years of data and our Forward Trading Timespan is 2 weeks: That makes 240 Walk Forward Windows. That means 500.000 tested parameter combinations per window would need 120.000.000 single simulations.

And then, remember that WFA relies on a trial&error principle, so you will most likely have to do this a few times.


You see? Evaluating the "real" picture would take very, very long, and therefore most WFA implementations are forced to only evaluate an very much cropped fraction of the actual parameterspace because it is not possible to evaluate the whole parameterspace (or a meaningfull sample of it) in a reasonably small timespan, because optimisation has to be done in every single WF-Window.

This means, WFA most likely does not evaluate 500.000 parameter combinations per window but only 10.000 or 50.000 or something like that. So eventually we already lose like 90% of all data in this step.


This is a problem that could be solved if the trader has lots of time for his/her analysis (which is not likely, especially based on the trial&error method), or with a more efficient design of these algorithms. Nevertheless, in praxis, this problem is ever-present.

For comparison: DATFRA, which is my private research project, only has to do one single simulation per parameter-combination, no matter how many WF-Windows it analyses. In the above example, that would already decrease the computing time by the factor 240.


Parenthesis: What kind of data do we look at when analysing trading systems, what is a "datapoint"

I will talk about "datapoints" and "data" quite frequently in this article and in my posts, so here is an explanation.

When analysing systems, it is always about a trinity of informations. Remember how WFA works:

http://s7.directupload.net/images/140225/qws9eczi.png

So a datapoint, of which 1 is generated per Walk Forward window, consists of:
* The performance in the RED optimisation window
* The performance in the GREEN forward trading window
* The used parameter-combination for this specific test

So, in our example, a WFA would generate 240 of them, whereas 120million (500k * 240) would be possible for our example system. That should already give you headache.



Walk Forward Analysis - tons of missed data during forward trading

Ok, now lets look at the second step of WFA, and the second problem: Missed data because of _wrong_ algorithm design and computing time concerns.

Now remember, a meaningfull sample of our trading system's parameterspace would be 500.000, and we have 240 WF-Windows. That would make a total of 120.000.000 optimisation-candidates. And out of this huge amount, a WFA algorithm takes the very best per window, 240 in this example.

That is 0,0002% of the total amount of all datapoints that we could use to describe/analyse this system and it's ability to produce good forward trading results, based on good optimisation results.

And then WFA takes these few datapoints and claims it gives a somehow realistic view on a trading system's performance / robustness.

Thats nonsense! You also would not judge a picture's colour by looking at 1 pixel, would you?



A word about fluctuations and why the "very best" parameter combination is not meaningfull

You could argue that it is not important if we forward trade all 500.000 candidates per window, because we are only interested in the top performers, as they are the ones we trade in realtiy.

Well this argument would _only_ works if:

* We would ignore the ~90% of data lost in the optimisation step
* The very best candidates would be meaningfull, which means that all candidates that are following (like the next 10 or 20 or 50, which is not much compared to 500.000) would behave in quite the same way.

But reality is different, the performance of the top candidates per window fluctuates quite much and taking the "very best" therefore leads to more or less random outcomes.



Experiment 1

Here are some examples, I plotted the forward trading performance of the best (left) and the next 4 candidates of some random strategies I created and evaluated with DATFRA. Most of the analysed WF Windows looked like these:

http://s1.directupload.net/images/140225/f9gi7dj9.png

These were just a few examples to illustrate my point of view, I could show hundreds or thousands of them.

So, for the real picture, you would AT LEAST need to evaluate a few hundred of the top candidates, not just one, as it does not show the "real" picture. It's performance is more or less random!

A perfect analysis algorithm would evaluate every single candidate that made at least 1$ profit during optimisation. That would give the real picture and most likely 1000 or 10.000 as many datapoints than what a WFA gives.



Experiment 2

Here are some more examples, this time I plotted the overall WFE (red) and the WFE of single windows (green) of some random strategies I created and evaluated with DATFRA.

WFE (Walk Forward Efficiency) is a measurement that compares in-sample and out-of-sample performance and is used as THE statistic about system robustness in WFA (google for it if you want to know more about it)

http://s1.directupload.net/images/140225/j4tugxf9.png

This clearly shows the flucutating nature of the results a WFA generates, and that the end result is not really telling much about your expected live trading performance.

Btw: To keep the plot scale in limits I did map all points > 2.5 to 2.5 and all points < -2.5 to -2.5, so reallity is even a lot worse. That is also the reason why the second image in the second row does not look "right"



A word about feasibility

Please do not think I only talk about grey theory here "as it is not possible to do this kind of simulations in a short enough amount of time anyway".

If the algorithm is designed well, one would not need a single further simulation in order to determine forward trading profit and not a new optimisation procedure for each WF-window.

So for the used example, DATFRA can generate 34.000.000 "Optimisation=>Forward Trade datapoints" in ~24 hours and on a mid-end PC (8GB Ram, quadcore 3GHZ).

Still not 120millions, sure, but compared to 240, I think its a very good result.

So it IS feasible to analyse a system with such a level of insight, even on today's hardware.




Some afterthoughts

To everyone claiming that backtesting strategies does not work: Well, in its current form it does not, but if you look at enough data, it does, and it can aid a trader in taking funded decisions.

To everyone using backtests/WFA: It does not work that way, you can never rely on your analysis results, and if your EAs/Trading Strategies make profit, then not because of the good tests, but because you did a very good job designing them!

In about 1-2 weeks I will post my next article, in which I will explain how an ultimative state-of-the-art system analysis algorithm works and what can be done with it. You will be stunned, promised!


"Are you just trying to sell stuff?"

People keep asking me this whenever I post stuff.

No, I post this because I want to discuss my concepts and thoughts with other advanced traders. The side benefit is the educational effect for everyone that is willing to learn more about algotrading.

And yes, I am developing an algorithm that is based on the concepts that I explain in my articles (especially the next one) and that is able to solve the issues discussed here. Well, basically I have already developed it, it's in first alpha version at the moment and works great.

But I am developing this for my private usage, so No, I am not here trying to sell you stuff, as most of the people reading this will not have the chance to purcase it.

It will only be sold to a few people, just enough so that I can fund my own trading accounts (I am young and therefore need the money ).

Most likely I will limit the amount of copies sold or only sell to expert traders or only to companys or charge enough money so most ppl won't want it or sell copies in silent auctions or ..... Well, I do not know yet how it will work out, I can just say that I will keep it private to a small circle of happy few, so do not read this article with the bias of "this guy just wants to sell me stuff", thanks.


-Darwin


PS: As always, just add me on Skype if you want to discuss further and/or want to have more informations about DATFRA: Darwin-FX is my SkypeID.

It actually didn't sound like a sales pitch until you added the section at the end "Are you just trying to sell stuff" and started musing about selling it.

It sounds like you're finally back to testing over the entire dataset and more in line with the ideas presented in Systematic Trading which from my experience are sound ideas. I think we all "try out" WFA at one time or other but breaking up is hard to do. Isn't what you describe essentially testing lots of parameter sets and then analyzing the data over a long period to make a more consistent selection?

WFA builds in instability by the constant parameter switching. It is difficult to get consistent results from an inconsistent process. In the process of chasing the past return, often results turn out worse, in many cases much worse than sticking with a single parameter set that has ups and downs over time but gradually recovers from the downs. WFA essentially guarantees that you will switch parameters after a down period, or when another parameter set has a good period. Good periods tend to be followed by bad and vice versa. None of these systems go straight up, yet that is what WFA implicitly assumes - that after a switch a system that has had a short-term upswing in equity will continue with the same upswing indefinitely. This assumption is unreasonable.

I know that I could have hidden this, but I do not like people with a hidden agenda, people that do not play with open cards. So why should I become one of them?
Also it does not make my arguments any less valid.

No, I am not back to testing over the entire dataset, quite the opposite. I went one step further!
Yes, I test lots of parameter sets, but I do not analyse their performance as a whole, but split them in "past" and "relative future" parts, just like WFA does.

I tackle the constant parameter switching in a simple way: I forward trade all of the parameter sets. This has the benefit of removing instability and it gives you a broader picture of your system than any normal analysis method, while still maintaining the good thing about WFA: It analyses system performance performance in relative future in relation to the past.
That way I can keep the benefits of WFA and tackle its weakness, and in the end I get a few million Past->Future Measurements (whereas WFA gets one of them per WF-Window).

And using this method I get enough informations to know when a system will (future!) work good or not, which is never possible with common systematic trading approaches, as they say nothing about past<->future relationships

Doing some first experiments with AI techniques based on this data (this is still in an alpha state) I am even confident that I can get a very straight curves up (on the cost of profit, because then you can only trade a strategy when it is very "save"). So I am able to predict when it will go down or up to a degree

You can not do all this with common analysis methods, so I will never "go back" to them.


I will explain that in my next article, this one is actually only about weakness of WFA. But I do not want to argue that normal testing over the whole dataset is better, it is a lot worse. If your systems work that way, only because you did a good job, because they are very, very unreliable.

-Darwin

I will say that what Darwin is proposing should make one more confident in trading a system. Nothing is fool proof, but this approach is the best I have seen so far compared to other approaches. It's difficult to debate otherwise. If any commercial vendor dared to apply this to its system, I suspect the results would be humbling or at least cause a change in parameters. One potential issue is that the approach lends itself well to systems that trade on bar open on higher timeframes, but that still leave limitless options. I would like to see more analyses of other systems to see if the analysis proves beneficial.

Thanks nonlinear.

Yes, sadly I can not apply the approach to systems that do not only trade at bar opening, as it would take too long to generate the data needed

Also, I would be happy to debate if someone thinks backtesting (wink to FXEZ) or WFA is somehow reliable, so bring it on

-Darwin

Looking forward to reading your next article. I wonder how you manage the combinatorial explosion of the parameters space. My first guess would be to use a variable density grid based on some curvature metrics. Additionally it would hint about the relevance of a parameter: where the grid is the less dense the parameter may be useless and where the grid is extremely dense the parameter is probably overfitted.

I am not sure what you mean with that, but I do not do anything to crop the parameterspace


I am able to evaluate it as a whole (or at least a meaningfull sample of it). I can evaluate 150.000 parameter-combinations, which makes like 35million Past->Future "Walk forward window"-like datapoints, in 24 hours.


-Darwin

As usual a picture is better...

150.000 is only ~53^3. That's not so much.
Say I have a pair of MAs for the direction (20-50 and 25-200). A pair of faster MAs for entry/exit (2-10, 5-20) --in the direction of the trend--. The ATR (5-50) and a multiplier (0.5-3.0 step of 0.5) for the SL. I already get 31*176*9*16*46*6 = 216.843.264 sample points in the parameter space per window. And it still is a toy example! Can you keep them all?

I really appreciate your example image, as always I did talk to a friend that works as mathematician about the mentioned topic, and I might be able to deliver such a grid based solution for more effective sampling of the solution space in further versions.

Tough, sadly, in alpha I can not, it only uses random sampling, so the question is what is a meaningful sample of such a huge parameterspace and how to spot it.

The current method I use is simple: I collect data for some time, and then see if the data forms clear trends or if it is still noisy
If it still looks noisy/random, I continue datamining until it shows clear trends. For example (left = not enough, right = enough):
http://s14.directupload.net/images/140226/4ohohvj3.png

The second thing to mention: As I do not analyse absolute performance of parametersets, but performance of them in regards to their Past<->Future-relationship, I can generate millions of datapoints out of these 150.000 initial measurements, which helps to get a more robust view with smaller amount of tested parametersets.
The above image, for example, was generated from the same initial measurements/dataset, but on the right side I did extract more past<->future datapoints.

Last thing, tough I did not verify its correctness in empirical tests: I think, if you analyse the above mentioned problem looking at 1 or 2 parameters at once, you do not need to evaluate the whole parameterset. For example, you could analyse ATR and multiplier at once, in relationship to forward trading profit.
Or two paired MAs (fast and slow), in relation to forward trading profit.
You will soon have a meaningfull sample of the 46*6 possibilities for your stoploss parameters, for example.



So what I want to say is that the actual amount of measurements you have to do is a lot smaller than the absolute size of your search space.
Nevertheless, of course, my above statement of analysing the whole parameterspace was wrong.

Thoughts?


-Darwin

Regarding optimization, perhaps this course will shed some light on better ways to do it (coming up in 5 days):

https://www.coursera.org/course/optimization

@Darwin Perhaps you can explain the difference between the following two cases:

Let's say in both cases 10,000 parameter combinations are being run.

Method 1 - Darwin's method runs 10,000 combinations by Walk forward window and keeps track of the stats on each smaller segment, and sums them up to the full period tested for final analysis.

Method 2 - Runs 10,000 combinations over the entire training period and keeps track of the stats. Further analysis is made by splitting up the equity curves and evaluating performance on smaller segments.

Maybe I'm missing something, but don't they both of these methods get to the same place?

I've found that you don't necessarily need to test every combination of parameters to get a good picture of what is going on. Also, proper parameter discretization can help cut down on the actual number of parameters that need to be tested.

For instance, if you are testing MA lengths from 5 to 500, you might select 5-49 by ones, 50- 150 by fives and above 150 by tens or higher. When you select large number parameters by ones you increase the chance of overfitting. It isn't so much of a problem for discrete numbers as it is for continuous numbers. For continuous (real) numbers make sure to use rounding to a significant digit for parameter ranges. For example in picking a standard deviation threshold, you might allow 2 decimal places so optimizing to 2.15 deviations makes sense, but 2.1502343343430343 doesn't. Otherwise your parameter space explodes with real numbers that prevent your optimizer from finding unique parameters!

Why do you say that the "approach lends itself well to systems that trade on bar open on higher timeframes"? Can you give an example so I can understand what you mean?

Darwin's new DAFTRA is set up to conduct a massive amount of WFA and then crunch those results. If you tried to run this type of analysis on an EA that only works on tick data, you might be sitting in front of your relatively "fast" computer for a few months or more waiting for the result. Its running brute computations in order, over and over, likely on one maxed out CPU. Running this type of test is only realistic on most accessible computers when you test on bar open on higher time frames. If you test on bar open M1, or M5, then you have the same issue. If you test on H1, 4H, or daily on bar open, then you can likely run these massive iterations in a reasonable amount of time.

But, I am just guessing here -- I do not have the new toy, yet.

I see what you mean. In my framework I first convert ticks to whatever bar type I'm looking at then run through the data bar by bar for the optimizations. I have a tick by tick option but you're correct, testing is really slow.

It is tough these days as everyone wants EAs that take a tons of trades and that are quickly in and out of the market. And many "hot" indicators work off of ticks. Simple systems can be really good. Not saying complex systems are losers, but the simple ones tend to endure if run correctly.

Splitting is faster And afterwards, I can split them in every way I want. So trying in-sample timespan of 1 year, 2 years and 3 years? No problem, can all be done without one single further backtest!

True, rounding is a good thing to do when dealing with floats/doubles, but due to random sampling it is more or less irrelevant, as it does not loop through every possible combination. So if it picks 2.150234 or 2.15, the results will be the same, and the datapoints will be treated in the same way (later, during analysis).


Exactly




Regarding fast in and out: Well, afaik most professional traders use higher timeframes, lower trade frequency and longer lasting trades, as they know that trading is not about doing as many trades as possible, but about exploiting inefficiencies in a robust and reliable manner.

True, complex, fast systems with high trade frequency do not have to be worse than others, but they are never as reliable and robust.
And I hope with the heavy analysis power of DATFRA I can make this statement even more true

Hey Darwin et al,

any thoughts on my WFA process...

http://www.forexfactory.com/showthread.php?t=472798 - - i created a new thread as didn't want to hijack this one.

would appreciate some insight



thanks

Hey Simon,

I will answer your questions later this day when I find some time. Until then, be aware that whatever WF Process you use, its tricky and results might be random / not meaningfull

-Darwin

Edit: Answered

On what basis will DAFTRA measure WFE? I see Prado measures WFE based on profit, but that seems lacking and misleading. Curious on different approaches to this. Also, how important is WFE?

WFE is measured as described in literature: summed oos profit per month / summed is profit per month.

It is not very important, in my opinion, as it is very fluctuating.

My WFA implementation has an additional way to describe a system that is more meaningful: expectancy of trades.
So after each entry, every candle (~200-300) is analysed, and then the average movement in our favour - or against us - is recorded.

For example: (blue line = average exit)
http://darwins-fx-tools.com/images/sc/mexp.png

I think this is a lot more meaningful than WFE, if the only data we have to analyse a system is a WFA

-Darwin