Walk Forward Analysis - the only logical successor to backtest | Page 2 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Is backtesting necessary before Walk Forward Analysis? 7 replies

Walk Forward Analysis - the process of testing 6 replies

Walk forward analysis for EAs 6 replies

Visual backtest result different from non-visual backtest 0 replies

Neuroshell Walk Forward Analysis, ATR Questions 0 replies

Attachments: Walk Forward Analysis - the only logical successor to backtest
Exit Attachments

Walk Forward Analysis - the only logical successor to backtest

  • #21
  • Nov 29, 2013 5:24am Nov 29, 2013 5:24am
Quote
Darwin do you know the genetic alogorithms? I think that's the way you're taking

Sure I do, and because my tool uses the Metatrader4-optimisation engine (which uses genetic algorithms), it is already in my tool.
Tough, there is the need for custom optimisation-algorithms that are not just able to optimise parameters but also the logic and other stuff, so it is not the end of the road.

Also, I started my journey into algo-trading with genetic programming algorithms that were able to generate EAs from scratch, but soon found out that the generation of EAs is an easy task, but the evaluation of them is the hard part.
So I came to WFA somewhen

Quote
Where do you find 100.000 independent samples? as soon as your training windows overlap their are no more independent.

Ok, lets say "more or less independent", as they overlapp of course. Nevertheless, they are also not the same, just overlapping to some degree.
Well, back to my initial post, you see how to get 100 samples, right?

Now, if you do not just analyse the "very best" candidate after every optimisation but the 1000 best, you suddenly have 100.000 samples.
If you then reduce the test-window to the smallest possible value, 1 trade, and an EA made like 1000 trades in the evaluated timespan, you can easily get 1.000.000 samples for analysis.
Which is quite good compared to the 1 sample a normal backtest+oos-test can give you


-Darwin
  • #22
  • Jan 28, 2014 12:47pm Jan 28, 2014 12:47pm
My question is: What is the point of back testing or WFA testing ?
In live, let's say, 1H-chart, you use a system with crossing of EMA to give you a signal BUY or SELL according to what crosses under what etc...so it may get a signal and a good one with 5 pip difference on the crossing (like 'before'), and then 30 minute later you see a complete reversal. When back-testing, you do NOT see the full story unless you module somehow with a 15-min chart. The EMAs are different when they start and end the candle.
I have never back-tested (i don' know how to use MT4), but is it possible to backtest using TICKED graphs so you see the movement of EMAs (as in my example) against time. I wanted to follow that up, but you cannot do it apparently
  • #23
  • Feb 6, 2014 1:08am Feb 6, 2014 1:08am
Hey Darwin,
Are you still developing this? If so I have a bunch of questions for you. Happy to ask them here.


cheers

Simon
  • #24
  • Edited 5:39am Feb 20, 2014 5:02am | Edited 5:39am
@laurentyves: You can use Tick data, but in algotrading it is usual to rely on systems that only act on bar opening. Because that way you can simulate them in like a second, instead of minutes. That gives you the advantage to make millions of measurements, for one system, in only 24 hours.
And as long as the system is designed to wokr that way, your simulations are as accurate as with tick data


@simonsc: This = WFA? No, I am developing a completly new way of system analysis atm, it works like WFA but instead of forward-trading the very best candidate per window, it does so with all of them.
That way, I can generate like 10.000 as many datapoints per system than WFA can, which gives a very powerful analysis framework
Using that huge amount of data, I think, you have a way of looking at the real characteristics. Because I found out that WFA, using only 100 or 200 datapoints, is still very much a product of chance.
But overfitting 100.000 (or even millions) of single past->future measurements is very unlikely. The same goes for beeing a product of chance!

Here is an example of a few hundred thousand past->future points plotted:
http://s7.directupload.net/images/140214/owgurmkk.png

See the clear trend? That is a strong indicator that the shown trend is "real". WFA can not provide that kind of insight, therefore I consider it as obsolete and unreliable

But ask your questions anyway!

-Darwin
  • #25
  • Feb 20, 2014 6:54am Feb 20, 2014 6:54am
Quoting Darwin-FX
@simonsc: This = WFA? No, I am developing a completly new way of system analysis atm, it works like WFA but instead of forward-trading the very best candidate per window, it does so with all of them. That way, I can generate like 10.000 as many datapoints per system than WFA can, which gives a very powerful analysis framework Using that huge amount of data, I think, you have a way of looking at the real characteristics. Because I found out that WFA, using only 100 or 200 datapoints, is still very much a product of chance. But overfitting 100.000...
Hi Darwin, I saw this reply and it now looks like you're headed in a slightly different direction than traditional WFA. I'm interested in more information regarding your research and direction. In particular can you expand on both the process and goals?

The picture you posted shows (my interpretation) that variance of system performance is increasing with rank. What is your interpretation? Does high rank indicate high system performance?

If you forward test all the candidates per window what data do you collect and what do you do with this data? Are you talking about combining multiple systems into a single signal of some sort? If so I'm very interested in your line of research as it has been on my radar for some time now. Are you using this data to evaluate overall system robustness or parameter selection? That's also an interest of mine.
  • #26
  • Edited 8:51am Feb 20, 2014 8:28am | Edited 8:51am
Hi FXEZ,

well, the core is the same as in WFA: analysing the correlation of past and relative future. The change in process is just the increasing of datapoints collected (like I said, 10k - 100k more data). But this allows a whole new way of analysing a system.
So the goal is to write a framework that first generates all possible past->future datapoints and then lets you dig into this data and look at all relationships one is interested in.

For example between a parameter and forward trading profit.
http://s1.directupload.net/images/140214/u6igit3c.png

Perhaps 3 in-sample fitness values in dependence of forward trading profit? No problem!
http://s1.directupload.net/images/140214/csogbczr.png

Or anything you could imagine.




Oh I should have mentioned, rank 0 = best, so variance is increasing with decreased in-sample performance.
Also it shows that taking the best candidate per wf-window only gives you about the 1500th best candidate in forward trading, which is very bad (but hey, the system analysed is the one that comes with every mt4 by default, "Moving Averages.mqh" or something).



Just like WFA, each past->future datapoint contains In-sample performance, Out-of-sample performance and used parameters.
But the difference to WFA is that you get many, many of them, and you can use it to evaluate system robustness in a way never seen before.. of course you can also see which parameters work best etc.

One thing to mention, instead of WFA, my approach is parameterless. You just need an EA, a Market (EURUSD/H4 for example) and the used timespan (2000-2013) for it.
EVERYTHING else, like different insample->outofsample timespans, or different parameter-ranges for the EA etc can then be extracted afterwards, without the need for re-doing any backtests.


-Darwin

PS: My skype name is "Darwin-FX", everybody that is interested in details can add me there. I have finished the alpha version 2 days ago and I am currently bugfixing and polishing stuff, so I did not really have time to play with it yet

PPS: I am currently writing 2 new articles, one in which I try to explain why I do no longer think WFA is a good analysis method and another that will explain how my approach works
Also I want to use my tool (DATFRA - Darwins algorithmic trading framework) to analyse some of the standard indicators we all use, and make a statement about what parameters work best in which markets.
  • #27
  • Feb 20, 2014 3:44pm Feb 20, 2014 3:44pm
Hi Darwin,

Interesting that you have modified your approach.
The general question i had was related to the number of passes you walk forward with. I am not sure if that is what you are now doing with datapoints? (I didn't quite follow what you meant by datapoints?)
My concerns around WFA, is that if based of an optimization run, if it only takes the best pass to move forward with, then this may make the WFA inaccurate. As we know the optimization process itself may be curve fitting, and so the most profitable pass may not be the best one in out of sample data. Therefore several passes/results from the optimization need to be run over the out of sample data at each stage.
So i guess my original question was, does your current version do this? If not is that what you new version will do?
Will it cater for multiple optimization parameters? I have reduced mine as much as possible but I still have about 4/5.
I am in the process of building a basic process to do this based on this article http://articles.mql4.com/1012, but it will still be a fairly manual process for now.
  • #28
  • Feb 20, 2014 7:46pm Feb 20, 2014 7:46pm
Quoting simoncs
Hi Darwin, Interesting that you have modified your approach. The general question i had was related to the number of passes you walk forward with. I am not sure if that is what you are now doing with datapoints? (I didn't quite follow what you meant by datapoints?) My concerns around WFA, is that if based of an optimization run, if it only takes the best pass to move forward with, then this may make the WFA inaccurate. As we know the optimization process itself may be curve fitting, and so the most profitable pass may not be the best one in out...

Darwin is proposing running a WFA for a ton of optimization parameter sets, which gives one a really good assessment if the set that you do trade going forward has any chance in _ell of performing like it did in the past. This is a much better method of assessing whether or not your chosen parameter set will perform well going forward. Darwin is just touching the benefits that could be garnered from these analyses. Serious stuff.
  • #29
  • Feb 21, 2014 5:27am Feb 21, 2014 5:27am
Quote
The general question i had was related to the number of passes you walk forward with. I am not sure if that is what you are now doing with datapoints? (I didn't quite follow what you meant by datapoints?)
My concerns around WFA, is that if based of an optimization run, if it only takes the best pass to move forward with, then this may make the WFA inaccurate. As we know the optimization process itself may be curve fitting, and so the most profitable pass may not be the best one in out of sample data. Therefore several passes/results from the optimization...

Exactly that is what I do, and you are right, if you only forward trade the "best" candidate, it is highly inaccurate.
Therefore, my new algorithm forward trades them _ALL_, and then just saves these forward trades.
And that is then a "datapoint", consisting of
a) the performance during in-sample test
b) the performance during out of sample test
c) the used parameters.

Afterwards, it gives you the tools to dig into this data.

For example, based on this data, you could "simulate" a normal WFA, but this time always choosing one of the top 5 candidates instead of the best, and that repeated over 20 times (for example):
http://s14.directupload.net/images/140221/haovv9r6.png
And because all the data was generated in a step before, it only takes a few seconds to do this.


But that is just a very basic way to use this data as it still chooses only a few "datapoints" out of the hundred thousand datapoints we have.
So it is better to use them all, and for example look into how well one fitness value (like profit) during optimisation is correlated with another (like profit, again) in forward trading (like in the image postet here: http://www.forexfactory.com/showthre...08#post7294008 )

So simulating WFA, even with randomisation in choosing the candidate for forward trading, is still only the worst idea to view a system. If you have so many data, you should also look at all the data









Quote
Darwin is proposing running a WFA for a ton of optimization parameter sets, which gives one a really good assessment if the set that you do trade going forward has any chance in _ell of performing like it did in the past. This is a much better method of assessing whether or not your chosen parameter set will perform well going forward. Darwin is just touching the benefits that could be garnered from these analyses. Serious stuff.

And it is not so much about forward trading all this parameter sets, it is more about the possibility of viewing the single parameters in dependency of, for example, forward trading profit (first image in my last post).
Or you could also view 2 parameters in dependency of forward trading profit:
http://s14.directupload.net/images/140221/5lirnlui.png


You see, there are no limits to the imagination, if the data is there, you can look at it in so many different ways, and the ones I came up with are just the beginning
For example, atm I am experimenting with Neural Networks in order to forecast live trading performance based on in-sample performance so you can adjust the risk you take per trade.

-Darwin
  • #30
  • Feb 21, 2014 5:49am Feb 21, 2014 5:49am
Hey Darwin Im picking up what your putting down - I have previously given EAs a shot and then proceeded to drop them because I found them BS...

But yeah you obviously have a head on your shoulders and ill need to reread some of your posts but expect to hear from me in the near future
  • #31
  • Feb 21, 2014 5:57am Feb 21, 2014 5:57am
EAs are not bullshit, but current analysis methods are.
You know, if you have an EA with a few parameters you soon end up with a possible parameterspace in the millions or billions - and then ppl try to judge these EAs by doing 1 (normal backtest + out of sample test) or 100-200 (walk forward analysis) measurements. That can not work

Sure, EAs can work nevertheless, but then just because the trader that wrote them did a good job - not because the analysis predicted good performance.


-Darwin

Ah and sure, whoever wants to discuss stuff with me, my Skype is Darwin-FX (again), just add me if you have something to say
  • #32
  • Feb 21, 2014 6:22am Feb 21, 2014 6:22am
One addition to the topic EA<->Manual Trading: Of course one can also just evaluate the "manual trading indicator setup" with this approach, looking for the best working parameters and stuff and then continue normal manual trading, but with heavy analysis in the background, instead of only relying on experience.

-Darwin
  • #33
  • Feb 21, 2014 7:13am Feb 21, 2014 7:13am
Hi Darwin,

Very interesting thread, thanks for sharing your work.

As an EA writer, off course I do some kind of testing on my current work but not in a such advanced manner like you and not looking for parameters giving me the best performance but the minimum draw-down instead, I always look to minimize the risk instead of maximizing profits, as long as I have at some decent profit as a result.

To validate my EA, I run a back-test of 3 years (2008-2010) of duration which results in about 3k trades, then I move forward 2 days and I repeat that same test, Last test goes from 2011-2013. And from all of this, I choose the parameters that produce the minimum of draw-down. My initial idea was that if even starting to run my EA at the worst moment, it should go trough without problem.

You see, my approach is a bit different, just wanting to know what you could think about it.
Simplicity is the Ultimate Sophistication.
  • #34
  • Edited 7:58am Feb 21, 2014 7:35am | Edited 7:58am
Hey broketrader

I would also try to reduce risk instead of maximising profit.
Also your very small forward rolling window of 2 days seems to be a good idea, buw what I do not get is if you analyse only parameters<->backtests or if you do some kind of parameters<->past<->future relationship tests.

If you don't, I would highly recommend this, as you are not interested in "how to pick the parameters with best drawdown in the past" but in "how to pick parameters, based on past performance (backtests), that will give low drawdown in the future".



Ah and btw, if one wants to reduce drawdown instead of maximising profit, of course that can also be done with DATFRA:
http://s7.directupload.net/images/140221/ueaklgig.png

But you see, low drawdown always comes with low profit, so I will have to come up with a fitness measurement that incorporates profit && (smootheness of curve || drawdown) in order to get better results, as min drawdown is not good if there are not real profits made


-Darwin

PS: My drawdown calculation function seems to be a bit screwed, so do not take the above image as "real", its just an example. Well, this is still alpha version, so that kind of stuff happens


EDIT: Give me a few minutes then I will post a Drawdown<->Profit plotting that should be "real"
  • #35
  • Feb 21, 2014 8:11am Feb 21, 2014 8:11am
In fact, I have a long term view which is 5 years, then I choose what I wanted as a minimum profit for that time frame and according to that I just have to get the settings producing the minimum drawdown for that minimum profit.

For the 2 days rolling window, I wanted to go with one day first, knowing that my EA trades the 5 min time-frame, but that was taking me too much test time.

To respond to your question, I only analyze parameters<->back-tests results. I have to say that I'm programmer but don't have sufficient scientific knowledge to generate that kind of past<->future relationship analysis .

Also as my tests terminate at the end of 2013 I'm starting to do simple forward test since beginning of 2014 at least to see that results are comparable and most important, that I don't go over my maximum drawdown.

I'm just editing, because I forgot to say that currently this EA is also live trading since a month. You can see the results in my signature, this is also why I forward test since 2014, to see if real trades correspond to what the EA does during back-tests. I'm looking for 1 year of live trading 24/7 to validate my work.
Simplicity is the Ultimate Sophistication.
  • #36
  • Feb 21, 2014 8:19am Feb 21, 2014 8:19am
Ok, so you make sure not to get very unprofitable results (with good drawdown) by defining a minimum profit level? That should do the trick if you analyse one specific system I guess

Yes I know that problem, I am currently even using 60 Days as forward trading timespan, to reduce data and test time, in order to be able to develop faster.
In the end I think something like 7 days should be fine Because then, in real trading, you could re-optimise your systems on the weekend and still be in-line with your testing approach.

Well, the secret of how this works should be in this article, so in my first post in this thread just like WFA!

So basically you choose your parameters and stuff on data up to 01.01.2014 and then verify them up to 21.02.2014?
That is a good idea, I have also implemented a method where some of the data at the end is reserved for a final out of sample test, in order to verify everything. I think not doing this is very dangerous!

-Darwin
  • #37
  • Feb 21, 2014 8:29am Feb 21, 2014 8:29am
Quoting Darwin-FX
Ok, so you make sure not to get very unprofitable results (with good drawdown) by defining a minimum profit level? That should do the trick if you analyse one specific system I guess
Yes it's only one specific system. which took me more than a year to develop

Quoting Darwin-FX
...in order to be able to develop faster
And my approach is that if I change a single coma in the code I go through all the tests again which is sometimes a bit painful .


Quoting Darwin-FX
So basically you choose your parameters and stuff on data up to 01.01.2014 and then verify them up to 21.02.2014? That is a good idea, I have also implemented a method where some of the data at the end is reserved for a final out of sample test, in order to verify everything. I think not doing this is very dangerous! -Darwin
Exactly, for my it's some kind of final verification test.
Simplicity is the Ultimate Sophistication.
  • #38
  • Feb 21, 2014 8:42am Feb 21, 2014 8:42am
I am not good in developing systems.. so I wrote an EA Builder.. But then I was missing a good analysis method, so I experimented with WFA and finally came up with DATFRA

And yes its a good idea to test every minor change, one does never know how it ends up in trading.

-Darwin

PS: Promised Profit<->Drawdown stuff comes in 15 minutes, still busy
  • #39
  • Edited 10:55am Feb 21, 2014 9:11am | Edited 10:55am
Ok, as promised, here is real dd<->profit comparison, one window shows forward trading profit, the other forward trading drawdown:

http://s7.directupload.net/images/140221/ld3ascv5.png
So you can see that small moving periods give good profit but bad drawdown.

So best would be to choose the moving periods around 9th field, they give good drawdown and good profit


-Darwin
  • #40
  • Feb 21, 2014 3:22pm Feb 21, 2014 3:22pm
Quoting nonlinear
{quote} Darwin is proposing running a WFA for a ton of optimization parameter sets, which gives one a really good assessment if the set that you do trade going forward has any chance in _ell of performing like it did in the past. This is a much better method of assessing whether or not your chosen parameter set will perform well going forward. Darwin is just touching the benefits that could be garnered from these analyses. Serious stuff.
Hi nonlinear,

Can you comment on what sort of data / analysis might be helpful in assessing OOS performance?

As a more general question, if you use the results from these tests to select a particular parameter set, whether this means accepting the chosen parameters (validation) or switching to a different set, then isn't this a form of data snooping?
0 traders viewing now