Walk Forward Analysis - the only logical successor to backtest | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Is backtesting necessary before Walk Forward Analysis? 7 replies

Walk Forward Analysis - the process of testing 6 replies

Walk forward analysis for EAs 6 replies

Visual backtest result different from non-visual backtest 0 replies

Neuroshell Walk Forward Analysis, ATR Questions 0 replies

Attachments: Walk Forward Analysis - the only logical successor to backtest
Exit Attachments

Walk Forward Analysis - the only logical successor to backtest

  • #41
  • Feb 21, 2014 1:29pm Feb 21, 2014 1:29pm
Quoting Darwin-FX
Ok, as promised, here is real dd<->profit comparison, one window shows forward trading profit, the other forward trading drawdown: {image} So you can see that small moving periods give good profit but bad drawdown. So best would be to choose the moving periods around 9th field, they give good drawdown and good profit -Darwin
The GUI and your framework is impressive. No doubt.
But when it comes to trading there is all about risk management and minimizing risk.

I think you already know that.

There is a reverse proportional dependency between "hit rate", "RR" and "signal frequency".

The higher the "hit rate" and the higher the "RR" then the lower the "signal frequency" and vice versa.

To balance those three variables over the time is the big art of trading. A high hit rate is all too often bought by a very meager RR during the trade.
  • #42
  • Feb 21, 2014 2:46pm Feb 21, 2014 2:46pm
@Naddmer:
Well, I would agree that trading is a lot about minimizing risk. And one of the best ways to minimize risk is to know every little detail about your strategy and its behavior/performance, measured in the most robust way possible. So minimizing risk is more or less the core of my approach.

You can also use DATFRA to measure "hit rate", "RR" and "signal frequency" of your system to see if it fits your needs.


Also I am currently experimenting with Neural Networks in order to reduce risk by only trading a strategy in the best possible moments!
They can be trained on the same data that is already generated for analysis, and their predictions can then, in return, be analysed with the normal framework.

They are doing a great job in predicting wether a strategy (that you have chosen for forward trading) will perform well or lose money during live trading / in the next OOS window.

That way you get some sort of AI-trading-coach, which shows very promising first results because it is tailored to the specific needs of every strategy/market and can be applied to every MT4 Expert Advisor in an automated fashion. That way it can help to reduce your risk to a minimum

Perhaps I will write a paper about this topic when I had more time for research.





@FXEZ:
I know your question was for nonlinear, but I will be shameless and answer, too.

That is why you should keep an unseen out of sample dataset, whatever you do, no matter how robust you think your approach is.
After you did all your analysis work and could come up with a plan how to"select a particular parameter set", you test this plan on the unseen OOS data in order to verify your findings.

If the OOS test fails, throw away everything. As soon as you begin to change your strategy/plan, to make it work well in OOS test, OOS becomes In sample and "seen", which will then lead to the data snooping problem.

Hope I could help


-Darwin
  • #43
  • Feb 22, 2014 2:15pm Feb 22, 2014 2:15pm
Quoting Darwin-FX
{quote} Exactly that is what I do, and you are right, if you only forward trade the "best" candidate, it is highly inaccurate. Therefore, my new algorithm forward trades them _ALL_, and then just saves these forward trades. And that is then a "datapoint", consisting of a) the performance during in-sample test b) the performance during out of sample test c) the used parameters.
Thanks Darwin that sounds excellent. I am very excited to see this!!!! Having this sort of tool available for MT4 will be a game changer for sure.
If you need any help with testing then please let me know.
  • #44
  • Feb 22, 2014 3:02pm Feb 22, 2014 3:02pm
Quoting FXEZ
{quote} Hi nonlinear, Can you comment on what sort of data / analysis might be helpful in assessing OOS performance? As a more general question, if you use the results from these tests to select a particular parameter set, whether this means accepting the chosen parameters (validation) or switching to a different set, then isn't this a form of data snooping?

My view is that if only a few parameter sets perform well on forward data, then its likely that your chosen system and set will fail sooner than a system where a large number of different parameter sets performed well on forward data. If given the choice between system (1) having an exceptionally well performing set that had only a few well-performing other sets versus system (2) having a moderately performing set that was buried among similarly performing sets, I would feel better about trading system 2. For system 2, I would try to identify a broad set range that produced the most similarly performing sets, and then trade the best set from that analysis (what defines best (profit, PF, profit per DD, etc) is different for everyone). If you choose a system and set based on this type of analysis, then yes I suppose that is a form of data snooping, but I don't mind as betting against statistics is a typically a losing game.
  • #45
  • Edited 8:35am Feb 24, 2014 5:55am | Edited 8:35am
Quoting nonlinear
{quote} My view is that if only a few parameter sets perform well on forward data, then its likely that your chosen system and set will fail sooner than a system where a large number of different parameter sets performed well on forward data. If given the choice between system (1) having an exceptionally well performing set that had only a few well-performing other sets versus system (2) having a moderately performing set that was buried among similarly performing sets, I would feel better about trading system 2. For system 2, I would try to identify...
@nonlinear Thanks for your answer. I suppose finding a "best" result buried among similarly performing sets is itself a way to limit data snooping. If an overfit result is finding the one outlier that has very little chance for success, then by focusing on a cluster of "good" performance you have limited the chance of choosing that one outlier, because all those "good" sets would need to be deceptive / based on luck, which is less likely than a single lone outlier being based on luck.

@Darwin Appreciate your comments and am happy you chimed in. Keeping some data out for final verification is a good idea, assuming you have sufficient data. I'm not necessarily a big fan of the "throw everything away" school of thought, however. I believe in recycling. It is possible that the problem in this case is the selection method and not the underlying strategy. So perhaps throw away / rework the old selection method, and choose a better fitness criteria for a new round of testing. Then once you think you have that "ideal" selection criterion based on validation, test it on many different samples to ensure that it isn't a one time fluke or overfit to the data.

I suppose my view on data snooping can be summed up with the following example: Let's say there is a good "ideal" system in a set of parameters, system A. And there is also a deceptive parameter set D that seems to perform well but is based on luck alone, also in the parameter set. The job of a selection process is to find A and not find D.

If data snooping is prevalent in your process, but your process still manages to find system A, and avoid finding system D, then system A will still be system A. In other words a corrupt process doesn't necessarily corrupt an individual system, but rather increases the odds of finding D instead of A.

If data snooping is absent from your process, but your process finds system D, system D will still be a deceptive system that will fall apart out of sample. So the lack of data snooping in your process doesn't ensure that you will find the quality system A but perhaps makes it more likely.

Thoughts?
  • #46
  • Feb 25, 2014 3:50am Feb 25, 2014 3:50am
Quoting simoncs
{quote} Thanks Darwin that sounds excellent. I am very excited to see this!!!! Having this sort of tool available for MT4 will be a game changer for sure. If you need any help with testing then please let me know.
Hello, thanks for the nice words. Yeah I am sure that will be a gamechanger, thats what it was designed to be
Sure, just add me on Skype Everybody else that is interested in alpha testing can do the same: Darwin-FX




Quoting nonlinear
{quote} My view is that if only a few parameter sets perform well on forward data, then its likely that your chosen system and set will fail sooner than a system where a large number of different parameter sets performed well on forward data. If given the choice between system (1) having an exceptionally well performing set that had only a few well-performing other sets versus system (2) having a moderately performing set that was buried among similarly performing sets, I would feel better about trading system 2.
Ah nice, thanks for the input. It inspired me to add new stuff to DATFRA's datapoints: Not only in-sample performance, out-of-sample performance and used parameters, but also some measurements that describe the whole in-sample window with all candidates. Like average performance or "how many of them made profit" etc. Good idea!




Quoting FXEZ
@Darwin Appreciate your comments and am happy you chimed in. Keeping some data out for final verification is a good idea, assuming you have sufficient data. I'm not necessarily a big fan of the "throw everything away" school of thought, however. I believe in recycling. It is possible that the problem in this case is the selection method and not the underlying strategy. [...] Thoughts?
Well, this is true for old analysis methods, however DATFRA can show you the ideal selection criterias, and if a system does not work with them in out-of-sample, throwing it away is the only good thing to do.

Also, even for old methods like WFA you have to be carefully, because the odds you choose system D get high quick as soon as you incorporate OOS data into your process in some way.

Btw: If you do not have enough data for final verification, you do not have enough data for analysis in my opinon. But if you decrease Out of Sample timespan that should not be a problem


-Darwin
  • #47
  • Last Post: Feb 25, 2014 8:01am Feb 25, 2014 8:01am
New Thread, new insights, happy to discuss with you: http://www.forexfactory.com/showthread.php?t=472656

-Darwin
1 trader viewing now