Is backtesting necessary before Walk Forward Analysis? | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

EA Walk Forward Optimization - To maximize the performance 14 replies

Quant - I will run a forward walk for you! Analyze your risk! 0 replies

Walk Forward Analysis - the process of testing 6 replies

Walk forward analysis for EAs 6 replies

Neuroshell Walk Forward Analysis, ATR Questions 0 replies

Is backtesting necessary before Walk Forward Analysis?

  • #1
  • First Post: Mar 28, 2015 1:17am Mar 28, 2015 1:17am
Hi everyone,

Is backtest necessary before WFA, or is it just a waste of time? Considering the fact that WFA will test the strategy's robustness and estimate its real time performance better, do you guys do backtests before WFA, or just put a strategy to test with WFA?

WFA:
http://forums.babypips.com/attachmen...7&d=1427385810
Done only with historical data. Not with live data.

Backtest:
Using all or part of historical data without any out-of-sample testing.

Thanks.
  • #2
  • Mar 31, 2015 2:17am Mar 31, 2015 2:17am
I think it is great. I have been experimenting with EAs for 2 years now and finally realized I was curve fitting. Just breaks a trader's confidence big time. This concept however of optimizing, then forward testing is like trading for real in the past because the out of sample tests is info I never saw or knew existed. So with decent software to compile the test reports such as EAAnalyzer and Walk Forward Analyzer for Metatrader, I can get a real view of results providing I have accurate data. A lot of work though. I have lost a lot of money over the years trading manually. Emotions have to leave. I read that approx. 1/3 of all stock market trades in Europe in 2006 were algorithmic. I think it is heading there.
  • #3
  • Apr 1, 2015 7:43am Apr 1, 2015 7:43am
Can you provide a link for EAAnalyzer?

Also my question was regarding if a full-sample (all of the historical data) backtesting (and consequent optimisation) being necessary if a WFA is going to be done.
  • #4
  • Apr 1, 2015 7:52am Apr 1, 2015 7:52am
Backtesting is obviously necessary. For me the main reason is not testing the goodness of your EA strategy but testing a large use case coverage. Verify that the EA is doing the required logic in a strict way is mandatory.
Because of the complexity of your EA logic you can face on several unpredicted behaviour that can be avoided to have in forward tests.
Algorithm is life
  • #5
  • Apr 1, 2015 8:58am Apr 1, 2015 8:58am
The link for EAAnalyzer. It compiles all the tests from the Walk Forward Analysis. You will have to dig and figure out how you will organize your data, but an absolute must if you wish to remove all emotion from the trading system and verify the robustness of your system. I have decided to work with a moving average system. I used Tickstory Lite to download all my data removing all volume which is not necessary and greatly increases the speed using ticks. Volume is more broker dependant. I also download an EA called search for gaps in the data. The key is to have real life data without holes and a robust simple strategy that can be optimized in small in samples and tested in out of samples several times over a period of say 9 years. I test using 720 days of intraday and walk forward 144 days resulting in 20% walk forward. I get 24 (out of samples) results merged together with EAAnalyzer. I do this with several timeframes and currency pairs. As I build my profiles, they also are merged into one using EAAnalyzer. This is like having a large portfolio in the end. All trades are sorted in chronological order and displayed in a chart. I am still waiting..............for matching results somewhat. Still a patience game regardless. Good luck! I am having fun trying to do like the big boys.
http://www.strategyquant.com/eaanalyzer/
  • #6
  • Apr 2, 2015 7:35am Apr 2, 2015 7:35am
Quoting CrazyCoder
Backtesting is obviously necessary. For me the main reason is not testing the goodness of your EA strategy but testing a large use case coverage. Verify that the EA is doing the required logic in a strict way is mandatory. Because of the complexity of your EA logic you can face on several unpredicted behaviour that can be avoided to have in forward tests.
I should too look for is some unpredictable thing your system does and take preventative actions. That is a good reason to do backtesting. Thanks for pointing that out. So my follow-up question is:
1) Do you go trade by trade to figure out if any unpredictable problem occured? Or do you look at some statistics like max drawdown or another statistic?
2) Do you do the same after you optimize your strategy (to see whether optimized parameters cause any unpredictable behaviour not seen in not-optimized-system backtest? Or you just skip optimizing and move on to forward testing?

Quoting rhhince
The link for EAAnalyzer. It compiles all the tests from the Walk Forward Analysis. You will have to dig and figure out how you will organize your data, but an absolute must if you wish to remove all emotion from the trading system and verify the robustness of your system. I have decided to work with a moving average system. I used Tickstory Lite to download all my data removing all volume which is not necessary and greatly increases the speed using ticks. Volume is more broker dependant. I also download an EA called search for gaps in the data....
Thank you for the link. Seems like a good analyzer. The EA that searches for gaps seems to be very helpful too. As to why might your live trades not match the WFA results: did you incorporate commissions, slippage etc? Just a guess.
  • #7
  • Apr 2, 2015 12:03pm Apr 2, 2015 12:03pm
Basically, my EA simply searches for everyday occurrences and use small stops and trailing stops and a few other things. I let strategy tester do its thing and choose the most profitable which the walk forward analyzer chooses as well using the interactive interface. What ever worked best for 720 days trades the next 144. That 144 days is data that I never knew about. I am an extremist and the resulting 1200 (out of sample) reports get compiled as a portfolio in the EAAnalyzer and I can further review the grouped strategies and decide if I should trade it. Long procedure, but I have lost many dollars due to curve fitting and manually go heavy or go broke attitude over the years that I want to pretend I never saw the data. That I believe is the confidence builder. I want to see if the next 144 delivers something close to the strategy tests. Slippage, spreads and carry over fees all affect the end, but if the system can capture 75% of the walk forward results I would be happy.
  • #8
  • Last Post: Apr 5, 2015 10:44pm Apr 5, 2015 10:44pm
Quoting kojinakata
1) Do you go trade by trade to figure out if any unpredictable problem occured? Or do you look at some statistics like max drawdown or another statistic?
Most of my EA are done outside of mql, in a dll or in a standalone trading system. Durin the development I create several unit tests with some scenarios. It's my first test campaign.
The second campaign is backtest. I check that I don't see crazy behaviour. I take time to check trade by trade when I see some strange behaviour. This check is only done on pointed out periods. I can't verify trade by trade on the entire backtest period. It can take too much time

Quoting kojinakata
2) Do you do the same after you optimize your strategy (to see whether optimized parameters cause any unpredictable behaviour not seen in not-optimized-system backtest? Or you just skip optimizing and move on to forward testing?
Because I don't develop for myself but for other, I let the trader to do parameters optimization. I can help him if he needs to understand what happened in some cases but it's really rare.
Algorithm is life
0 traders viewing now