Why backtests are useless & EAs are flawed [discuss]

.
.
.
ATTENTION: THIS THREAD IS OUTDATED
Please go to http://datfra-expert-advisor.com/ in order to find the final result of all the research!





















Hi, I am Darwin and I am new here.

tl;dr In the next weeks, I am planing to release my private Walk Forward Analyzer, but before I do this, first I want to make sure that also the beginners in mechanical trading understand why common backtests are quite useless and why walk forward analysis is better in so many ways, so I decided to start this discussion.

And as people keep asking why I release free stuff and why I post this thread on multiple forums (actually, I want to post it on all of them, so tell me if I have missed one), let me explain my motivations:

I am targeting a job in the trading-economy, and this is not easy to get. Beeing known in the community would help me a lot to accomplish this goal, so I decided to release some of my private tools for free and/or open source to make myself a name over the next months. I am not here to sell you stuff, keep that in mind, please!


For the record: I do not claim that WFA is the holy grail in system testing, but I argue that it is way better than normal backtests.




THE GENERAL APPROACH TO SYSTEMATIC TRADING

Take a trading logic => Generate an EA that trades exactly like it => Optimise Parameters => Do a backtest => Trade it live => $$$

BUT: Everybody that has ever had an EA that was a money-printing machine in the backtests (and this is very, very easy to be done), knows that a good backtest does not imply profitable live trading.
And that is the problem. We need a test-method on whichs evaluation we can rely!




STATEMENT 1: Your parameter choices are not good

Well, every EA consists of 2 things:

1. the logic/script itself
2. the parameters (like periods for moving averages, stop loss values etc. Just everything that can be adjusted!)

The first thing is static and "given". And a lot traders only focus on this part.
But, an EA can behave in very different ways, depending on the parameters. And there can be billions of possible parameter-combinations (=parameterset). There are 3 approaches to determine the parameterset for live trading.

1. the intuitive approach (non adapting):
The trader just chooses the parameters based on his expert knowledge.
But, there are just too many possible parametersets. You can't just "guess" them without evaluation and testing.
It does not matter how good you are, you are never good enough to "think through" this huge amount of possibilities in a reliable fashion!

2. the "optimise on all data" approach (non adapting):
The trader chooses the parameters based on all past data.
So, the configuration/parameters that worked best for the last 12 years, for example, are chosen to be traded live.
But, to stay up-to-date it is not a very good method to optimise the parameters on so much and therefore OLD data.
Because the market today is not the same as it was 12 years ago.

3. the "optimise on the last few years" approach:
The trader chooses the parameters based on the last few years of data.
But you can not just take this approach without testing how this "optimisation method" would have worked in the past.
And this is exactly what a walk forward analysis will do: "Optimise on the last few years", but it tests this approach on all data in the past!




STATEMENT 2: A non-adapting EA can never make longterm profits

I mean, there is an infinite amount of possible trading-systems AND an infinite amount of ways the market can change AND an "infinite" amount of possible parameters for your trading-system.

Considering this, would you really want to bet money on the fact that you have a trading system that will always work, in the same way, without adaption, on all future market conditions? I would not.

"But why shouldn't I be able to put a traders knowledge into a script and trade it?"
Its simple: A trader always learns, he takes input from many sources, he has knowledge about the markets and therefore he adapts his trading strategy so he can always be as close to the markets as possible.
A simple EA script can't do this, a Walk Forward Analysis can (to some extent).

Reason #1 why there are many "profitable" EAs out there
The EA really works, it has a sound strategy and was developed properly. But the markets always change, and they can change in infinite ways.
So, at some point, the traded market-inefficiency WILL(!) change, and the EA can not adapt to this change and therefore will lose its profitability.

Reason #2 why there are many "profitable" EAs out there
is that some of them, sound or unsound systems, are just lucky.
If you send 10.000 people to a casino and let them play for a while, just due to chance, some of them will make profit over some time.
The same is for EAs, if there are enough of them, some will really make profit (even if they are in fact useless).
But as they keep trading, they will lose, as the probabilities are against them (same as in a casino).

Reason #3 why there are many "profitable" EAs out there
is the huge risk some EAs take (grid trading, martingale systems etc).
These EAs take very huge risks to make small profits. But somewhen, the risk strikes, and at that point, the EA will to lose (all) money.

Reason #4 why there are many "profitable" EAs out there
is the small timespan on which they are profitable. Its no magic to make profit for months, but its hard to make it for many, many years.




STATEMENT 3: A backtest does not tell you anything about the future performance

Well, don't get me wrong, I strongly hold the opinion that simulations on the past are the only way to really test a trading system.
But what does a backtest tell you? Just that your system performed well on the past.

But is trading really about having a system that represents the past?
No, trading is about designing a system on the past and then trade it in the future.

THIS IS A FUNDAMENTAL DIFFERENT QUESTION THAN WHAT A BACKTEST CAN TELL YOU!
You want to answer the question "how good is my live trading performance" with the answer to the question "how good did my system perform in the past". That logic is flawed, of course.

"But if I use out-of-sample data to verify my backtest..."
Out of sample testing is a good idea. But you only have one optimisation-dataset and one test-dataset, which is not very reliable. Walk forward analysis, somehow, is out-of-sample testing on steroids. It uses the same method, but generates 10-1000 opti/test dataset-pairs.

"So, are all backtests useless?"
No, it can help you to get sound and good trading systems, if done right.
But my point is that a good performance in a backtest does not make sure the system is also good in live trading.

Trading is a game of probabilities, and the probability that an EA is profitable, only based on a good backtest, is very, very low.

"But why should a system that performed good over the past simply stop working"
The main reason is, and that might be the most important one, OVERFITTING.
That means, the EA tackles an "random" inefficiency of the market that was just there in the past, but that was no real thing.

I found a very good picture that explains overfitting:

http://i.imgur.com/CIqiSuF.jpg

You see? There are infinite of such useless "relationships" in the price data, and most EAs tackle one of them which gives them very good backtest results, but that was just due to luck, and these "relationships" will not hold in the past.
And as there are a lot more of these overfitted solutions than sound ones, the chances are very high that you optimised towards such an overfitted system.

The other reason is that the markets always change. So even if you have a non-overfitted, good performing backtest, you will run into serveral problems that you can read in Statement 2



THE SOLUTION: WALK FORWARD ANALYSIS

A Walk Forward Analyser is an external program that can take EVERY EXPERT ADVISOR and, using Metatrader4, do the analysis.
SO YES; YOU CAN CONTINUE TO USE YOUR ALREADY WRITTEN EAs!

Well, some of you will already know how a walk forward analysis works. For the others here is a short overview, as I will explain the process in more details in a future thread.

A WFA takes, for example, the data from 2000 to optimise your system, then tests it on 2001 (which is, from the point of optimisation, the "future" or "live trading").
Then it walks forward and optimises the system on 2001, test it on 2002. Then optimise on 2002, test on 2003 etc.
Do this until you walked through your whole data and only consider the "live trading" in your evaluation.

You see? With this simple tactic you can tackle the 3 problems I have described above, the lack of adaptability, the need for a evaluated parameter selection process and the uselessness of a "past-performance"-backtest.


And then, when you want to trade the system live, you do the exact same thing that you have tested:
1) Optimise on the last available data, with the same optimisation settings you tested in WFA
2) Trade it in the future, with the same procedure you tested in WFA

And that is the main difference to normal backtests: YOU TRADE YOUR EA IN THE EXACT SAME WAY AS YOU TESTED IT!!!



So, this is what I wanted to say, now feel free to tell me what you think about this topic, please
- Darwin

Spot on Darwin. I have done many backtets and optimised my EA on the backtester, only to find all those great results are meaningless. I understand the concept of walk forward and have done that using backtester but you are proposing a somewhat different tester than the MT4 one? Or is it just the process tat you are tweaking?

Note there is a little danger with the WFA:

1- You write an EA
2- You WFA it over your E/U data
3- Results suck!

4- You change this and that in your code
5- You WFA it over your E/U data
6- Results are slighly better

7- You change this and that in your code
8- You WFA it over your E/U data
9- Results are impressive. Holy grail found!

Of course the reality is that you overfitted your E/U data! By misusing the WFA you turned it into a usual backtest...
Always use ANOTHER dataset when trying the EA. This second dataset shall be as few correlated with the first one as possible.
NEVER spot "problems" on the second dataset, just go blind. Otherwise you'll overfit it as well!

... and isn't the History Centre data artificial anyway? It does not necesarily match the brokers data/spread that they apply. One EA vendor suggest that to overcome this you should collect data by pressing the LH arrow key and scrolling back for several months on each of timeframes, ie 1 min, 5 min, 15 min etc. A very tedious process especially for the 1 min data set! And do this when the markets are open, ie not on weekends.

The premise that backtests are useless and EAs flawed is only 100% true assuming the strategy itself has no sound edge to begin with. Unfortunately, from my limited experience, that seems true of 99ish% available EAs. Hence, backtests (and likely WFA too) are indeed very close to useless when it comes to predict future performance. I say this because the edge soundness being qualitative in nature cannot be 'fixed' by quantitative tools like BT and WFA. Those tools can only address robustness.

Assuming a sound edge exists, and equally well done BT/WFA, then WFA should be better at least in theory.

Looking forward to read more on this from you guys.

Well, at the moment i just tweak the process and, most importantly, make it automated.
The WFanalyzer serves as proof-of-concept of a way more sophisticated algorithm that I am currently writing (and releasing in a few months, here, free), so I decided to use the MT4-optimiser.
Tough, the new algo has an interface for custom optimisation engines that are then not only able to optimise parameters but the whole logic of systems.
But more on this project when there is more to show



----------------------------------------------------------



Ah a very good point, yes, I will have to make sure everybody using my tool understands this and the danger coming from it. Thanks for mentioning it



----------------------------------------------------------




Thats the underlying problem with these small timeframes! You will never get very reliable and stable results on them, because, as you said, the brokers differ.
So, to use past data to optimise and test a strategy, I would never go below H1, just to make sure to get reliable results



----------------------------------------------------------



Good points, and thats true. The thing is, it is easier to use a WFA in a sound way than using a backtest in a sound way, as the WFA is, in its nature, not so vulnerable to get good results out of unsound techniques.
But yes, you are right, the danger is always there, we can just lower the probabilities for it to happen.

do you have any results to show?

Absolutely. That being precisely the reason why I mentioned WFA in theory should be better than conventional backtest.

Hey, well, there are no "results to show" as it is an analysis-method, so it depends on the Expert Advisor you test if the results are good or not
And as I am still working on my own Expert Advisor Builder (an algo, of course) I only used the default "Moving Average"-EA that shipps with every mt4-installation to make some tests.




In theory, yes. Sadly I can not "force" the users to use the tool in a sound way, so let's hope most users will understand enough of EA development to not get into this trap

do you have an expert advisor that performed well for 1 year using your method?

Well, I fear that you do not understand the problem, it would be totally meaningless if I present you such an result (even if it would be 1 year of live trading), as the method I describe is just an analysis-method, and the performance depends on the used Expert Advisor. Garbage in -> garbage out

So whatever result I show, it would just represent the performance of the EA, not of my tool.


But to answer your question, no, it's a new tool, a concept, there was no live testing done yet.

-Darwin

whell you can have a new tool and concept but you made the statment that it's better than backtesting. i'm asking you not to make statements which you cannot back up with results.

The concept of a WFA is, in its nature, better than a backtest, because of 4 facts

a) it is more like live trading as any backtest, as it simulates optimising on the past + trading in the future
b) it gives the EA a new feature - adaptability - which is essential in ever-changing markets
c) a (backtest + out-of-sample-test) gives you 1 datapoint to judge from, a WFA gives you 100-150
d) get a good backtest out of an useless EA is easy, get a good WFA out of an useless EA is hard

So the statement that it is better than backtesting is one that can be proven by logical arguments, no need to "test" it, as "testing" would mean you need MANY EAs and A LOT of forwardtesting with both methods to get statistically relevant results.
Also it would need a reproducable approach towards both methods.

If you want to take that effort, please do so, but I don't have the time and the need for this.
Tough, I am quite sure I can win EVERY argument if someone argues that a wfa is not better than a backtest.

You see? "showing results" is not a suitable method here.

-Darwin

yeah sure you do. you can win an argument with stupid people. smart people always want to see results because life is all about results. results are a healthy respect towards reality. if you don't know why it's always best to respect reality i'll let you go.

Well, of course it's always best to have reliable figures and tests and experiments, I agree with you 100%.
I just say that it would need a massive amount of work, time and data to get statistically relevant numbers that show "reality", as you say

As we do not have these numbers (I would also love to have them), we can just work with logical arguments and counterarguments to prove the point.
And I am sure I can win an argument about the pro and contra of a WFA, also against smart people, just because the concept itself is better than the concept of a backtest - even without fixed numbers

Hope you agree with this

-Darwin

why a massive amount of work? work towards 1 succesfull ea that was developped through your concept and that would be your best resume. at least 6 months of consecutive non losing ea.

Ok, what would that prove? Only that this particular EA was good for a particular 6 month dataset.

Sorry, but I think you don't get the problem with un-reliable numbers.

What you suggest would mean nothing at all. WalkForwardAnalysing one EA and say "that concept is good" is the same thing than doing a single backtest and say "this EA is good". (drawing conclusions using a statistically non-relevant amount of datapoints... and 1 datapoint is for sure not relevant).

Also, this is NOT AN EA DEVELOPMENT METHOD (like you suggest with "1 succesfull ea that was developped through your concept"), it is an method that ANALYSES an EA (+ its optimisation-methodology)... which would make this "test" even more non-saying

So please, take the time an read what I have written already. It is just not possible, on so many levels, to "show one EA that proves this concept works"

-Darwin

1 it will prove that the ea worked for 6 months live testing and it will give you an edge over 900000 market participants.
2 you want the concept to be accepted? then don't do 1 ea. do 100. do 1000.

The walk forward analysis is a... hmm... an ANALYSIS tool! It is not a EA factory.
Please stop spoiling what could become a good very thread.

go f yourself.