Another Reason Not to Trust Backtesting

Okay, so this is probably beating a dead horse to some of you, but I wanted to share an example I encountered with back testing, that will make some of you laugh, and hopefully be an eye opener for others.

Way back in the day when I still believed in the "Holy Grail" of trading system, I spent a fair amount of time surfing the web searching for that special system that would make me wealthy beyond my wildest dreams. I specifically remember coming across on EA in particular that when back tested seemed to turn mere pennies in to thousands of dollars. Now lucky for me, I was wise enough to test it on a small live account and quickly found this to be too good to be true.

Fast forward to today, know more than I did than, (hopefully ) I went back and found that system I back tested. Know that when I first tested it I most likely did not spend the time to optimize the test results and use the most accurate data (back then my computer was very slow and would take literally hours to run one test). Now thanks to modern technology, I present to you two EXTREMELY different views of the same system.



In case that all was too long for you and you just skipped to the bottom... here is the short and sweet

DO NOT TRUST BACK TEST RESULTS

Just because you do not have a clue on how to backtest properly does not mean that backtest results can not be trusted. Neither of the 2 screenshots have any value since the curves have not been genereated with tick data. Judging from the sharp drop in the equity on the second sreenshot and the smooth curve before the drop this is some sort of a martingale EA. They work great until that one time that wipes the account. This however does not mean that backtesting can not be trusted or be reliable, but merely that a martingale aproach to the market can not be trusted.
When you run a backtest you need accurate data and you need to understand the system you are backtesting. You need to know what the systems edge is and how it takes advantage of that edge. You need to look at where the trades are placed and closed and make sure that your logic is being implemented corectly by the EA. If all of this conditions are met the results are quite reliable.

Oh yeah, and im very curious as to what your other reasons not to trust backtesting are?

I think your taking this post out of context. The whole premise was to show how easy it is to manipulate data when back testing and how even a small change in quality of the back testing data can significantly change actual results.

Although back testing does serve some use in developing and tweaking systems, it is still never truly trust worthy. If it was, you could eventually create a system through numerous back testing trials that would out perform the market, however that is not the case.

The only real results that can be trusted are live audited results, however even then system typically only work for so long, before they end up becoming junk. The ultimate problem with back testing is simply, it back testing.

Past results do not guarantee future performance. That is where I fault back testing.

I am not taking anything out of context. All i am saying is "garbage in garbage out". You have shown 2 equity curves of a garbage system, tested on inacurate garbage data and somehow that is supposed to support your claim that backtesting is unreliable. That is the problem with public forums, too often people talk about things they dont really know much about as if they have made this big discovery and want to share it with the fellow traders! You took a system from a public forum backtested it on a few months of bad data went live with it expecting to make money. When the system went bust you decided it was the backtesting that was faulty. If so let me ask you this:

Did you know the logic behind the system?
Did you spend any time thinking why the system may work?
Did you check to make sure that trades were placed and closed based on the above logic?
Did you test with 1 min or les data, on a timeframe >1h?
Did you forward test?

This is only a part of a process someone that wants to create a succesfull system goes through, and backtesting is only part of that process. It is not bad or good, its just a tool. And as any tool it is only as good as the person using it. Give a hummer to a clueless 2 year old and he'll be smashing hes fingers all the time crying about the horible hummer.

You still really are missing the point... I am not trying to display my trading skills or lack of or what ever you want to derive from this one post. This was something that happen many many years ago, but even if it happened today, it is besides the point...

All I am trying to do is display how SOME back testing results can be manipulated and how a system might look profitable, even when it is actually not. If for some reason you qualms with that, then I would welcome your critiques, but honestly why so hostile?

Backtesting is one of the best methods to assess risks and rewards of a trading idea. It is important to do tests properly before forming a conclusion. Cherry's post reflects natural tendency/urgency of people to form believes based on a few observations. I think unrealistic expectation is one of the factors contributing to the development of superstitions....

Backtesting is an essential part of trading, but it can act as a double edge sword. The data collected from backtesting can be false in many cases due to the EA's code or methods of testing. If you're a programmer, of course its easier to catch these problems. I wouldn't say that backtested results are useless, but can be misleading without understanding the concepts behind the system.

Visual backtesting can also be misleading because you have hindsight all over the place, and many times you wouldn't even realize it.

IMO the point is that, despite the disclaimer 'past peformance doesn't guarantee future results', all methods nonetheless trust the premise that past behavior, cycles, patterns etc are more likely, than not, to repeat themselves. Whether it's a MACD crossing, a candle breakout pattern, or a rising trend in a nation's GDP, the analyst pins his hope that the effect it had on price historically will occur once again. If that were not the case, all analysis would be futile.

Sure, the value of backtesting results is eroded by poor data quality and/or test conditions. But even today's forward testing, run against live data and (if necessary) rubber stamped as authentic by an independent auditor, will become history tomorrow. To what extent can it reliably be used to forecast future price behavior? Well, I guess that brings us back to the premise in my first paragraph.

I favor elements in Merlin's approach to system development (more here). If I understand correctly, he believes that one must find a real inefficiency - or better yet, an underlying reason for price movement - and only then use the data to prove the validity (or otherwise) of the theory. As opposed to using the testing itself to create a method that's been fitted around the test data.

I believe that there are different types of potential 'edges', and that they differ qualitatively. The most robust edges are somehow rooted in the fundamental forces that drive price - heavyweight player agendas, collective 'sentiment', an understanding of mass psychology around certain key levels, order flows, and other aspects of 'market knowledge'. At the next level, these drivers supposedly create chart patterns - 'abstracts' like trends, consolidation, support/resistance, congestion, momentum, volatility, and so on, that can be exploited by the skilled technical trader. Less robust (IMO) concepts involve statistical averaging, or using indicators to measure cycles, placing the trader one further step away from the 'real' action. The indicators must be fitted to historical cycle lengths, and the assumption made that this 'optimization' will retain its accuracy into the future.

Any method will remain profitable only to the extent that the underlying driver, behavior, pattern or cycle continues to prevail. Part of the difficulty underlying all methods is that it's impossible to ascertain who the participants are at any given moment, let alone their intent. I've read some interesting comments where the writer notes how (for example) a pinbar on a D1 chart led to a huge move that lasted several weeks. IMO this is misleading. The pinbar may have helped to initiate the first part of the move, but a zillion imponderables will have meanwhile occurred, whose collective effect has somehow sustained the move.

Anyway (as usual, LOL) I've drifted off topic.

Cherry, you did say (initially) "do not trust backtesting results". Perhaps it would have been better to make a less sweeping statement like "not all backtesting results can be trusted". Just my 2c worth.

Good post hanover.

Another issue is that most people (here) performs "backtests" without having a clue about how the backtest engine really works. How can you assume that a test is trustworthy or not if you don't even know how the test is performed?

This system incorporates martingale, and therefore doomed to failure. In MetaTrader backtesting does not simulate margin calls for trades that are not closed, so such EAs could 'win' forever. It is true for the PipBoxer EA (one of the very early versions). It would simple not close orders but enter new ones in opposite direction. As long as they were not closed, it would keep them and wait for a profit. Then it would close them so it appears profitable. (Whole thing was unintentional, I believe, but it gave a false picture)

Backtesting is also unreliable because many people don't have high-quality historic data.

Another major failure, is over-optimization... the most deadly of all, I would say, as I system developer.

From Eden and up to and including now,every new day has always been a function of the old day.That naturally negates the power of the statement that defies the place of yesterday on today,and today on tommorrow.It is story that becomes history.Once again,we need to put on our thinking cap.Truly,backtesting is not absolute in determining the efficiency of a system.But it is the only available zero risk method.Yes backtesting fails to recognize the place of trailing stops,but its place can never be over-emphasized.

As for me,backtesting shows me that gbpusd must move atleast 40 pips in a given direction in any session.So,I build my trading styles around market open.One of the mysteries of life is that histories naturally repeat themselves.

Hi Folks

I have follwoing point about problem we see in Back testing is due to following

 

1. We all download history data from MetaQuest, regardless of our Broker.So technically, we are getting away from our broker data and testing on other data, and when we test it in real, the results are found different.
2. There are in-consistencies in some of the Brokers, which results in wrong results for our strategy as well.

My opinion, we could Back-test reliably if we use the feed from our broker only. And that could only be found, if our broker provide the history data, which majority of the broker do not. So the choice we have is only to back test around two months data that we find on our LIVE chart.

Export this data to CSV and then import it in Offline mode and test your strategy offline mode. This would give you at least a feeling that how your strategy would perform on your Broker.

I am open for suggestion and sharing my experience on this subject.

Regards
ForexZAK

Anyone have any comment about how news releases affect and possibly even corrupt back tested data.

Or in forward testing would a trader discount all news activity and follow the back tested method regardless?