How useful are EA backtest results? | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Free Expert Advisor with positive backtest results 5 replies

Visual backtest result different from non-visual backtest 0 replies

MT4 EA - strategy tester - backtest results inaccuracies 15 replies

Unhappy Weird results in backtest build 200.. need help! 7 replies

Backtest results! 7 replies

Attachments: How useful are EA backtest results?
Exit Attachments

How useful are EA backtest results?

  • #1
  • First Post: May 29, 2014 7:30pm May 29, 2014 7:30pm
I'm currently trying to learn how to code EA's. In learning, I've created various scripts that perform very simple tasks. One of these entered a buy trade everytime a Doji candlestick appeared, with a SL of 25 pips and a TP of 50 pips. When I ran it on the tester though, it gave me a nice steady equity curve (seen in the image below):
Attached Image (click to enlarge)
Click to Enlarge Name: Equity curve.PNG Size: 34 KB


Obviously you'd be mad to trust a single penny to such a system, so what should be made of the fact that it actually seems to have turned a steady profit? Does this mean that the tester is not a very good indicator of whether an algo is likely to be profitable?
  • #2
  • May 29, 2014 7:33pm May 29, 2014 7:33pm
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting Ponzi Jr
Does this mean that the tester is not a very good indicator of whether an algo is likely to be profitable?
The purpose of the backtester is only to check that your strategy is following your rules ... nothing else ... and definitively not the profitability.

The only valid result about profitability is forward on a live account.
MT4/MT5 EAs/Indicators/Alerts coder
  • #3
  • Edited 11:33am May 30, 2014 4:50am | Edited 11:33am
Quoting FerruFx
The purpose of the backtester is only to check that your strategy is following your rules ... nothing else ... and definitively not the profitability. The only valid result about profitability is forward on a live account.
Thanks for the response. How long would you recommend running an EA live for (on a demo account, I assume) before deciding that the results are meaningful enough to put real money on it?
  • #4
  • May 30, 2014 4:57am May 30, 2014 4:57am
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting Ponzi Jr
{quote} Thanks for the response. How long would you recommend running an EA live for (on a demo account, I assume) before deciding that the results are meaningful enough to put real money on it?
The best would be to go Live directly but with a tiny account.

Of course once the backtesting is done and rules are confirmed to be coded right.
MT4/MT5 EAs/Indicators/Alerts coder
  • #5
  • May 30, 2014 11:29am May 30, 2014 11:29am
Quoting FerruFx
The best would be to go Live directly but with a tiny account.
Why not demo? I would the result not be the same?
  • #6
  • May 30, 2014 1:08pm May 30, 2014 1:08pm
Backtesting is a trap if you believe it too much. I have created agro futures trading systems years ago which showed better than +200% profit per year, over a period of 10 years. The system completely crashed and burned the 2nd month I used it on real money. It experienced its worst historical drawdown right at the moment I attempted to actually use it. Other similar things seem to happen to many other people.

A backtest is a summary of past luck, not a validation of a technique. You can backtest to get some insights, but until you put your ideas into action with real money you know nothing about what is about to happen.

In general, beware of ANY historical results. Even top gun traders who have made millions for the past 20 years can crash and burn the second you invest with them.

"Past performance is NOT a guarantee of future results" is the #1 thing for newbs to remember. That applies to all trading systems and all traders, no matter how excellent they appear. You will always be at risk. Get used to that. Get comfy with it.

It is a better use of your time to learn to trade rather than learning to create mechanical systems. Mechanical systems all fail eventually.
  • #7
  • May 30, 2014 7:02pm May 30, 2014 7:02pm
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting Ponzi Jr
{quote} Why not demo? I would the result not be the same?
Because that's the only "real" trading environement.
MT4/MT5 EAs/Indicators/Alerts coder
  • #8
  • May 30, 2014 7:16pm May 30, 2014 7:16pm
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting Fractalist
Backtesting is a trap if you believe it too much. I have created agro futures trading systems years ago which showed better than +200% profit per year, over a period of 10 years. The system completely crashed and burned the 2nd month I used it on real money. It experienced its worst historical drawdown right at the moment I attempted to actually use it. Other similar things seem to happen to many other people. A backtest is a summary of past luck, not a validation of a technique. You can backtest to get some insights, but until you put your ideas...
Excellent ...
MT4/MT5 EAs/Indicators/Alerts coder
  • #9
  • May 30, 2014 7:16pm May 30, 2014 7:16pm
Quoting Ponzi Jr
{quote} Why not demo? I would the result not be the same?
Typically, I wouldn't disagree with Ferru...and really, I'm not now...

But, I do have a bit of a caveat...

It's true that backtests are completely 100% garbage as far as statics and data on profitability is concerned and that the only way to truly know how any EA will react to market is in a live environment...

Demo?

Well...ya really have to work at this, but it can be done!

I have tested scads of brokers for live vs demo feeds and I believe what Ferru is saying, is that demo and live do not run the same, so expectations will vary from demo to live. Along with this, Demo Servers have issues and these may not line up the same as issues you may suffer from with live accounts. Ya know...where price isn't available, or widening spread, or broker just goes off line for no apparent reason...that kind of stuff...

I have only found 1 broker that I trade with, and have run live side by each on demo and had the results be pretty close to spot on...the differences that I noticed from live to demo, were nothing more than a commissions difference. As I have signed up through an IB, my commissions on live are cheaper than what is on demo...besides this...the bulk of the trades line up...spot on...

It can be done..but even this step, takes some effort.
Money Can't Buy Happiness. Poverty Can't Buy SHIT! You Choose!
  • #10
  • Jan 5, 2015 2:49am Jan 5, 2015 2:49am
Hello All,

Although I agree with the comments below, for me the benefit of backtesting is that it allows one to quickly assess a particular trading idea and decide whether it is worth pursuing further. Some initial ideas which I thought would be successful proved to be complete rubbish when backtested and exposed to a reasonable sample of historical data (5 - 10 years). In contrast, other ideas which I thought probably wouldn't work actually showed some merit.

I agree that nothing beats testing a strategy on a live account but for me the drawback with this approach is time and the sample size of data that you are running it on. If, for example, you had a strategy for one particular FX pair that traded daily bars, if you tested it on a live account, after 1 year real-time you would have only tested the strategy on just 1 year of data (365 bars). In this example, given the relative small data size that the system has been exposed to I'm still not sure whether I would then have enough confidence to trade the system with a reasonable lot and account size.

Leading on from this, I would really be interested in how others validate their own trading ideas. I'm more interested in those who base their strategies on 1 to 4 hour bars or higher as opposed to say, scalpers who are trading very small timeframes and can therefore test on a larger data size for the same testing period (1 min bars compared to daily bars over the same 1yr testing period results in a much large testing sample of data to draw a conclusion from).

From initial idea conception to live trading, what testing/validation steps do you take?

Any feedback welcome....

Cheers,

Chorlton



Quoting FerruFx
{quote} The purpose of the backtester is only to check that your strategy is following your rules ... nothing else ... and definitively not the profitability. The only valid result about profitability is forward on a live account.
Quoting Fractalist
Backtesting is a trap if you believe it too much. I have created agro futures trading systems years ago which showed better than +200% profit per year, over a period of 10 years. The system completely crashed and burned the 2nd month I used it on real money. It experienced its worst historical drawdown right at the moment I attempted to actually use it. Other similar things seem to happen to many other people. A backtest is a summary of past luck, not a validation of a technique. You can backtest to get some insights, but until you put your ideas...
  • #11
  • Jan 5, 2015 3:42am Jan 5, 2015 3:42am
They are useful. From them you can get several indications:
If the system can generally be profitable
What the profitability would be( winning v.s. losing trades)
See your drawdown

However, you cant rely on backtests 100%. In backtesting you get the best possible results, for instance, if one currency works better than other - it was just the best in that period, if the best results are in london session, again the same thing. Basically if the system was -15% over the year, but in december and july +40%, by adding the right filters, you would get, that the system produced 80% over the year and you would assume that it will produce similar results in future.

My point is that it's useful, but you have to think about what filters you are adding and think if it could be just luck. If the system shows profitability and you have thought it over, that it's at least a bit accurate, then you can try it on demo or small live account for a few months. trade it, you will get used to it and make adjustments accordingly.
  • #12
  • Jan 5, 2015 6:18am Jan 5, 2015 6:18am
I would say back test results are very useful, especially if the system is which is tailored towards scalping. Many people say that back test is useless, primarily because "The results are so good in back test, but in live it isn't." Although that is a very TRUE statement, we as a trading community do not look at the reason as to why that holds true. As to why when a system is placed on a live account, all of a sudden it bombs. The answer is rather simple as to the latter. Human intervention.
Most, if not 100% of the times, we over ride our EA's with our bias opinion to what a currency is going to do, and then all of a sudden we get into problems. We try to pick tops and bottoms, and simply place trades that the EA was never designed to take and that is where is the problems start.
  • #13
  • Jan 5, 2015 6:46am Jan 5, 2015 6:46am
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting KmBioDmoneda
I would say back test results are very useful, especially if the system is which is tailored towards scalping. Many people say that back test is useless, primarily because "The results are so good in back test, but in live it isn't." Although that is a very TRUE statement, we as a trading community do not look at the reason as to why that holds true. As to why when a system is placed on a live account, all of a sudden it bombs. The answer is rather simple as to the latter. Human intervention. Most, if not 100% of the times, we over ride our EA's...
With all due respect, I disagree 100% with your statement ... You are comparing backtest with an EA and 'manual' trading ... while the point (at least mine) was to compare both 100% automated. No human intervention here.
MT4/MT5 EAs/Indicators/Alerts coder
  • #14
  • Jan 5, 2015 8:49am Jan 5, 2015 8:49am
Quoting FerruFx
{quote} With all due respect, I disagree 100% with your statement ... You are comparing backtest with an EA and 'manual' trading ... while the point (at least mine) was to compare both 100% automated. No human intervention here.
It would appear to me that many people post backtest results of an EA, then all of a sudden when placed on a real account, the account isn't able to come close to the backtest results. I believe that happens because although the EA may be running, the account manager places manual trades, which end up bombing the account. At which point you would not be able to know if that one trade was caused by the EA, or by manual trading.
  • #15
  • Jan 5, 2015 8:55am Jan 5, 2015 8:55am
Not very useful. I remember when I made my first EA and saw %100,000. Thought I was going to be rich, then I live tested my EA...
  • #16
  • Jan 5, 2015 9:11am Jan 5, 2015 9:11am
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting KmBioDmoneda
It would appear to me that many people post backtest results of an EA, then all of a sudden when placed on a real account, the account isn't able to come close to the backtest results. I believe that happens because although the EA may be running, the account manager places manual trades, which end up bombing the account. At which point you would not be able to know if that one trade was caused by the EA, or by manual trading.
You didn't get it ... Even 100% automated in Live like in the backtest, the results will be different. Don't need human intervention ... just that the market conditions are far different.

You can use the EA on Live for 1 month ... then backtest this exact same period, you won't have the same results.
MT4/MT5 EAs/Indicators/Alerts coder
  • #17
  • Jan 5, 2015 9:21am Jan 5, 2015 9:21am
Quoting Ponzi Jr
I'm currently trying to learn how to code EA's. In learning, I've created various scripts that perform very simple tasks. One of these entered a buy trade everytime a Doji candlestick appeared, with a SL of 25 pips and a TP of 50 pips. When I ran it on the tester though, it gave me a nice steady equity curve (seen in the image below): {image} Obviously you'd be mad to trust a single penny to such a system, so what should be made of the fact that it actually seems to have turned a steady profit? Does this mean that the tester is not a very good indicator...
good points have already been discussed here. backtest has only limited value. it can act as a filter (i.e. ignore strategies that dont work even in backtest) but wont guarantee profit in live. also there is overfitting bias that comes when doing many backtests (and even when doing forward tests multiple times).

but looking at you graph here, i think it is a programming error. taking every trade on doji cant possibly produce such an equity curve even in backtest. i think there is look ahead bias or some other very fundamental error. check your code.
Think : Trade : Live Life
  • #18
  • Edited 9:43am Jan 5, 2015 9:24am | Edited 9:43am
Back testing can be a very useful exercise...but you do need to know what you are doing. Remember, during the back test, you want to simulate a live environment as close as possible. You should simulate price...you should simulate different market conditions...you should simulate slippage...you should simulate internet connectivity issues...you should simulate rapid price changes...you should simulate periods of low/high volatility...you should simulate latency. The more of these "intangibles" you can simulate, the closer to the live environment you will get. The point here is that you want to bombard your system with everything you can think of to see how it handles it. You want to stress it to the extremes to see where it breaks down. Again, much easier/quicker back testing than live.

A good back test has to start with good data. For most systems this means you need accurate tick data. Without good data your back test is not going to properly simulate the live environment and your results will not be optimal.

One of the biggest benefits of a back test is that it allows you to determine the efficiency (profitability) of an automated system during different market conditions much quicker than live trading. Now, of course the age old adage that "past performance does not guarantee future results" is true...everyone knows that...it's been said a million times. But the back test can give you a good idea of how the system has behaved in different circumstances.

Now...how accurate can back testing be? Well, the answer depends on how accurately you can simulate the live environment. Here is a great exercise that I learned a while ago. This exercise will allow you to determine how efficient your back testing is and teach you how to optimize your back tests (no, not curve fitting...optimizing). Having used this method for a few years now, I usually see a 98+% accuracy in back tests results compared to live results.

Run your automated system in a live environment for some period of time (could be a few weeks, months, whatever). Keep a good record of your trades and the performance of the system. Then, get your hands on some good data and back test your system over the same period of time. You can then compare the back test results to your live results and take note of the discrepancies. You can use these discrepancies to improve your back testing so that it can be as accurate as possible. When you have done this a few times over different periods of time and over a wide range of market conditions, you should have a solid back testing mechanism that should give you very accurate results.

Remember...the more market conditions you can simulate in a back test, the more accurately you can predict future results with the system. Most people do not take the time to identify the different market conditions and then simply don't back test enough of those conditions to make the back test as accurate as it could be.
  • #19
  • Jan 5, 2015 9:35am Jan 5, 2015 9:35am
Quoting FerruFx
{quote} You didn't get it ... Even 100% automated in Live like in the backtest, the results will be different. Don't need human intervention ... just that the market conditions are far different. You can use the EA on Live for 1 month ... then backtest this exact same period, you won't have the same results.
It would appear that the creator has not been able to PROGRAM into his EA, the ability to trade only certain market conditions. You simply use....... say... MA.. Then take the avg of the price over 30 mins of a 5m candle, and only wager on those formations, it doesn't matter of the market spikes 100 pips in 1 second, you are not exposed to the market. It only activates over the engulf of a candle within the avg of time. Break is defined by PRICE in relation to TIME..... Not just price. What is the price without time? No time and you have no X axis on your chart am I correct. Most people hold over night, you never hold over night. Why should you? If you SL why aren't you taking the reverse trade?
  • #20
  • Jan 6, 2015 3:17am Jan 6, 2015 3:17am
All the replies so far about backtesting are valid and I think most would agree that there can be a place for it but it does have its limitations.


Rather than create a new thread, I would like to continue with this thread and ask everyone who has contributed one question:


From your initial idea conception to live trading, what steps do you personally take to validate a trading idea before trading real money?

I'm really interested in knowing how you create the necessary confidence to trade a particular idea....
1 trader viewing now