How useful are EA backtest results? | Page 2 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Free Expert Advisor with positive backtest results 5 replies

Visual backtest result different from non-visual backtest 0 replies

MT4 EA - strategy tester - backtest results inaccuracies 15 replies

Unhappy Weird results in backtest build 200.. need help! 7 replies

Backtest results! 7 replies

Attachments: How useful are EA backtest results?
Exit Attachments

How useful are EA backtest results?

  • #21
  • Jan 6, 2015 3:43am Jan 6, 2015 3:43am
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting Chorlton
one question: From your initial idea conception to live trading, what steps do you personally take to validate a trading idea before trading real money?
It's now few years that all my testings are on LIVE account (real money). Of course the risk taken for testing isn't like when I trade ...
MT4/MT5 EAs/Indicators/Alerts coder
  • #22
  • Jan 6, 2015 5:07am Jan 6, 2015 5:07am
Quoting FerruFx
{quote} It's now few years that all my testings are on LIVE account (real money). Of course the risk taken for testing isn't like when I trade ...
Thanks for your comment FerruFx.

Although I understand the benefit of doing this I'm just a little uncomfortable about going directly to a live account (albeit a small account trading smallest possible lot sizes) simply due to the amount of time needed to provide any useful results as to whether an idea is worth pursuing further.
  • #23
  • Jan 6, 2015 6:55am Jan 6, 2015 6:55am
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting Chorlton
{quote} Thanks for your comment FerruFx. Although I understand the benefit of doing this I'm just a little uncomfortable about going directly to a live account (albeit a small account trading smallest possible lot sizes) simply due to the amount of time needed to provide any useful results as to whether an idea is worth pursuing further.
Yes I understand ... but I have far more convicence about the results given by live trading.

BTW, that's a great training for those who need to work their discipline ... which is definitively not involved in backtesting ...
MT4/MT5 EAs/Indicators/Alerts coder
  • #24
  • Jan 6, 2015 7:58am Jan 6, 2015 7:58am
Interesting results on the doji! I have developed EAs that show roughly 50% for dojis and most other candle stick patterns (Hammer, Harami, Morning/even star, engulfing, hang man, meeting lines, piercing lines etc). This is with filters for S&R as well as a few other simple indicators.

It wasnt until I did these tests that I realized my only way of learning the markets was understanding why the market actually moves.

Volume and Price!!! Buyers and Sellers!!!SUPPLY and DEMAND!!!
Learn, a forex trader must, unlearn and relearn he will.
  • #25
  • Jan 6, 2015 8:55pm Jan 6, 2015 8:55pm
Quoting FerruFx
{quote} You didn't get it ... Even 100% automated in Live like in the backtest, the results will be different. Don't need human intervention ... just that the market conditions are far different. You can use the EA on Live for 1 month ... then backtest this exact same period, you won't have the same results.
Hello, What I failed to mention yesterday is the following. 1) You are correct from what is back tested doesn't mean in live it would of worked. How many of you traded YEN pair when CNN said that the Nuc Power plant was about to explode. It caused what to occur to the YEN. For how long? The chart will show about 330 pips in 15 mins. Yet no order would be able to be activated. As the spread became 120 pips at one tick then 6 ticks on the next, and back and forth for 15 mins. A dead chart will never show you the ASK. Thus more then likely margin calling your account, and you being upset and revenge trading. You can counter that trap by the market by programming it to never open a pending (stop,limit) if the spread is more then X amount of pips. That also protects you from market gaps, and market moving shifts like NFP. You also hide your stop,limit orders. That is also possible to script into your EA. So if you know how to code, and know the counters of the market, you can lower your exposed currency amount.
  • #26
  • Jan 7, 2015 11:07pm Jan 7, 2015 11:07pm
A person’s opinion on the efficacy of backtesting will depend largely on their market philosophy.

Those who believe the market is predictable will naturally find some value in backtesting since they will assume that certain long-term trends/cycles will continue.

However, according to the random walk hypothesis, such an assumption is fundamentally flawed. Those who believe market prices are purely random and evolve according to a random walk will not find any value in backtesting. I am not sure to what extent traders on this forum subscribe to the random walk hypothesis, but I do not imagine there are very many since I see a lot of discussion regarding technical analysis.

The only way one would be able to effectively test a trading strategy would be if they ran the system through multiple stochastic/random walk simulations using Monte Carlo Method (to test for many different possible outcomes).

If you do it any other way you will likely fall victim to curve fitting.
"What glory would attend the discovery..."
  • #27
  • Jun 24, 2015 3:58pm Jun 24, 2015 3:58pm
Guys please tell me, if you are doing backtests, do you use Dukascopy data, or default Mt4 data?
  • #28
  • Aug 8, 2015 2:52am Aug 8, 2015 2:52am
Interesting thread, thought I would offer my 2 and a half cents.

Of course backtesting works, providing you can clone your ea's moves. Remember, over 100 trades, if your entry is on average out by even 1 pip to the ea's then thats in and out 200 pips lower right there, your trades have to match the ea or trade the ea itself. (bot trade)

If it worked well on backtest it will work well on live money account BUT somebody above mentioned the BE effect. What I call Backtest Exuberance Effect. This effect is when a trader finds a really great backtest result, 10,000 percent per week or something, then go live and immediately halve their money within a few days of trading. Then give up and probably kick a cat.

What is the BE effect and why does it occur so often. Everyone has experienced this and the first thought is you have a broker on the other side of your screen, playing your chart like a video game. Everyone immediately has that thought but after checking out other price sources (for the millionth time) you know thats not true...so what is it and how and why does it happen?

The simplest explanation is always the most probably correct one. You sit on your hands watching this remarkable ea that you so cleverly discovered, wondering what to do with it. Should you dare trade it? You have been here before and burned money, dont want that again. The simple answer is you wait until it runs through a maximum cycle up tick in the equity curve, this is precisely when the equity curve correction will return. But you dont think of that. Your emotions and ambition understandably carry you away and you put hard money down, just at the point when you should be expecting the correction right at its height of gains. You play the correction all the way down until you give up on it. Had you continued the correction would have completed and a new and stronger uptrend would have resumd.

Markets dont change that easy, what its doing now, going up and down is what it will do tomorrow. Unless somebody declares war on somebody or a change in that nature occurs, the market fairly is stable in its nature which ever market you favor.

The solution to the be effect is to wait intentionally for a decent correction in your equity curve, wait for the first few winning trades to start coming back, then put your money down.

cheers.
  • #29
  • Oct 21, 2015 9:41pm Oct 21, 2015 9:41pm
Quoting argofunds
They are useful. From them you can get several indications: If the system can generally be profitable What the profitability would be( winning v.s. losing trades) See your drawdown However, you cant rely on backtests 100%. In backtesting you get the best possible results, for instance, if one currency works better than other - it was just the best in that period, if the best results are in london session, again the same thing. Basically if the system was -15% over the year, but in december and july +40%, by adding the right filters, you would get,...
My question is how accurate is backtesting for the period tested. I.E., of you backtested an EA for 12 months, and it showed excellent profit, are you guys saying that would not have happened on a live account? If not why? (excluding the effects of swap, which can't be backtested)
Thanks.
  • #30
  • Oct 21, 2015 10:40pm Oct 21, 2015 10:40pm
  •  FerruFx
  • Joined May 2007 | Status: MT4/MT5 EAs/Indicators/Alerts coder | 6,542 Posts | Online Now
Quoting EventHorizon
{quote} ...are you guys saying that would not have happened on a live account?
Yes exactly.

You will never have in backtesting the real market conditions : spread, liquidity, etc ...
MT4/MT5 EAs/Indicators/Alerts coder
  • #31
  • Feb 3, 2016 4:48pm Feb 3, 2016 4:48pm
Hey guys, I need some help here.
I just don't understand what is wrong with this tester in mt4?
Here is what happened:
I'm doing some simultaneous tests (backtesting for the last 16 years), multiple algo systems (over 30 systems) each with hundreds of unique patterns, all with different variables. It's basically data mining operation using the default optimization feature in the tester........So as you know this takes a lot of processing power even on multiple CPUs, so I left the simulations to run for several hours...... And when I came back to check the results, I found something crazy! Out of thousands simulations(all pretty much showing useless results), one simulation had very interesting results:
Profit Factor 0.3
Win Rate 52%
RR 1 : 0.35
Expectancy per trade -19 pips
At this point I was like...,hmm this is interesting...
So I quickly wrote the strategy that is exact mirror image.(reversing the entry exit rules) and then started backtest for the last 10 years. I was shocked when the result came out
Oh boy,.....From 2005 to 2016 it turned 10 000 into over 144 000 000 . For a moment I tought to myself - this is it! I've found the holy grail.
Attached Image (click to enlarge)
Click to Enlarge Name: EA 5DDCHv3.png Size: 26 KB



But is this result real? Maybe I missed something?
The code seems fine, no errors, no error in the journal.....there is no mistake.....
The strategy is LFT(no more than 1 trade per day) the system is based only on D1 High, Low, Open, Close data.

And here comes the problem!
From 2005 - 2016 backtesting on control points results are just insane (1 440 000%) !
From 2015 - 2016 backtesting on control points, the result is over 200% for 1 year.
Same test from 2015 - 2016 backtesting on every tick (90% modeling quality) result is over 160% for the year.......so the difference between control points and all ticks is not very significant only 20%... I manually checked the charts, and all entries are almost identical!
But when I test on tick data from 2005 to 2016 the drawdown is 99% So the result is completely different? WHY? Why this happens? Any ideas?
Even if there is some deviation between the entry/exit points, they are not significant. Even testing with 10 pips spread it still gives positive result because the expectancy per trade is 15 pips.

So what is wrong here?
  • #32
  • Edited 5:40pm Feb 3, 2016 5:25pm | Edited 5:40pm
Quoting alphaomega
Hey guys, I need some help here. I just don't understand what is wrong with this tester in mt4? Here is what happened: I'm doing some simultaneous tests (backtesting for the last 16 years), multiple algo systems (over 30 systems) each with hundreds of unique patterns, all with different variables. It's basically data mining operation using the default optimization feature in the tester........So as you know this takes a lot of processing power even on multiple CPUs, so I left the simulations to run for several hours...... And when I came back to...

Hi A

Check that you do not have an entry and exit on the same day. If so, the method using control points can throw up very dodgy data. Control points is fine if you can avoid multiple events on the same bar, but get's dodgy for the shorter timeframe strategies.

The price between control points (open/high/low/close) as you know is random, so when you have a low and high on the same bar, you can't definatively tell which came first in time order. If your entry and exit is therefore on the same bar, this could be significant to the strategy as it may not accurately represent the trade process on that day. Also when using a low historic fixed spread which is not realistic, the result can be significantly material to the strategy particularly if the actual spread spans across multiple control points. This gave me some fantastic equity curves that got my hopes up........ until I forward tested them to find the result was totally bogus. :-(

This is because MT4 uses the lowest available timeframe to construct control points. Where you don't have low timeframe historic data, it will use the lowest available timeframe as a proxy to construct control points such as D1. The farther you test back in time, the less accurate the result if your strategy is dependent on an entry and exit on the same bar.

Check the difference in total trade number between tick by tick and control points over a historic timeframe (say 5-10 years ago) and see if any material difference shows up.
  • #33
  • Feb 3, 2016 6:44pm Feb 3, 2016 6:44pm
Since you are using D1 data you are probably testing in the tester on Period: Daily

If you have exits on the same day as entry using control and open data it's not going to factor in when you entered and what direction the price went after entry, it only assumes that if your target is met, it's a profit, however it doesn't know if your SL was hit first, since it is only seeing OHLC.

Try testing on a lower TF using Period: M1 or M5

And try testing variables for algo prices on M1 through D1 data instead of just D1.

The lower TF you drill down to the closer to "real" price action the testing will be.

You may not have all that 1min data, and it will take forever to test, but 5mn data or at least 15m is not half as bad and should be sufficient in most cases.

With 5m data tested, your results will be within 5mn of actual data at worst. Assuming your algo is properly written, your test results can mimic live results entry and exit and vice versa within a few ticks, less than a pip, assuming the individual trade is not on a news spike when the spread can widen 25.0 pips or more.
  • #34
  • Feb 3, 2016 7:00pm Feb 3, 2016 7:00pm
It depends on the system and how it trades exactly, but I would not rely on much from the backtester. If you do it should be with a large grain of salt and from high modeling quality.
Attached Image (click to enlarge)
Click to Enlarge Name: Untitled.png Size: 20 KB
  • #35
  • Feb 4, 2016 4:41am Feb 4, 2016 4:41am
Quoting Ponzi Jr
{quote} Why not demo? I would the result not be the same?
Because you have not traded until you have ridden the rage-train to irresponsible risk town.
Price-Action Extremist
  • #36
  • Aug 29, 2016 1:21pm Aug 29, 2016 1:21pm
Quoting zznbrm
Back testing can be a very useful exercise...but you do need to know what you are doing. Remember, during the back test, you want to simulate a live environment as close as possible. You should simulate price...you should simulate different market conditions...you should simulate slippage...you should simulate internet connectivity issues...you should simulate rapid price changes...you should simulate periods of low/high volatility...you should simulate latency. The more of these "intangibles" you can simulate, the closer to the live environment...
This is precisely what I have been doing with my EA. I spent considerable time back testing, but didn't know if I could trust the results. I decided to start running it live and then back test the same period and compare the results - it's the only way I could determine if the back test results were accurate. The results aren't exact due to slippage and what not, but trades that won, won, and trades that lost, lost. That is all I need to know. A few pips difference here and there don't bother me. The fact the live results produce profitable trades when it is supposed to gives me confidence in the back test results. Now, if there were discrepancies between the live and back test results like a winning trade on back testing that wasn't a winning trade live then I could not trust the back testing. So far 100% of my live results - in terms of trades won and lost - match the back testing results for the same period.
  • #37
  • Aug 29, 2016 10:50pm Aug 29, 2016 10:50pm
I disagree with a lot of 'live test only' crowd in here ... if your bot can't profit 2006 - 2011 there is no point running it live, simple ... if you're bot did well in this period and in the following 5 years, you're probably on to something.

Also I've often produced back test results that mimic a forward test with very little divergence.

I think we're people go wrong is usually the spread - they either use the standard setting (current spread) or even worse the broker's advertised minimum ... you really need to do backtesting with an average spread + a point or so if you want realistic results. Slippage on other hand is an issue and much harder to simulate - but this will effect scalpers more than 1h/4h/daily bots.

Quoting Spoof
{quote} This is precisely what I have been doing with my EA. I spent considerable time back testing, but didn't know if I could trust the results. I decided to start running it live and then back test the same period and compare the results - it's the only way I could determine if the back test results were accurate. The results aren't exact due to slippage and what not, but trades that won, won, and trades that lost, lost. That is all I need to know. A few pips difference here and there don't bother me. The fact the live results produce profitable trades when it is supposed to gives me confidence in the back test results. Now, if there were discrepancies between the live and back test results like a winning trade on back testing that wasn't a winning trade live then I could not trust the back testing. So far 100% of my live results - in terms of trades won and lost - match the back testing results for the same period.
Exactly.
  • #38
  • Feb 20, 2017 6:46am Feb 20, 2017 6:46am
Quoting zznbrm
Back testing can be a very useful exercise...but you do need to know what you are doing. Remember, during the back test, you want to simulate a live environment as close as possible. You should simulate price...you should simulate different market conditions...you should simulate slippage...you should simulate internet connectivity issues...you should simulate rapid price changes...you should simulate periods of low/high volatility...you should simulate latency. The more of these "intangibles" you can simulate, the closer to the live environment...
Very good concept about backtesting. Simulation of the live environment is what actually happening.

But what do you think current possibilities could offer good simulation? In MT4 or cTrader is there good options to backtest? If there would be better solution (I mean faster, easier, more effective, more complex) would people use it?
  • #39
  • Feb 20, 2017 9:17am Feb 20, 2017 9:17am
I think back testing enables the EA developers to make sure that the strategy they have coded do work accordingly..
  • #40
  • Feb 20, 2017 6:13pm Feb 20, 2017 6:13pm
One thing I notice on Optimisied baktesting an EA on MT4, is that over a year of backtesting, you may have had a decent one with DD of say 40%. But if you pick the best backtest, and START it at a date into the year, not at the begining of the 12 months optimisation you did, the account might crash. This happens to me even though I have compounding on the EA, which should avoid later false readings. It seems that just because you are out of sinc of the optimised test, it crashes. So after optimisation, Get the best result, and backtest it starting from each month into the 12 months to be sure.
0 traders viewing now