Why does Backtesting differs so much ? | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Loss calculation does not match loss in backtesting MT4 - why? 6 replies

How much backtesting is enough? 41 replies

Does backtesting work? 58 replies

SL and close price differs 2 replies

does backtesting include pip spread? 1 reply

Attachments: Why does Backtesting differs so much ?
Exit Attachments

Why does Backtesting differs so much ?

  • #1
  • First Post: Dec 5, 2009 12:07pm Dec 5, 2009 12:07pm
HI all


i just backtested an EA .. on MT4 alpari demo and on MT4 FxPro demo

the results a totaly different ? but why is this ?

is it because of the spread ?

because iam green on MT4 alpari (30 pips spread)
+144K net profit

and the bankroll busted @ FXPro (50 pips spread)
-9K net profit

does 20 pips do such a damage ? .. or is the backtesting that flawed ?


thnx cheers
  • #2
  • Dec 5, 2009 12:32pm Dec 5, 2009 12:32pm
It can take a lot less than 20 pips to go from taking profit (TP) to a SL taking you out of a trade.

dave
Is it Break Time yet?
  • #3
  • Dec 5, 2009 12:33pm Dec 5, 2009 12:33pm
here are screenshots from the tests

Graphic Alpari

http://a.imagehost.org/t/0760/GraphAlp.jpg

Report Alpari

http://a.imagehost.org/t/0954/RepAlp.jpg

Graphic FxPro

http://a.imagehost.org/t/0662/GraphFX.jpg

Report FxPro

http://a.imagehost.org/t/0171/repFX.jpg
  • #4
  • Dec 5, 2009 12:34pm Dec 5, 2009 12:34pm
Quoting dkhinckley
It can take a lot less than 20 pips to go from taking profit (TP) to a SL taking you out of a trade.

dave

mate i dont use TP points and no SL points only Trailing stops = 30pips

is it still possible then ?

thnx
  • #5
  • Dec 5, 2009 12:44pm Dec 5, 2009 12:44pm
Without tick simulation every backtest is pretty much useless.
  • #6
  • Dec 5, 2009 12:53pm Dec 5, 2009 12:53pm
Quoting Tjpld
Without tick simulation every backtest is pretty much useless.

HI , i will get back with tick by tick simulation.. ASAP

cheers
  • #7
  • Dec 5, 2009 1:04pm Dec 5, 2009 1:04pm
Quoting PrymeTyme
HI , i will get back with tick by tick simulation.. ASAP

cheers
As for your other question:

Your Trailing Stop is 30 Pips. Your spread difference between both brokers is 2/3 of your Trailing Stop it's obvious that it will effect your results.
  • #8
  • Dec 5, 2009 1:18pm Dec 5, 2009 1:18pm
Quoting Tjpld
Without tick simulation every backtest is pretty much useless.
This statement in its absoluteness is wrong. Every strategy that doesn't use Ticks (a strategy that will only trade or place or change orders on bar open) can very well be tested in open-prices-only mode. The only thing that could go wrong here would be bars big enough to hit SL and TP within the same bar.

However, the thread starter uses a trailing stop which is calculated on every tick, so this must be tested with ticks. Metatrader offers no real ticks for backtest, so the backtest would most likely be wrong anyways.

The profit curve on the alpari backtest looks too good to be true, so it can safely be asumed to be wrong.
  • #9
  • Dec 5, 2009 1:58pm Dec 5, 2009 1:58pm
HI All


ok here is a tick by tick backtest .. i used a different timeframe this time as i have to test this EA anyhow..


but waht teh test on alpari produced is still to good to be tru ..

i mean look at the steep consistant balance line.. it cant be


this time FxPro was profitable aswell . but also pretty good ... but still

i cant belive these results are legit .. seems like backtesting is useless .. because of inacurate outputs..

waht u guys think ?

Graphic Alpari

http://a.imagehost.org/t/0960/GraALP.jpg


Report Alpari

http://a.imagehost.org/t/0669/RepALP.jpg


Graphic FxPro

http://a.imagehost.org/t/0977/GraFX.jpg

Report FxPro

http://a.imagehost.org/t/0774/repFX.jpg


greetz
  • #10
  • Dec 5, 2009 2:28pm Dec 5, 2009 2:28pm
Quoting PrymeTyme
seems like backtesting is useless
1. You will always have weekend spreads when backtesting at weekends (make a copy of the metatrader folder when market is opened and dont login anymore with this copy of mt4 to have a version with "normal" spreads over the weekend for backtest).

2. Backtesting EAs with trailing stops tend to be extremely inaccurate.

Try to change the strategy to act on candle open only and don't use trailing stops at all. This kind of EAs will have much more realistic backtests.

PS: I still haven't had the nerves and the time to care about the thing you PM'd me last week. May take some more time.
  • #11
  • Dec 5, 2009 3:27pm Dec 5, 2009 3:27pm
Quoting PrymeTyme
HI All


ok here is a tick by tick backtest .. i used a different timeframe this time as i have to test this EA anyhow..


but waht teh test on alpari produced is still to good to be tru ..

i mean look at the steep consistant balance line.. it cant be


this time FxPro was profitable aswell . but also pretty good ... but still

i cant belive these results are legit .. seems like backtesting is useless .. because of inacurate outputs..

waht u guys think ?

Graphic Alpari

http://a.imagehost.org/t/0960/GraALP.jpg


Report...
To get better Modelling Quality you have to get all sub Timeframes that are in the test. For H1 test you need complete M30 M15 M5 and M1 Data.

If you have M1 Data since March 2009 you will get 90% MQ if you test from March 2009. If you test from Febuary 2009 you will only have M5 M15 and M30 Data for that month so the MQ will drop below 90%. 90 is the best you can get with the backtester.

To get the data simply zoom out on your chart as much as you can disable auto scroll and press and hold Pos 1 on your keyboard.
  • #12
  • Dec 5, 2009 3:45pm Dec 5, 2009 3:45pm
Quoting 7bit
1. You will always have weekend spreads when backtesting at weekends (make a copy of the metatrader folder when market is opened and dont login anymore with this copy of mt4 to have a version with "normal" spreads over the weekend for backtest).

2. Backtesting EAs with trailing stops tend to be extremely inaccurate.

Try to change the strategy to act on candle open only and don't use trailing stops at all. This kind of EAs will have much more realistic backtests.



Hi mate



thnx for ya replys , and tips i will test the EA further ..


Quote
PS: I still haven't had the nerves and the time to care about the thing you PM'd me last week. May take some more time.

no problem , take your time and thank you .. cya soon

greets
  • #13
  • Dec 5, 2009 3:46pm Dec 5, 2009 3:46pm
Quoting Tjpld
To get better Modelling Quality you have to get all sub Timeframes that are in the test. For H1 test you need complete M30 M15 M5 and M1 Data.

If you have M1 Data since March 2009 you will get 90% MQ if you test from March 2009. If you test from Febuary 2009 you will only have M5 M15 and M30 Data for that month so the MQ will drop below 90%. 90 is the best you can get with the backtester.

To get the data simply zoom out on your chart as much as you can disable auto scroll and press and hold Pos 1 on your keyboard.

Thnx Tjpld ,, i wil take care of this in the next test
cheers
  • #14
  • Dec 6, 2009 10:16am Dec 6, 2009 10:16am
Metatrader backtesting assumes perfect fills at the Bid or Ask and this is not always the case in live trading

Backtesting in Metatrader is done using interpolated ticks and it's quite unreliable.


Especially scalpers EA that close large numbers of trades with less than 20 pips of profit or loss produce grossly unrealistic profits in backtests which will perform worse and not profitable at all in live trading.


http://www.forexfactory.com/attachme...1&d=1259677120


Forward test is still the best method for EA testing, but if you want to do backtesting, you should do it with tick data to get 99% modeling quality.

Read the attach file and visit these links to know how to do it :

http://www.cubesteak.net/mt4-multi-b...-data-project/

http://www.forex-tsd.com/expert-advi...ata-tross.html
Attached File(s)
File Type: doc Backtesting with 99 percent modelling V1.doc   345 KB | 1,146 downloads
  • #15
  • Dec 6, 2009 10:25am Dec 6, 2009 10:25am
Metatrader backtest interpolation approach
Attached Image (click to enlarge)
Click to Enlarge Name: flaw.jpg Size: 118 KB
  • #16
  • Dec 6, 2009 10:50am Dec 6, 2009 10:50am
HI All


Thnx for Pointing me in the right direction for optimizing the backtest..

@wieb thnx .. i will do the next backtest according to your links next weekend.

i cant forward test it cause i have only time to get to my pc at weekends.. cause of the work i have .. iam only at home on weekends.. for at least the next 4 weeks..

anyway i did some new backtests wich gained mostly 90 percent modeling..

pair EUR/USD trade 0.1 lot size @FxPro 30 pips spread

Tick by tick modeling

No stop loss , take profit and trailing stops where used
the EA simply entered and exit a trade when system indicated it
Trend following system

from 2000.1.1 to 2009.12.6

EUR/USD Daily

http://a.imagehost.org/t/0134/EURUSDdaily.jpg

Daily Report

http://a.imagehost.org/t/0861/EURUSDdailyRep.jpg

EUR/USD H4

http://a.imagehost.org/t/0910/EURUSDH4.jpg

H4 Report

http://a.imagehost.org/t/0225/EURUSDH4rep.jpg

EUR/USD H1

http://a.imagehost.org/t/0793/EURUSDH1.jpg
  • #17
  • Dec 6, 2009 10:50am Dec 6, 2009 10:50am
H1 Report

http://a.imagehost.org/t/0398/EURUSDH1rep.jpg
  • #18
  • Dec 6, 2009 1:13pm Dec 6, 2009 1:13pm
Heya All


I have build this EA based on this system

http://www.forexfactory.com/showthread.php?t=203433

with this web based EA builder
http://sufx.core.t3-ism.net/ExpertAdvisorBuilder/

and it looks like it has merith .. but several options i would like to include cant be put in that simply and iam looking for someone who is interrested in finishing the EA , there are no rules set in stone either and any good advice to get the best out of it is appreciated ..

so if ya interrested drop me a PM

greetz
  • #19
  • Dec 6, 2009 1:36pm Dec 6, 2009 1:36pm
Do your backtest during market times. Otherwise you will have redicoulus spreads. With 30 Pips Sprads on FxPro do you mean 30 5 digit Pips? so 3.0 Pips actually? The backtester always takes the current spread for its test.
  • #20
  • Last Post: Dec 6, 2009 8:19pm Dec 6, 2009 8:19pm
Quoting PrymeTyme
HI All

Thnx for Pointing me in the right direction for optimizing the backtest..

@wieb thnx .. i will do the next backtest according to your links next weekend.

if your EA has worse performance on backtest, definitely it will not survive on forward testing.

But if it profitable on backtests, its only the indicate that your EA might be profitable on forward test as long as it not has very tight SL and TP (i.e. scalpers EA).

If your EA is trend follower and has good enough performance then spreads is not a big problem.

Only scalpers EA which has very tight SL and TP consider spread differences.


After successfully forward testing on demo, do forward test again on small real mini account.

This again give different results than demo forward testing.

Always remember, market changes everyday and it will effect to your EA performance.

The good EA is the EA which always profitable no matter the market condition.
0 traders viewing now