MT4 Backtesting Inaccuracies

So I'm back-testing an EA that I wrote, within the MT4 'Strategy Tester'. Without getting into details of the code, I have my EA running on hourly 1.5 years in Strategy Tester that will consistently give me the same very positive result when using particular input settings. Now when I go into the input settings and make 1 single minute adjustment to a setting that should not result in any significant change from the prior settings (the small adjustment is to do with how much the trailing stop loss trails by once the profit is already locked in, which I'm only changing by 1%) -- the end result is drastically changed, to the point that it doesn't even finish the test because the account is wiped barely 1 month in.

Now I've found this kind of thing to happen a lot when making seemingly minute adjustments to input settings or code, where you'd really not expect any significant change in the final test result.

I'm still open to the idea that there is perhaps some logical reason for it (to do with input settings/code), but more and more I'm becoming suspicious of the reliability of the MT Strategy Tester itself.

Just wondering if many others here who do a lot of back-testing within MT4 with their own EAs also notice this?

Bump. Anyone??

yes and it is even worse if you BT during market close on weekends. Even if you manually input weekday spread values

my 2 cents? Treat BT as a way to prove your EA working as expected. If you want a brief idea of profitability, run the BT at same setting for a few times across the weeks. Also best to do some manual backtest to cross check results.

How it actually perform can only be proven in forward test. So I usually put it on demo (spend a few bucks on VPS, see it as investment) and run for a month or so. If the stats look fine, then I put it in small Live acc with $1000 or less (whatever you feel comfortable to blow, if not winning) to keep it running and collect more data.

[quote=navieko;13240720]So I'm back-testing an EA that I wrote, within the MT4 'Strategy Tester'.

Yes, ditch the MT4 tester because it’s just inaccurate. Many Algo trader has given up trading EA’s just because of the tester, hours and hours of wasted runs only to find the draw down figure is 100% out and two runs done hours apart often give different results. Demo trade 5-8 versions of a EA after no more than a weeks testing is a good option.

[quote=navieko;13240720]So I'm back-testing an EA that I wrote, within the MT4 'Strategy Tester'. Without getting into details of the code, I have my EA running on hourly 1.5 years in Strategy Tester that will consistently give me the same very positive result when using particular input settings. Now when I go into the input settings and make 1 single minute adjustment to a setting that should not result in any significant change from the prior settings (the small adjustment is to do with


And must use 99% data. Free one from Stratquant is pretty good but Birts Data Suite better

MT4 Strategy Tester has been very helpful to me. I used it to discover lots of market statistics, and to junk strategies whose optimization results are not impressive.

I've experienced one of the things Rick mentioned- getting a different result when run at a later date over the same range of time. I wasn't sure why, since I often make changes to the code between optimizations. Perhaps, what you're experiencing precisely.

A few practices I perform to improve the situation:

I first ensure the history data is downloaded for the period before I start. I must be able to view the entire period on the chart when I scroll back using the Home or Page Up/Page Down key, while the chart is zoomed to the smallest bar size (so it can scroll faster).

I deleted the content of cache folder under the tester folder each time before running optimization, just to be sure. It sometimes makes a difference in the result.

I would sometimes delete the history file (.fxt) in the tester folder as well before starting.

If the strategy references other instruments history data, (say, I'm trading EURJPY using OHLC data or EMA of EURJPY, USDJPY and EURUSD), I will extract the data using print statement in a script. I print the Bar Time (iTime) beside each pair's data on the same line, then import into spreadsheet and check that there are no missing bars. The BarTime must be the same for each line/bar.

Alao, if I download history data from a different broker, I would get a very different result. Therefore I seek to generalize the Optimization results I obtain.

I also have reason to believe that the history data sometimes change after I've started optimization. (I run Optimization on VPS, so it could have downloaded new data at some time.)

I should expect double the drawdown seen in tester report when I run it live.

I think there's fraud behind MT4 strategy tester itself. I wonder if MT5 is any better.

I once tried to find out who owns MetaQuotes and which country it's based. The and might interest you.

//------

odds are the trouble lies in the code/logic associated with that change or the cascading effect small changes can have.....

the mt4 backtester is reliable when back testing code that is back testable..... some ea's are not..... some ea's are.....

it's the code.....h

For this type of tests, i totally agree with @tommydoginti - a VPS should be used to ensure uninterrupted run of the tests!

Was there once, trying to use broker's data. So many issues with incomplete/inaccurate/limited data. If you Google and read some of the articles, the historic data doesn't carry the tick by tick data, so MT4/Strategy Tester creates synthetic/inaccurate ticks for smaller timeframes

The conclusion I came to was that I must use accurate data. I chose Tick Data Suite. It's really a slick product. Tick data carries the spread and also it will show slippage on stop orders and also when you exit the market. The difference in profitability running it through TDS and broker data, especially on older data, is massive.