MetaTrader BackTesting, Reliable or Not?? | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Reliable Volume data feed for Metatrader 14 replies

How to get 99% tickdata for metatrader backtesting? 30 replies

MetaTrader Expert Advisor BUILDER for backtesting and auto trading 4 replies

Seeking for a reliable CFD broker using MetaTrader 2 replies

metatrader- bad backtesting or bad broker quotations? 0 replies

Attachments: MetaTrader BackTesting, Reliable or Not??
Exit Attachments

MetaTrader BackTesting, Reliable or Not??

  • #1
  • First Post: Edited 10:43am May 30, 2006 10:41am | Edited 10:43am
I have noticed quite a lot of people are using MetaTrader 4. I am sure most people who use MetaTrader and Strategy Tester to back test EA’s would already now how to import good quality one minute data. However it remains a tedious process but it is worth it to increase the modelling quality. Attached is a pdf file that guides you through the whole process. I was surprised myself when comparing the Alpari data and the demo feed. Worth a look. This link is just the html version of the document:

Strategy Testing in MT4

I guess it begs me to ask the question how reliable is back testing?
Attached File(s)
File Type: pdf Strategy Testing in MetaTrader 4.pdf   452 KB | 5,854 downloads
"a fool and their money will easily part"
  • #2
  • May 30, 2006 12:09pm May 30, 2006 12:09pm
Quoting mackdodger
I guess it begs me to ask the question how reliable is back testing?
I don't quite follow are you asking about the reliability of back-testing on MT4 specifically? or are you questioning the reliability of back-testing in general?
  • #3
  • May 30, 2006 8:08pm May 30, 2006 8:08pm
I am asking about the reliability of MT4 back testing capabilities. Specifically the modelling quality parameter. In some cases with the optimised Alpari data the modeling quality is 70% to 80%. Is it possible to make any conclusions about the usefulness of a strategy with such low modeling qualities?
"a fool and their money will easily part"
  • #4
  • Jun 20, 2006 11:33pm Jun 20, 2006 11:33pm
Thanks Mack,

The article is very informative and well-written.

The period converter is taking forever on my terminal. I'm still waiting for it to finish converting from 1m to 5m as I am writing this. Not having any indicator of the progress does not help. If it's not done in the next 20 minutes, I'm gonna have to kill the process and try again.
  • #5
  • Jun 21, 2006 1:03am Jun 21, 2006 1:03am
It is a pity that MT4 does not have a progress indicator. Genrally it shouldn't take 20 minutes to convert to the one minute period. This is very subjeective however dependent on the speed of the computer. You can tell what is going on with the period converter by clicking on the experts tab as shown below:
http://mackdodgey.googlepages.com/initJournal.jpg
You can see that it cleary states that the period converter is attached. When the period converter is finished and can safeley be removed you will see at the top of the terminal ":xxxx Record(s) written" see below
http://mackdodgey.googlepages.com/finished.jpg
Hope this helps
"a fool and their money will easily part"
  • #6
  • Jun 23, 2006 2:47am Jun 23, 2006 2:47am
Thanks!

After checking out the 'expert' tab, I've realized that the period converter script was successfully executed eventhough the process kept consuming 100% of my CPU time.

I've successfully converted all the periods and now getting a 90% modelling quality. It's good to see the results being somewhat more reliable after being enlightened by your article.

I have yet to do any forward testing. There's where the real prove will be.
  • #7
  • Sep 19, 2006 3:54pm Sep 19, 2006 3:54pm
Quoting mackdodger
I am asking about the reliability of MT4 back testing capabilities. Specifically the modelling quality parameter. In some cases with the optimised Alpari data the modeling quality is 70% to 80%. Is it possible to make any conclusions about the usefulness of a strategy with such low modeling qualities?
Me personally, I'd like to know what actually influences that parameter? ie. what can I do to get it to approach 100%? and if I have loaded the M1 data then why the heck isn't it coming close to 100%?

Or maybe in another words, what the heck is this parameter actually measuring?
Just my 2 pips worth
  • #8
  • Sep 19, 2006 5:18pm Sep 19, 2006 5:18pm
Quoting saltcafe
Me personally, I'd like to know what actually influences that parameter? ie. what can I do to get it to approach 100%? and if I have loaded the M1 data then why the heck isn't it coming close to 100%?

Or maybe in another words, what the heck is this parameter actually measuring?
Modeling quality — the quality of ticks modeled during testing, in per cents. Modeling is shown elementarily as a band in the next line of the report. The band can be of one of three colors:

 

  1. Gray — this part of available data was not involved into testing. Gray color can appear if there was a date range specified in the testing settings;
  2. Red — modeling was not performed in this space because of missing available data of a less timeframe. At that, only data of the timeframe selected in the testing settings were used;
  3. Green — modeling was performed in this space. And the brighter is the color, the higher modeling quality was. For example, at testing on the H1 period, the deep-green band means that data of M30 period were used for testing, and the brightest one does that M1 data were used;

Attention: If the fastest method ("Open prices only") was selected in the testing settings, the entire band will be red. At that, "n/a" (no modeling was performed) will be written in the "Modeling quality" field;

I'm not insane; my mother had me tested.
  • #9
  • Sep 19, 2006 5:26pm Sep 19, 2006 5:26pm
MT4 backtester does not include spreads and commissions that can turn a winner into a loser.
  • #10
  • Sep 19, 2006 8:31pm Sep 19, 2006 8:31pm
Quoting richx7
MT4 backtester does not include spreads and commissions that can turn a winner into a loser.
And the other way too. Since even using 1 minute data does not have all the realtime ticks, MT4 guestimates what ask/bid prices may have been available at any particular time. A system could possibly be MORE profitable.

What may have been a long entry at 1.2345 can end up being a slipped entry at 1.2353 (as an example) in the strategy tester. Or if you specify 0 or 1 pip slippage it won't even register an order, it'll be a miss, since the tester never gets to see the exact price point/area your EA wanted to enter at.

You can only get your EA to a rough point of completion using the strategy tester. You then have to test it realtime in demo... And then, if successful, test it using micro leverage on a real account before committing to it.
Bundy's status today: "Waiting..."
  • #11
  • Sep 19, 2006 11:09pm Sep 19, 2006 11:09pm
When testing for a number of months is there a way to get the modeling quality higher than 90%? Does anyone out there get into the mid to upper 90s?
  • #12
  • Sep 19, 2006 11:33pm Sep 19, 2006 11:33pm
MT4 build 197 has a new feature in backtesting called visual mode that will give you frame by frame view of the trading which is like watching a movie. Selecting visual mode on the bottom left corner when you backtest an EA will open a new chart that shows each frame as it trades showing buys, sells, and exits. Its far from perfect but is a vast improvement over previous versions.
  • #13
  • Sep 20, 2006 4:45am Sep 20, 2006 4:45am
Quoting RoBiK
Modeling quality — the quality of ticks modeled during testing, in per cents. Modeling is shown elementarily as a band in the next line of the report. The band can be of one of three colors:

  1. Gray — this part of available data was not involved into testing. Gray color can appear if there was a date range specified in the testing settings;
  2. Red — modeling was not performed in this space because of missing available data of a less timeframe. At that, only data of the timeframe selected in the testing settings were used;
  3. Green — modeling was performed in this space. And the brighter is the color, the higher modeling quality was. For example, at testing on the H1 period, the deep-green band means that data of M30 period were used for testing, and the brightest one does that M1 data were used;

Attention: If the fastest method ("Open prices only") was selected in the testing settings, the entire band will be red. At that, "n/a" (no modeling was performed) will be written in the "Modeling quality" field;

Now if only that text were to tell you HOW to ACTUALLY IMPROVE your modelling results, then it would actually be useful.

I have read the help file and found no information on actually WHY it is 90% or 80% or 70% or ... What the help file is missing is information on what is the algorith used to calculate the value and what can be done to improve it.

For example if I have M1 data for a particular currancy for the entire duration of the test period then at the moment I can't get any more accurate data into MT4. Or can I?

Any gurus out there that know better?
Just my 2 pips worth
  • #14
  • Sep 20, 2006 5:15am Sep 20, 2006 5:15am
Quoting saltcafe
Now if only that text were to tell you HOW to ACTUALLY IMPROVE your modelling results, then it would actually be useful.

I have read the help file and found no information on actually WHY it is 90% or 80% or 70% or ... What the help file is missing is information on what is the algorith used to calculate the value and what can be done to improve it.

For example if I have M1 data for a particular currancy for the entire duration of the test period then at the moment I can't get any more accurate data into MT4. Or can I?

Any gurus out there that know better?
Modelling quality' is calculated according to the following formula:

ModellingQuality = ((0.25*(StartGen-StartBar) + 0.5*(StartGenM1-StartGen) + 0.9*(HistoryTotal-StartGenM1)) / (HistoryTotal-StartBar))*100%;

where:

 

  1. HistoryTotal - the total amount of bars in history;
  2. StartBar - the number of bar with which the testing was started. Modeling starts at at least 101st bar or the bar corresponding with the initial date of test limits;
  3. StartGen - the number of bar with which the modeling on the nearest timeframe started;
  4. StartGenM1 - the number of bar with which the modeling on minutes started;

at that:

 

  1. The distance between the beginning of modeling of databases for the nearest timeframe and the beginning of modeling on the nearest timeframe data has a weighting factor of 0.25;
  2. The distance between the beginning of modeling on the nearest timeframe data and the beginning of modeling on minutes has a weighting factor of 0.5;
  3. The distance between the beginning of modeling on minutes and the end of history data has a weighting factor of 0.9;

The following colors are used in the color diagram:

 

  1. Lime - modeling on minutes
  2. Deeper green colors show modeling on large timeframes
  3. Pink color - pure fractal modeling of the databases of smaller timeframe
  4. Gray color - modeling limitation by date

From the formula you can see, that 90% is the maximum modelling quality possible.

I'm not insane; my mother had me tested.
  • #15
  • Sep 20, 2006 9:15am Sep 20, 2006 9:15am
Quoting RoBiK
Modelling quality' is calculated according to the following formula:

ModellingQuality = ((0.25*(StartGen-StartBar) + 0.5*(StartGenM1-StartGen) + 0.9*(HistoryTotal-StartGenM1)) / (HistoryTotal-StartBar))*100%;

where:



 

  1. HistoryTotal - the total amount of bars in history;
  2. StartBar - the number of bar with which the testing was started. Modeling starts at at least 101st bar or the bar corresponding with the initial date of test limits;
  3. StartGen - the number of bar with which the modeling on the nearest timeframe started;
  4. StartGenM1 - the number of bar with which the modeling on minutes started;

at that:

 

  1. The distance between the beginning of modeling of databases for the nearest timeframe and the beginning of modeling on the nearest timeframe data has a weighting factor of 0.25;
  2. The distance between the beginning of modeling on the nearest timeframe data and the beginning of modeling on minutes has a weighting factor of 0.5;
  3. The distance between the beginning of modeling on minutes and the end of history data has a weighting factor of 0.9;

The following colors are used in the color diagram:

 

  1. Lime - modeling on minutes
  2. Deeper green colors show modeling on large timeframes
  3. Pink color - pure fractal modeling of the databases of smaller timeframe
  4. Gray color - modeling limitation by date

From the formula you can see, that 90% is the maximum modelling quality possible.

GREAT information. I've wasted a lot of time trying to get the modeling quality above 90%. Your information on backtesting is very helpful.

Thanks again
  • #16
  • Sep 21, 2006 3:01am Sep 21, 2006 3:01am
Quoting RoBiK
Modelling quality' is calculated according to the following formula:

ModellingQuality = ((0.25*(StartGen-StartBar) + 0.5*(StartGenM1-StartGen) + 0.9*(HistoryTotal-StartGenM1)) / (HistoryTotal-StartBar))*100%;

where:

 

  1. HistoryTotal - the total amount of bars in history;
  2. StartBar - the number of bar with which the testing was started. Modeling starts at at least 101st bar or the bar corresponding with the initial date of test limits;
  3. StartGen - the number of bar with which the modeling on the nearest timeframe started;
  4. StartGenM1 - the number of bar with which the modeling on minutes started;

at that:

 

  1. The distance between the beginning of modeling of databases for the nearest timeframe and the beginning of modeling on the nearest timeframe data has a weighting factor of 0.25;
  2. The distance between the beginning of modeling on the nearest timeframe data and the beginning of modeling on minutes has a weighting factor of 0.5;
  3. The distance between the beginning of modeling on minutes and the end of history data has a weighting factor of 0.9;

The following colors are used in the color diagram:

 

  1. Lime - modeling on minutes
  2. Deeper green colors show modeling on large timeframes
  3. Pink color - pure fractal modeling of the databases of smaller timeframe
  4. Gray color - modeling limitation by date

From the formula you can see, that 90% is the maximum modelling quality possible.

Is this something that is located in the help file? or are you one of the developers of the system?

At least now I can feel better that at least I've done all that is possible to improve the results.

Reading this ... I understand that if I have M1 data for the entire period then that's the best that I can do, and I should be concentrating on the actual system and not worry about the modelling quality number.
Just my 2 pips worth
  • #17
  • Sep 21, 2006 5:00am Sep 21, 2006 5:00am
Quoting saltcafe
Is this something that is located in the help file? or are you one of the developers of the system?

At least now I can feel better that at least I've done all that is possible to improve the results.

Reading this ... I understand that if I have M1 data for the entire period then that's the best that I can do, and I should be concentrating on the actual system and not worry about the modelling quality number.
my first post was from the help file, the second was obtained by the art of googling (wich probably tells you, that i am not the developer )

if you have M1 data, than yes, that is the best you can do in metatrader. The question is, if the best you can do is good enough. The shorter the timeframe your system results depend on, the more likely it will not be good enough. For example, to test a scalping system on a resolution of 1 minute is useless... there is too much price action lost, price action that makes a significant difference for a scalping system.

cheers
RoBiK
I'm not insane; my mother had me tested.
  • #18
  • Last Post: Sep 22, 2006 2:48am Sep 22, 2006 2:48am
I'd say that there is little chance of testing a scalping system unless you had access to tick data (which I don't). For my purposes M1 is more then adequate. I'm looking for a system that may place 1-3 trades a day for 4-24 hour durations on around 10 ccy pairs, so I'm not looking at testing firebird which trades like crazy, but something significantly more subdued.

Thanks for the info.
Just my 2 pips worth
0 traders viewing now