Major problem with metatrader | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Metatrader 4 vs Metatrader 5! 23 replies

Metatrader problem...help!!! 6 replies

Metatrader 5 is going to wipe out your Metatrader 4 indicators / systems 13 replies

Strategy Builder FX Metatrader Problem?? 3 replies

usd/cad problem on MetaTrader 3 replies

Attachments: Major problem with metatrader
Exit Attachments

Major problem with metatrader

  • #1
  • First Post: Edited 4:29am Nov 22, 2009 4:09am | Edited 4:29am
Hey,

I have a major problem with metatrader that I noticed a few weeks ago and it seemed so serious that I tried to pretend it wasn't happening, but it is, and I can't avoid it any longer.

If you plot a high period MA , such as 600 EMA, from an EA, and use Print() to give you the value of that MA at a specific time, the value that Print() returns and the value of that same EMA on the chart will differ substantially.

For example, look at the following picture. I'm using a 1296 EMA, and on the 5-minute chart, it says that at on September 14th, 01:35, the EMA is at 1.65624.

But if you look at the Print() function for the same exact time, it gives a value of 1.66074.

But don't just take my work for it, try this code yourself and you'll see what I mean.

Inserted Code
 extern int baseperiod=1200; 
 extern double TF=PERIOD_M5;
 extern string tim="01:35";
 
 
 double ZZ;
 
 int start() { 
 
 if(ZZ!=iTime(NULL,TF,1)) {
         if(TimeToStr(TimeCurrent(),TIME_MINUTES)==tim) { Print("MA: "+iMA(NULL,TF,baseperiod,0,MODE_EMA,PRICE_CLOSE,1)); }
         ZZ=iTime(NULL,TF,1); }}

http://fiat-luxx.com/trading/problem.jpg
Attached File(s)
File Type: mq4 Test.mq4   < 1 KB | 420 downloads
  • #2
  • Nov 22, 2009 9:14am Nov 22, 2009 9:14am
could you for debugging purposes write an expert (or modify the existing one) that would create an arrow (or other object) in the chart at the exact price the iMA() function returns (on every bar, so the arrows will form a curve) and compare that to the manually attached MA for comparison? Maybe this will help understanding whats going on.

I am absolutely sure there is an error somewhere in your logic, not in metatrader.
  • #3
  • Nov 22, 2009 9:56am Nov 22, 2009 9:56am
Quoting 7bit
could you for debugging purposes write an expert (or modify the existing one) that would create an arrow (or other object) in the chart at the exact price the iMA() function returns (on every bar, so the arrows will form a curve) and compare that to the manually attached MA for comparison? Maybe this will help understanding whats going on.

I am absolutely sure there is an error somewhere in your logic, not in metatrader.
As you can see, my logic is only 3 or 4 lines long, not much room for error there. If you're so sure the error is in my logic, please point it out.

As for arrows, that would involve getting the values from Print() but keeping it in the EA, and I'm not sure thats possible. It's not a complicated concept, what metatrader plots and what metatrader thinks it plots are two different values. The big problem is, which is right? What it's plotting, or what it thinks its plotting?

And the larger the period, the more pronounced the difference. Most people use small periods (less than 100) so they don't notice the difference but it's still there. On larger periods though, the difference is huge, on the order of 20-50+ pips.
  • #4
  • Nov 22, 2009 10:03am Nov 22, 2009 10:03am
Since the market is not currently open, I compiled your code as a script. To ensure that I got some output, I deleted the if() statements, and left only the
Print("MA: "+iMA(NULL,TF,baseperiod,0,MODE_EMA,PRICE_CLOSE,1));
statement in the start{} section

I attached the script to a GBPUSD,M5 chart, having already attached an EMA(1200). I pressed Ctrl-D to open the Data Window, and placed the mouse cursor over the second last candle (candle #1). Inside the Data Window, the reading for the MA was 1.66581 (see attached screenshot)

I then looked in the log file and the output was:
04:58:56 Test GBPUSD,M5: loaded successfully
04:58:56 Test GBPUSD,M5 inputs: baseperiod=1200; TF=5; tim="01:35";
04:58:56 Test GBPUSD,M5: MA: 1.66581195
Attached Image (click to enlarge)
Click to Enlarge Name: aicccia1.jpg Size: 180 KB
  • #5
  • Nov 22, 2009 10:07am Nov 22, 2009 10:07am
Okay so why do I get something different?

I believe this only occurs during a backtest, and the closer to the beginning of the backtest that you check the price, the bigger the discrepancy.
  • #6
  • Nov 22, 2009 10:29am Nov 22, 2009 10:29am
Changed my code to read:
_______________________

extern int baseperiod=1200;
extern double TF=PERIOD_M5;
extern string tim="23:59";
double ZZ;

int start() {
if (ZZ != iTime(NULL,TF,1)) {
if(TimeToStr(TimeCurrent(),TIME_MINUTES)==tim)
{ Print("MA: "+iMA(NULL,TF,baseperiod,0,MODE_EMA,PRICE_CLOSE,1)); }
ZZ=iTime(NULL,TF,1);
}
}
_______________________

because my last tick was at 23:59.

This still tests OK, proving that it's not the if() test that's the problem.

The only differences now (that I can think of) are:
-- you're running it as an EA, and I'm running it as a script
-- you're checking data at a 01:35 candle, me at 23:59
-- the data on my broker's server is likely to be different to yours

I assume you realize that the ,1 (highlighted in red in the code) refers to the candle prior to 01:35 (i.e. should be 01:30 on an M5 chart), rather than the 01:35 candle itself?

EMA value will always be different with a different starting point, due to the way it's calculated. But of course we would expect that the Print() statement and the chart itself use the same data set.

Those are the only ideas that I can think of. I can't seem to reproduce your problem.
  • #7
  • Nov 22, 2009 10:50am Nov 22, 2009 10:50am
How do we know this isn't a rounding error? Large EMAs tend to give a very long decimal value.

How far off is the EMA value if you use something like 20?


http://www.iexplain.org/ema-how-to-calculate/
  • #8
  • Nov 22, 2009 11:05am Nov 22, 2009 11:05am
Quoting aicccia
As you can see, my logic is only 3 or 4 lines long, not much room for error there. If you're so sure the error is in my logic, please point it out.

As for arrows, that would involve getting the values from Print() but keeping it in the EA, and I'm not sure thats possible.
I'm not sure where the error is because i havent tried your script yet. I only believe with 99.9% certainty that there is an error. It is the kind of error that will only be obvious after having found it.

Making the Arrows is easy. Instead of printing the value just do the following: (remove the restriction to a certain time, do it on all bar opens)

Inserted Code
double ma = iMA(...);

// untested, but should work
ObjectCreate("arrow_" + Time[0], OBJ_ARROW, 0, Time[0], ma);
ObjectSet("arrow_" + Time[0], OBJPROP_ARROWCODE, 251);

This should plot a nice curve of arrows on your chart when running forward.
  • #9
  • Edited 1:27pm Nov 22, 2009 12:15pm | Edited 1:27pm
I have extended your example for better visual debugging output. I have not changed anything, except formatting it properly for better readability and storing the ma value in a variable before printing it and only added some Object*() functions to visualize Times and prices in the chart:

Inserted Code
extern int baseperiod=1200; 
extern double TF=PERIOD_M5;
extern string tim="01:35";
 
double ZZ;
double value;
 
int start(){ 
   if(ZZ!=iTime(NULL,TF,1)) {
      value = iMA(NULL,TF,baseperiod,0,MODE_EMA,PRICE_CLOSE,1);
   
      if(TimeToStr(TimeCurrent(),TIME_MINUTES)==tim) { 
         Print("MA: " + value);
          
         ObjectCreate("arrow_" + Time[0], OBJ_ARROW, 0, Time[0], value);
         ObjectSet("arrow_" + Time[0], OBJPROP_ARROWCODE, 5);
         ObjectSet("arrow_" + Time[0], OBJPROP_COLOR, Yellow);
         
         ObjectCreate("line_" + Time[0], OBJ_VLINE, 0, Time[0], 0);
         ObjectSet("line_" + Time[0], OBJPROP_COLOR, Yellow);
      }else{
         ObjectCreate("arrow_" + Time[0], OBJ_ARROW, 0, Time[0], value);
         ObjectSet("arrow_" + Time[0], OBJPROP_ARROWCODE, 4);
         ObjectSet("arrow_" + Time[0], OBJPROP_COLOR, Blue);
      }
      
      ZZ=iTime(NULL,TF,1);
   }
   
}

and it will produce the following output:
Attached Image (click to enlarge)
Click to Enlarge Name: test.png Size: 45 KB


The green line is the moving average directly attached as indicator to the chart, the blue "line" are the blue dashes produced by arrowcode 4 on every bar and the yellow line and label (code 5) is a visualization of the same data that is printed to the log (a visual print() so to speak).

Now please try to reproduce your problem with excatly this extended version of your script and then post the screenshot again.
  • #10
  • Edited 1:40pm Nov 22, 2009 1:39pm | Edited 1:40pm
Quoting aicccia
Okay so why do I get something different?

I believe this only occurs during a backtest, and the closer to the beginning of the backtest that you check the price, the bigger the discrepancy.
For backtesting purposes, the server models the time. The problem is how can server accurately provide a value - it can't. What you are seeing is the correct MA for another time and you are assuming it is the time you want. I discovered the same thing when I tried to generate random numbers using a current time seed. I was certain their Rand function was broken until I thought about it.

You won't ever duplicate this problem in forward testing.
  • #11
  • Edited 2:01pm Nov 22, 2009 1:50pm | Edited 2:01pm
Quoting CodeMeister
For backtesting purposes, the server models the time.
Current time in backtest will be the time on the historic chart.

It will always return the time that corresponds to where (in time) the backtest currently is. Otherwise my above example would not be able to trigger the creation of the vertical lines at the right times and would produce a completely wrong curve (the blue line) for the iMA() return values.

Note that i didnt make any changes to the original script regarding how times are handled and how the iMA() value is obtained. I just added some visualization on the chart and it produces correct output.

Even in the original form without my changes it produced correct output. I could not reproduce the problem.

Maybe aicccia's history files are somehow corrupted or something else is completely messed up.
  • #12
  • Edited Nov 23, 2009 5:56am Nov 22, 2009 10:20pm | Edited Nov 23, 2009 5:56am
Quoting 7bit
I have extended your example for better visual debugging output. I have not changed anything, except formatting it properly for better readability and storing the ma value in a variable before printing it and only added some Object*() functions to visualize Times and prices in the chart:

[code]
extern int baseperiod=1200;
extern double TF=PERIOD_M5;
extern string tim="01:35";

double ZZ;
double value;

int start(){...
thank you for the code, I tried it and this is what I got with a period of 600 and 1200. The difference with 600 is almost negligible, 3 to 5 pips. With 1200 the difference is more substantial, about 25 pips.

I suppose the moral of the story is to not use large EMAs, but it's still unsettling to me.

http://fiat-luxx.com/trading/600.gif

http://fiat-luxx.com/trading/1200.gif
  • #13
  • Nov 23, 2009 8:39pm Nov 23, 2009 8:39pm
Most likely rounding error as RR said. double variables have this problem especially with exponents... perhaps you could solve it by changing the algorigthm used for the EMA calculation, but I'm not sure it's worth it...

BTW, are you making sure you have enough past bars for the indicator before calling it?

Quoting aicccia
thank you for the code, I tried it and this is what I got with a period of 600 and 1200. The difference with 600 is almost negligible, 3 to 5 pips. With 1200 the difference is more substantial, about 25 pips.

I suppose the moral of the story is to not use large EMAs, but it's still unsettling to me.
  • #14
  • Nov 24, 2009 6:12am Nov 24, 2009 6:12am
Quoting challenger78
Most likely rounding error as RR said.
The iMA() call should simply return the value of the indicator buffer and that should be the same as the one visible on the chart. If there are rounding errors they would be in the indicator itself.

I suspect that the indicator when initialized by the first call to iMA() is a different instance (and initialized with a different number of available history bars) than the one put onto the same chart (this time it has the complete chart) after backtesting is over.

Unfortunately there is no way of specifying the number of history bars that need to be in the chart when backtesting starts (the initial value of the variable Bars and the number of available bars in the time series buffers Close[], Open[], etc.), usually these are only a few hundred. This would not affect live trading since there should be always some 1000s of bars in the history, and for backtesting maybe always skip the first few thousand bars until there are enough bars:

Inserted Code
int start(){
  // simply do nothing when not enough bars on the chart
  if (Bars < big_enough_number){
    // ignore the ticks and immediately return until there are enough bars
    return(0);
  }

  // here we can start our work
  foo = iMA(...);
}
  • #15
  • Nov 24, 2009 6:25am Nov 24, 2009 6:25am
7bit,

MT4 is smart enough during backtesting to get the proper values--provided you give it enough historical data.
  • #16
  • Nov 24, 2009 6:31am Nov 24, 2009 6:31am
Quoting Ronald Raygun
7bit,

MT4 is smart enough during backtesting to get the proper values--provided you give it enough historical data.
How do you interpret the difference then?

How much is "enough" and with how much history will the backtest be initialized? And how much available history will be ignored when initializing the backtest? (I'll write an EA to find this out in 30 minutes when my currently running optimization is over)
  • #17
  • Nov 25, 2009 12:52am Nov 25, 2009 12:52am
Well, I would imagine every indicator checks number of bars available before starting the calculations... and usually require bars > period, this is standard indicator programming... I would be very surprised if MetaQuotes didn't do this for the MA!

On the other hand, the errors occur only on large values of period, which means rounding errors accumulate more. You will be surprised with how fast the errors compound! double is not a very powerful variable type for this kind of calculations.

Finally, it could be just sloppy way of plotting the thing on screen...
  • #18
  • Nov 25, 2009 1:16am Nov 25, 2009 1:16am
Quoting 7bit
How do you interpret the difference then?

How much is "enough" and with how much history will the backtest be initialized? And how much available history will be ignored when initializing the backtest? (I'll write an EA to find this out in 30 minutes when my currently running optimization is over)
As I've understood metatrader, if it doesn't have the data, it models the data. Thus, it's possible that the historical data was modeled in order to generate the really large EMA value.

If I were to run a backtest from 1/1/1970 to the present, there is absolutely no way the EA would be able to get the value of a 1500 period EMA at the very beginning of the test without guessing. What we're seeing there IMO is metatrader's guess.

Before we continue this debate, I will say that I do not consider myself to be an expert programmer at all. I consider myself to be more a trader than a programmer.

Quoting challenger78
Well, I would imagine every indicator checks number of bars available before starting the calculations... and usually require bars > period, this is standard indicator programming... I would be very surprised if MetaQuotes didn't do this for the MA!

On the other hand, the errors occur only on large values of period, which means rounding errors accumulate more. You will be surprised with how fast the errors compound! double is not a very powerful variable type for this kind of calculations.

Finally, it could be just sloppy way of plotting the thing...
They should, but they don't always. We don't really know what's happening in the iMA() function.
  • #19
  • Edited 8:07am Nov 25, 2009 7:56am | Edited 8:07am
Quoting Ronald Raygun
As I've understood metatrader, if it doesn't have the data, it models the data. Thus, it's possible that the historical data was modeled in order to generate the really large EMA value.
yes, but the question is why does the same indicator show different values when attached manually to the chart and when used via iMA(). Available history data should be the same in both cases, My theory is that in backtest only a maximum of x bars is in the history at the beginning of the backtest, no matter what amount would be theoretically available from the history database. So when iMA() is called at the beginning of a backtest it can only ever be calculated from a maximum of x bars. When you attach it manually to a chart it will use all available bars in the chart instead.

I can not imagine any other explanation for the observed behaviour.

Edit:

Maybe the reason is how EMA is calculated. You will get different values if you start calculating an x period EMA only at exactly x bars before the current value than if you start the EMA calculation long before that point in time (attached to the chart the line starts at the very beginning of all history, starting the backtest at time T the calculated EMA line in the iMA() function will only start at T-x, just enough bars that backteter thinks is needed for an x period moving average)
  • #20
  • Nov 25, 2009 10:36am Nov 25, 2009 10:36am
Quote
Maybe the reason is how EMA is calculated. You will get different values if you start calculating an x period EMA only at exactly x bars before the current value than if you start the EMA calculation long before that point in time (attached to the chart the line starts at the very beginning of all history, starting the backtest at time T the calculated EMA line in the iMA() function will only start at T-x, just enough bars that backteter thinks is needed for an x period moving average)
from aiccia's post, it seems the EMA is attached to backtester's chart. and both backtester's chart and EA should have the same available bars. on my mt4, EA and indicator that were attached on backtester's charts have 1002 bars when they started.
0 traders viewing now