Hi all,
A quite common issue with Metatrader is having bad history data.
This leads to very misleading results when viewing charts in various timeframes, without even noticing it yourself immediately.
It may take some time to discover why the indicators you have plotted on the chart don't make sense.
Moving your mouse on the chart will quickly tell you that indeed the timestamp between those 2 bars is way longer than just 1 bar...
The only explanation you can find to this is: there are BARS MISSING in the chart.
This is due to missing data in your history.
The other effect of missing bars is that EA backtesting results may become highly unreliable when run using the regular history data when there are huge gaps.
Most of the time, using the 'Refresh" command will solve the problem (right-click on chart>Refresh).
Sometimes you can also fix that by downloading the history from the History Center (F2).
But sometimes none of these 2 methods will work, the missing bars will remain missing, whatever you do...
If you are using Alpari-UK MT4 setup, then you may have already discovered that there is a huge 2-weeks gap in the history data for GPBUSD between 2010.03.12 and 2010.03.25...
And the history center won't help: these bars are definitely not in their history!
For this particular case, there is a 3rd way to get around it, explained in this thread . But that's only for the Alpari-UK installation...
I fell into those issues several times, and still fall into them regularly...
So i decided i needed a way to clearly identify if there are gaps in my history data, how many, and where.
So i developped 2 tools that you will find attached below.
- a script : "sqCheckHistoryGapsScript.mq4":
this script (goes into your mt4expertsscripts folder) will report into the Terminal "Journal" tab the full list of gaps for ALL TIMEFRAMES on the chart on which you will drop it.
You can view the Journal file by right-clicking on the Journal tab of the Terminal window, and select "Open", then open the yyyymmdd.log file with the Notepad for instance, and scroll to the end of it.
You will discover a full report for each timeframe, with statistical reports as well to tell you the quality of your history.
- an Indicator : "sqShowHistoryGaps.mq4"
this indicator (goes into your mt4expertsindicators folder) will draw a vertical red line at each gap found in your current chart, and will display how many bars are missing at those locations.
This is an easy way to locate gaps in the charts at locations you are currently watching.
If you refresh your history data while the indicator is already loaded, you'll have to refresh the indicator as well for it to update the missing bars shown on the corresponding chart.
These tools are NOT a SOLUTION to the issue itself, but at least now you know why your indicators or EAs act crazy...
There are some known issues i have identified with these tools: they may give you false missing bars areas when the markets are closed longer than saturday+sunday, due to a bank holiday, like Christmas or New-Years day, but you will easily spot those dates.
If anyone wants to contribute to improve these indicators, feel free to modify them.
I will just ask you to post your improvements back into this thread, so everyone can benefit from these improvements.
Thank you all.
SQ
Update -- 05 nov 2010: sqCheckHistoryGapsScript.mq4 updated with DST and broker/timezone auto-detect;
A quite common issue with Metatrader is having bad history data.
This leads to very misleading results when viewing charts in various timeframes, without even noticing it yourself immediately.
It may take some time to discover why the indicators you have plotted on the chart don't make sense.
Moving your mouse on the chart will quickly tell you that indeed the timestamp between those 2 bars is way longer than just 1 bar...
The only explanation you can find to this is: there are BARS MISSING in the chart.
This is due to missing data in your history.
The other effect of missing bars is that EA backtesting results may become highly unreliable when run using the regular history data when there are huge gaps.
Most of the time, using the 'Refresh" command will solve the problem (right-click on chart>Refresh).
Sometimes you can also fix that by downloading the history from the History Center (F2).
But sometimes none of these 2 methods will work, the missing bars will remain missing, whatever you do...
If you are using Alpari-UK MT4 setup, then you may have already discovered that there is a huge 2-weeks gap in the history data for GPBUSD between 2010.03.12 and 2010.03.25...
And the history center won't help: these bars are definitely not in their history!
For this particular case, there is a 3rd way to get around it, explained in this thread . But that's only for the Alpari-UK installation...
I fell into those issues several times, and still fall into them regularly...
So i decided i needed a way to clearly identify if there are gaps in my history data, how many, and where.
So i developped 2 tools that you will find attached below.
- a script : "sqCheckHistoryGapsScript.mq4":
this script (goes into your mt4expertsscripts folder) will report into the Terminal "Journal" tab the full list of gaps for ALL TIMEFRAMES on the chart on which you will drop it.
You can view the Journal file by right-clicking on the Journal tab of the Terminal window, and select "Open", then open the yyyymmdd.log file with the Notepad for instance, and scroll to the end of it.
You will discover a full report for each timeframe, with statistical reports as well to tell you the quality of your history.
- an Indicator : "sqShowHistoryGaps.mq4"
this indicator (goes into your mt4expertsindicators folder) will draw a vertical red line at each gap found in your current chart, and will display how many bars are missing at those locations.
This is an easy way to locate gaps in the charts at locations you are currently watching.
If you refresh your history data while the indicator is already loaded, you'll have to refresh the indicator as well for it to update the missing bars shown on the corresponding chart.
These tools are NOT a SOLUTION to the issue itself, but at least now you know why your indicators or EAs act crazy...
There are some known issues i have identified with these tools: they may give you false missing bars areas when the markets are closed longer than saturday+sunday, due to a bank holiday, like Christmas or New-Years day, but you will easily spot those dates.
If anyone wants to contribute to improve these indicators, feel free to modify them.
I will just ask you to post your improvements back into this thread, so everyone can benefit from these improvements.
Thank you all.
SQ
Update -- 05 nov 2010: sqCheckHistoryGapsScript.mq4 updated with DST and broker/timezone auto-detect;
Attached File(s)
sqShowHistoryGaps.mq4
7 KB
|
1,373 downloads
sqCheckHistoryGapsScript.mq4
9 KB
|
906 downloads
|
Uploaded Nov 5, 2010 4:16am