Finding Gaps in History Data | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Live History vs Demo History 2 replies

Finding Yield Data 10 replies

Backtesting and Historic Data Gaps 0 replies

gaps in data mess up my "correlation" indicator 1 reply

History of the Stock Exchange on history channel today 0 replies

Attachments: Finding Gaps in History Data
Exit Attachments

Finding Gaps in History Data

  • #1
  • First Post: Edited Nov 5, 2010 1:17am Sep 13, 2010 10:07am | Edited Nov 5, 2010 1:17am
Hi all,

A quite common issue with Metatrader is having bad history data.

This leads to very misleading results when viewing charts in various timeframes, without even noticing it yourself immediately.
It may take some time to discover why the indicators you have plotted on the chart don't make sense.
Moving your mouse on the chart will quickly tell you that indeed the timestamp between those 2 bars is way longer than just 1 bar...
The only explanation you can find to this is: there are BARS MISSING in the chart.

This is due to missing data in your history.


The other effect of missing bars is that EA backtesting results may become highly unreliable when run using the regular history data when there are huge gaps.


Most of the time, using the 'Refresh" command will solve the problem (right-click on chart>Refresh).

Sometimes you can also fix that by downloading the history from the History Center (F2).

But sometimes none of these 2 methods will work, the missing bars will remain missing, whatever you do...

If you are using Alpari-UK MT4 setup, then you may have already discovered that there is a huge 2-weeks gap in the history data for GPBUSD between 2010.03.12 and 2010.03.25...
And the history center won't help: these bars are definitely not in their history!
For this particular case, there is a 3rd way to get around it, explained in this thread . But that's only for the Alpari-UK installation...


I fell into those issues several times, and still fall into them regularly...
So i decided i needed a way to clearly identify if there are gaps in my history data, how many, and where.

So i developped 2 tools that you will find attached below.

- a script : "sqCheckHistoryGapsScript.mq4":
this script (goes into your mt4expertsscripts folder) will report into the Terminal "Journal" tab the full list of gaps for ALL TIMEFRAMES on the chart on which you will drop it.
You can view the Journal file by right-clicking on the Journal tab of the Terminal window, and select "Open", then open the yyyymmdd.log file with the Notepad for instance, and scroll to the end of it.
You will discover a full report for each timeframe, with statistical reports as well to tell you the quality of your history.



- an Indicator : "sqShowHistoryGaps.mq4"

this indicator (goes into your mt4expertsindicators folder) will draw a vertical red line at each gap found in your current chart, and will display how many bars are missing at those locations.
This is an easy way to locate gaps in the charts at locations you are currently watching.
If you refresh your history data while the indicator is already loaded, you'll have to refresh the indicator as well for it to update the missing bars shown on the corresponding chart.


These tools are NOT a SOLUTION to the issue itself, but at least now you know why your indicators or EAs act crazy...



There are some known issues i have identified with these tools: they may give you false missing bars areas when the markets are closed longer than saturday+sunday, due to a bank holiday, like Christmas or New-Years day, but you will easily spot those dates.



If anyone wants to contribute to improve these indicators, feel free to modify them.
I will just ask you to post your improvements back into this thread, so everyone can benefit from these improvements.


Thank you all.


SQ

Update -- 05 nov 2010: sqCheckHistoryGapsScript.mq4 updated with DST and broker/timezone auto-detect;
Attached File(s)
File Type: mq4 sqShowHistoryGaps.mq4   7 KB | 1,500 downloads
File Type: mq4 sqCheckHistoryGapsScript.mq4   9 KB | 1,028 downloads | Uploaded Nov 5, 2010 1:16am
  • #2
  • Sep 13, 2010 10:16am Sep 13, 2010 10:16am
Here is a tip to quickly find and spot gaps in your history data once you have loaded the "sqShowHistoryGaps.mq4" Indicator on a chart:

- display the Objects drawn on your chart: ctrl-B
- all objects drawn by the indicator have their names starting with "sqShowHistoryGaps-..."
For instance:
"sqShowHistoryGaps-GBPUSD/M15:2010.03.26 22:30"

If you want to find the latest gap:
- scroll down the list until the last entry with the "sqShowHistoryGaps" prefix
- highlight it (click on it)
- click "Show"
- your chart will be moved to the location of the that particulr gap.

Enjoy... (if we can ever "enjoy" such MT4/brokers disaster...)

SQ
  • #3
  • Sep 13, 2010 10:41pm Sep 13, 2010 10:41pm
Squalou,

As always you've done a great job on this.

Thanks,

Johnny
  • #4
  • Sep 14, 2010 1:37am Sep 14, 2010 1:37am
Quoting JohnnyBSmart
Squalou,

As always you've done a great job on this.

Thanks,

Johnny
Cheers Johnny...

This actually explains why some of the 3Tier LBO boxes were not positionned where they should have in some of your screenshots (and in some of my charts as well!!!)...
Damned MT4 that won't tell you about those missing bars!!!

SQ
  • #5
  • Sep 15, 2010 3:36pm Sep 15, 2010 3:36pm
Why don't you simply use this method?

http://eareview.net/tick-data
  • #6
  • Sep 15, 2010 5:10pm Sep 15, 2010 5:10pm
Quoting watcherkb
Why don't you simply use this method?

http://eareview.net/tick-data

This is very good for backtesting (99% modeling quality).

But Dukascopy updates its tick files with 2 hours delay or so.
I update my tick files from there once a day, not more.
It takes ages to "crunch" those tick files into CSV, and then into FXT+HST files...

Thanks anyway.

SQ
  • #7
  • Oct 26, 2010 6:00am Oct 26, 2010 6:00am
WOW!!!
thanks!!!
running your script on metaquotes history data gives 1,990,489 missing 1 minuite bars which is 52% missing bars!!
unbeleivable

when i downloaded 1m data from dukascopy using JFOREX platform
it still give many wrong bars.
is alpari.ru ok?
  • #8
  • Oct 26, 2010 10:38am Oct 26, 2010 10:38am
Quoting lolik2020
WOW!!!
thanks!!!
running your script on metaquotes history data gives 1,990,489 missing 1 minuite bars which is 52% missing bars!!
unbeleivable

when i downloaded 1m data from dukascopy using JFOREX platform
it still give many wrong bars.
is alpari.ru ok?
52% is very high...
Try updating the history data from the history center and re-run the script.

In fact the script has problems with fridays/mondays depending on the GMT shift of the broker.
You will see that there can be high blocks of missing bars showing around the week-end when in fact there are not, due to the week opening/closing hours shift.

But these are very easy to spot and "mentally" eliminate.

You will also experience similar glitches on holidays (like dec 25, jan 1, etc) when these days are week days.

At most this could create perhaps show as max 20% "virtually" missing bars, but not as high as 52%.

But this is still very useful to spot real gaps in history.
Then you can update your history data, and re-run the script to see if it improved things.

Metaquotes has a big gap from 2010.03.12 till 26 for instance (on EU and GU at least).
Which you hardly see when scrolling your chart (appart from a big price gap!).
But the script will spot it nicely.

SQ
  • #9
  • Edited Oct 27, 2010 2:49am Oct 26, 2010 11:47pm | Edited Oct 27, 2010 2:49am
actually 2/7 is 14%
if the code is not taking into account the weekends. then
14% missing bars is ok.!!
so all we have to take into account is 11 US holidays... for XXXUSD, USDXXX symbols

i wrote a part of code but do not know where to put it. can you help?

PHP Code
              int mnth = Month();
              int day   = Day();
              /*//252 market days per year without 11 US holidays
               if (   
                      (mnth == 1  && day ==  1) || // 01 January    is new years     Day (all markets are closed)
                      (mnth == 1  && day == 19) || // 19 January   is Martin Luther King Day 
                      (mnth == 2  && day == 16) || // 16 February  is President     Day 
                      (mnth == 5  && day == 25) || // 25 May        is Memorial      Day      
                      (mnth == 7  && day ==  3) || // 03 July         is Bank      HoliDay      
                      (mnth == 7  && day ==  4) || // 04 July         is Independence  Day      
                      (mnth == 9  && day ==  7) || // 07 September is Labour        Day      
                      (mnth == 10 && day == 12) ||// 12 October    is Columbus      Day
                      (mnth == 11 && day == 11) ||// 11 November  is Veteran’s     Day
                      (mnth == 11 && day == 26) ||// 26 November  is Thanksgiving  Day
                      (mnth == 12 && day == 25)    // 25 December  is Christmas     Day (all markets are closed)
                    )
                     gap = false;
              */
  • #10
  • Oct 27, 2010 1:00am Oct 27, 2010 1:00am
"(mnth == 10 && day == 12) || // 25 December is Columbus Day
(mnth == 11 && day == 11) || // 25 December is Veteran’s Day"

October and November.

24th of December ends earlier than midnight (at 19:00)
  • #11
  • Oct 27, 2010 1:37am Oct 27, 2010 1:37am
Quoting lolik2020
actually 2/7 is 14%
hehe... not quite... 2/7 is 28% ...
oh well, a rough estimate is good enough anyway, it's the main idea that is important, not the exact figures.

Quote
so all we have to take into account is 11 holidays...
Well, markets are not closed all over the world just because of USA bank holidays...
The only days where markets are closed worldwide for sure are dec 25, and jan 1.
I went thru my Dukascopy tick database and these are the only 2 days where markets are closed appart from sat-sun.
There are days however where markets are closing earlier than usual, thanks Smilee89.

The main issue with the script is that the week-end start/end hours are hardcoded to be from friday 22:00 to monday 00:00, regardless of the broker time zone shift.
This is already wrong, as a normal market week is EXACTLY 5 days long (5*24 hours), so week-ends should really be saturday 00:00 to monday 00:00, adjusted to broker time zone shift and DST.
Then we should take out dec 25 + jan 1, and finally cut off the few hours when markets close earlier than usual on dec 24 and dec 31, as smilee89 pointed out.
I think this should be exhaustive.

So, the script and indi will wrongly show the week-end days as being missing bars when the broker time zone is not within GMT+0, GMT+1, or GMT+2 or so.

I will try to fix that.

Thanks for your contributions!

SQ
  • #12
  • Edited 9:18am Oct 27, 2010 4:41am | Edited 9:18am
Thanks
if i can help i will
it is a liitle bit hard for me to get into someone's elses code...

if anybody can add GMT support so it will work with any broker.
suppose to be something like this...(hope there is no bug)
PHP Code
int BrokerGMT = 0;//should be extern parameter
if (! (TimeHour(timeCurr+BrokerGMT) == 0 && TimeMinute(timeCurr) == 0) ) gap = true;
one more thing.
it is better to put the missing bars to a .csv file (instead of into log file)
so we can add the missing bars to chart later with history center.
i think that we have to fill in the missing bars with O=H=L=C AUTOMATICALLY.
Without this many scripts/indicators/experts fail miserably!
OHLC can be taken from:
1) previous bar before the hole.
2) first bar after the hole.
3) average/intermediate values between the previous and the next bar.

so someone have to add this function somewhere in code(hope there is no bug :-))
PHP Code
    int handle;
void Add2Csv(int i, int timeframe, datetime timePrev){
      FileSeek(handle, 0, SEEK_END);
      int chars = FileWrite( handle
                            ,TimeToStr(timePrev+timeframe*60,TIME_DATE|TIME_MINUTES)
                            ,iOpen  (NULL, timeframe,i)
                            ,iHigh  (NULL, timeframe,i) 
                            ,iLow   (NULL, timeframe,i) 
                            ,iClose (NULL, timeframe,i) 
                            ,iVolume(NULL, timeframe,i)
                  );
      if (chars < 0) {
         string gs_error = GetLastError();
         Alert("History Log: Error " + gs_error );
         FileClose(handle);
         return (false);
      }
}
  • #13
  • Nov 5, 2010 1:15am Nov 5, 2010 1:15am
I coded the DST and w-e stuff...

It should now properly detect week-end hours whatever broker and timezone you are in, and also apply DST shift automatically.

So, now there should be very few "missing bars" around the w-e.
Hopefully...

I have put that code in the script for the moment, i will also add it to the Indicator.

Update in post 1.

SQ
  • #14
  • Jan 1, 2011 5:24am Jan 1, 2011 5:24am
Hi Squalou,

Thanks for your great work on this.

I've got the indicator to work but not the script.

The log says: "loaded successfully" quickly followed by "removed".

Searching the forum others have had similar problems with scripts that were fixed by setting Options on the "Experts Advisors" tab. I've done that but still no joy.

If you or others have any ideas what could be causing the problem I would be grateful for your input.

Regards

Pravah
  • #15
  • Jan 2, 2011 9:22am Jan 2, 2011 9:22am
Hi

This is normal behaviour for scripts: they are loaded, run once, then removed (unless the script does not "return" from the "start()" function).
The "Journal" tab will only tell you that.

This script outputs all its reports in the "Experts" tab of the "Terminal" window (that's where all "Print()" calls end up in MT4).

To read the full report confortably, right-click on the "Experts" tab of the "Terminal" window, then "Open", then open the latest .log file with the notepad for instance, and scroll to the end of it, you'll find all the reports.

Sq

Quoting Pravah
Hi Squalou,

Thanks for your great work on this.

I've got the indicator to work but not the script.

The log says: "loaded successfully" quickly followed by "removed".

Searching the forum others have had similar problems with scripts that were fixed by setting Options on the "Experts Advisors" tab. I've done that but still no joy.

If you or others have any ideas what could be causing the problem I would be grateful for your input.

Regards

Pravah
  • #16
  • Jan 3, 2011 3:50pm Jan 3, 2011 3:50pm
Quoting squalou
Hi

This is normal behaviour for scripts: they are loaded, run once, then removed (unless the script does not "return" from the "start()" function).
The "Journal" tab will only tell you that.

This script outputs all its reports in the "Experts" tab of the "Terminal" window (that's where all "Print()" calls end up in MT4).

To read the full report confortably, right-click on the "Experts" tab of the "Terminal" window, then "Open", then open the latest .log file with the notepad for instance, and scroll to the end...
Thanks a million Squalou,

My problem was that I was looking in the Journal log rather than the Experts Log for the output. It's now all working as advertised.

Thanks again

Pravah
  • #17
  • Mar 15, 2012 3:07pm Mar 15, 2012 3:07pm
thanks alot for the nice efforts pointing this gap problem
i know it is an old thread .. i was just wondering if there has been any new modifications

thanks again for the nice work squalou
  • #18
  • Last Post: Jul 28, 2012 7:28am Jul 28, 2012 7:28am
Quoting lolik2020
WOW!!!


when i downloaded 1m data from dukascopy using JFOREX platform
it still give many wrong bars.

1 minute data from jforex has missing bars too. Not a shock. How about compared to the tick data? same issue?

great work here. I've been using this script for a long time and it's a part of my default template to ensure my indicators are functioning as they should and also for visual backtesting.
0 traders viewing now