Bad historical data | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Darwinex historical data (only hst files) no tick data needed 0 replies

Pinnacle Data Historical Futures Data 0 replies

Playback of historical data - simulated Data feed 0 replies

metatrader- bad backtesting or bad broker quotations? 0 replies

Historical chart to track historical news events. 0 replies

Attachments: Bad historical data
Exit Attachments

Bad historical data

  • #1
  • First Post: Jul 24, 2010 7:07am Jul 24, 2010 7:07am
Hi all.
I have recently written a simple MQL script for the only purpose of testing the historical data of Metaquotes. Basically, it simply spans the whole historical data of a given pair, checks if there are data gaps in the time and price domain and then writes a text report. The results tell that the quality of the data is really poor. This means that with this data the result of any kind of backtest is completely erroneous, even for long term strategies. Anyone dealing with such issue?
  • #2
  • Jul 24, 2010 7:10am Jul 24, 2010 7:10am
Don't use metaquotes data.

www.forextester.com/data/datasources.html
  • #3
  • Jul 24, 2010 7:12am Jul 24, 2010 7:12am
Quoting Ronald Raygun
Don't use metaquotes data.

www.forextester.com/data/datasources.html

Even in this data there's plenty of holes.
  • #4
  • Jul 24, 2010 7:30am Jul 24, 2010 7:30am
Quoting asasa
Even in this data there's plenty of holes.
Let me show you an example regarding the data you have suggested, give a look at the picture below...
Attached Image
  • #5
  • Jul 24, 2010 7:41am Jul 24, 2010 7:41am
Quoting asasa
Even in this data there's plenty of holes.
A gap in the time domain:
Attached Image
  • #6
  • Jul 24, 2010 8:40am Jul 24, 2010 8:40am
I agree completely with you.

However, this is to my knowledge the best free data source out there.
  • #7
  • Jul 24, 2010 9:21am Jul 24, 2010 9:21am
Quoting asasa
Let me show you an example regarding the data you have suggested, give a look at the picture below...
That spike is real. Here's a look at M1 candles built from GAIN Capital tick data from that day. The event happened at 12:30 UTC, which given that it was a Friday and summer time (DST on) is very likely a NFP:

http://dl.dropbox.com/u/190212/ff/eu...2004_08_06.png

The other tick sources I have (FXCM, dbFX, Dukascopy and TrueFX) only go back to 2007, so I can't check with them.

The image is from my own backtester/market research software I'm writing at the moment.
  • #8
  • Jul 24, 2010 9:30am Jul 24, 2010 9:30am
Quoting Adal
That spike is real. Here's a look at M1 candles built from GAIN Capital tick data from that day. The event happened at 12:30 UTC, which given that it was a Friday and summer time (DST on) is very likely a NFP:



The other tick sources I have (FXCM, dbFX, Dukascopy and TrueFX) only go back to 2007, so I can't check with them.

The image is from my own backtester/market research software I'm writing at the moment.

Almost one hundred pips of spread in that moment?
Can it be real for the Forex?
  • #9
  • Jul 24, 2010 9:46am Jul 24, 2010 9:46am
Quoting asasa
Almost one hundred pips of spread in that moment?
Can it be real for the Forex?
You are confusing price with spread.

Here are the actual GAIN ticks:
632283,"EUR/USD",2004-08-06 08:29:58,1.207000,1.207300,"D"
632292,"EUR/USD",2004-08-06 08:29:59,1.207200,1.207500,"D"
632416,"EUR/USD",2004-08-06 08:30:05,1.215100,1.215400,"D"
632462,"EUR/USD",2004-08-06 08:30:08,1.215400,1.215700,"D"
632468,"EUR/USD",2004-08-06 08:30:08,1.217100,1.217400,"D"

As you can see, at this bucket-shop the spread was 3 pips all the time. However, they stopped quoting prices for 5 seconds, between 08:30:00 and 08:30:05, the period in which the price jumped and when the market was extremely volatile.

Here's a similar event from last year, which I'm sure a lot of people remember:

http://dl.dropbox.com/u/190212/ff/eu...2010_03_18.png

dbFX ticks from that time showing a 200 pip jump in 44 seconds (from the first tick to the last):

Inserted Code
20090318-14:16:58.492,1.31106,1.31147
20090318-14:17:01.134,1.31105,1.31146
20090318-14:17:03.443,1.3117,1.31211
20090318-14:17:03.834,1.31158,1.31199
20090318-14:17:03.942,1.3114,1.31181
20090318-14:17:04.458,1.311,1.31141
20090318-14:17:04.750,1.31103,1.31144
20090318-14:17:04.961,1.31109,1.3115
20090318-14:17:05.043,1.31112,1.3115
20090318-14:17:05.343,1.31273,1.31361
20090318-14:17:05.379,1.3128,1.3134
20090318-14:17:05.581,1.31273,1.31351
20090318-14:17:06.096,1.3133,1.31371
20090318-14:17:06.570,1.31323,1.31371
20090318-14:17:06.776,1.31335,1.31371
20090318-14:17:07.113,1.3138,1.3144
20090318-14:17:07.284,1.31282,1.31371
20090318-14:17:07.332,1.3127,1.31371
20090318-14:17:07.776,1.3168,1.3172
20090318-14:17:08.667,1.3137,1.31451
20090318-14:17:09.204,1.3168,1.3174
20090318-14:17:09.266,1.317,1.3174
20090318-14:17:09.616,1.3167,1.31751
20090318-14:17:09.983,1.3169,1.31771
20090318-14:17:10.145,1.31977,1.32018
20090318-14:17:10.393,1.31935,1.31975
20090318-14:17:10.416,1.31977,1.32018
20090318-14:17:10.578,1.3188,1.3193
20090318-14:17:10.650,1.31886,1.31927
20090318-14:17:10.996,1.31882,1.3192
20090318-14:17:11.388,1.31923,1.31927
20090318-14:17:11.584,1.31876,1.31917
20090318-14:17:12.083,1.3203,1.3208
20090318-14:17:12.870,1.3204,1.3208
20090318-14:17:14.095,1.3203,1.32101
20090318-14:17:14.110,1.32073,1.32091
20090318-14:17:14.111,1.3204,1.32091
20090318-14:17:14.112,1.32075,1.32111
20090318-14:17:14.114,1.3208,1.3213
20090318-14:17:14.668,1.32075,1.32171
20090318-14:17:14.750,1.3206,1.32171
20090318-14:17:14.777,1.32065,1.32171
20090318-14:17:15.740,1.321,1.3216
20090318-14:17:15.749,1.3211,1.32171
20090318-14:17:15.751,1.32095,1.32171
20090318-14:17:16.359,1.32123,1.32171
20090318-14:17:16.361,1.3223,1.32241
20090318-14:17:16.790,1.32123,1.32241
20090318-14:17:17.443,1.32232,1.32241
20090318-14:17:17.888,1.3213,1.32221
20090318-14:17:18.969,1.32472,1.32533
20090318-14:17:20.361,1.3249,1.3256
20090318-14:17:21.391,1.3248,1.32499
20090318-14:17:22.064,1.3239,1.32499
20090318-14:17:22.066,1.3242,1.3254
20090318-14:17:22.067,1.32453,1.32494
20090318-14:17:22.069,1.32448,1.32489
20090318-14:17:24.545,1.32458,1.32489
20090318-14:17:25.559,1.3245,1.32489
20090318-14:17:25.560,1.3248,1.32489
20090318-14:17:25.680,1.32485,1.32526
20090318-14:17:25.806,1.32502,1.32541
20090318-14:17:25.807,1.32509,1.32541
20090318-14:17:25.816,1.32505,1.32546
20090318-14:17:25.819,1.32515,1.32556
20090318-14:17:25.827,1.32552,1.3257
20090318-14:17:25.829,1.32617,1.32658
20090318-14:17:25.830,1.32616,1.32657
20090318-14:17:25.831,1.3261,1.32651
20090318-14:17:25.878,1.32614,1.32655
20090318-14:17:25.988,1.32604,1.32645
20090318-14:17:26.195,1.32582,1.32601
20090318-14:17:26.339,1.32594,1.32601
20090318-14:17:26.438,1.32619,1.3266
20090318-14:17:26.841,1.32768,1.32863
20090318-14:17:27.277,1.32795,1.32865
20090318-14:17:27.428,1.32792,1.32839
20090318-14:17:27.886,1.32832,1.32879
20090318-14:17:28.150,1.32842,1.32889
20090318-14:17:28.776,1.32865,1.32889
20090318-14:17:28.778,1.3289,1.32911
20090318-14:17:28.798,1.32862,1.32899
20090318-14:17:28.873,1.32852,1.32899
20090318-14:17:29.108,1.3286,1.32899
20090318-14:17:29.278,1.32865,1.32899
20090318-14:17:29.408,1.32892,1.32929
20090318-14:17:29.803,1.32888,1.32909
20090318-14:17:29.840,1.32889,1.3291
20090318-14:17:30.190,1.32888,1.32909
20090318-14:17:30.498,1.32889,1.3291
20090318-14:17:30.517,1.32842,1.32899
20090318-14:17:30.521,1.32839,1.3286
20090318-14:17:30.541,1.32875,1.32945
20090318-14:17:30.558,1.32899,1.3292
20090318-14:17:30.925,1.32912,1.32949
20090318-14:17:32.026,1.32916,1.32951
20090318-14:17:32.949,1.32902,1.32959
20090318-14:17:32.988,1.32892,1.32941
20090318-14:17:32.989,1.32913,1.32954
20090318-14:17:32.991,1.32915,1.32951
20090318-14:17:33.028,1.32901,1.32942
20090318-14:17:33.029,1.329,1.32941
20090318-14:17:34.839,1.32892,1.32926
20090318-14:17:34.978,1.3289,1.32911
20090318-14:17:34.980,1.32854,1.32869
20090318-14:17:34.982,1.32842,1.32869
20090318-14:17:34.983,1.32829,1.3287
20090318-14:17:34.984,1.32917,1.33024
20090318-14:17:37.318,1.32907,1.32984
20090318-14:17:37.320,1.32924,1.32965
20090318-14:17:37.321,1.32957,1.32965
20090318-14:17:37.322,1.32924,1.32965
20090318-14:17:37.323,1.32984,1.33001
20090318-14:17:37.324,1.32974,1.33015
20090318-14:17:37.326,1.32975,1.33016
20090318-14:17:37.327,1.33051,1.33092
20090318-14:17:37.397,1.32991,1.33032
20090318-14:17:37.460,1.32979,1.3302
20090318-14:17:38.091,1.33067,1.33141
20090318-14:17:38.094,1.32979,1.3302
20090318-14:17:38.096,1.32951,1.32992
20090318-14:17:38.098,1.32961,1.33002
20090318-14:17:38.306,1.32956,1.32997
20090318-14:17:39.018,1.32955,1.32996
20090318-14:17:39.019,1.32957,1.32996
20090318-14:17:39.020,1.32976,1.33017
20090318-14:17:39.037,1.32971,1.33012
20090318-14:17:39.038,1.32991,1.33032
20090318-14:17:39.388,1.32951,1.32992
20090318-14:17:39.572,1.32941,1.32982
20090318-14:17:39.682,1.32936,1.32977
20090318-14:17:39.858,1.32937,1.32977
20090318-14:17:40.088,1.32936,1.32977
20090318-14:17:40.189,1.32956,1.32995
20090318-14:17:40.246,1.32941,1.32982
20090318-14:17:40.340,1.32946,1.32987
20090318-14:17:40.538,1.32941,1.32982
20090318-14:17:40.945,1.32931,1.32972
20090318-14:17:41.078,1.32937,1.32972
20090318-14:17:41.245,1.32961,1.33002
20090318-14:17:41.747,1.3296,1.33001
20090318-14:17:42.152,1.32973,1.33014
20090318-14:17:42.536,1.3298,1.33021
20090318-14:17:42.737,1.32983,1.33024
20090318-14:17:42.842,1.33017,1.33024
20090318-14:17:43.156,1.33051,1.33092
20090318-14:17:43.308,1.33087,1.33092
20090318-14:17:43.481,1.33111,1.33151
20090318-14:17:43.890,1.33119,1.33151
20090318-14:17:44.498,1.33118,1.33141
  • #10
  • Jul 24, 2010 9:55am Jul 24, 2010 9:55am
Quoting Adal
You are confusing price with spread.

Here are the actual GAIN ticks:
632283,"EUR/USD",2004-08-06 08:29:58,1.207000,1.207300,"D"
632292,"EUR/USD",2004-08-06 08:29:59,1.207200,1.207500,"D"
632416,"EUR/USD",2004-08-06 08:30:05,1.215100,1.215400,"D"
632462,"EUR/USD",2004-08-06 08:30:08,1.215400,1.215700,"D"
632468,"EUR/USD",2004-08-06 08:30:08,1.217100,1.217400,"D"

As you can see, at this bucket-shop the spread was 3 pips all the time. However, they stopped quoting prices for 5 seconds, between 08:30:00 and 08:30:05, the period...

In your screenshot of the chart dated 06-08-2004, we see no price for almost 100 pips, meaning there was no offer for that price range (that is why I called it spread).
In any case, I can be with you for NFP days.
Apart from those, in the historical data I see a lot of holes in the time domain (missing 1M data for large time intervals).
  • #11
  • Jul 24, 2010 10:10am Jul 24, 2010 10:10am
Well, I've decided to use exclusively tick data, and then compare the data from different brokers to filter spikes and ensure they match. That was a good idea, since this way I discovered that the timezone for FXCM and dbFX ticks changed from EST to UTC in september 2009. I noticed this by suddenly seeing 100 pip differences between FXCM/dbFX ticks and Dukascopy ticks. After changing the timezone for the affected ticks, everything matched again.

Now I test my strategies with at least two data sources, and only from march 2007 on, since I don't have good tick data before that.

My life trading system (work in progress) will check the ticks from my broker against online tick providers (like http://rates.fxcm.com/RatesXML or http://webrates.truefx.com/rates/connect.html?f=csv) to ensure I don't try to trade on spikes.
  • #12
  • Last Post: Jul 24, 2010 10:50am Jul 24, 2010 10:50am
Quoting Adal
Well, I've decided to use exclusively tick data, and then compare the data from different brokers to filter spikes and ensure they match. That was a good idea, since this way I discovered that the timezone for FXCM and dbFX ticks changed from EST to UTC in september 2009. I noticed this by suddenly seeing 100 pip differences between FXCM/dbFX ticks and Dukascopy ticks. After changing the timezone for the affected ticks, everything matched again.

Now I test my strategies with at least two data sources, and only from march 2007 on, since I don't...
You have given me some interesting ideas to work on.
0 traders viewing now