Assessing the quality of historical data

Hi all,

I'm investigating the quality of a few free sources of data, measuring it by calculating bar gaps with a matlab script I made. Here you can look at M1 data for EURUSD on Alpari UK:

>> checkcsvgaps('EURUSD1.csv', 30000)
Preallocating structured data for better performance... done (elapsed time: 0.032 s)
Reading file into the main memory (ETA: 6.00 s)... done (elapsed time: 6.922 s)
Checking file for gaps... done (elapsed time: 0.609 s)

Bars read: 30000
Gaps: 24710 (82.37 percent of bars)
Total gap: 48256.00 pips
Average bar gap: 1.608 pips
Standard deviation (in pips) on average gap: 2.1752

http://img505.imageshack.us/img505/4370/alparigaps.jpg

and this is the same thing with FXDD:

>> checkcsvgaps('EURUSD2.csv', 30000)
Preallocating structured data for better performance... done (elapsed time: 0.032 s)
Reading file into the main memory (ETA: 6.00 s)... done (elapsed time: 6.093 s)
Checking file for gaps... done (elapsed time: 0.469 s)

Bars read: 30000
Gaps: 6025 (20.08 percent of bars)
Total gap: 6183.00 pips
Average bar gap: 0.206 pips
Standard deviation (in pips) on average gap: 0.4662

http://img403.imageshack.us/img403/350/gapdistr.jpg

So that suggests FXDD data is better (a LOT better in this sample).

This is the tick data for 1 month of AUDJPY at GAINS capital data, which I had previously converted to M1 bars:

checkcsvgaps('AUDJPY.csv', 30000)
Preallocating structured data for better performance... done (elapsed time: 0.015 s)
Reading file into the main memory (ETA: 6.00 s)... done (elapsed time: 5.563 s)
Checking file for gaps... done (elapsed time: 0.578 s)

Bars read: 26904
Gaps: 24087 (89.53 percent of bars)
Total gap: 25826.00 pips
Average bar gap: 0.960 pips
Standard deviation (in pips) on average gap: 1.0771

(it's less than 30k bars because in a month there are =~ 24*60*5/7 M1 bars)

http://img411.imageshack.us/img411/1957/gainm1.jpg

However, if you think about it, a 1 pip gap between one bar and the next isn't really an error in tick data! Still, looking at the 2 pip gaps I'd say the best is FXDD.

My question to you: what other metrics would you use to assess the quality of historical data, apart from bar gaps? Can you tell me of other free data sources I should test?

Actually, because FX doesn't trade at an exchange, it's actually important to get data from the broker that you will trade from (but this is sometimes easier said than done, and the data you get may not be clean/reliable).

In general, the shorter your time-frame for trading, the more sensitive you will be to bad data. Simulating market orders is also tends to be less robust than simulating limit orders.

You have checked the price-dimension of your data. Perhaps you should also check the time-dimension and see if you have gaps in time. Also, you may want to check your two data sources AGAINST EACH OTHER and list/investigate the largest differences.

Also gaps in the FX markets may be less usual than in other types of markets, but it's certainly possible. How do you discern that a gap is "not real"? There may be a gap at one broker, but no gap at a more liquid broker...

Thanks for the reply...

yes, I thought about that and I'll definitely check the three (one of the pictures isn't showing up) against each other. Unfortunately there is no benchmark to compare the data to, so if they are very different there's no way for me to know which one is wrong (apart from the case of gaps). Maybe I should get 4-5 sources and check them all against each other, that way it'll be easier to see if there's an outlier.

When I try to evaluate data, I really just check for gaps in time. Price gaps can happen over the weekend.

Like I said, it depends on your frequency of trading - if you intend to trade at the seconds or <1min time-frame, then >really< clean data is crucial (as close to the trading source as you can).

If you are going to trade on a daily basis, then what you are doing may be overkill. Remember, even if your data is 100% clean, you are still simulating the past, which doesn't need to repeat.

Also, there will be many other assumptions in the simulation engine which may cause greater error than a pip in the data - for example, if you are doing a triangle arbitrage between currencies, your data and clocks had better be in sync, or you will have problems. Although depth/liquidity is usually not an issue in FX, it could be, if you run simulations for institutions. If you do simulate a Market Order, how fast will that order be sent and recognized? In a fast moving market, the answer could be crucial.

That makes sense Ronald, but compare Alpari UK data below to the one from FXDD above:

Bars read: 30000
Gaps: 24710 (82.37 percent of bars)
Total gap: 48256.00 pips
Average bar gap: 1.608 pips
Standard deviation (in pips) on average gap: 2.1752

http://img194.imageshack.us/img194/4370/alparigaps.jpg

I don't think there's any doubt from this pictures that Alpari UK data is worse than FXDD, so I wouldn't completely dismiss bar gaps as a metric -- they can be important for your order fills, too (I tend to trade on the short term, even though I'm taking adequate precautions against bad data).

It is a great idea to perform some statistical calculations on the backtest tick data, because some characteristics can make a big difference when using a scalping EA on the low timeframes. I agree that the historical data does not represent the future quotes, but it can still show a good indication what to expect in forward testing.

I have invested some time and effort to automate the downloading and cleansing of the Gain capital tick data bank, which is indeed huge. Run some statistical analysis and one of the properties checked was the time gaps between successive ticks. Now I see the stats results, but still in dilemma about what to consider as normal and which are the bad gaps. The attached stats does not contain the normal weekend gaps of 2 days.

The ticks are not coming in regular time intervals but sometimes there are even 4 ticks per second, other times there are more than 10 minutes gaps between two successive ticks. The question is what to consider as normal time gap and where to draw the line of bad gaps. What do you guys think… what is the largest time gap between ticks that can be considered as normal, regular feature of real trade?

Another question is whether these gaps are the same in live trade as well (which I would expect, since each tick has its unique Tick ID number and the difference in Tid does not indicate missing ticks). It would be good to perform the same stats on commercial tick data considered to be of good quality (which I haven’t got yet). If there is not much difference in gaps and bad quotes then no need to spend on expensive data.

This is my first post here. Will try to attach 3 diagrams showing the gap stats for Gain Capital GBP/USD 2008 tick data.

Cheers.

Forgot to mention that the horizontal scale on the diagrams represents minutes and the vertical is the number of gaps of specific duration in the batch of data downloaded from the 2008 folder. For example there are 6 gaps of 15 minutes duration. This contains some ticks from the end of 2007 as well so it is little more than 12 months data.