HDF5 for Tick Data

hi,

does anyone of you use HDF5 instead of a Relational database like (MySQL, PostgreSQL)

If so, what is your experience with it for working with Forex Data?

I have investigated it for tick storage and found it sub-optimal (at least for me). The main issue is that it's designed for massive data which does not change. Meaning that you must specify the size of the array (ie: 4 mil ticks). Adding new ticks requires resizing the array (which may be fast). It is possible, but I don't like working this way. BTW, I only investigated it for a couple of hours, so I may be wrong.

I have downloaded all the tick data from 4 brokers: FXCM, dbFX, Dukascopy and GAIN Capital. I have created a custom compressed tick storage format for storing them. I have 10 GB of tick data, which would take 200 GB if they would be stored as uncompressed CSV files. My binary structure and compression achives an average of 2 bytes/tick. I store ticks using the minimum number of bytes required (4 bytes for timestamp, 3 bytes for bid, 2 bytes for the spread, ask being bid + spread). I compress using LZMA (one of the compressors from 7-Zip). I store ticks from each day in a separate file, like FXCM/EUR_USD/2009_01_28.tick, ... I also put in the header of each file all the information I have - broker name, API used, pair name, digit count, date of retrieval, so that if the files get mixed up somehow, I won't have a problem resorting them. I have 100.000 files now, but I want to aggregate them, by storing them in monthly archives.

I tried using SQLite at first, but it doesn't scale, not even for M1 candles. PostgreSQL will handle more, but it certainly can't handle the amount of ticks I have.

At work (I'm a computer programmer) one of our teams use HBase for massive data storage. You may want to investigate that, but it requires a cluster for optimal use (they have 10 machines right now and store around 2 TB of data, 1 billion records).

EDIT: Screenshot of my tick downloaders in progress - http://dl.dropbox.com/u/190212/public/tick.png

Thanks Adal,

looks quite interesting.

For the time being I decided to go with HDF5.

Has anyone has any new input of your tick storage experience? I'm looking into this as well. Thanks!