Tick Database

1. I'm not sure how to do that.
2. It continuously uploads ticks as they happen. Metatrader terminal does not store ticks as far as I know.
3. The length of time varies. For example: 1 million ticks of NZDJPY is 3 months. 1 million ticks of GBPJPY is only one month. (Yes, this is very dense)

ok
wow i didn't know the tick frequency was so high..

Not really.

It averages out to about 16 ticks a minute, which is about what you normally see on a chart.

Ronald,

Congratulations on the effort.

I stopped working with ticks years ago (pre-internet), because of problems with the various softwares that were available at the time. Not having looked at the new and improved products out there, and therefore I have no idea if the issues I faced are relevant in todays world. I thought I would share some of the walls I hit, so that you can insure you don't smack your head against the same ones. Although I will number them everything related to one issue.

1. Ticks were not time stamped to the fraction of a second.
2. Order of historic ticks in the creation of or build of a bar was not as received or viewed. Prices within the bar would move if the data load was refreshed. The order of tick display was controlled by some scheme, proprietary to the company.
3. The values of indicators or "signals" (EA types of programs) produced one result live and created a different result historically. In many cases, signals would vanish, always benefiting a historic result over a live one.
4. Historic tick bars were built using open, high, low and close only.

Additional considerations

I assume the data is bid ticks. Is the tick created with a transaction or when the quote charges? Work previously done had significant result differences based on this issue alone.

Depending on the above answer, not having a centralized exchange could mean a significant difference in tick counts between brokers.

Good luck,
BR

Thank you for your concern bigred:

 

1. Since this is more for testing purposes as opposed to analytical purposes, all that is important is the order of the ticks. I don't think I need the time down to the millisecond. I know metatrader has a function to generate milliseconds, the problem is lag between the terminal and the receiving website and compensating for that lag.
2. I don't think this is going to be a problem, but I'll keep an eye out.
3. This will always happen no matter what I do. The best thing to do in this situation is to build in a buffer for forward testing. If results were +/- 5% of the expected result, then I'd say it's a good test.
4. This is in the strategy tester of MT4. We're trying something a little different.

The data we're collecting is Bid, Ask, Time, and broker.

All the data is centralized against one central database. Command central as it were.

I could write you a perl or python script to do this, at least into csv. Another way would be to dynamically stream the file with a cgi/php/whatever script. We could make a nice little web api to gain access to the data also...