Tick Database Version 2

After going through a few months with the first Tick Database, there were a few glaring flaws that I noticed:

 

1. Retrieving the data was hard for any end user.
2. Query collisions were frequent.
3. Downstream users were sometimes crashing the server.

With that in mind--and the fact that my hosting agreement was almost up--I took the opportunity to recreate the database and hopefully make it more usable for everyone to have usable tick data.

The database is currently set up for 20 pairs:

 

1. AUDCAD
2. AUDJPY
3. AUDNZD
4. AUDUSD
5. CADJPY
6. CHFJPY
7. EURAUD
8. EURCAD
9. EURCHF
10. EURGBP
11. EURJPY
12. EURUSD
13. GBPCHF
14. GBPJPY
15. GBPUSD
16. NZDJPY
17. NZDUSD
18. USDCAD
19. USDCHF
20. USDJPY

If there are any other pairs you want to add to the database, let me know. I just ask that if you want to have those pairs, that you please contribute to them.

Every Saturday, the previous week's data will be exported to a csv file and saved in a folder on the site. I'm currently in the process of migrating the old database into the new one, so previous tick data will be available soon.

Tick Loader 4.mq4 is the primary method of contributing ticks to the database. All you need to do is apply it to a chart, and make sure that DLLs and EAs are enabled.

The other method to contribute ticks is through some of my specially marked EAs. Any EA released by me with the [TD] designation has the capability to contribute to the tick database. Those EAs will have a parameter marked "TickDatabase", set it to true to enable the EA to send ticks.

There are only 5 pieces of information I am collecting:

 

1. Broker Time
   1. Knowing the broker's time will be useful in trying to line up tick feeds together.

2. Bid
   1. Need the bid price

3. Ask
   1. Need the ask price

4. Broker Name
   1. It will be useful to know where a tick came from, especially if--after analysis--it becomes known that a broker's tick feed is bad.

5. UserName
   1. In the event that the costs to run the database becomes prohibitively high for me to support it myself, I may turn the website into a membership site which allows downloads based on amount paid, or ticks contributed. By associating a tick with a user, I can keep track of who contributed how many ticks.

Nothing will be personally identifiable. I am not collecting account numbers, account margin, etc.

For any developers who want to utilize the tick database, or a slightly-delayed tick stream, please contact me either via PM, or my Support Ticket Center and I will get you set up.

This is a great idea Ron, mt4 backtest data is poor at the best of times. I missed your original thread on it.

Many won't like to enable dll due to the hacking potential, am I assume you would allow your source to be read by a programmer to confirm nothing untoward is going on?

Also, to contribute, do I just need to add it to 1 chart or to 1 of each of the pairs I wish to contribute?

Source code is all up there. I didn't write the dll, microsoft did. It's the same DLL that Fxcal uses. It's basically a mini-web browser.

1 EA for each pair that you wish to contribute.

Cheers Ben!

Is the EA still sending each and every tick separately?

Yes.

I updated to version 4, can you verify if it's working. I no longer see IBFX information updating on the web site for GBP/CHF.

I switched back to version 3 and I see the updates, I'll switch back to 4 and let me know if you see anything on your end.

Yep, it's working. Check your PM for more details.

Nice to see the 2nd thread..yes these things get better with time.

Do you have any way to filter demo ticks?
I dont think demo tick accumulation in between the live ones.. will be good for the databse. We all know brokers dont care about demo stuff..and spreads are quite different.
or tickloader doesnt work on demo?

Regards

It works on demo, that's not really a problem, and yes, there is a delimiter between demo and live ticks, so they can be filtered out.

I have a slightly larger problem.

The response was far bigger than I had expected, and my webhost shut the website down. I still have backup ticks feeding at another server, so no data is technically lost.

Here are my options:

1) Limit contributors.
2) Dedicated hosting.

Limiting contributors is not really something I want to do, but it seems that I might just have to do it.

I could also go the dedicated hosting route, but that will cost a minimum 149/month (as opposed to the $3/month I currently pay). Will anyone be willing to contribute fiscally?

Twee, I know FF has considerable resources, would you and the rest of the FF dev team consider a partnership? The entire infrastructure and backbone of the tick collection system is set up, I just need to find a home capable of running it.

I'm holding a video/audio conference to discuss this problem and what options you would rather see me take.

You can join me here: http://www.tokbox.com/conf/x62kodgz9pbu8ctg

Ok, here's the situation...

In the past 3 days, thanks to your contributions, the database collected nearly 3 million ticks. That averages to about 11 ticks per second, or 660 ticks a minute. The server the primary database was hosted on could not handle the load and the website was disabled.

My backup database is still collecting ticks, so there is no loss in data continuity.

I'm not sure how I'm going to distribute the data, but I am certain that it will be a long time before I am able to provide any sort of tick feed. Instead, I am in the process of exporting data to .csv files, and uploading them en-masse to various data-hosting websites such as Rapidshare, megaupload, etc.

I'm currently working on building a local mysql server to handle the incredible amount of data that you all contributed.

If anyone else here understands LAMP or can help build something similar, I would love to speak with you.


I should also note that all of the [TD] enabled EAs will still work. The exception is HiLo Trader [TD]. That was the only EA which relied on a stream of ticks downloaded from the database. The EA will still work, but it will only be using bar data.

3 million ticks! That must take up a lot of space!

What if you split it up each day? Like create a cronjob that runs every 24 hours to save the last 24 hours worth of ticks to a csv file and deletes the data from the database? Then whoever wants to download can download all the csv files and compile them however they wish.

Working on it.

The problem with running a daily cron job is the risk of skipping some data. I had set a weekly cron job to put the ticks in a csv file. It was supposed to run saturday at midnight (my time) but obviously won't run until I can find a better more reasonable host.

3 million ticks is about 100MB if you're trying to scale it. The main problem I ran across was too many simultaneous incoming streams to one database. Do you know anything about delaying or redistributing the load elsewhere? (E.g. another database on a different server?)

Just came back.
A good quality Shared host, i know of starts with 2.5/month...may be 2 accounts on it? or What about the Gogrid trial thingy..thats approx 100days with smallest ram server (incoming bandwidth is free on it)

As regarding saving CSV files and bandwidth issues with them... many good (YES GOOD,even non deleting) free hosts are around. So we dont need bandwidth actually for it.

The discussion seems to be over as per the link. Any decisions?

Currently, I'm trying to get a MySQL server set up in my dorm. I'm working with UITS (campus IT department) to approve this project.

I considered multiple accounts. But there's a problem...how do I program the load-balancing?

Go-Grid is my last resort, although... if I have a ready market to buy tick data, I might consider going this route: http://www.gogrid.com/bizspark/

As I'm seeing it: People who contribute to the tick database can retrieve x multiple of the ticks they contributed. I doubt any of you would like that. I would also consider charging people who don't contribute at all...

Orr... set up the tick database as a membership website, where the membership fee is a straight cut of whatever my costs that month are. I'm looking at ~$150/month for a fully dedicated server. I really only need a dedicated MySQL server, the web interface can be run on a shared host.

If anyone can find a nice economical mysql hosting platform, I'd certainly be interested.

I'm anticipating usage at 2,000 INSERT requests (ticks) a minute based on my estimates on how many people are actively using my EAs.

I could possibly host this, how much bandwidth are we talking? How much cpu useage etc do you estimate?

Bandwidth should be minimal, around maybe 10GB/month uploading ticks and exporting them to a free host.

CPU usage might be a problem.

When JustHost suspended my account, they told me I was using 30% of the CPU. The database was around 800 insert queries per minute when that happened. If you'd like to run some benchmarks let me know and I'll send you the source code.

I just opened an new account with WebHostingPad.com

Hit it with everything you got!

Its a quad core, lets see where this goes, maybe I can pop on a VM for you.

Hi RR.. won't the basic Cloud server with those trial $$$ will help?
The basic Ram of 0.5gb and all incoming traffic free amounts to about free 40-50days. Those who are really involved & interested can sign up individually totaling to about atleast some 5-6months..that's enough i think, at least as of now

Anyway...the new server u got is which one? basic shared hosting?
Dont think it can take it ....though its holiday season...lets c

Any ways, i am shooting with 12 pairs on 2 Live brokers..Alpari Uk and FxPro

I think you should accept/add 2 out of Gold/Oil/Silver.
They are commonly traded and quite often on many brokers.
Let me know..if you can.

Regards