Tick Database Version 2 | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

An approach to get MT4 tick data into a PostgreSQL database 10 replies

Tick Size, Tick Rate, Tick Spread, and Tick Logger indicators 36 replies

Tick Database 25 replies

Community Tick Database 5 replies

Brand-spanking new tick database. 4 replies

Attachments: Tick Database Version 2
Exit Attachments

Tick Database Version 2

  • #41
  • Dec 29, 2009 11:12am Dec 29, 2009 11:12am
Ahh. Calculate the difference up front, I take it?
  • #42
  • Dec 29, 2009 11:25am Dec 29, 2009 11:25am
Difference is calculated locally within the metatrader terminal. The terminal uploads both times to the server anyway, so I can see which broker changes for daylight savings and when.
  • #43
  • Jan 1, 2010 3:16am Jan 1, 2010 3:16am
Quoting Ronald Raygun
Microsecond level would be nice. It's very possible to capture time at the milisecond level within metatrader. The problem is the lag between the contributing EA, and my webserver.

I figure what is more important is the order of the ticks as opposed to the absolute specific time they happened.
This is a bit off-topic, but how would you propose to get to the millisecond level within MetaTrader since the highest resolution I am aware of is from TimeCurrent() which is in seconds.
  • #44
  • Jan 1, 2010 6:18am Jan 1, 2010 6:18am
There might be a DLL that can do it. I know there's a dll that does clock ticks.
  • #45
  • Jan 1, 2010 9:01am Jan 1, 2010 9:01am
Quoting algotrader
This is a bit off-topic, but how would you propose to get to the millisecond level within MetaTrader since the highest resolution I am aware of is from TimeCurrent() which is in seconds.

Relatively Easy:

int GetTickCount( ) The GetTickCount() function retrieves the number of milliseconds that have elapsed since the system was started. It is limited to the resolution of the system timer.

 

  1. Get the current time when the EA was started
  2. Figure out how many seconds passed,
  3. Add the remainder to the timestamp sent to the server.

At most, you will be 1 second off.

I'm not sure if this is a good idea, but as I understand it, time is only used to determine the order that ticks arrived. Is there any reason to use time for time itself?

  • #46
  • Jan 1, 2010 12:10pm Jan 1, 2010 12:10pm
Quoting Ronald Raygun
I'm not sure if this is a good idea, but as I understand it, time is only used to determine the order that ticks arrived. Is there any reason to use time for time itself?
I'm no expert either, but for some volatility computations it could be useful (ie, a 5 pip increase in 3 ticks in 20 ms is different from a 5 pip increase in 3 ticks in 800 ms). On the other hand if you need this kind of information, you probably don't trade through a MT4 broker and you already have high quality ticks.
  • #47
  • Jan 1, 2010 12:48pm Jan 1, 2010 12:48pm
Well, ping times are 50/100ms... so... a lot will depend on your connection.
  • #48
  • Jan 1, 2010 1:21pm Jan 1, 2010 1:21pm
I see. I was considering building a tick database collecting the time in milliseconds, bid, ask for a given symbol through an EA with a connection to a database over an imported DLL, but I wasn't able to get the functions of the DLL to be callable from MT4.

@Adal: high quality ticks? from an institutional brokerage? or a FIX server? Please elaborate.
  • #49
  • Jan 1, 2010 1:47pm Jan 1, 2010 1:47pm
algotrader, I have a mysql dll which may be useful to you if you are creating a local tick database.
  • #50
  • Jan 1, 2010 4:48pm Jan 1, 2010 4:48pm
Quoting Ronald Raygun
algotrader, I have a mysql dll which may be useful to you if you are creating a local tick database.
Is it publicly available? I was going to PM since this is getting off-topic, but I can't since I'm still a junior member.
  • #51
  • Jan 1, 2010 5:23pm Jan 1, 2010 5:23pm
Quoting algotrader
I see. I was considering building a tick database collecting the time in milliseconds, bid, ask for a given symbol through an EA with a connection to a database over an imported DLL, but I wasn't able to get the functions of the DLL to be callable from MT4.

@Adal: high quality ticks? from an institutional brokerage? or a FIX server? Please elaborate.
From my searches this is considered the highest quality FOREX tick data available. http://www.olsendata.com/data_products/historical_data/ Check out the price for 3 years of EUR/USD tick data. BTW, you can only get tick-level data if your an institution, they don't sell to individuals (probably because they fear it will be leaked). For another €4000 you can get the name of the bank which provided each tick (this is interbank data).

http://imgur.com/FQqJy.png

For my own use I have first downloaded Gain Capital tick data, but it has some issues. Now I'm working with FXCM tick data (taken from a demo account using their API), which is much denser (some seconds have 12 ticks in them). I'm also investigating Dukascopy tick data, which is even denser, timestamped to the millisecond, and it also has bid/ask depth available for each tick.

I don't believe in extracting tick data locally, since you don't know how fresh it is (latency). Also, it's really hard to collect continuously (ie, you got to reboot from time to time).
  • #52
  • Jan 1, 2010 6:52pm Jan 1, 2010 6:52pm
@Adal: I see your point. It would be nice to have high quality tick data. Have you considered that although you can get higher quality tick data, the methods on which you derive your trading from should be derived from tick data that you can actually trade with. In other words, you can collect all the high quality tick data that you want, but you don't have access to trade on that "higher quality market".

I looked at the latency to the MT4 servers on my account and it's around 100-200 ms on my current connection. I plan on finding a VPS host to record the tick data if I ever get it to work that would address the continuity and latency issues most likely.
  • #53
  • Jan 2, 2010 7:00am Jan 2, 2010 7:00am
Quoting algotrader
@Adal: I see your point. It would be nice to have high quality tick data. Have you considered that although you can get higher quality tick data, the methods on which you derive your trading from should be derived from tick data that you can actually trade with. In other words, you can collect all the high quality tick data that you want, but you don't have access to trade on that "higher quality market".
That's why I'm using FXCM data, since I plan on dealing with them. That's also why I'm still not using the Dukascopy tick data, even if it seems better, because I don't think that they are as regulated as a UK broker.

However, for slower types of trading (H4 or daily candles), you don't need tick data from your actual broker. Any decent data is good enough.
  • #54
  • Jan 16, 2010 12:30pm Jan 16, 2010 12:30pm
Here is one site where you can find a lot of information about PHP.
  • #55
  • Jan 30, 2010 9:25am Jan 30, 2010 9:25am
The first batch of ticks have been delivered.

www.RonaldRaygunForex.com/TickDB/Archive

Currently, it's unrestricted, but if demand becomes too high, I will either restrict the download speed, the number of simultaneous connections allowed, or both. As an option, will someone take the responsibility of uploading the files to rapidshare, megaupload, or similar websites? That way, we can at least host these files in a cost-effective way.
  • #56
  • Jan 30, 2010 2:38pm Jan 30, 2010 2:38pm
Quoting Ronald Raygun
The first batch of ticks have been delivered.

www.RonaldRaygunForex.com/TickDB/Archive

Currently, it's unrestricted, but if demand becomes too high, I will either restrict the download speed, the number of simultaneous connections allowed, or both. As an option, will someone take the responsibility of uploading the files to rapidshare, megaupload, or similar websites? That way, we can at least host these files in a cost-effective way.
I take that responsibility, as have experience in file hosting and with various hosts.
But which files are to be uploaded? the biggies with no time stamp in their name. right?
Are the files with date stamps are dynamic (weekly) and for current month only and after month end, they are added to main biggies right?
Let me know
Regards

P.S. i have not been able to feed tick loader on mt4..throwing errors like anything?what about that?
100 Fold Challenge->Interested? ->https://www.forexfactory.com/thread/32152
  • #57
  • Jan 30, 2010 2:52pm Jan 30, 2010 2:52pm
The files with no timestamp are from my private database, and should have data back to september.

Timestamped data is from this version of the database using [TD] enabled EAs.

The most recent TickLoader should work. The important part is that it is pointing to RonaldRaygunForex.com

Thanks for your help ForexSaint!
  • #58
  • Feb 1, 2010 6:20am Feb 1, 2010 6:20am
Don't you think these files can be trimmed down a bit by just storing
Fxpro instead of whole "FXPRO Financial Services Ltd; " &
CFx/CllFx instead of "The Collective FX LLC; " &
likewise
May be a a weird thought.. i am yet to use this data!
Will soon pm.
100 Fold Challenge->Interested? ->https://www.forexfactory.com/thread/32152
  • #59
  • Feb 1, 2010 10:05am Feb 1, 2010 10:05am
It's possible to abbreviate the broker name. But there are so many brokers...
  • #60
  • Feb 1, 2010 11:06am Feb 1, 2010 11:06am
Quoting Ronald Raygun
It's possible to abbreviate the broker name. But there are so many brokers...
I think harnessing proper broker feeds (by proper, i mean those who are into business from some good time now) from 10-20 brokers is enough. isn't it?
Otherwise there are many uber-bucketshops (of course proper ones are also to some extent), that they come and go..how and when no1 knows.
Further the feeds from those brokers will be nicely manipulated too.

I think you should, in fact, limit feeding to some fixed brokers only trash other tick feeding just like demo account feeding.
To name most (Abbv. in brackets) :
Fxpro (use FxPro)
Collective Fx (CFx)
MbTrading (MBT)
NordMarkets (Nord)
ATC (ATC)
Ducascopy (DC)
InterbankFx (IFx)
AlpariUK (AiUK)
AlpariUS (AiUS)
BrocoTrader (BT)
FxcmUK (FUK..lol)
FxcmUS (FUS)
Fxdd (FxDD)
ODL (ODL)
GoMarkets (GM)
MiG (MiG)
Tadawul (TW)
are enough?

I hope i am not talking something improper. Those big names constitute almost 33% of one line record. Can be trimmed nicely imo.
Regards.
100 Fold Challenge->Interested? ->https://www.forexfactory.com/thread/32152
0 traders viewing now