Tick Database | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

An approach to get MT4 tick data into a PostgreSQL database 10 replies

Tick Size, Tick Rate, Tick Spread, and Tick Logger indicators 36 replies

Tick Database Version 2 90 replies

Community Tick Database 5 replies

Brand-spanking new tick database. 4 replies

Attachments: Tick Database
Exit Attachments

Tick Database

  • #1
  • First Post: Edited Sep 24, 2009 12:05am Sep 23, 2009 11:34pm | Edited Sep 24, 2009 12:05am
Thanks to Rangebound, I finally have a means to allow ForexFactory members to contribute to, and access a tick database. As of right now, there are ~6-7 million ticks collected across 20 currency pairs listed below.

 

  1. AUDCAD
  2. AUDJPY
  3. AUDNZD
  4. AUDUSD
  5. CADJPY
  6. CHFJPY
  7. EURAUD
  8. EURCAD
  9. EURCHF
  10. EURGBP
  11. EURJPY
  12. EURUSD
  13. GBPCHF
  14. GBPJPY
  15. GBPUSD
  16. NZDJPY
  17. NZDUSD
  18. USDCAD
  19. USDCHF
  20. USDJPY

The EA only collects the following data from your terminals:

 

  1. Broker's Name
  2. Broker's Time
  3. Bid
  4. Ask

I will post the source code here and invite any other programmer to analyze the code and confirm that I am not trying to take personal information.

Over the next few weeks I'll be working on developing scripts and applications alongside other developers which will utilize the tick data to improve backtesting results, enable EAs to trade off of historical tick data, and create a whole new class of indicators dedicated to using tick data.

To use the EA:

 

  1. Enter the desired username: I collect the username so I can keep track which contributor contributed which ticks into the database. It does not have to be your ForexFactory username.
  2. Enable Dlls: This EA only uses the wininet.dll. It's the exact same dll as used by the FFCal indicator. Documentation below...

    1. http://msdn.microsoft.com/en-us/libr...8VS.85%29.aspx
    2. http://msdn.microsoft.com/en-us/libr...8VS.85%29.aspx

  3. Attach the EA to the pairs you want.

Attached File(s)
File Type: mq4 Tick Loader 3.mq4   7 KB | 914 downloads
  • #2
  • Sep 23, 2009 11:50pm Sep 23, 2009 11:50pm
What a great idea. I'm definitely on board this one!

I do think that working off tick data is definitely the way forward. If we want to take advantage of market inefficiencies, this is likely best done at the sub 1-minute stage.
  • #3
  • Sep 25, 2009 12:01am Sep 25, 2009 12:01am
how can we use these ticks

unless i am mistaken, a few years ago i discovered you cannot test on old builds of mt4, i tried importing ticks, i ran into a dead end, how do you use these ticks in tester ronald?

after a certain version, importing ticks is no longer a feature
  • #4
  • Sep 25, 2009 12:05am Sep 25, 2009 12:05am
i see a possible way, but id think its to much

are you recording ticks, then during the test, making the ea trade the ticks from a file, on a per bar basis

i thought about doing that a long time ago, but i never did
i figured mt4 would run out of memory and crash
  • #5
  • Sep 25, 2009 1:02am Sep 25, 2009 1:02am
I haven't played with it yet, but I believe that the most recent build which allows this is build 208. That build allows you to use your own .fxt files which you can generate using your own tick data, thus achieving 99% modling quality .

As a small insight into my most recent research, I have been writing a program to write a neural network for me. It takes 3-5 seconds to process each tick, what I'm currently doing is feeding ticks into my own mysql database, and then having the neural network process each tick as they arrive.

In my delayed tick processing, the NN is showing some improvement. It's just a matter of speeding up the time to process each tick. With my introduction to CUDA and Open CL, I'm trying to take advantage of the enhanced processing power to reduce tick processing times to 10 miliseconds or less.
  • #6
  • Sep 25, 2009 1:11am Sep 25, 2009 1:11am
3-5 seconds is very slow

i really hope u succeed

most of the stuff you said is Spanish to me

if you need any help on the mql side, lemme know
  • #7
  • Sep 25, 2009 7:46am Sep 25, 2009 7:46am
I guess the first thing I would like to ask Trendchaser...

Is there anything that can be done to make the code more efficient/faster?

I can modify the receiving web page if you need me to do so.
  • #8
  • Sep 25, 2009 8:10am Sep 25, 2009 8:10am
Hi Ronald,

seems to be an interesting project.
Have found some time ago the attached package with MT4 scripts.
They are collecting tickdata and you can convert them to the standard mt4 history file format. So it should be no major problem to use the tickdata for backtesting ....

Regards,
MaMood
Attached File(s)
File Type: zip temp.zip   69 KB | 570 downloads
  • #9
  • Sep 25, 2009 9:26am Sep 25, 2009 9:26am
use http get, to open the file, write the data to cvs, then open the cvs to use the data

Quoting Ronald Raygun
I guess the first thing I would like to ask Trendchaser...

Is there anything that can be done to make the code more efficient/faster?

I can modify the receiving web page if you need me to do so.
  • #10
  • Sep 25, 2009 9:47am Sep 25, 2009 9:47am
i looked at your code, i see now your uploading ticks, not downloading

i had many beers last night ronald

the code looks cool, a bit over my head even
iv always used ftp to upload

is this faster than ftp?

with my trade copy ea, its hard to tell how long it takes, maybe a few seconds

il post my code for http get, maybe itl be useful?
well its not my code, i found it in the mql database and modified it

this goes up top
Inserted Code
#import "shell32.dll"
int ShellExecuteA(int hwnd,string lpOperation, string lpFile,string lpParameters,string lpDirectory,int nShowCmd );
#import
#include <http51.mqh>

heres a function that gets the file, then writes the data to cvs
Inserted Code
void trade_copy()
{
//------------------------------------------------
 int get;
 string params [0,2];
ArrayResize( params, 0);int status[1];addParam("Bid",Bid,params);addParam("Ask",Ask,params);
string req = ArrayEncode(params);
string res = httpGET("http://trendchaser.org/updates/trades.cvs?"+ req, status);
get=FileOpen("trades.cvs", FILE_CSV|FILE_WRITE, ';');
if(get>0)
{
FileWrite(get, res);
FileClose(get);
get=0;
}
//------------------------------------------------
}

heres a link to 2 files, one goes in the include
the other in libraries
dll in libraries
http://trendchaser.org/updates/files.zip
  • #11
  • Sep 25, 2009 1:16pm Sep 25, 2009 1:16pm
Yes you can use your own .FXT tick files with build 208.

For those that want it ... download the RAR file at the bottom of this forum thread. (You'll have to login to TSD to download)

http://www.forex-tsd.com/expert-advi...ild-208-a.html
  • #12
  • Sep 30, 2009 9:01am Sep 30, 2009 9:01am
R-B: Ok, got build 208 installed ok, but it will not finish registration. Do we have to register in a newer version and copy the files over, or how to get this going.. Any ideas? Thanks, Ken.
  • #13
  • Sep 30, 2009 10:59am Sep 30, 2009 10:59am
Well you can copy any .SRV file into the config folder and then you'll have a choice of servers to connect to. (theres a thread here on FF that has a load of differnt SRV files ready for download)

I dont run build 208 connected to a server. I just used the strategy tester which can be run without a connection. You'll need to copy the data you want (.HST files) to the history folder.

As a test I copied the .SRV file for Alpari Demo from a newer (live running) install (from its config folder) to the build 208 config folder and created a new Alpari account no problem.

Just a note though there may be issues running 208 on some or all servers, and youll obviously have to close the live upddate window everytime you start.

Anyway I'd advise :

1) If you really need to get a newer/better symbols list then create a new account with your preferred broker using a new SRV file.
2) Then block the build 208 App with a firewall or screw up the login settings
3) run another (newer) online version to download your chart data
4) copy the .HST data to the history folder of your build 208


finally to use your own FXT file you'll have to (easiest way)

1) launch strategy tester specifying date range+ $pair and tick the recalculate box.
2) Quit the test you now have a modelled .FXT file (MT4 just created it)
3) code an App (could build a MT4 script) to build a new FXT file, copying the Header from the FXT file above, and then inserting your own tick data after it
(for file structure info press F1 from MT4 then look in :
Auto Trading->strategy testing->history files in FXT format

4) restart the strategy tester with the same date range + $pair but with the recalulate box unchecked.


HTH
  • #14
  • Sep 30, 2009 3:28pm Sep 30, 2009 3:28pm
R-B: Thanks! That got me going. . ;=)) I wonder why they dropped this feature.. ???
(Just a note for some others doing this: I've noticed before that I got an error on minimum lot size (0.01) when not connected to the server, but connected it worked ok. This was not a MarketInfo call, just placing the 0.01 in the lots to open. Of course the way around that is to just bump up the lots until it works.)
  • #15
  • Oct 2, 2009 8:29pm Oct 2, 2009 8:29pm
Hello Ronald,Pleased to meet you here. Sorry to be off subject here but I wonder if you know of an alarm for an Envelope touch high and low. Something like alert me when it hits the top and again when it hits the bottom.Thanks in advance and thankyou so much for all you do here at FF for all of us newbies. I have looked for a while at FF and can't seem to find any. Thanks again. Good trading to all Mr D.
Jesus my Lord
  • #16
  • Oct 7, 2009 1:12pm Oct 7, 2009 1:12pm
Maybe you have already heard about Nvidia's personal supercomputer. If not, this might be interesting to you.

Quoting Ronald Raygun
With my introduction to CUDA and Open CL, I'm trying to take advantage of the enhanced processing power to reduce tick processing times to 10 miliseconds or less.
  • #17
  • Nov 14, 2009 3:15am Nov 14, 2009 3:15am
not that expensive either....
Nihil inimicus quam sibi ipse Marcus Tullius Cicero
  • #18
  • Dec 2, 2009 12:28pm Dec 2, 2009 12:28pm
how do we get access to the tick data?
Can you post exports from mysql so we can download?
  • #19
  • Dec 2, 2009 1:10pm Dec 2, 2009 1:10pm
I'm working on a way to get users to interact with the database without bringing it down. I have a temporary but very crude way to download ticks.

go to this page: http://forex.offers4u.biz/Tick%20Request%20Form.html

Enter in the currency pair and the index. Each pair has between 1 million and 3 million ticks. Ticks are downloaded through this form in batches of 50,000.

So, if I wanted ticks 0-50,000 from EURUSD I put:

Currency Pair: EURUSD
Tick Index: 0

I then get ticks 0 to 50,000

If I want 50,000 to 100,000...

Current Pair: EURUSD
Tick Index: 50,000


I realize this is incredibly crude, but I currently am trying to figure out the load balancing required. If anyone knows how to set cron jobs to automatically back up the database into a set of excel files (that I can put on a download page on the website) that would be most appreciated.
  • #20
  • Dec 2, 2009 2:44pm Dec 2, 2009 2:44pm
Quoting Ronald Raygun
I realize this is incredibly crude, but I currently am trying to figure out the load balancing required. If anyone knows how to set cron jobs to automatically back up the database into a set of excel files (that I can put on a download page on the website) that would be most appreciated.
Hi Ronald.

I don't know. But .xls? are u sure about that? I would think pure ascii code would be easier to handle..

btw, I do not fully understand: for the tickLoader, will it continuously grab ticks as I have MT4 open, or is it kind of a one-time grabbing of old data stored in ticks in my history? I guess the first. but does it have any impact on my stations performance?
What is 2-3millions ticks approx. worth in years? 1 year? 3?

Lowen
0 traders viewing now