Tick Database Version 2 | Page 4 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

An approach to get MT4 tick data into a PostgreSQL database 10 replies

Tick Size, Tick Rate, Tick Spread, and Tick Logger indicators 36 replies

Tick Database 25 replies

Community Tick Database 5 replies

Brand-spanking new tick database. 4 replies

Attachments: Tick Database Version 2
Exit Attachments

Tick Database Version 2

  • #61
  • Feb 1, 2010 11:34am Feb 1, 2010 11:34am
The main reason behind including the broker name is for a few reasons:

 

  1. With several broker feeds (bucket shops too) we can calculate an average price between brokers and further figure out what happens when a broker moves away from the norm (does the price retrace?)
  2. We can detect broker spikes, record them, and see which brokers 'cheat'

If you can find a universally agreeable way to abbreviate the broker name I'll go ahead. I'm not sure how you're handling the .csv files, but I would imagine a simple "Find and replace" should be sufficient.

  • #62
  • Feb 1, 2010 11:47pm Feb 1, 2010 11:47pm
Quoting Ronald Raygun
The main reason behind including the broker name is for a few reasons:

 

  1. With several broker feeds (bucket shops too) we can calculate an average price between brokers and further figure out what happens when a broker moves away from the norm (does the price retrace?)
  2. We can detect broker spikes, record them, and see which brokers 'cheat'

Yes agree. But we can find all that info from simple abbreviations too.
"FxPro" instead of "FXPRO Financial Services Ltd" will be more than enough.

Quote
If you can find a universally agreeable way to abbreviate the broker name I'll go ahead.
I think you are doing it (this tick feeding Dbase idea)for the first time,so why we need everyone to agree. You can just abbreviate the common brokers (excluding "too new" or 'Uber-bucketshops") as per your liking & being bit meaningful (or based on my last post); Like everyone can make out what is FxPro or ATC or MiG or AiUK or MBT and likewise.
Still for some others brokers, you can quote this all into the tick feeder itself as some info or in first post itself.

Quote
I'm not sure how you're handling the .csv files, but I would imagine a simple "Find and replace" should be sufficient.
Yes..i have done that to decrease file size...(i am using EditPlus)
but if the tick feeder itself does it to the Dbase entries..it would be lot better.

Regards
100 Fold Challenge->Interested? ->https://www.forexfactory.com/thread/32152
  • #63
  • Mar 11, 2010 11:10pm Mar 11, 2010 11:10pm
Quoting Ronald Raygun
The main reason behind including the broker name is for a few reasons:

 

  1. With several broker feeds (bucket shops too) we can calculate an average price between brokers and further figure out what happens when a broker moves away from the norm (does the price retrace?)
  2. We can detect broker spikes, record them, and see which brokers 'cheat'

If you can find a universally agreeable way to abbreviate the broker name I'll go ahead. I'm not sure how you're handling the .csv files, but I would imagine a simple "Find and replace"...

Excellent job!
Could you add a checksum file?
or maybe just zipped the text file ? that would save lots space for your hosting
  • #64
  • Mar 12, 2010 7:34am Mar 12, 2010 7:34am
If you know how to zip files using php code, I'd be more than open to implementing it into the system. As it stands, downloads from the repository have been slow at best, and that's probably because metatrader can't take tick data.

I've been doing some statistical research at hour starts using the tick data. E.G. on GJ, if the spread is less than 7 pips, and the price moves 5 pips in one direction, there is a 60% chance that the price will continue in the same direction for another 10 pips.

I've been busy trying to create a large distributed computing platform using metatrader terminals to try and collect this information, so the community can benefit from it.
  • #65
  • Mar 12, 2010 3:17pm Mar 12, 2010 3:17pm
Hi all,

I am enchanted by the idea, this is ingenious!

However it seems to me like the implementation is struggling. Would you be interested in some professional IT help?

PHP + MySql is a huge overhead over the task of collecting ticks, I think we can put together, rather easily, a solution that will drop the CPU load 10x or more.
  • #66
  • Mar 12, 2010 3:20pm Mar 12, 2010 3:20pm
I'm all ears!
  • #67
  • Mar 12, 2010 3:47pm Mar 12, 2010 3:47pm
I can suggest a custom file format .NET based solution for the storage. I have a module (open source LGPL) that can do about 1 mil entries per second in this mode. And reading speed is also in that range.

I think the best solution would be to dump all the incoming data on the HD, and than have a secondary process that processes it, and extracts some CSV or other format from it. This way the runtime load can be very low, also the processing/filtering can become better over time, and still be applicable to previously collected data. Also this process can be easily stored on another machine, so that the receiving web server is as free as possible (low hosting payment).

If you need to do sequential operations on large arrays of data, like tick data, a relation database is really not the way to go. There are another type of DB known as ODBMS (Object based data base), much faster than RDBMS, however a custom binary file solution wins over them as well many times over.

Sorry if some of this is redundant, I am not fully familiar with your existing infrastructure.

I can also provide the coding for this for free, if you go this way.
  • #68
  • Mar 12, 2010 3:59pm Mar 12, 2010 3:59pm
If you can call me via skype or the phone number in my profile, I'd love to go over this with you.
  • #69
  • Mar 12, 2010 4:43pm Mar 12, 2010 4:43pm
Sure, I will try to get in touch with you in the next couple of days.
  • #70
  • Mar 14, 2010 4:36am Mar 14, 2010 4:36am
agree with drmkkk, I'm checking the "FXTicksCollector" these day, To binary file really more efficient than recording the ticks to RDBMS.Ronald , I don't familar with php either . but seems create a md5 string for a file is quite simple: http://www.php.net/manual/en/function.md5-file.php
  • #71
  • Mar 14, 2010 1:52pm Mar 14, 2010 1:52pm
Let us know how its going guys, I haven't been able to use the information properly yet.
The greatest force on earth is compound interest.
  • #72
  • May 23, 2010 4:18pm May 23, 2010 4:18pm
Quoting Ronald Raygun
and that's probably because metatrader can't take tick data.
I just ran into this problem, is there a method for adding it to mt4?
  • #73
  • May 23, 2010 4:32pm May 23, 2010 4:32pm
There is now: http://eareview.net/tick-data
  • #74
  • May 23, 2010 10:39pm May 23, 2010 10:39pm
Well, I don't know what your EA does nor DLL. If you would post those so they could be examined would be a help. Firstly, I think the tick data could be sent with a HTTP Post to the server. On the server a simple C++ program would append it to the symbol/trader/source file. Not a .NET program since that limits your server to MS. Probably best would be appending to a file on the client and send a days worth of data via FTP. Yeah, thats probably best. Instead of sending ticks, send a days worth in a file.

How's that grab you?
  • #75
  • Nov 26, 2010 7:07am Nov 26, 2010 7:07am
crashes :-(

Inserted Code
There has been a critical error
Time        : 2010.11.26 13:06
Program     : Client Terminal
Version     : 4.00 (build: 226, 24 Mar 2009)
OS          : Windows 2003 Domain Controller 5.2 Service Pack 2 (Build 3790)
Processors  : 4 x X86 (level 6)
Memory      : 2096124/1177816 kb
Exception   : C0000005
Address     : 001A44FC
Access Type : read
Access Addr : 00000000

Registers   : EAX=00000000 CS=001b EIP=001A44FC EFLGS=00010246
            : EBX=0000005C SS=0023 ESP=04D5F804 EBP=0019CD58
            : ECX=00000057 DS=0023 ESI=702226D8 FS=003b
            : EDX=00000000 ES=0023 EDI=001A43F0 GS=0000

Stack Trace : 6D75636F 00000000 00000000 00000000
            : 00000000 00000000 00000000 00000000
            : 00000000 00000000 00000000 00000000
            : 00000000 00000000 00000000 00000000

Modules     :
          1 : 00400000 004BB000 c:program filesmetatrader - alpari uksl75tp50h1terminal.exe
          2 : 027A0000 00009000 c:windowssystem32normaliz.dll
          3 : 02BB0000 00017000 c:windowssystem32odbcint.dll
          4 : 038B0000 002C5000 c:windowssystem32xpsp2res.dll
          5 : 403F0000 000E6000 c:windowssystem32wininet.dll
          6 : 40A90000 001E8000 c:windowssystem32iertutil.dll
          7 : 48890000 0003D000 c:windowssystem32odbc32.dll
          8 : 4B3C0000 00050000 c:windowssystem32msctf.dll
          9 : 4DC30000 0002E000 c:windowssystem32msctfime.ime
         10 : 5F270000 0005A000 c:windowssystem32hnetcfg.dll
         11 : 60970000 0000A000 c:windowssystem32mslbui.dll
         12 : 6F350000 00133000 c:windowssystem32urlmon.dll
         13 : 70200000 00094000 c:program filesmetatrader - alpari uksl75tp50h1expertslibrarieswininet.dll
         14 : 71AE0000 00008000 c:windowssystem32wshtcpip.dll
         15 : 71B20000 00041000 c:windowssystem32mswsock.dll
         16 : 71B70000 00036000 c:windowssystem32uxtheme.dll
         17 : 71BB0000 00009000 c:windowssystem32wsock32.dll
         18 : 71BF0000 00008000 c:windowssystem32ws2help.dll
         19 : 71C00000 00017000 c:windowssystem32ws2_32.dll
         20 : 73EB0000 00121000 c:windowssystem32mfc42.dll
         21 : 75E60000 00027000 c:windowssystem32apphelp.dll
         22 : 76190000 00012000 c:windowssystem32msasn1.dll
         23 : 761B0000 00093000 c:windowssystem32crypt32.dll
         24 : 76280000 00005000 c:windowssystem32msimg32.dll
         25 : 76290000 0001D000 c:windowssystem32imm32.dll
         26 : 762B0000 00049000 c:windowssystem32comdlg32.dll
         27 : 76AA0000 0002D000 c:windowssystem32winmm.dll
         28 : 76F50000 00013000 c:windowssystem32secur32.dll
         29 : 77380000 00091000 c:windowssystem32user32.dll
         30 : 77420000 00103000 c:windowswinsxsx86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.3790.4770_x-ww_05fdf087comctl32.dll
         31 : 77670000 00139000 c:windowssystem32ole32.dll
         32 : 77BA0000 0005A000 c:windowssystem32msvcrt.dll
         33 : 77C00000 00049000 c:windowssystem32gdi32.dll
         34 : 77C50000 000A0000 c:windowssystem32rpcrt4.dll
         35 : 77D00000 0008B000 c:windowssystem32oleaut32.dll
         36 : 77E40000 00102000 c:windowssystem32kernel32.dll
         37 : 7C800000 000C2000 c:windowssystem32ntdll.dll
         38 : 7C8D0000 007FF000 c:windowssystem32shell32.dll
         39 : 7D180000 00052000 c:windowssystem32shlwapi.dll
         40 : 7D1E0000 0009C000 c:windowssystem32advapi32.dll

Call stack  :
  • #76
  • Nov 26, 2010 7:23am Nov 26, 2010 7:23am
just tried again after a reboot, exactly same with a different address
  • #77
  • Nov 28, 2010 10:11am Nov 28, 2010 10:11am
Thanks for the update ben. I'm looking into it.
  • #78
  • Nov 28, 2010 10:31am Nov 28, 2010 10:31am
Quoting Ronald Raygun
Thanks for the update ben. I'm looking into it.
are you still running/using this? i grabbed the dukas ticks as yours stopped in August
  • #79
  • Nov 28, 2010 10:47am Nov 28, 2010 10:47am
Ooh

Another bug. I need to check the code that updates the repository. Thanks again for the update!

And yes, I do use this database every day. But I store everything in a private database so I can query it at will.
  • #80
  • Jan 4, 2011 8:37am Jan 4, 2011 8:37am
Great project.

Sorry for what is probably a very stupid question but how do I import the tick data into Metatrader to be used for testing an EA?
1 trader viewing now