MT4 Historical Rates - Backtest

20131208
I will not anymore update/support this tool: reason: Going Platform independent! Going pyhon!
I stopped following this thread.

LATEST INFO: UPDATE: 20131007

MT4 Historical Rates


THIS is an SCRIPT for Metatrade MT4: which will help producing/converting/importing historical rates data.
- multiple file conversion (WHOLE FOLDERS)
- CSV file conversion
- MT4 HST file conversion
- Multiple Period generation from M1 files: (M1, M5, M15, M30, H1, H4, D1, W1, MN1)
- data filter
- Report Log
- runs also in unconnected state


I offer this free of charge for anyone who wants (only .ex4 file)



DISCLAIMER: !!!USE AT OWN RISK!!! You assume <full responsibility> for all risk associated.

COPYRIGHT: Copyright 2011,2013 Ati;


Cheers

ATI



++++++++++ Before reporting any Errors please check out the last version: anyway I do not promise to fix any errors

UPDATE: 20131007
- Upload file: sGenerateHistoryFiles_ATI_20131003_OBF.zip
- Helpfile small correction (added missing info about hst. file generation for Period W1, MN1
- Fixed ERROR: if one used in the sourcefile for example 50 files at once (1 got after the 30. processed file an Error: too many files open)
- added Dukascopy JForex-Default IN.CSV.Header.Format for easy copy




UPDATE: 20130816
- removed TimeHourShift completely
- added Zero Price value filter
- added support for all MT4 Periods (TimeFrame Interval TFI W1, MN1 also supported)
- First and Last Result bars are not written anymore: as they could have only partial data if the source starts/ends in the middle of e.g. month


UPDATE: 20130621
- re-uploaded: README file change

UPDATE: 20130611
- small update

UPDATE: 20130529
- first uploaded




sGenerateHistoryFiles_ATI_20130816_OBF.zip 62 KB | 56 downloads | Uploaded Aug 16, 2013 11:17

ok here it is what I do at the moment. If you have a better working solution - other data sources please feel free to post - but please explain it step by step.

Or use tick data and use CSV2FXT to create all timeframes hst files. I've never tried it with M1 data, it might work.

I will give it a try.... but this are huge files to download. Anyway what is your opinion on the best source data.

Best ? best in what way ?

sorry for not being too clear:

e.g.
no huge time gaps, no huge wrong spikes
available time span
to some extend also how easily they can be downloaded/prepared for MT4

Ah, I see, I'm not really qualified to answer your question. I have downloaded Dukascopy tick data and Pepperstone data, I found some problems with the Pepperstone data where it had duplicated some records, I fed this back but I don't know if they ever fixed it. It was easy to fix as they presented their data in 1 month chunks. I wrote some batch files to concatenate the chunks into one file and then ran it through CSV2FXT. The Dukascopy data didn't present any issues, and from what I have read it is pretty good regarding gaps . . . I found this posted elsewhere:

Thanks.

I'm giving this two a try - don't want to go through a lof of pain as I do not really bother about tick data (at least not at present M5-1 is good enough for me)

 

1. The Tickstory is a free all-in-one tool that not only downloads the Dukascopy tick data but also processes it into an FXT. If you select this route, you no longer need to go through the CSV2FXT conversion step (you can still export a CSV and do it, though) ....
2. The StrategyQuant Tick Data Downloader is an application that automates tick data downloading and CSV creation. It’s free and very easy to use.

will report back how it goes

Tickstory

- does not run under my linux (wine) but in a Virtual Windows XP

- seems to be quite easy to use - some extra features like





- weekend filter, time adjustment ect.. if it works might be quite handy

did not check out all of them - only downloaded AUDJPY for a test

StrategyQuant Tick Data Downloader

- runs under my linux (wine)

- seems to show the date of the earliest data available for download

- generates a .CSV file like this for AUDJPY

2007.03.30 16:01:15.663,95.24,95.27,4.00,19.50
2007.03.30 16:01:16.014,95.24,95.26,4.00,19.10



FAZIT: both do the job of easily downloading the files:
but I prefer for my purpose the Tickstory (I need only the final M1-H1 hst files) maybe with weekend filter and timeoffset


compared to the M1 solution of FXDD-HistoricalData or FOREXITE(forextester)-HistoricalData this is going very slow very slow and a lot to download:

FXDD most M1 files are about 25MB for 5 years - I finished not even 1 year with Tickstory and have more than 100 MB but on the plus side they have quite a number of other Symbols too.

Anyway: I will do FXDD, FOREXITE(forextester) and at least some from dukascopy

convert each to MT4 M1 HST files and compress them: using lzma compression.

Maybe it will fit on one or two DVD: than I can use something like: sPeriodgen to produce the final hst files as needed.

After all the Tick solution is not worthwhile for my needs

nearly 800MB per Symbol to download.

So I will invest a bit of time and code an all in one solution for my needs -- will update when done.

FINISHED my solution for: Historical rates Period Generation from M1 CandleStick data in HST or CSV format

FXDD-HistoricalData
FOREXTESTER(forexite)-HistoricalData
DUKASCOPY-HistoricalData

Will first use it myself and than updated the first post. see: sCSV_HST_ToMT4PeriodsHST_ATI

Mistake: I actually restarted quite from scratch: it was ok but because by now I have already invested much time: I decided to code a bit of a better one:

in case anyone is interested I attached a Helpfile to the first post: along this lines I'm planning to do it: might still change here and there

See: first post: __HELP__sHistoryFileManipulation_ATI__README.txt

I did today already the new: broker Historical rates: Good History Rate Downloader EA

there are a couple of issues I saw: e.g.

In HistoryCenter hst files might have multiple bars with the same open time:




this do show up as 2 bars with the same opening time in the data window.

theoretically such should not be possible - how MT4's bar timeline is layed out.

One can also not enter such bars within the HistoryCenter.



so my assumption is that such hst files might have been written with an external program script (converted from any csv files for instance).

In the above case: the reason the two bars show up is: that the second one seems to be 5 seconds later: which one can not see in History center or the data window


In my opinion such should not happen and one more reason to write my own conversion tool.

uploaded also related: Good History Rate Downloader EA

UPDATE: 20130529
- first uploaded

what I do:

Original Broker data: download using my other download EA:
Good History Rate Downloader EA


1. download all historical data M1 from:
FXDD-HistoricalData
FOREXTESTER(forexite)-HistoricalData
DUKASCOPY-HistoricalData I signed up for a demo and downloaded it with JForex I downloaded M1 bid data

2. Convert/PreFilter downloaded M1 data ALL (.hst and CSV files) to CSV files: (only filter I use is: b.SkipBarNoPriceChange which is not 100% MT4 comform)#
- using the scrip from the first post.
- this will result in smaller files and also CSV files seem to compress better than MT4 hst files

3. compress each separately with lzma compression (inclusive the logfile)

4. store them on DVD


---- for actual backtesting I Convert the M1 CSV files to the MT4 hst files Periods M1-D1: apply any other filter like TimeShift ect..
- using the scrip from the first post.

Note: FOREXTESTER(forexite)-HistoricalData are CSV files but the filename ends in .txt this must be renamed to .csv read the HELP files for more info.

Note: DUKASCOPY-HistoricalData the CSV header structure depends on the particular files: e.g. JForex one can choose the datetime format - so do not follow blindly the template: read the HELP file

Also always skip through the specific log file (in the folder where the ResultFiles are) it shows sometimes if anything is unusual

In case someone else wonders: As of today: FXDD-HistoricalData USD / TRY is a complete mess even with negative values for close and volume in between.