Metatrader 5's distributed optimization

I've been looking for some documentation on this, but I've unfortunately come dry.

If where any way we can somehow hack together a similar version of metatrader 5's distributed optimization for metatrader 4? I have some self optimizing EAs which optimize against in excess of 4 million permutation. Problem is using my post powerful computer, it will still take me 4 months to optimize against the previous month.

Ronald,

I have run 6 MT4 sessions in parallel on my core i7 (8 cpus om 4 cores with 6Gb RAM) successfully. I just copied the MT4 folder 5 times and started the terminal directly from each new folder, then ran the optimisation over different values.

Doing this I have found that if I write a lot to the journal or to a output file during the test it can become I/O bound at the disk. So try to turn any disk output off by commenting out Print's etc. Also if you write to file, set it up so that only happens at the end of the run. I have a routine that fires off from deinit that goes through every trade and writes a heap of information out based on the EA. I guess an SSD or 6 would help too :-)

That brings up another point. You might find it more useful to put in bigger steps, write out some analysis data (see above) and then load it into a data analysis tool that does pivot style analysis. OLAP type stuff. I would only suggest using Excel for this if you also have SQL Server (or other ODBC DB) to load the data into, or if you are using the new 64-bit version of Office (millions of rows in a spreadsheet !!!!), even then, store the data in a data base.

By using the cube style analysis you can then find areas of interest and optimise those separately in more detail.

Pardon my ignorance, but why not convert the code to MQL5, and then use the MT5 backtester?

Sounds easier to me than creating a distributed tester for MT4!

Jim

MT5 backtester still has a few flaws (0 spread, wrong executions, and unusual swaps).

I've written a self-optimizing EA. The problem is it takes me ~120 days to test each and every permutation. I'm working on a distribution method so I can effectively 'crowdsource' the optimization part across brokers and pairs. That's why I'm curious to the workings of the MT5 backtester.

I'm not entirely sure how they are handling spreads in the MT5 strategy tester now, but MetaQuotes do at least seem to have fixed the zero spread issue.

I finally got MT5 Remote Optimization working last night and this is by far one of the greatest improvements that MQ has done. I am able to run 10s of thousands of iterations of my EA Manage The Pip 5 times faster than without the distribution of the load. I have it running on 5 different laptops and a job that took 30 hours to run now only takes 6.

I am having some issues connecting to a PC that has windows 7, for some reason MT5 can't see the MetaTester program, I am guessing its a security or firewall issue.

Now for the issues with MT5 there are plenty, and it is not a simple conversion of a complicated EA from MT4 to MT5. I equate it to going from VB 6 to VB.Net. I like the way they are heading but it will take some time to convert.

I also do not like the fact there is no visual mode in the backtester in MT5, this is a severe oversite, which sounds like they might fix, but I think it should of been in the initial release. If anyone has questions on the remote agents let me know.

Thanks,
Brian

crowdsourcing would be a mess -- youd have to make sure everybody has the same data and gmt offset- It could be done, yes, but not easily.

you could look at a combination of batch files and amazon EC2 cloud instances -- if you look at the startup (command line) options for mt4 youll see that you can make a batch file that will start mt4, do an optimization run, save the report and exit.

make an applet that generates all the batch files, and then have the various machine instances request the batch files from a queue and execute them, saving the report to a shared location. combine the results and you're done.

you'd probably architect the whole thing so that every mt4 instance accesses the same data.

I am having one issue with my ManageThePip EA in the back tester I can connect to 6 different PC's one has 2 cores and it allows me to seperate connections and works great. However, I have a quad core PC that I want to use as well, giving me a total of 10 remote agents running the back testing (So cool). Now I can get one of the ports to work on this quad core PC but the other three will not connect for some reason. Any ideas?

I did have one issue with EA running on the remotes, in that I checked to make sure DLL's was enabled and if wasn't I would exit the EA, this would cause the remotes to not run properly even if you had the setting correct in MT5. So I un-commented the code and it works great now, besides DLL's won't work via the remotes anyway.

This EA has so many money management settings that I have never been able to have all them on at once and iterate through the different values, now I can.

Thanks,

4 months of optimisation time seems a little too much! There are only two ways to speed this up; increase the speed or reduce the number of calculations.

I have heard that the MT4 engine is very slow; I've read about FF members using database engines and doing the equivalent of 1 hour of MT4 optimisation in a few seconds in an offline application and database. Perhaps it's worth a thought?

Also, there may be some commands in your code that are very processor intensive. I haven't seen any info about what functions are most efficient/inefficient in MT4, but if you are able to find some of this information out you could really speed things up.

In terms of distributed processing, I once solved a complex physics problem using SQL server and a c++ command line client running on multiple computers in parallel. The SQL server contained the original data to process, which was loaded into each program at start-up to prevent too much traffic on the database. Each program simply queried the database to get a range of variables to test, and updated the database so that another program didn't try to do the same calculations (i.e. just store the last tested values, you don't need to store every value and have millions of rows!). Once the range of data had been processed, it would compare the optimum value calculated with one stored in the database; if it was more optimised than the one in the database, it would store it as the "one to beat" with all its parameters. This minimised network traffic and kept things very simple. I ran the command line program on three machines to triple the calculation speed (which still took hours). I also tried my best to optimise the code so that the same work could be done in less processor instruction cycles, which also provided reasonable speed gains.

One of the benefits of using a database for this sort of thing is because they are very robust and are used to being talked to remotely by multiple clients; hence a lot of the work is done for you. Also, if any of the computers/clients crash, simply restart them and they query the database and continue on from where they left off. This is important if you don't have the time to re-run an optimisation if things go wrong for one reason or another.

Out of interest, what type of trading system requires so much optimisation? Does this work well for you in forward-test mode? I've written a few of these types of EAs but did not have much success with them.

Cheers,

Michael

My theory--if it can be called one--states that the perfect trading system for a given time in a certain market is different from a perfect trading system for a different time in the same market. The changes are not necessarily significant, but at the same time enough to throw off any optimized system.

Thanks for the info. Good luck with the strategy and optimisation, and if you get a chance please let us all know how you choose to speed it up.


Cheers,

Michael

One other thought; processing using your computer's graphics card processor is becoming more popular. The processors are setup for handling large datasets and doing lots of floating point calculations simultaneously, exactly what is needed for running backtesting. You should see huge speeds when compared to a pc core. You can also gang up multiple cards relatively inexpensively to further speed up processing.

If you are interested, check out Nvidia's website. They have a developer's area with lots of information about how it all works, plus some examples. Apparently some banks and financial institutions are already using them. It does look quite involved to setup though.

Cheers,

Michael