DATFRA - Mt4 EA Builder & Walk Forward Analyser

ATTENTION: Please go to http://datfra.com/ for the version of DATFRA that is currently in development. The version here is not!




















Hello fellow traders

Today I can finally release my algorithmic trading framework. I have been working on this for many months, and now its time to share it.
Its an algotrading framework, which builds on top of Metatrader4 and MQL Expert Advisors, and it is meant to analyse, optimise and manage our EAs.

Features:

1. Parameterspace Analysis: An analysis method 10.000 times as powerful as Walk Forward Analysis - based on millions of 'backtests'
   1. Use it to find the very best parameters for every trading system
   2. Use it to find out how to optimise your system properly

   
   
   
   

2. Paper Trader: Playback the market data, as fast and as often as you want, and practice your manual trading on it! Within Metatrader4 and with full DATFRA Integration ( HowTo: http://darwins-fx-tools.com/manual.php?ppt )
3. Indicator -> EA Wizard: generate an Expert Advisor out of any Metatrader Indicator automatically. Use it to backtest indicators (even if you are a manual trader without coding skills) or to speed up EA development ( tutorial: http://www.forexfactory.com/showthread.php?t=494895 )
4. EA Builder: A simple to use EA builder allows you to let your computer search for new trading systems - without human interaction. Either using normal Backtests - or full Walk Forward Analysis!
5. Walk Forward Analyzer: Analyse your trading systems with Walk Forward Analysis - fast, effective, multithreaded.
6. Metatrader4 Integration: DATFRA works on top of Metatrader4, you do not have to change your preferred trading enviroment to use it!
7. MQL Library: Many MQL Functions and EA templates come with DATFRA, everything you need to immediately start EA programming!
8. System Management: The internal Database allows you to manage your EAs, parametersets and various kinds of analysis reports - neat and central
9. Data Manager: DATFRA can import history data from Metatrader4, Metatrader5 or CSV files.

Planned Features:

1. AI-Adivsor: Try hundreds of entry-filters and exit rules for any system (paper-traded or WFAnalysed or backtested) on the fly
2. Professional & Flexible Builder: This will allow you to use any MQL Function, any Metatrader indicator and any Expert Advisor for system building - full freedom for professional traders
3. Portfolio Creation: Analyse correlation of trading systems - and build the ideal portfolio
4. Live Optimisation: Let DATFRA re-optimise your EAs for the current market conditions - based on your findings during walk forward- or parameterspace analysis.
5. Monte Carlo Simulations: Determine the real statistical characteristics of a trade-distribution

This is an alpha version, so please be so kind and report all bugs to me via skype, pm or e-mail: http://darwins-fx-tools.com/?contact
As long as this is a tiny project, everybody is also invited to contact me for general help or questions about the framework.

The software is free - and I want to keep it that way. However, I am not just sharing this because I am such a nice person, but also because I want to make some valuable contacts and reputation through this project. So, if you like it, pay back through sharing this with as many traders as possible

As this is under heavy development, I will update this thread frequently, whenever I change something or add new features - so subscribe if you do not want to miss it!


Screenshots:

http://darwins-fx-tools.com/images/sc/addsys.png
http://darwins-fx-tools.com/images/sc/build_simple.png
http://darwins-fx-tools.com/images/sc/b3.png
http://darwins-fx-tools.com/images/sc/wfa_1.png
http://darwins-fx-tools.com/images/sc/overview.png



Download & Instructions:

Download the tool here: http://darwins-fx-tools.com

LISTEN! I profit from debugging this project as much as you profit from my help setting it all up and troubleshooting stuff. So, if you did read the manual, but still have ANY problem, please do both of us the favour and contact me! Either through skype or email.
I even provide free teamviewer assistance, as I profit from the help in debugging.


ATTENTION: There is a manual linked on my website, READ IT ALL! You will not even be able to install this software if you do not carefully follow the instructions (thats not a 'just click on Next' kind of installation routine!)
The manual is constructed out of slideshows with a bit of text, so it wont take long. But you MUST read every text below every image, there are sometimes very important instructions written there.


Here is a collection of the manual and all my articles about trading system analysis, thanks to x26s: http://darwins-fx-tools.com/dl/DATFR...the_manual.pdf


-Darwin

Looks like this is going to be a very important and useful software. Hope you keep this free and come up with a version to handle Metatrader 5 too.

Good Luck, Darwin!

P.S. I had tried to download it a few days ago, but the "agree" button on the disclaimer on your webpage was hidden behind the addon bar of my firefox (I figured it out today by accident). Today, for some reason, I was skimming through my collection of EAs and decided to try again. I zoomed out the page and got to see it. You might lose many testers/fans because of the jquery disclaimer window being misplaced. Check if its a real issue in different browsers.

Hey x26s, thanks for the nice words

Could you perhaps send me a screenshot of that website bug? Thanks!

-Darwin

Hi Darwin,
Check the attached files:
Visible part: That which is visible on firefox by default
Fullpage~: Full page captured by qsnap plugin. This is the entire page that is not normally displayed on the browser window.

Additionally, I am attaching a pdf file compiled using "this forexfactory page", "the articles from your site" and "the installation manual slides", with bookmarks! Delete it if you don't like. Else, it will serve as a easy reference for your alpha version.

By the way, the key expiration every month might be a deterrant to users unless you plan on updating at least once a month and keep it updated. If for some reason you stop updating the keys or the site, wouldn't that mean the end of this software on every computers that it is installed?
On the other hand, it could be a good idea to keep users forced to come back to your site so that they may not miss any changes.


Enjoy!
x26s

I just saw the link to "[Free & Open Source] Darwin's Walk Forward Analyzer v0.1" with 5 replies in the similar Threads widget on the left side. Maybe I could have added some of the pages from your research site to the pdf.
A bit too late now. Moreover, there are several threads on this tool on forexfactory.
Anyway, the material on your software looks very interesting!
Its getting late here. I will begin installing and testing your software in a couple of days.

Thank you very much for the pdf, I linked it in my post and I added it to the .rar file, now I finally have a docu to ship haha.

Yes, I will update the key each month, forcing the users to update and come back to my site, to read whats new
It was a massive amount of work to make this (DATFRA + website is 75.000 lines of code right now), so I guess it's ok to prevent that the tool just spreads uncontrolled.

-Darwin

!!NEW FEATURE!!

DATFRA can now generate an Expert Advisor out of any Metatrader Indicator automatically!
Use it to backtest indicators (even if you are a manual trader without coding skills) or to speed up EA development

TUTORIAL: http://www.forexfactory.com/showthread.php?t=494895


-Darwin

I am looking for some people that are willing to help me test a new feature, parameterspace analysis. With this, it is possible to analyse the general performance of an indicator or expert advisor, and to determine their best parameters!

If interested, please contact me through PM or Skype (Darwin-FX)

Thanks,
-Darwin

Hey there,Darwin. Thanks for this. It's people like you that make FF a good place to visit between trades.

DS

Holy crap Darwin, as a full time system developer / MAM manager, this is something I was always looking for with the ugly limitations MT4 brings. I am in for a heavy test-ride and will give all the feedback you are looking for Big thanks!

You are welcome
Yes, MT4 is really not taking its potential to the limits, but it does not matter, I will

-Darwin

Currently at the install, good documentation I have to say, although we clash with some of our perspectives (e.g. optimization on all data can´t be profitable while I found that this is the only way we have been constantly making money for the last 2 years), I am eager to see the optimization possibilities your software has. You have put some massive work into it as I can already tell, very amazing! One thing that is not clear to me yet: does your software overcome the input amount limitations that MT4 imposes? E.g. if you try to optimize a to wide parameter range it won´t start? Also, do you allow more than the 10496 passes for genetic optimization that MT4 imposes?

I have been playing with it and see it works a bit different than I expected, e.g. for optimization MT4 is still used and hence the limits of input variables and genetic passes do still apply. However, I am letting the EA builder run currently and that is a seriously powerful thing. In general what you have done there deserves BIG respect and that you are giving it away for free at the moment too - it must have been countless hours of work to create this and especially to use MT4 along with it. Amazing what I see there! Thanks a lot for making this!!

Hello geektrader, thank you very much for the feedback and nice words

When I say optimisation on all data can't be profitable, I was more referring to algorithmically built systems, as they have a hard time regarding overfitting and such problems (unsound systems that show nice backtests "by chance"). I should make that a bit more clear I guess, and not generalize.



What you are currently seeing is the Walk Forward Analyzer, and yes, it uses MT4 for optimisation, and therefore has the same limitations.
BUT the parameterspace analysis, which is the actual core of that framework, uses its own algorithms to do its job. Yes, it uses MT4 to simulate the trades, but then takes the raw trade outputs and everything else is then done internally, which means the limitations you mentioned do not apply.
I hope to release that part in a few weeks.



The builder you see right now is a very alpha thing, and its the simple version. That means, it only builds based on templates and is more usefull for people very new to trading. (also, it needs veeery much time to find really promising systems!)

The actual builder, on the other hand, is much more powerful, as it can take ANY mql function to generate its rules and ANY expert advisor to put them into.
That means, full freedom for an EA developer to let the builder test ideas for an in-development EA, which is the really helpful part.
As the current builder suffers from the same stuff i mentioned above: randomness in outcomes, unsound systems that are nice "by chance". Thats why it takes some time to find really good ones.

But with an expert managing and supervising the whole building process, in terms of inputs and checking the soundness of outcomes, this is a whole different game


-Darwin

Edit: You should however better use higher steps in optimisation, as parameterspaces grow exponentionally, no algorithm can optimise well on spaces that are too wide

Great work Darwin, that´s all I can say and I very much look forward to the new features, sounds VERY promising.

As for optimization steps, I keep them as high as possible but as low as needed. Though the step size was rather a problem for old genetic optimization algos. Nowadays there are very good ones where step-size doesn´t matter to much at all. You might want to look into "Covariance Matrix Adaptation Evolutionary Strategy".

For scientific background see:

https://www.lri.fr/~hansen/
and
http://en.wikipedia.org/wiki/CMA-ES

According to scientific benchmarks outperforms nine other, most popular evolutionary strategies (like PSO, Genetic and Differential evolution).

From AMI Broker documentation at https://www.amibroker.com/guide/h_optimization.html (and I can confirm that):

It should be noted, as it is the case with many continouos-space search algorithms, that decreasing "step" parameter in Optimize() function calls does not significantly affect optimization times. The only thing that matters is the problem "dimension", i.e. the number of different parameters (number of optimize function calls). The number of "steps" per parameter can be set without affecting the optimization time, so use the finest resolution you want. In theory the algorithm should be able to find solution in at most 900*(N+3)*(N+3) backtests where "N" is the dimension. In practice it converges a LOT faster. For example the solution in 3 (N=3) dimensional parameter space (say 100*100*100 = 1 million exhaustive steps) can be found in as few as 500-900 CMA-ES steps.

This is by far the best optimization algo I´ve ever used. Might be interesting for your program too?

Btw, just to point this out to the users here: you can´t run more than 32 MT4 instances SYSTEM-WIDE (a stupid new "protection" Metaquotes added to all builds > 600), so take care when building the cluster to not use to many instances as they otherwise won´t launch (except I´ve missed the fact that Darwin-FX uses a runtime patch with his loader to remove that limitation?

What I´d also really like to see is that the EA builder can avoid the usage of indicators for exit. My best systems are the ones with a ATR based trailing start trigger that, once reached, begins to trail from there. So would it be possible to just use ATR Trailing Stop (and ATR or fixed SL of course) for Exit instead of indicators?

This is backtracking search optimization algorithm (BSA), a new evolutionary algorithm better than CMA-ES http://www.pinarcivicioglu.com/ds.html

Sorry for the many posts, but I am getting heaps of errors in the log:

HTML_REPORT DOES NOT EXIST!

mt4_instance::backtest ERROR XYZ_DATFRA_4504777418606

could not execute mt4, err = 11; PARTLOG =

HTML_REPORT DOES NOT EXIST!

ERROR! ATTENTION! BACKTEST REPORT DOES NOT EXIST!

MT4 IDLES, DAMN! cpuusage = 4; mem = 210928; caches = 0. Its ok when this happens from time to time.

When it happens often, please start the MT4 from which you build the clusters, let it update, close it and rebuild the mt4-cluster!

Same thing when 'User Account Control' pops up sometimes.

MT4 ERROR NUM == 11

HTML_REPORT DOES NOT EXIST!

mt4_instance::backtest ERROR XYZ_DATFRA_4504662870626

could not execute mt4, err = 11; PARTLOG =

HTML_REPORT DOES NOT EXIST!

ERROR! ATTENTION! BACKTEST REPORT DOES NOT EXIST!

I am using 10 MT4 instances in the cluster on a 32 CPU core machine (non hyperthreaded) and 16GB RAM, so the machine is powerful enough. MT4 is updated to the latest build already and no UAC is running + everything has admin privileges (it´s run on Win 2k12 Server), but still it gives those "MT4 idles errors" which would be acceptable if there weren´t those "HTML_REPORT DOES NOT EXIST!" problems which obviously lead to that your program doesn´t receive the backtest results it wants from the related MT4 instance, right?

What´s wrong here? I use this machine for optimization for MT4 day in and day out without any problems.

Update: above problem with the missing HTML reports / MT4 "idling" does seem to only happen if use longer than 1080 days max for the optimization time-range (I did try from 365 to 4000 days instead of 1080 days max since I have 14 years of data and wanted to use more for the initial test). Anyhow, if setting to 1080 days max, the error happens a lot less and not at all sometimes.

Good find, so for what do you use this algo, is it your own financial software or how do you know it´s better than CMA-ES for the task of optimizing financial systems? I´ve seen CMA-ES in action in AMI-Trader, but I never heard BSA is used in any application like that. So how do you know it outperforms CMA-ES? According to the author of BSA, it´s success is similar to CMA-ES. Not all algos are equally good for the same task, for CMA-ES we at least do know it´s doing great for trading systems.