Expert Advisors optimization & backtesting

Fixed major bug in EA when using open price modeling. It turned out that real ticks deals didn't open on exact minutes/seconds, for example when used M5 in backtests/optimization then go on real ticks then deals opened for example, not at 9:45:00 but 9:48:34, etc. That gave huge differences.

Now it is fixed and the new EA version (with included additional strategies) attached to the post nr. 1

Now Darwinex and Admiral provide similar (but still quite different) results (Darwinex is ECN and use commissions, therefore different results). See Admiral Markets and Darwinex backtests with real ticks (but EA was optimized with open prices just on Admiral Markets data):



But as you can see similarity now is notable - balance ups and downs is correlating for Admiral vs Darwinex:

Darwinex EURUSD:


Admiral EURUSD:


And different story for SP500:

Darwinex SP500:


Admiral SP500:


As we can see SP500 is not correlating at all...

Here GBPUSD - quite corelated:

Darwinex GBPUSD:


Admiral GBPUSD:


Maybe if a symbol is not "popular" (not "Major"), then it would be hard to find similarities across different brokers?

Uploaded the new EA version - see the post-Nr 1 - with cheetah implementation.

There are sprinters and there are marathonists. Cheetah is a sprinter. It hunts very fast with maximum power but a short time.

The same here in strategy - increase trade volume on every successful deal and do in n times in a row. It's anti-martingale - classical martingale strategies increase the next deal volume on loss to cover previous negative deals. When you get a long series of negative deals you quite quickly accumulate a disaster and stop out.

With cheetah strategy, we accumulate success.

Here are cheetah parameters in EA for optimization:



When optimizing we are looking for at least 500 deals in history. Because we need to be sure that cheetah will survive and pattern exists. Usually, the best cheetah setups are with negative risk-reward ratio because there are more large series with positive deals. And the minimum account is 1000 USD or use a cent account.

See in this thread my comments about the cheetah approach and see my post with EA examples.

Have a good hunting season

Here one idea how we can proceed with collaboration:

1) You post here synthetic asset (consists of at least 10 various symbols/EAs setups/strategies using EA from post Nr.1) - will clarify conditions for expected synthetic asset later on - we can discuss here what should be the best parameters how to construct good/consistent synthetic asset
2) Open FXCopy ECN account in Roboforex (20% commission for you) - get EA free key, setup VPS and EAs
3) I will grant 300 USD as an investor to your account - but will trade it as "synthetic asset" - for example, wait for X% drawdown to enter (according to historical data for at least 10 years backtests you will provide)
4) Will publish my investor account here in this thread - so everybody could follow

So you will get me as 1st investor and you can use social networks to attract more investors. You could refer to this thread so investors will understand all this concept. So here will be win-win for everybody:
1) You - get fine backtested by yourself EA, get 1st investment
2) Me - as an investor, leverage backtesting time to find out various "synthetic assets" to invest in
3) Investors - could invest in synthetic assets - all will be listed here in this thread post-Nr.1
4) Broker - get profits from successful traders, attract more investors
5) ForexFactory - we will target traffic of users to this forum from our social networks

Everything here is transparent, all parties could verify everything. Not fast money - it's all longterm. But should be stable. For most investors 20-30% yearly profits are very good. Because you could use compounding and then earn even more.

So what you think about this idea? What should be considered/clarified? Other alternatives than Roboforex?

Is it bad if FF will decide this as commercial? Can we continue to freely run this thread as commercial? Think if we will publish here links to synthetic assets FXCopy then it will become a commercial thread... is this problem?

Here is Excel where play around with random sequences and you will see that the idea "after 1 loose profitable deal probability increases" is false and here is article regarding that.



Conclusion: it's not a good idea to implement in EA feature to raise after some lose series. In the longterm, that won't work out.

BUT... here another idea about "synthetic assets":

Look at all objects what you see with your eyes - at the table, at the chair, at your computer, at anything... Do you see? ALL IS RANDOM because molecules move chaotically.

AND... all physical objects still are stable!!! Why? Because of the law of compensation! How to use this in trading? You see all the charts seems to move chaotically and maybe it is so.

When you combine n EAs as a whole system - all that system is more stable than single EA taken. Every EA will act on a "randomly" object but combined in a package the order and stability will arrive.

Now let's look at attached Excel magic on random series... You see, if some process is stable with 60% wins and 40% looses in longterm. Then we can get profit when it fluctuates in interval from 61% - 62%. And we get looses when it fluctuates from 58% to 59%. Thats all fight we have there... using any strategy, any EA.. all the question is about those 1-2%... in longterm. There will be periods when EA will loose and periods when win. Those periods can be even more than year.

When you re-calculate that random Excel you will see that sometimes it is 58%, sometimes 61%, very rare 64% or 57%... It is not guaranteed that even in longterm it will be close to 60% wins vs 40% loses. Nothing is guaranteed. Complete "randomness"..

And here comes idea about moleculas... thats core of "synthetic asset" when we talk in a simplified manner.

Take look at real account results in my Trade Explorer used early version of EA "Candles trend" from this thread. You see there 13 EAs running. Each EA were backtested vs 10 years data. Was chosen 13 the best winners (setups). Yes there exist patterns - that's in our favor. And you see that all that system behave according to 10 years backtests (summed up all 13 EAs data):



You see some are winners some are loosers. All loosers did not reach the bottom yet. Drawdowns for each looser is only 50% of backtested results. Still I see that whole system is according to backtests. Lets see another 6 months at this system results. journaled with this post to comapre after 1 month how all goes on. This is my first "synthetic asset". Waiting for drawdown to enter in with another account.

Uploaded new EA version to post Nr. 1

Implemented protection from server power off - you can set how many pips to add for TP and for SL with parameters add_TP_pips and add_SL_pips.

So with tp_money and sl_money you set TP and SL (in money, not pips). EA will calculate total TP and SL and append additional pips for safety reasons. So if the server goes power off then all opened deals will be safely handled.

And one more parameter you can use to protect account - max_down_perc which is from 1 to 100. The default is 100 (no protection). It is percents of account money. For example, you can set 50. In case EA will lose 50% of the account then it will shut down.

Added one more strategy named DISTANCE. Very primitive strategy - when the price was going in one direction X pips starting from given time safe_h_from (1-23) till end time safe_h_to (1-23) then:
A) Continue the same path and open deal in the same direction
B) Open deal in the opposite direction

Backtest with parameter is_reverse_logic to see which approach performs better for each symbol.

See the attached example with USDCAD.

Additional protection on VPS power off - set up autologin and autostart all MTs instances on admin login. So when server will restart - MTs will auto-reopen and EAs will continue running.

Was not easy so write here at least for myself to not to forget how to do that:

1) Download and run AutoLogon - otherwise, MTs will restart in the background and no visible UI

2) Make a .bat file - enter all MTs paths like in this example (with notepad) and save with .bat file extension

start "" "C:\Program Files\Admiral Markets MT5\terminal64.exe"
start "" "C:\Program Files\Admiral Markets MT5 2\terminal64.exe"
start "" "C:\Program Files\Darwinex MetaTrader 52\terminal64.exe"
start "" "C:\Program Files\Darwinex MetaTrader 5 Darwin\terminal64.exe"
start "" "C:\Program Files\Darwinex MetaTrader 6\terminal64.exe"
start "" "C:\Program Files (x86)\RoboForex - MetaTrader 4\terminal.exe"
start "" "C:\Program Files (x86)\RoboForex - MetaTrader 42\terminal.exe"
exit

3) Setup the Scheduled Task as described here

Now when VPS will be restarted, the Administrator user will automatically be logged in and all MTs instances automatically open with visual interfaces.

Profit 959 USD with max drawdown 67 USD in 20 years and 3600 deals - perfect USDJPY EA setup with strategy "Bollinger spike". Find 10 such setups, and will have an excellent synthetic asset. Which will perform by itself about 50% yearly (add compounding and get even more) and will be super stable (because of so many deals - that's confirming strong statistical pattern in longterm).



Noticed that for backtests published here we need to add parameter setups too because from Excel file timeframes are coded as not-understandable values.



And here .set file, rename .txt to .set and load in EA to see how this setup performs with 20 years data on your broker account.



So we could make various synthetic assets packages for example: each EA setups with 200 deals in 20 years, with 500 deals, with 3000 deals etc.

Now Candles trend synthetic asset went out of drawdown:



And here profits/looses distribution by 13 EAs setups:



You see there are 1 massive winner with 19% which covers 2 most losers with -9% each. Then there are 2 strong winners with ~13% each. And one regular winner with 8%. This is clear idea of synthetic asset package - you never know which EA will profit and which lose in any time period. But all system of 13 EAs as a whole is compensating and with profit. Let's see after 1 year how it will perform.

so now according to backtests will be some more profits... and about after 3-4 months, we can expect here again some drawdown which I am waiting to jump in with my investor account B which is already setup and backed with money.

Meanwhile continue to build another package for synthetic assets. And yep - still waiting for collaboration my dear readers We need a lot of synthetic assets to diversify diversification and to apply to invest and to apply cheetah style trading.

If you are a coder and like your own coded EA then you can make your own and offer a package of your backtests as a synthetic asset. It is ok too! This thread and my TEs you can see in my profile (actually some of them are synthetic assets too but not so many EAs setups - from 3 to 5 on 1 account) are all about synthetic assets.

The new version of EA attached to the post Nr. 1 - implemented an additional strategy based on MCAD and implemented Neural Network-based "brains" (Artificial Intelligence).

How to use neural network "brains":

 

1. First optimize as usual - pick up a strategy and set is_neural_on = false

1. Then choose some profitable setup where deals more than 400 then clear all optimization checkboxes and check those which under section "Neural brains" - see an example:

Keep in mind that all w's must be optimized from -1 to 1 and step 0.1.

Neural brains are based on RSI indicator.

Parameter rsi_neural_bars means how many RSI values will be used to identify patterns - optimize it from 1 to 10.

Here 2 neural parameters are for BUY/SELL decision making. We have 3 possible decisions - BUY, SELL, DO NOTHING:

1. decide_low is from -1 to 0 step 0.1
2. decide_up is from 0 to 1 step 0.1

Parameter decision_behavior can be any number, I optimize from -100 to 100 step 1. What it is? Well... divine mystery... I could say that the "character" of neural "personality"... be brave or be conservative maybe... no scientific explanation in this case... just implemented by intuition in decision-making function as additional criteria.

You see neural networks is anyway black box for us humans. We never will understand patterns that AI is capable to identify and use. AI has its own "eyes" and see what we humans don't see. But if AI gives results why not use? See here my experimets with AI.

Here attached example to see the benefit of AI.

First optimized without AI - we get quite a bad results (recovery factor 4).

Then optimized with neural brains and... ola la - we see almost 3x better results because AI filtered out a lot of bad deals and now we have 2x smaller drawdown and even more profits (recovery factor 11):






What kind of assets we can build, for example:

1) EAs running on the night and no longer than X hours. So you wake up and "all is done". If you wish you can set up such package of EAs and that could be "no worries" asset working while you sleep

2) EAs with high win rate (risk/reward ratio negative) - if you prefer to get for example 80% profitable deals and feel good 80% time.

3) You can mix up everything - long term deals, short term, positive/negative risk/reward, with or without AI etc

Yep, all going according to backtests. At this moment the package of 13 EAs reached 33% profit (see details in my profile TE):



Here distribution by EAs:



GBPUSD reached maximum drawdown level according to 10 years backtests. Let's see how it will go further. We newer could guess which EA will be profitable. But if we make a package of n EAs then statistical fluctuations compensate.

Have made 2 EAs (signal sender and receiver) to copy deals from MT5 to MT4 to:
1) Make possible trading synthetic assets using primary and secondary accounts;
2) Port my MT5 trades to MT4 which can be published in Trade Explorer (only MT4 supported now);
3) Port MT5 trades to MT4 which can be published in FXCopy to attract investors (only MT4 supported now).

But results on RoboForex demo accounts were not good...



Even if everything is going on 1 computer - there are still some seconds delay and even if no delay then still differences.

Will run this on a demo account to see results in longterm.

For now, it seems FXCopy service is most appropriate for synthetic asset trading because deals are copied on 1 server instantly. But this is supported only for MT4

Attached MT5 to MT4 in TE and attached Candles Trend TE to this thread too. Both TEs result from this thread EA.

At this moment seems MT5 to MT4 copier is making differences because of delay and/or different brokers (Admiral on MT5 and RoboForex on MT4). Believe there will be situations when copied trades will perform better and worse but in the longterm (1-3 months) will compensate and should be a similar result on MT4 as signal provider account MT5. Let's see - will inform stats after 1 month about this.

Now working on synthetic assets based on neural network optimization.

Here one example of how to find good setup "from zero to hero":

Here USDCAD before neural optimization - very bad:





And here after neural optimization:





And again - this is only 1 setup but we need a package of at least 10 setups to run on 1 account.

Noticed one issue with my 1st synthetic asset where I am now in "trade" (see post here when timestamped entry) - it is hard to catch the exact best drawdown point because there is always some trade already opened.

Therefore synthetic asset must be made in this way Nr1 to be "tradable":

1) All trades are closed before 23:00
2) No trade longer than 1 day - so if trade opened then must be closed on the same day

Or this way Nr2:

1) All trades are closed on Friday 23:00

So in the case of Nr1, it is possible to "trade" (turn on copy for account B) every weekday. But in the case of Nr2, it is possible to turn on copy for account B on weekends.

And here goes "tradable" synthetic asset build in this way:

1) Use only major pairs - USDCAD, EURUSD, GBPUSD, USDJPY, because if going for big money maybe it's better to trade the most liquid pairs

2) All trades opened on Monday and closed on the same day - no later than 23 (depending on backtests results)

3) Backtested 20 years (started from the year 2000)

4) Used one strategy "DISTANCE"

5) Used at least 2 TP/SL setups for each pair to diversify (positive Risk/Reward and negative)

6) First optimized without neural brains - then with neural

See attached backtests with all .set files in this ZIP:



But... it's not very good...

As you see it's not stable. Yes it is profitable in longterm but there is 1 year where almost no profit... and it's 2019...



And here are drawdowns stats - we enter "trade" (start copy from account A to B) when 10% drawdown:

2000-01-03 04:00 - 2000-01-17 18:00 down 14 from 100 was up 0
2000-02-14 18:00 - 2000-03-20 19:00 down 20 from 103 was up 17
2000-04-17 18:00 - 2000-07-03 20:00 down 11 from 116 was up 34
2000-11-13 18:00 - 2000-11-27 19:00 down 14 from 145 was up 39
2001-06-11 20:00 - 2001-08-13 18:00 down 21 from 189 was up 58
2002-04-01 19:00 - 2002-06-24 13:00 down 19 from 195 was up 28
2002-07-08 16:00 - 2002-07-29 12:00 down 21 from 197 was up 20
2002-10-28 20:00 - 2002-12-16 16:00 down 12 from 248 was up 72
2002-12-30 19:00 - 2003-02-03 18:00 down 19 from 254 was up 19
2003-05-05 18:00 - 2003-06-09 14:00 down 11 from 304 was up 68
2003-06-23 20:00 - 2003-09-29 13:00 down 30 from 312 was up 18
2003-10-13 16:00 - 2003-11-17 16:00 down 21 from 324 was up 42
2004-01-05 11:00 - 2004-01-26 12:00 down 18 from 326 was up 23
2004-07-26 13:00 - 2004-09-27 13:00 down 18 from 395 was up 87
2005-02-07 13:00 - 2005-02-07 23:00 down 10 from 411 was up 35
2005-02-14 22:00 - 2005-02-28 13:00 down 13 from 413 was up 12
2005-08-29 18:00 - 2005-09-26 15:00 down 21 from 453 was up 53
2006-07-10 13:00 - 2006-09-18 17:00 down 18 from 574 was up 142
2006-11-13 19:00 - 2007-03-12 20:00 down 28 from 575 was up 19
2007-07-09 22:00 - 2007-10-15 20:00 down 19 from 580 was up 32
2007-10-22 20:00 - 2007-11-05 21:00 down 12 from 612 was up 51
2008-02-18 19:00 - 2008-03-03 18:00 down 15 from 718 was up 118
2008-03-10 22:00 - 2008-03-24 17:00 down 22 from 724 was up 21
2008-04-21 19:00 - 2008-05-19 13:00 down 12 from 745 was up 43
2008-09-29 13:00 - 2008-09-29 20:00 down 12 from 882 was up 149
2008-10-20 17:00 - 2008-10-27 13:00 down 11 from 959 was up 90
2008-12-22 18:00 - 2008-12-29 12:00 down 16 from 1114 was up 165
2009-01-05 15:00 - 2009-01-12 17:00 down 12 from 1146 was up 48
2009-01-26 16:00 - 2009-02-02 13:00 down 13 from 1213 was up 80
2009-03-09 13:00 - 2009-03-16 10:00 down 14 from 1304 was up 104
2009-04-20 19:00 - 2009-06-01 19:00 down 28 from 1366 was up 75
2010-03-15 09:00 - 2010-04-12 18:00 down 16 from 1498 was up 161
2010-05-10 18:00 - 2010-05-24 19:00 down 15 from 1547 was up 65
2010-07-05 11:00 - 2010-07-26 18:00 down 14 from 1569 was up 37
2010-08-30 20:00 - 2010-09-13 19:00 down 15 from 1595 was up 39
2011-07-18 10:00 - 2011-07-25 14:00 down 13 from 1807 was up 228
2011-08-01 04:00 - 2011-08-01 16:00 down 10 from 1810 was up 16
2011-08-15 19:00 - 2011-09-05 17:00 down 18 from 1838 was up 38
2011-11-14 19:00 - 2011-12-12 19:00 down 21 from 1869 was up 49
2012-08-13 18:00 - 2012-09-17 21:00 down 11 from 1954 was up 106
2014-01-13 16:00 - 2014-06-02 18:00 down 13 from 1994 was up 50
2014-09-22 18:00 - 2014-11-10 18:00 down 14 from 2000 was up 19
2014-12-15 20:00 - 2015-01-12 23:00 down 17 from 2011 was up 25
2015-03-23 16:00 - 2015-04-13 17:00 down 11 from 2040 was up 45
2015-05-25 22:00 - 2015-06-01 18:00 down 22 from 2063 was up 34
2016-02-08 12:00 - 2016-02-08 19:00 down 10 from 2121 was up 80
2016-03-21 11:00 - 2016-04-11 16:00 down 15 from 2130 was up 19
2016-06-27 18:00 - 2016-08-01 18:00 down 26 from 2165 was up 50
2016-11-21 16:00 - 2016-11-28 13:00 down 10 from 2165 was up 26
2017-09-25 13:00 - 2017-12-04 13:00 down 13 from 2255 was up 100
2018-02-05 22:00 - 2018-03-12 18:00 down 15 from 2257 was up 15
2018-09-24 18:00 - 2018-11-19 19:00 down 23 from 2303 was up 60

As we can see we would have about 51 entries in 20 years. It is ~2-3 entries per year. But no entries for 2019... and only 1 for 2001, 2013 and 2017...

Well... as total drawdown is just -30 usd I will add to this asset more strategies still using major pairs only. As you can see... not so easy and a lot of analysis needs to be done to get a good "tradable" asset.

Exit investing with 18% profit in 2 weeks. Not bad at least for me - this is my 1st synthetic asset "trade" operating with 2 my accounts (A - signal account see attached to this thread as "Candles Trend 10 years backtest", and B - investor account, attached to this thread "Synthetic asset investor").



Seems MT5 to MT4 copier EAs working +/- ok. There were small differences but ok. Now I am in my 2nd synthetic asset "trade" this time not using RoboForex investing features but my trades copier/receiver EAs.

Have made another synthetic asset with 10 EAs setups using EA from this thread. Backtested 20 years. Running now on my MT5:





I am entering "trade" now because drawdown on signal account A is 28% now from peak:



This trade is quite risky... well this asset is not so good/stable as "Candles Trend"... I am ready to survive additional drawdown. My investing capital is 3x more than the signal account - I am ready to get 15% drawdown on investing account (according to 20 years that's maximum drawdown) and get profit 10%. So it should be ok after 1-2 months.

Don't try to do this at home by yourself

Have to clarify my mistake that it's not molecules but free electrons which moves chaotically but didn't entirely leave solid object:

https://en.wikipedia.org/wiki/Solid

Metallic solids are held together by a high density of shared, delocalized electrons, known as "metallic bonding". In a metal, atoms readily lose their outermost ("valence") electrons, forming positive ions. The free electrons are spread over the entire solid, which is held together firmly by electrostatic interactions between the ions and the electron cloud.[7] The large number of free electrons gives metals their high values of electrical and thermal conductivity. The free electrons also prevent transmission of visible light, making metals opaque, shiny and lustrous.

About "chaotically"... If we don't understand free electrons movements maybe it does not means there is no order? We just not see it, perhaps..? Because even mathematical function RANDOM actually is not "random"... it's pseudo "random".

Anyway, idea is that "random" objects still can form a stable system. The same should be with synthetic assets here in this thread. Will see after several years if this idea is working in the longterm.

As you can see in TE had made a mistake. I decided to turn on additional copy from another my EA which runs only with 1 instance (usdjpy) and one strategy (breakouts) on H4 timeframe... was backtested with 3 years data... source account C was in drawdown 20% (according to backtests it could be 60%)... but because of previous ups/downs stats decided to copy trades with my investing account B (in parallel working trades copy from A which is still in progress).



And that's ok... the decision was right... but here mistake I made because it is not possible to backtest RoboForex investor copy functionality... Problem was that B account has 3x more money than the C account. And turned on proportional copy and set the parameter to 3 (so my idea was that will have 3x more deal volume). But it turned out my deals went with x7 value because RoboForex "proportional" switch means something different... formula is somehow calculated not very clear to me at this moment... will investigate... but there are some "protection" built-in and some "proportional accounts" raising etc

So yes, my investor account went in big unexpected drawdown now 53%... If I would not make mistake drawdown would be about 23% what would be fine. No one to blame except me... education costs some money.

Let this post remind me of this lesson:

When noticing a mistake, ok accept 1 losing trade but not believe "it must go back because of already big drawdown" (my case)

I still have 70% of my initial balance for investor account - still some working capital available, painful lessons learned.

That C went in so huge drawdown 40%... it can go an additional 20% down according to 3 years backtests... So I reset copy strategy off from "proportional" to "standard"... so will continue to copy.

have the temptation to keep a copy with the bigger volume on... because "it must go up..." but nop. turning down to standard. accepting looses.

Hello, I have one question for you guys. So I had an EA made according to my strategy. I use 99% accurate tick data. But when using the backtester the optimization doesn't work, I can't even put in the parameters to test. Does anybody have any idea what might be the problem?
I wanna improve the results which are already pretty good. The graph is for the past year.

What platform using - MT4 or MT5?

If MT5...