Expert Advisors optimization & backtesting

In this thread, I will write about Expert Advisor based on various strategies and ideas of optimization.

Believe that investment capital should be divided into several parts for various strategies/expert advisors. In that way can be diversified risks.

Automating trading for me is one I can trust because I can't trust my emotions when trading manually.

You will get the MT5 Expert Advisor for free if you will help with backtesting.

The goal of this thread is making an ecosystem of "synthetic assets". In short - "synthetic asset" is a package of at least 10 EAs (various strategies and/or symbols) combination running on 1 account, backtested with at least 10 years data and having no more than 20% total drawdown and earning at least 80% (without compounding) every year within 10 years backtesting. So in real life (when everything is twice worse because that's life), it means no more than 40% total drawdown and at least 40% profit every year.

How to trade "synthetic assets": we use minimal capital operating with 0.01 volume (in case of currencies) for each deal just to have real data on account A which is set up as signal provider (using some services like Roboforex or Darwinex etc). We wait for a drawdown from 10% - 20% (according to backtests) - if so, we turn on investor account B which copies trades. Turn off B when A recover loses (+/- X% according to your risk appetite).

We need a lot of "synthetic assets" because normally there are only 3-5 entries per year for one "asset".

To get EA running on real account:

1. Post here a backtesting result (minimum 10 Excel files as the whole package of "asset") - using EA from this post (see at the end for download)
2. Backtesting period must be at least 10 years with open prices modeling (set parameter is_open_prices_only to true)
3. Recovery factor for every single Excel file must be greater than 5 (more than 10 is ideal)
4. In every single Excel file with backtesting must be at least 200 trades (more than 700 is ideal)
5. The total drawdown of all Excels combined no more than 20%
6. "Asset" must work with 300 USD capital on account (and have no more than 20% combined drawdown)
7. Each year must be with at least 80% profit (without compounding)

You could use EA of this thread on 1 your live account once you post 1 valid "synthetic asset". Then write to me private message providing your account number and will get free key to enter in EA parameter "free_key".

Keep in mind - no warranties. It's not my goal to earn money by selling EA. EA here is for free and all on your own risk.

The expert advisor published here can be executed on MT5. MT5 has much better backtesting & optimization possibilities compared with MT4.

The main benefit of MT5 optimization/backtesting with open prices modeling compared to MT4: much faster optimization performance compared to MT4, for example, it is possible to optimize more than 15 parameters all at once (not recommended for real ticks, use so much parameters optimization only with Open Prices modeling EAs)

Tips for optimization:

1) Optimize various days and hours for various symbols:

1.1) It is a good idea to optimize one symbol for 5 days individually each with separate EA setup because there exist patterns for weekdays. And what is working for Monday not always works fine for other days;

1.2) The same for hours optimization - there are different market conditions and different sessions (Tokyo, London, New York). And there are different conditions for working day start and end hours;

2) Optimize BUY and SELL - there might be different SL/TP ranges depending on the deal type. It can strongly be noticed for indexes like SP500, DJI30, NQ100 (longer TP ranges for BUY than for SELL)

3) Any EA I develop no matter what is an algorithm for entering in the deals I make opposite parameter switch is_reverse_logic - this should be optimized as well. For example, EA has logic that when the Stochastic oscillator value is less than 10 then open BUY (because oversold). Hint: optimize with SELL too by turning on a switch in EA input parameters. If you don't know the name of that switch parameter - ask me here.

10-20 years optimization might run several hours (depending on computer resources). When I'm very excited about an EA idea and wish to see results as fast as possible I use MQL5 Cloud Network service but it cost some money. Mostly I run my optimizations at night while sleeping. So in the morning, I have a surprise - some very good setup found.

All strategies in this thread consolidated in one EA - use only this EA, other EAs in this thread won't be supported and are outdated:



Edited on 8 November 2022:

More than 350 donwloads of EA.. 1 real contributor but realy great coder/person/spirit!!! We had good working sessions. We shared EAs source codes, tools, and ideas and did really good results. Thank you for your collaboration, buddy, you know who you are

For all the rest - this thread is obsolete. We managed to build good helper tools to analyze data. No more contribution is needed.

Profit 636 USD with max drawdown 79 USD in 3 years. That's only one setup with DJI30 index for Bollinger Bands strategy.

Strategy idea:

We use the Bollinger Bands indicator to detect a trend. Indicator setup: period 20, deviation 2.0.

We open BUY when:
1) previously closed candle[1] crosses upper Bollinger Band line - opened below and closed above
2) Bollinger Band is expanding - previous upper line value[2] is less than value[1]

We open SELL when:
1) previously closed candle[1] crosses lower Bollinger Band line - opened above and closed below
2) Bollinger Band is expanding - previous lower line value[2] is bigger than value[1]

You can use the attached Expert Advisor on MT5 to backtest DJI30 or find out with strategy tester optimization some good setups for other symbols.

In this DJI30 setup (see attachment), we entry only BUY deals.

Idea is to find out some 5-10 other setups and run them on the same account. Each setup earns about on average 2% monthly. Combined there can be more than 10% monthly.

Profit 1202 USD with max drawdown 125 USD in 3 years. That's setup with DJI30 index for the modified "1-2-3" strategy described here.

Sometimes I pick up a strategy, try to develop an Expert Advisor, and miss or adjust some original strategy idea and as a result, there is made a new strategy.

So, this time strategy idea:

We put BUYSTOP on candle[0] open price when:
1) previously closed candle[1] low is less than candle[2] low (lower low)
2) current candle[0] low didn't breakout previous low therefore candle[0] low is higher than candle[1] low
3) current candle[0] open price is higher than current candle[0] low
4) distance from current candle[0] open price and current Ask price is more than X pips

We put SELLSTOP on candle[0] open price when:
1) previously closed candle[1] high is higher than candle[2] high (higher high)
2) current candle[0] high didn't breakout previous high therefore candle[0] high is lower than candle[1] high
3) current candle[0] open price is lower than current candle[0] high
4) distance from current candle[0] open price and current Bid price is more than X pips

You can use the attached Expert Advisor on MT5 and DJI30 3 years optimization Excel and some good setups attached too.

You can try to optimize other symbols and post here good setups.

Profit 5880 USD with max drawdown 110 USD in 10 years. That's setup with 13 currencies pairs for the modified "Follow the candles" strategy described here.

As you can see in the original strategy thread I'm not welcomed with ideas of how to improve strategy. So posting modified strategy Expert Advisor here.

We open BUY on candle[0] opening in H1 chart when:
1) MN1 chart (monthly) current candle[0] is bullish (up)
2) W1 chart (weekly) current candle[0] is bullish (up)
3) D1 chart (daily) current candle[0] is bullish (up)
4) H1 chart (hourly) current candle[0] is bullish (up) or bearish (down) - depending on EA parameter is_retrace_candle value
5) and there is the right time - we do this at the very beginning of the specific hour (individually for each symbol according to best backtesting results)

We open SELL on candle[0] opening in H1 chart when:
1) MN1 chart (monthly) current candle[0] is bearish(down)
2) W1 chart (weekly) current candle[0] is bearish(down)
3) D1 chart (daily) current candle[0] is bearish(down)
4) H1 chart (hourly) current candle[0] is bullish (up) or bearish (down) - depending on EA parameter is_retrace_candle value
5) and there is the right time - we do this at the very beginning of the specific hour (individually for each symbol according to best backtesting results)

We close deals when one of these conditions is true:
1) time left according to deal_open_bars parameter
2) take profit or stop loss happen according to parameters close_on_profit and close_on_loose (parameters are money, not pips) - this will be checked when new candle starts.

About this parameter is_retrace_candle - if a candle opens it already have a difference in open and close values. Small but still difference. Let's treat this as "candle impulse".

You can backtest this EA on H4, H1, and smaller timeframes.

You can see there are raising parameters as well.

You can use the attached Expert Advisor on MT5 and some good setups attached too. Please tell if with your broker data those setups work as well?

You can try to optimize other symbols and post here good setups in case they are better than mine. I hope somebody could find out better-combained setups package to eliminate drawdown below 100 USD with a better recovery factor than mine 53.45. And hope you will find better setups for the 2019 year too.

Most important - this EA works with open prices! So you can backtest it very fast using modeling "Open prices only" - that's the huge advantage of this EA (no need for precise and slow every tick modeling).

Profit 11927 USD with max drawdown 350 USD in 10 years. That's setup for "Follow the candles" strategy with GBPUSD timeframe M15 optimized for each weekday.

Drawdowns:
2011-06-28 13:15 - 2011-07-11 16:15 down 304 from 3687 was up 2687
2013-05-21 17:00 - 2013-06-25 04:15 down 336 from 5925 was up 2542
2016-11-11 15:15 - 2017-01-11 12:15 down 318 from 10538 was up 4950
2018-09-06 09:15 - 2018-09-25 04:30 down 350 from 11705 was up 1485

Yep, for smaller timeframes there exist much more profitable setups than on H1. If you look at GBPUSD H1 setup then there recovery factor is 10.44 but when we split optimizations for each weekday and use M15 timeframe then combined setups recovery factor is 34.08. 3x profitable than H1!

Well.. there is a lot of work to be done for all the rest 12 symbols from H1 package. Will optimize each on M15 and all weekdays. In the end, we will have 13x5 = 65 instances of EAs working on 1 account.

See GBPUSD M15 setups attached and combined Excel with all setups results in one pivot table.

My fault - parameter is_retrace_candle actually is using the previous candle of the current chart. So you can decide to open a deal after retrace or by trend. For example, the month is bullish, the week is bullish, the day is bullish and the current chart H1 last closed candle is bearish (or bullish - see what performs better).

Now working on the new EA version and noticed this small bug in the description.

Profit 145 USD with max drawdown 29 USD in 3 years. That's setup for "Conservative Grid" strategy optimized and backtested with SP500 index.

Mostly grid systems are running with martingale and no stop loss. "Conservative" means we use Stop loss in this Expert Advisor and we do not increase volume when opening new levels.

You can use the attached Expert Advisor on MT5 for optimizations and backtests. You can try to optimize other symbols and post here good setups in case they are better than mine.

Hy... i think your work is fantastic but i see no many following....
One of the best EA that could be useful test with your method could be blessing one .... I don't know if is possibile to convert it in mt5 cause it's quite strucured but i think that the power of mt5 with the paower of blessing is a really good gun.
Thanks
Regards

Thank you! Added a poll to this thread so maybe we will clarify why nobody posts here any optimization/backtest results so far.

Let me know the strategy idea and will see If we can somehow collaborate.

Blessing is more then strategy is a full solid Ea.... really don't know if is possible or simply port it in mt5....
But i think is the best to backtest....
here the 3d in ff
https://www.forexfactory.com/showthread.php?t=792598
Thanks
Regards

Thank you. Reviewed mql4 source code - it has 2904 code lines... I have read the Blessings manual (https://www.jtatoday.com/blessing.html). Well, it is a grid system and it uses volume increasing as more levels are opened. Recently I tested EA on a demo account with 7000 USD and it went quite good (earned 85% in 2 months) and then it all was gone because of bad conditions.. and it used volume increasing similar as I read in Blessing manual (raised by 1.45) - here you can see how bad it was: https://www.fxblue.com/misc/EmbedExa...spx?id=2548224

But ok, need more time to clarify if for Blessing is possible to use Stop loss. Maybe it can be profitable with 5 levels or something like that and accept some losses. Yes, this can and should be optimized/backtested. It will require about 8-16 hours to port mql4 to mql5 on weekends. And so it could take about 4-5 days in total, so it's about 1 month (not all weekends I can dedicate to FX).

I saw in Blessings source code a lot of things which I am not familiar with and didn't take into account in my EAs.. porting Blessing will definitely give me new knowledge of mql possibilities. I see Blessing is based on serious work.

Thank you for the idea. I will let you know - but not soon.

Last modified version by Richard ( a great and kindly member) maybe it's more clean.... I think you could try to contact him... Just a thought....
Thanks for your effort.
Best regards

Profit 145 USD with max drawdown 19.22 USD in 3 years. That's setup for "Bollinger retrace" strategy optimized and backtested with EURUSD - broker Darwinex.

We use the Bollinger Bands indicator with period 20 and deviation 2.0.

Conditions for BUY:
1) Distance between Bid and Bollinger lower line is equal or greater than X pips which is the best EA parameter spike_pips value found in optimization/backtesting



Conditions for SELL:
1) Distance between Ask and Bollinger upper line is equal or greater than X pips which is the best EA parameter spike_pips value found in optimization/backtesting



Optimized on 2 brokers' data and found different results. See attached backtesting Excels and EA.

Think there could be risks because those "spikes" might be because of the news.. and in real word deals maybe won't open/close so smoothly like in backtests. Anyway, I will give a try with real money for this.

You can try to optimize this EA for other symbols/timeframes and post here good setups in case they are better than mine.

Good work Joyny .

Still not many traders are using MT5 compared to MT4.

Are you planning to provide MT4 versions of your work?
You know tick data is available for MT4 (using TickStory for example), and many people are using it.

Thanks for the hard work,
Khalil

Yes, MT5 is already 10 years in the market but still, MT4 is used a lot more. Maybe because of this. I have read in old forums that people were disappointed about MT5 early versions bugs and missing some features from MT4 but then I saw all bugs were solved and features added.. but you know not so easy to get back trust after n bugs, perhaps that's one reason too.

Yes, it is not possible to use MT5 in FF Trade Explorer, and not possible to copy trades in Roboforex etc but I found that Darwinex introduced that MT5 can be used to attract investors too. That's good news.

I already ported some of EAs to MT4 and running on my real money accounts (see my profile with Trade Explorers). Bus as I found Darwinex possibilities seems not interested anymore to port to MT4. It is always risky to backtest in MT5 and then port to MT4.. some mistakes can be made... better to run on MT5 exactly EA version which was backtested.

Tnx for pointing out. Yes real ticks seems is not the main reason to go for MT5. But for regular users, it is complicated to get reals ticks in MT4. In MT5 you can get real ticks starting from 2017 almost for any broker who supports MT5 by default.

Here are 2 main reasons why MT5 is much better for backtesting/optimizations than MT4:



With various timeframes, you have more options for optimizations. You can find good setups with H8 for example.

It is not possible to optimize Neural Networks with MT4. Too many optimization parameters. See here my experiments with Neural Networks.

abokwaik - thank you for comments. You are my inspiration. You have well-structured threads and a lot of EAs developed. What a great work you are doing!!

Here is an example of optimization of my new EA which will publish in case good results will found. Its adjusted version of "Boilinger Retrace" - implemented that timeframe is as parameter and deviation - so it will be possible to optimize the best timeframe too. And here you see how long it takes (100 hours... 4 cores, I'm using Contabo VPS):



So far the best result for GBPNZD is for H3 timeframe - no such timeframe in MT4...

Will publish this new EA version here soon.

Contabo VPS was forced to restart by Contabo stuff. And all my 6 instances of MT terminals with real money went down. And optimization running already 80 hours was gone too...

Checked my email and yes they have notified shortly before:

"With this e-mail, we would like to inform you that there is an immediate need for a short maintenance for your VPS S SSD. Our technicians will do everything in their power to finish all tasks as quickly as possible. Unfortunately, there was no way to announce this maintenance earlier, which is why we apologize for any inconvenience.

The maintenance will take place on Tuesday, 21st January at 09:00 UTC+1 in a time window of 60 minutes.

The offtime of your VPS is expected to last no longer than 15 to 30 minutes."

This is what must be considered when using EAs on VPS... Yes, I turned off Windows auto-updates and auto restarts.. but can't predict what VPS stuff are doing. Several EAs run when SL is controlled by EA... it means - all opened deals were on freeride...

Well, it means in EAs must be implemented some kind of protection. For example, if SL controlled by EA, then still put some wider SL just in case when opening deals. Then need to develop EA management... use 2 more independent VPS. On VPS Nr.3 will be stored PHP web application which will store VPS Nr. 1 EAs statuses, then VPS Nr 2. will have the same EAs but not doing anything - just asking from VPS Nr 3. PHP application if all ok with VPS Nr. 1. If VPS Nr. 1 down, then VPS Nr 2. takes over opened deals and continues operating. Each VPS controls if other VPS are up and running - in case of shut down sends email notification.. well a very complicated and developing-expensive environment must be set up.

Or need to change VPS provider... but such "emergencies" could happen with any VPS provider...

Seems figured out how to deal with this with 2 VPS and DropBox - so todo for my development:
1) Implement new parameter in EA "is_primary_EA"
2) On VPS Nr1 EA set parameter is_primary_EA = true, on VPS Nr2 EA set parameter is_primary_EA = false
3) Setup DropBox on VPS Nr1 and VPS Nr2 so that 1 common folder is shared via DropBox
4) Both VPS EAs write in a file latest time when EA was running
5) VPS Nr2 EA checks VPS Nr1 file and in case of logged time > X seconds then consider that VPS Nr1 is power off (send email notification) and overtake deals handling
6) VPS Nr1 EA checks VPS Nr2 file just in case and sends an email if time > X

Developing new EA which will consist of n strategies and switch to turn on/off open prices only mode. Because it would be hard to maintain/enhance n EAs versions (see previous posts about server power off solution). So decided to consolidate all logic and strategies in 1 EA and would be possible to maintain various security/protection enhancements. Regrouped EA parameters as well.

See/use what is at this moment already done - but it is not the latest version of that unified EA (at this moment still without power off solution and without neural network):



Yes and now available Boilinger Spike strategy with open prices. Quite good results on M5 for DJI30 and DAX - but yes buy and sell should be optimized separate for indexes to get best results:



See DAX - 1900 deals in 10 years. If combine such 5-10 various setups will see that total drawdown wont be more than 20% from the capital but total profit will be 5-10 times more.

See all sheets in file optimiz1.xls. See pivot report and then sheet "drawdown" - here is only DJI at this moment. Will add WTI, GOLD and GBPUSD, EURUSD, USDJPY. But you can build your own package of various combinations. And will be possible to add various strategies in 1 package.



What I am doing - I am building "synthetic assets" (will write more about that further). It is a combination of various strategies/symbols setups. Using open prices modeling for that - because then very fast optimizations for considerably long history at least 10 years (neural network only with open prices would be possible to optimize - too many parameters).

When you have "synthetic asset" then you can turn on all EAs on different account B when account A has total drawdown for example -50 USD. So you don't rely on single forex currency/index but instead, you build up your own ASSET. And you are the King - you define rules of your asset. And see from history that asset behaviour which is then predictable. That's all science of financial mathematics/statistics.

And yes... I see that on various brokers there are various results because of slightly different prices... So I cant use your backtests results and you cant use mine except if we stay on one broker (or at least one of 3: Roboforex, Admiral Markets, Darwinex). That's a problem for collaboration idea (you give backtests, and get EA for real account). don't know how to proceed with this? Any ideas?

See a lot of downloads but no feedback... strange. But yes optimization/backtesting is time-consuming. We all have day-time jobs... family responsibilities and time for analysis is limited.