How to build a successful EA from scratch

This thread is not for everyone. We all have different personalities at FF, and my philosophy will conflict with other philosophies out there. Just realize I am doing this in good faith, and will always try to present accurate and truthful information to help your ideas along.

Remember, 90% modeling quality in your backtests (in most cases) or all bets are off. Scalping EAs at 90% MQ are likely worthless. Please read the backtesting sticky here:

http://www.forexfactory.com/showthread.php?t=7637

To be successful, you will have to dedicate time to coding in MQL or some other trading language.

In the spirit of the James16 chart thread, I have decided to give back to the FF community. The purpose of this thread is to convey some lessons I have learned as I built my own EA.

Well the first thing to do is collect M1 data of the pair you are building for. Each currency has its own "personality," and what works for one pair may not work for another. I found free M1 data back to 2001 from Micheal Saganski, an euntreprenuer:

http://www.forexfactory.com/showthread.php?t=22963

The website has since been shut down. You can pay for backdata, or you can Google for hours like I did until you find it. My database has 2 million minutes of EURUSD data in it. It is my testbed, and it is what I use MT4 to backtest with.

Bottom line:
YOU WANT AS MUCH DATA AS POSSIBLE TO BACKTEST ON. A year or two just isn't a large enough population to base your decisions on when your hard-earned money is at stake.

The worst thing you can do when building an EA is create one that will lie to you in the backtests. What do I mean? It is possible to code MQL4 to create entries that fall off the price line. In other words, these are fantasy fills that would never happen in reality. A beautiful equity curve can be created with ease.

See this thread:
http://www.forexfactory.com/showthread.php?t=23087

ALWAYS OPEN TRADES ON A CLOSED BAR! This means that if you are using RSI, your MQL code should actually shift backwards one bar (for example: iRSI(NULL,0,96,PRICE_CLOSE,1)) Note the 1 at the end.... that pushes the RSI value to the closed bar. If you do not do this, an open candle that meets your TA criteria will open a trade. Sometimes this works for you, and sometimes against (if the candle reverses before close.)

To parlay this point further, your backtester CAN NOT give accurate results if you use the HIGH or the LOW of the candle to open trades with, because it is impossible to know the top or bottom until the candle closes.

I am not a scalper. Scalping EA backtests often do not compensate for rollover premiums or spreads! Those smooth equity curves would be eaten alive in a real environment. Be skeptical of them when you see them.... and those that promote them.

My trades are always for 100+ pips each, with tight stops. I only trade the EURUSD, because it is the most liquid (fastest fills,) tightest spreads, and least whipsaws with nice trends.

Okay, here is the meat and potatoes of building an EA from scratch. This is all my own philosophy, and it may not be the best way to do it. But it is all based on science and reason.

In hindsight, every chart is a pattern. You can observe trends and sideways markets in hindsight. With tons of backdata, you can see the "shapes" the price formed on the charts over time. You can guarantee yourself that 5 years from now, you'll look at 2007 as nothing more than the shape on a chart.

Forget news. Forget speculation. All you care about is shape and SOME curvefitting. Note* Merlin has a thread warning about curvefitting... I disagree with Merlin here, as I believe enough data moves you beyond curve fitting. So your job is to find TA that works with those shapes. Now let me tell you some pitfalls to watch out for:

1) In 2002, there was a 3000 pip uptrend on the GBPUSD. If your EA is a MA trendfollower, you might have made a 3000 pips gain. A move like that is RARE! It is like betting that a hurricane will come to your town every year. You've got to watch out for anomolies like that.

2) Sometimes strategies will overlap the exact same trend. For example, maybe your EA enters on a Donchian channel and ALSO on a MA cross. So you could have 2 entries that capture 3000 pips in example 1. But what if the trend reversed there? You'd have 2 losers. Try to avoid the same types of entries... overlaps can kill your account.

In the words of Rob Booker, preservation of capital is important. Even me.... who tries to compound the living daylights out of my account (with a small initial risk) has to be concious of a complete wipeout during a losing streak. Fixed r is a nice approach... best I've found to date.

3) AVOID OVERLEVERAGING. People trading without stop losses often get rude awakenings. Search around on other threads. I can name a few people at FF that are "cruising for a bruising." They scalp for a few pips, and think they can do it infinitely... until the market turns against them JUST ONE TIME. Want more information about this? Just search for people that used the RobinHood EA.... worked well in the 2006 MQL Championships, then lost a few in recent times.

I find the best practice to have 2 EAs. One is for the collection of finalized strategies, the other is a "testing" EA. You try your ideas on the testing EA, and when it meets your criteria (high pips, low drawdown) you insert it into the "real" EA. The real EA is kind-of like a pizza.... you just add the toppings (good strategies) as you discover them.

Okay, again I present my own opinions here.... and there is no blueprint for the perfect system.

I realize that any system will fail from time to time.... but having multiple strategies gives me a chance to cover my drawdowns with other strategies that are doing well. If your EA has multiple strategies drawing down simultaneously, you are in a pickle.

But you knew that, right? You knew there were going to be both good and bad periods.... and you planned for it.

My weapon against losing periods is compounding. I have a finite amount I am willing to lose, but an infinite amount that I can gain. See this:

http://www.forexfactory.com/showthread.php?t=44083

So I personally try to compound the hell out of my account over six month periods.... with 10% of my bankroll...... and if the system gets crushed over the six months, I am down 10%. Then I restart with 10% of my remaining account..... 90%..... which is now 9% of the original.... a fixed r of sorts.

So if the EA has gone ape-sh*t over the last six months, I pull all gains off the table, and restart with 10% of the new balance.

As stated before, a chart is nothing more than a "shape" or a pattern. Your job is to create rules for CERTAIN patterns..... and know that when the pattern occurs, you will be rewarded with trending markets, or disciplined by sideways markets.

Let's look at a chart, shall we? This nice EURUSD trend occured earlier this year. We are seeing an H1 view..... and our job is to find a pattern in that trend to "drop our anchor." We want to catch part of that move.

Then we backtest our idea against YEARS of data, and see if it catches trends in other markets. We are betting that the personality (for lack of a better word) won't change, so that this "predictive" pattern will continue emerging in trends.

I can't tell you my patterns, but I can tell you that the James16 chart thread is a nice place to start.

Just about everything I talked about in the previous posts is applied in this thread:

http://www.forexfactory.com/showthread.php?t=22302

You will see:

1) A finalized graph of 7 strategies combined
2) Drawdowns of 2 strategies
3) The effect of cancelling 1 strategy's drawdown with profit from another

I don't post there anymore, but I think it is useful information.

PS) Don't cross Jacko on his own thread, or you'll have a bee hive of visitors dropping 1 star ratings on your own thread

Let's say you are building an EA with tight stops. You realize that every day you are long in the EURUSD (for example,) the swap is draining your account by a pip at 5PM (all you have to do is figure out your $/pip, and compare it to the swap.) You realize on Wednesday you are being charged a triple swap.

So, at 17:00 each day, your EA performs a modify to EVERY ORDER that is long to move the SL up a pip, except Wednesdays where you move it 3 pips up. You see, you are preventing yourself from stopping out for more of a loss than you originally planned.

Suppose you are prepared to lose 5% of your account with a 35 pip stopout. 11 days goes by, so you will likely move your stopout up around 20 by the end of the trade. Your take profit will not change but will suffer too, due to the negative swap. In reality, you are choking your trade in order to keep from losing more than originally planned.

The bright side (although not really that bright) is that every third day, you can move BOTH your TP and SL up a pip for your short positions, because of positive swap. So a short trade (where you bank your swap) will give you a larger stop and shorter TP as time goes on, though not equally as fast moving as the longs.

So you have been spending hours searching for a Holy Grail EA. Just remember, hours spent searching for other people's EAs are taking time away from what you can be doing with your own EA. No doubt, I visit plenty of threads to see what other systems people are putting together, because there might be a nugget I can apply in my own work.

Now I can tell you that your confidence will grow once you start backtesting successful strategies, as you will not have to worry about the "false positives" I spoke of in Post #2. I suppose there is also a sense of accomplishment too. Some people will not have time to build an EA from scratch, and will be forced to take backtests of other EAs in good faith. Just keep in mind about the rollover premiums and spread removals mentioned in Post #3.

Okay, so now you have found some patterns that appear to be profitable in the backtests. Can you think of a reason for that particular pattern? Is there something that could have happened in the real world that caused the banks to make the price act the way it did? I find that if I can rationalize my patterns, I can feel more confident in their predictability.

A perfect example of this is the Nonfarm Payroll that comes out every 1st Friday of the month. No doubt, traders are straddling this news with ECNs to get a free ride off of the move. That is just a reality that causes a predictable price movement.... a news release that causes the banks to react.

Even at 90% modeling quality, you can get false positives in your backtests without scalping. That's why it is important to verify the entries (arrows) on the chart after your backtests completes. Eyeball the chart, and make sure the entries are where you planned for them to go.

A specific example of a backtesting anomoly happened to me about 4 months ago. I was straddling the price after a particular pattern, and the backtester was randomly ignoring some entries for no apparent reason. My code was sound. Luckily, the latest release of MT4 has fixed this problem.

Excellent stuff you have here! Thanks for the "secret" strategy

Returning to post #6, I want to hit on this point a little harder, as it is the foundation of my trading.

I realize that all methods an EA trades should periodically fail miserably. Drawdown will happen to the best of us (unless you are doing some amazing trading.) The key is to make the most of your run-ups.

I risk 10% of my bankroll every 6 months. Whatever my balance is on January 1st and July 1st, I take 10% of that to trade with.

If my EA is losing consistently, I will lose no more than 10% of my account. I have a finite loss that I am willing to take.

But what if the EA is winning a lot? I have unlimited upside, because I am compounding my account with extreme risk.

Bottom line.... after an extremely profitable 6 month period, ALL PROFIT is removed, and a new 10% calculation is applied to the balance.

In the "Launching 1kt" thread (no longer in existance) philmcgrew made a very profound point.

Randomness of trades can fool you. The two graphs shown are the exact same trades, but in different order.

This assumes that your method will produce the same frequency of winners and losers (which is highly unlikely) but it is still a good point.

This topic seems to be pretty popular on this forum these days.

It really matters what type of trader you are. For example, Jacko is a trend trader that holds positions until this trailing stops stops him out.

I am not a long term trend trader. I have a predefined stop and target, and do not scale out of my trades. In fact, my stops will widen over time as the trade goes into profit.

Unconventional for sure, but it suites my style.

http://www.forexfactory.com/showthre...t=22454&page=3

this is also my approach. when implementing my strategy I found out that there were some contingencies where my EA alone could not handle (due to my very limited programming skill). so I developed a new EA with the same logic to work together with an indicator. the indicator generates signals and write them to a file, the EA reads the file and issue orders. so far it has been working like a charm, if only I could solve the profitability problem http://www.forexfactory.com/images/icons/icon12.gif

this is like the Montcarlo analysis I've done for my trades. the analysis re-arranges my trades randomly tens of thousands of times and produces a report on returns, drawdowns, etc. of the different combinations. I've found it particular useful to try out different money management methods. the results showed that fixed ratio is probably the most reliable MM method to use with any EA.

anyone have, or know where I can get a generic master ea.. I dont know if there is something out there like this. Example: a ea that turns on another (slave) ea. The master may turn on another or several ea's when the master's lines up. For example the slave may be ready to scalp on a 15 min. But the master's settings dont let it. For example the master uses macd on the day. A master ea with several oscillators and adx. Would give the slave better odds for winning trades. The master also should be able to set multiple time frames. Let's say the slave only shorts when the master 4h rsi is above 125, and the 5 minute master adx is trending strong. The slave is then allowed to trigger on the 15 min. The master should have the slave's magic number to allow trading.

A master like this should improve most ea's. If made, adjustable. Where we can add the stock indicators or some custom. Where we can have 3 or 4 time frames. It will have less trades. But more quality trades. Then we could trade more pairs. It may make some of the crappy ea's winners.

Thanks for listening.

The master can be a fully functional ea on its own. A "swiss army knife" ea. That can use the slave to filter each other. It would make any slave ea less demanding to build. The main blueprint is the masters criteria. The slave could be fine tuning. This will filter out a lot of trades. But the quality should be better. One could just increase/decrease the size of trades or not (make more $). One could fine tune the slave to more pairs. The master being a "swiss army knife" could on its own be tuned to trade. But if one has a good ea already. Why not filter some bad trades out. I think there is a big demand for a master. You don't have to back test it. Too many advantages to name. You may be able to limit all trades to the master, manuel and ea. The master can be optimized to the best trading times, spread, and numerous other criterias. If you want to buy after a certain news. Tell the master. The master will do it of the other criteria is met. If you want to buy a certain level, tell the master. Go to sleep, the master will do it. Everyone is a fan.