Does backtesting work?

Hey guys,

I have this EA that made a killing when doing a backtest in MT4. It basically went from 1000€ to 90000€ in 3 months. The drawdown was only 16%. But when testing it live it didn't perform anywhere near that. What is the problem and what is the point of doing a back test when there is such a huge difference?

go forward .test in real time.and you will be rewarded .

Backtesting should be used only to confirm that the stretegy is trading as per the rules ... Never to check its profitability.

Back-testing has its purpose. There are pros and cons to it.

Know what you are back-testing and what you look for when you back-test a 'trading system'.
When I back-test a technical trading method, there two things which I look for:-
- the frequency of the trade setups occurring and the time period when those setups occurs and the market session.
- test my entry and exit strategy for each of the timeframes I would like to trade, H1, H4 and D1.

Back-testing also allows you to test your money management strategy.

It works but not all the time. Markets are dynamic and thus you cannot say 100% what worked yesterday will work tomorrow. However, overall backtesting can be crucial in coming up with a strategy.

mt4 does not have all the data for all the price movements inside a candle (tick data), so what it does is just predict what happened inside a candle which is ofcourse not accurate and your results are not to be relied upon.

Take time to read this if you want a real answer.

One of the hardest things that new traders have to grasp about the charts they are viewing is this: the only information each candle holds in its database is:

1. the candle Open price.
2. the candle High price.
3. the candle Low price.
4. the candle Close price.
5. the candle open time and date.

That's it folks. Now think of the things that are vital to a viewing the likely outcome of trading that is not held in each candle's database:

1. the spread. Think about this for a few seconds, folks. By default, CrapT4's Crapergy Tester uses the current spread. Ouch. If you are 'paper testing' by manually going back over the charts, you probably won't even bother to include any kind of spread calculation. Backtesting in crapergy tester ignores the spread at the time the trade would have been taken because it has no record of it. Consider this:
   1. if the trade was taken at the time a major news event was released and the spread jumped by a factor of 10, this will not show up in your backtested trade result.
   2. spreads widen at the weekend, sometimes by a factor of 20 or more. Imagine what that does to EA craptesting results.
   3. EA's craptesting their results use the current spread. This changes all the time even without what I previously described. A craptesting run at different times will give different results.

2. swap costs. I am not sure that Crapergy Tester factors this in at all when craptesting an EA. For sure, it would take a really dedicated paper trader to factor them in, even if he/she knows what they were. Just imagine the difference that this makes to the headline profit/loss.

Have a quick think about the implications of a candle only holding the OHLC in its database, on looking back to see what indicators are showing you. They are working with each candle's close price. Ok, so they might be working with their open price, or some combination of HL, but the big thing to remember is this; what an indicator shows you on a chart is a fixed point in time, which is the candle close time.

Take an indi that we are all familiar with - Stochastic. The damn thing might have zoomed up and down like an elevator on drugs during the course of a candle and played merry Hell with your trading. This will not show up on your chart because Stoch can only work the the candle close price. Imagine what this does to an EA that depends on Stoch to trigger trade opening and closing.

So, if you run your Crapergy Tester in visual every tick mode, where do all those ticks come from? Suppose you are craptesting an EA on the H1. You will see the "Using M1", "Using M5" etc messages. Crapergy Tester is taking the OHLC of each candle it has in its database and constructing a grid of ticks based on the direction these candles took. It is guesswork.

Fine/ish if you are testing a long term strategy that only makes trading decisions at the close of each candle. Bit of a sod if you are craptesting an M5 scalping system. Absolute sod if your long term trading includes a tick-by-tick trailing stop.

And then we get to the issue of missing candles. CrapT4 ships with incomplete candle OHLC data, let alone missing tick-by-tick data. Noobs do not tend to know this.

So guys, this is why back testing is informally banned at SHF and why trying to discuss it will get you into trouble an any of my threads, and most that I merely contribute to.

I use it to test that EA code is working. I would not even bother to have nightmares about using it to test an EA's potential profitability - that is a waste of time.

Always curious to read about someone who continues to use a trading platform that he thinks is "crap" when so many others are available. Very curious.

surfac - You can read my response to a similar question here :

http://www.forexfactory.com/showthre...99#post7974099

Remember, you are trying to simulate real trading. There are many factors that Strategy Tester in MT4 does not take into account. You might want to try a different testing mechanism so you can compare the results from your MT4 tests.

You can google Steve Hopwood (that someone) and read about him on his on fourum.

MT4http://cdncache-a.akamaihd.net/items...rrow-10x10.png itself is not a crap, backtesting is a crap.

As FeruFX said

Depends on the backtesting facility and how close you can set it up to mimic real trading condition, backtesting can show the strategy's profitability as well as its validity.

Hi, Surfac

I am impressed you have built an ea that works, that is well outside of my scope.

You have been given lots of good advice here. Just because the ea does not trade as well live, don't ditch it. The backtest should only be part of the process to create a robust live strategy.

The key now is to work out in what conditions it is not trading well and whether the win/loss ratio and return to risk is realistic. Most systems don't work in all market conditions. For example, it may work well in the euro session and not in the asian session.

If you ask for some help, I am sure someone will give you some help.

MT

Then why do you call it CrapT4?

Does backtesting work?


NO

I'm making my own robot ( profitable in real too ) and i can resolve you doubts.

Backtesting is useful for understand how you robot will run ( if you use visual backtest ). But you can't get profit numbers as real. And why many robots profitable in backtest fail in real is easy. Normally robots are made trying simplificate rules. That means you normally work with open/close candle values. And that is great until you test demo and you see this:

1) Inside candle movements affects much indicators used for filter and can give fake signal inside candle doing your robot close/open trades.
2) Even if you wait open/close moment, many times in that moment, indicators are in a edge limit and can show other state ( different to posterior ticks ), creating same fake signals like inside candle movements.
3) Even if 2 points i said you are not problem, you have others things can affect robot ( spread, etc ).
4) Even if all points are right, you have the big problem. Rules robot are very simple for control all possibilities.

Can you have profitable and consistent robot?. Yes, but you need have very good mix of filters ( indicators ) for do job fine. And that is the most complicate task. My experience say you need limit robot to entry in the best cases for him. And many times this means you will do little money because robot only have a few "best cases".

Bullshit!

Okay. Hope you will understand everything.

The material above (about why backtesting does not work) was copy/pasted from the Steve Hopwood's forum.

I PERSONALLY do not think that MT4 is a crap. MAY BE Steve Hopwood does, but I DO NOT.

Therefore, I tried to make it clear for surfac why Backtesting might not work - from the point of view of experienced trader (which is Steve) and I am not claiming anything what is written THERE.

Yes it does work

Nik13... thank you for the informative post(s).

Hope it helps, if so, then always welcome.

Back testing has it's place. Most developers will back test to ensure EA's are following the rules set out.

Back testing with the stock OHLC data should not be used to test profitability. I mean, you could do it but you may likely be misled.

A more accurate way to test a strategy's profitability using back testing is to use accurate tick data.

Fortunately, Dukascopy has a large archive of tick data for free. Their OHLC data is pretty rubbish though (sometimes off by a few hundred pips!).