Automation and backtesting

I'm trying to get my head around the idea of why backtesting (I assume regardless of the strategy as I tried 3 different strategies with the same outcome) does not work.

Here's the scenario:

1. Download historical data (regardless of the source) as OHLC and it must be EOD data
2. Write/program the code ( and no it is not MT4, it's Jave code.
3. Do the backtest

So far so good, based on the duration date 1 to date 2, I get a positive pips.


Here's the problem

Let's assume the period I am using is 15 years, I try to test random periods that is random start date and random end day within the 15 years (for example, period one is 200 days, period 3 1000 days, etc)

Out of 100 runs with random start date and random end date about 20 periods have negative outcome.

So 20% runs are negative, though the initial run (15 years are positive).

What is going wrong? Why a strategy would work on a period but when trying it on a random (sub) period it fails 20% of the random runs?

I'm no expert, but if you're running a 15 year test, that is inevitably going to include wins and losses... Have you calculated the win/loss percentage of trades over the entire period?

Even if the overall test is profitable, it will be made up of profitable and loss-making periods.

It depends on your strategy, but what I tend to do is group results by month, week and day so that I can see very roughly how many profitable days I could expect in a week, how many profitable weeks I could expect in a month and how many profitable months I could expect in a year. None of it is proof of anything going forward, but it gives me a rough idea of the system so that if it starts deviating enormously in live trading, I know to pause and reassess to see if anything has gone wrong.

So here's a random 100 runs (random start date and random end date from 15 years data) and based on 10000 account with 2% risk


[money 9907.01] [pips: 0.0054] [trades: 353] [max loss: 7] [win% 0.23]
[money 5277.57] [pips: 0.0035] [trades: 908] [max loss: 12] [win% 0.22]
[money 9810.1] [pips: 0.0295] [trades: 3625] [max loss: 26] [win% 0.26]
[money 13948.34] [pips: 0.0681] [trades: 1927] [max loss: 18] [win% 0.3]
[money 13106.92] [pips: 0.0187] [trades: 2617] [max loss: 22] [win% 0.26]
[money 7612.52] [pips: 0.1149] [trades: 3864] [max loss: 26] [win% 0.27]
[money 8988.92] [pips: -0.0102] [trades: 1872] [max loss: 18] [win% 0.27]
[money 8005.78] [pips: 0.0388] [trades: 1156] [max loss: 14] [win% 0.27]
[money 7921.84] [pips: 0.0629] [trades: 1405] [max loss: 15] [win% 0.29]
[money 4805.19] [pips: 0.0325] [trades: 3211] [max loss: 24] [win% 0.26]
[money 7389.6] [pips: -0.0061] [trades: 1725] [max loss: 17] [win% 0.22]
[money 4946.69] [pips: -0.032] [trades: 2643] [max loss: 22] [win% 0.24]
[money 8941.76] [pips: 0.0077] [trades: 2037] [max loss: 19] [win% 0.26]
[money 9857.13] [pips: 0.0304] [trades: 411] [max loss: 7] [win% 0.29]
[money 22397.12] [pips: 0.1395] [trades: 3366] [max loss: 24] [win% 0.27]
[money 7909.46] [pips: 0.0543] [trades: 3634] [max loss: 25] [win% 0.25]
[money 8812.79] [pips: 0.02] [trades: 2242] [max loss: 20] [win% 0.26]
[money 9630.73] [pips: 0.0504] [trades: 3670] [max loss: 26] [win% 0.25]
[money 8373.65] [pips: 0.0071] [trades: 195] [max loss: 5] [win% 0.21]
[money 7424.97] [pips: -0.0295] [trades: 243] [max loss: 6] [win% 0.19]
[money 5578.43] [pips: -0.0252] [trades: 2335] [max loss: 21] [win% 0.24]
[money 7130.87] [pips: -0.0207] [trades: 931] [max loss: 12] [win% 0.28]
[money 6976.04] [pips: 0.003] [trades: 717] [max loss: 11] [win% 0.3]
[money 5132.42] [pips: 0.0094] [trades: 3041] [max loss: 23] [win% 0.25]
[money 13441.06] [pips: -0.0151] [trades: 234] [max loss: 6] [win% 0.18]
[money 14409.29] [pips: 0.035] [trades: 1076] [max loss: 13] [win% 0.27]
[money 11452.8] [pips: 0.0279] [trades: 2970] [max loss: 23] [win% 0.26]
[money 8460.53] [pips: 0.0321] [trades: 920] [max loss: 12] [win% 0.32]
[money 16166.57] [pips: -0.0072] [trades: 722] [max loss: 11] [win% 0.22]
[money 11963.85] [pips: 0.1424] [trades: 4863] [max loss: 29] [win% 0.27]
[money 6650.21] [pips: 0.0306] [trades: 1868] [max loss: 18] [win% 0.25]
[money 7128.77] [pips: 0.0114] [trades: 1007] [max loss: 13] [win% 0.31]
[money 6088.53] [pips: 0.0251] [trades: 1270] [max loss: 14] [win% 0.25]
[money 4122.75] [pips: 0.0409] [trades: 2141] [max loss: 19] [win% 0.25]
[money 23286.06] [pips: 0.2193] [trades: 4037] [max loss: 27] [win% 0.28]
[money 11076.66] [pips: 0.0929] [trades: 4657] [max loss: 29] [win% 0.26]
[money 18746.5] [pips: 0.0629] [trades: 1468] [max loss: 16] [win% 0.29]
[money 10285.76] [pips: 0.0489] [trades: 2028] [max loss: 19] [win% 0.28]
[money 6643.44] [pips: -0.0107] [trades: 903] [max loss: 12] [win% 0.28]
[money 7965.12] [pips: 0.0398] [trades: 876] [max loss: 12] [win% 0.3]
[money 10178.61] [pips: -0.0075] [trades: 1371] [max loss: 16] [win% 0.22]
[money 6615.12] [pips: 0.0913] [trades: 3793] [max loss: 26] [win% 0.27]
[money 7499.12] [pips: 0.0212] [trades: 969] [max loss: 13] [win% 0.31]
[money 7155.41] [pips: -0.0115] [trades: 1831] [max loss: 18] [win% 0.22]
[money 9544.27] [pips: 0.0336] [trades: 604] [max loss: 10] [win% 0.32]
[money 5717.35] [pips: 0.0561] [trades: 3668] [max loss: 25] [win% 0.26]
[money 7241.96] [pips: -0.0069] [trades: 1892] [max loss: 18] [win% 0.23]
[money 22628.29] [pips: 0.0481] [trades: 2226] [max loss: 20] [win% 0.27]
[money 8191.5] [pips: 0.0046] [trades: 2368] [max loss: 20] [win% 0.24]
[money 11430.26] [pips: 0.03] [trades: 432] [max loss: 8] [win% 0.3]
[money 8853.27] [pips: -0.0084] [trades: 1707] [max loss: 17] [win% 0.27]
[money 7893.72] [pips: -0.014] [trades: 950] [max loss: 13] [win% 0.2]
[money 7633.12] [pips: 0.0001] [trades: 878] [max loss: 12] [win% 0.31]
[money 17102.01] [pips: 0.1078] [trades: 2548] [max loss: 21] [win% 0.28]
[money 21318.01] [pips: 0.1566] [trades: 2789] [max loss: 22] [win% 0.29]
[money 4122.48] [pips: -0.0022] [trades: 2696] [max loss: 22] [win% 0.24]
[money 11908.08] [pips: -0.034] [trades: 599] [max loss: 10] [win% 0.22]
[money 8400.79] [pips: 0.0038] [trades: 3091] [max loss: 24] [win% 0.25]
[money 11261.62] [pips: 0.0271] [trades: 418] [max loss: 8] [win% 0.3]
[money 7251.4] [pips: -0.0149] [trades: 1916] [max loss: 19] [win% 0.23]
[money 10370.43] [pips: 0.0066] [trades: 1668] [max loss: 17] [win% 0.24]
[money 8360.05] [pips: 0.0169] [trades: 1299] [max loss: 15] [win% 0.28]
[money 9444.79] [pips: 0.0475] [trades: 3702] [max loss: 26] [win% 0.25]
[money 15462.97] [pips: 0.1258] [trades: 2600] [max loss: 21] [win% 0.29]
[money 9419.2] [pips: -0.0007] [trades: 130] [max loss: 4] [win% 0.21]
[money 18708.2] [pips: 0.0816] [trades: 3260] [max loss: 24] [win% 0.27]
[money 11619.14] [pips: 0.1535] [trades: 4340] [max loss: 28] [win% 0.27]
[money 11163.57] [pips: 0.047] [trades: 138] [max loss: 4] [win% 0.39]
[money 8151.43] [pips: 0.0152] [trades: 3481] [max loss: 25] [win% 0.25]
[money 4490.24] [pips: 0.0349] [trades: 3951] [max loss: 27] [win% 0.26]
[money 11091.21] [pips: 0.063] [trades: 711] [max loss: 10] [win% 0.32]
[money 11518.53] [pips: 0.0331] [trades: 392] [max loss: 8] [win% 0.33]
[money 11271.57] [pips: 0.1492] [trades: 4315] [max loss: 28] [win% 0.27]
[money 22635.06] [pips: 0.1174] [trades: 1442] [max loss: 16] [win% 0.29]
[money 7746.22] [pips: 0.0454] [trades: 3795] [max loss: 26] [win% 0.25]
[money 8473.92] [pips: 0.0309] [trades: 2529] [max loss: 21] [win% 0.25]
[money 5020.8] [pips: -0.0367] [trades: 486] [max loss: 9] [win% 0.13]
[money 6377.39] [pips: -0.0067] [trades: 993] [max loss: 13] [win% 0.28]
[money 4728.98] [pips: -0.0586] [trades: 443] [max loss: 9] [win% 0.14]
[money 9138.94] [pips: 0.0737] [trades: 3966] [max loss: 27] [win% 0.26]
[money 9313.54] [pips: -0.0359] [trades: 1345] [max loss: 15] [win% 0.27]
[money 5714.01] [pips: 0.0206] [trades: 1362] [max loss: 15] [win% 0.25]
[money �] [pips: 0.0248] [trades: 2962] [max loss: 23] [win% 0.25]
[money 4642.84] [pips: -0.0471] [trades: 968] [max loss: 13] [win% 0.23]
[money 6565.66] [pips: -0.0112] [trades: 2075] [max loss: 19] [win% 0.23]
[money 21708.92] [pips: 0.1386] [trades: 1804] [max loss: 17] [win% 0.3]
[money 9331.8] [pips: 0.0687] [trades: 4460] [max loss: 28] [win% 0.26]
[money 10312.65] [pips: 0.1122] [trades: 4060] [max loss: 27] [win% 0.26]
[money 6428.83] [pips: -0.0057] [trades: 2187] [max loss: 20] [win% 0.25]
[money 48433.15] [pips: 0.209] [trades: 3185] [max loss: 23] [win% 0.3]
[money 6366.96] [pips: 0.0931] [trades: 3919] [max loss: 26] [win% 0.27]
[money 10146.94] [pips: 0.0118] [trades: 589] [max loss: 9] [win% 0.29]

What I am trying to get my head around, is why results varies when periods are sub part of the main period that is 15 years.

It's amazing the relation between money and pips, you can have negative pips and almost double the account or positive pips but end up with half account.

More interestingly,

Same pip count (or very close) for different two periods: one with positive money and one with negative money.

Well, based on my experience, testing is important but not HUGELY important. For me, having a "logical" system is/was most important closely followed by if you believe in your idea, just fund it with a few hundred dollars and then run it forward from there...

1. Do something that just "logically works": hopefully you have a system that uses "logic" (logical to you at least) and is not just a pure black box to you because I think that makes this all work:
   1. IE something like trend following. The more it intuitively makes sense, the more likely you are to follow it in good and bad times.
   2. IE start on a higher time frame to set the trend, then dial down the time to M1 or whatever and use another part to scale in on the trend direction.

   
   

2. Start small, use like 10:1 leverage.
   1. If you start to win, say making 1%, you get 10% wins each time because of the leverage

   
   

3. Start small (part two), start running your model with a few hundred $.
   1. If you are not willing to even at least try your model out with a hundred dollars or two, why are you even doing this?
   2. About the above: Getting actual execution of whatever you are doing is essential since real vs. demo do have differences, especially if you are working on quick/small timeframes. The experience alone with a few hundred $ seems to be worth a lot.

   
   

Basically wanted to mention this. My pet peeve is the people that just never fund and test and test and test and test... You could give up something or 5 dinners out ($20 a pop) or something and fund yourself with $100 or whatever... then as you gain confidence in your abilities, sacrifice and add more.

ALTHOUGH, I'm just getting more successful so take with a grain of salt: I have been studying since 2008, trading with real money since beginning of 2017 (breakeven) and demo trading since around 2014.

Thanks for the steps and input, yes I agree with trying the system with small account That is my next step.

I'm not trying to fit curve the system otherwise I wouldn't be doing 100 runs on random periods.

I'm just trying to see if I am missing something before forwarding the system.

Algo traders and backtesters know best but are all systems due to fail in the future ?

It's hard to get definite answer at this stage.

What I'm trying now is to play with the money management and see if I can get better results as results above are based on 2% risk.

Something is telling me that 2% rule are doomed to fail or it's not the optimal way at best

Okay 4 MM
1- MM only 2% (min run was 3698.65 max 37340.82)
2- MM 2% reduced when above 10000 (min 9461.97 max 10845.11)
3- MM 2% reduced when below 10000 (min 6623.54 max 16481.01)
4- MM 2% reduced when above and below 10000 (min 6868.08 max 16409.94)





Which one would you choose?

Backtesting is tough but gives you an idea of how the strategy will work in the real market. Things to consider slippage, swap rates, commission and high impact news events. Finding a system that works in history is great but running the system for x period in forward testing is really the only way to determine if the system will prevail.


Learning from history, you will but learning from history is all you will learn.

How do you define X for forward testing? Is it 20,100 or 1000?

Many would say 100 trades but where the 100 came from?

What if fails by X and succeeded by X-Y or X+Y?

Even more interestingly what if it succeeded by X and failed by X+Y?

I think your answering your own question with regards to backtesting. Is it 15 years vs 14 years and 9 months vs 1 year.

The market changes, and new things are presented daily, can your EA adapt. When will you feel comfortable with it, to trade live. This is the real question.

For my backtesting my dataset is 15 years.

The system generated roughly 3000 pips from 7000 pips range.

Once completed, I ran 100 random test (random start date and random end date from within the 15 years). Out of the 100 runs, 20 runs were negative (each run has its own positive and negative trades but total pips where negative).

Now if I decided to forward test the system on a live account, how long the forward testing should be for?

Hi thats the question [about forward testing and how long it should be] which most people do not have any valid or definate answer or will have a kind of answer such as 'it atleast requires 5 months or a year or 3 months' of forward testing in real market conditions.It means that for the majority of the people the time period or the numbers of trades will be relative.Because every one will be thinking differently according to their belief system and results they got after analysing the backtested strategy.

Now to answer your main question about

So 20% runs are negative, though the initial run (15 years are positive).

What is going wrong? Why a strategy would work on a period but when trying it on a random (sub) period it fails 20% of the random runs?

there could be two reasons
1] inaccurate price data [which i do not think it would be but just a possibility]
2] The strategy which you have developed might work very well in certain market conditions and would fail in other conditions because markets are ever changing dynamic structure and it might have changed and thats why the system didnt work on the random runs which you performed.

Back-testing is only for checking to see if the "edge" you are testing is viable. It should in no way make you believe it would work in a live market. Back-testing gives you the benefit of hindsight and also assumes you take every trade given by your "edge". In a real world environment that never happens.

As for forward testing, you should ALWAYS start in demo. Once you fully understand how your "edge" works and you are satisfied, move on to a small live account. Over time you will either become more confidant in your trading, in which case you can slowly add funds in a manner that does not put your overall financial health in jeopardy, i.e. only use money you can afford to lose, or you will lose confidence in which case you go back to the drawing board ans start again.

Good Luck

- Ric

Thanks for the input.

I guess it depends on the strategy some have few trades a day other have few trades a week.


For 15 years I had roughly 800 trades.

The other part of your reply. Yes that is exactly my point, backtesting will never have an answer if a strategy has positive expectancy or not.

More stats but these are based on the day of the week, the system would trade only Tue, Wed, Thu.

For 15 years: Tue,3469.16,Wed,2707.15,Thu,917.55

For 100 runs (random start date, random end date):

Here's how to do the 100 random runs:

Focusing on Tue only:

In 100 runs: 34 were minus but in the 48 runs (where total was below 10000) we get 15.

So 30% roughly.

What do we get if we run 1000 times?

380 out 1000 runs are negative so 40%

I am not sure why a system would work for a period from X to Y but when trying different periods either within X and Y or overlapping, it fails 30% percent of the time.


Any one has same conclusion?