Backtesting discussion | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Backtesting programs 20 replies

Best backtesting Period 3 replies

good backtesting software 5 replies

Custom backtesting code 9 replies

Backtesting vs. Thought Process 10 replies

Attachments: Backtesting discussion
Exit Attachments

Backtesting discussion

  • #1
  • First Post: Feb 25, 2016 1:32pm Feb 25, 2016 1:32pm
Hi everyone!

I wanted to start my first thread about the question that i'm facing right now, The question after how long do you think a ''system'' is worth calling a system.
How long backtesting do you consider necessarily before you decide if the system WORKS

To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for now.
Do you consider that the system is working or just lucky month???

Curious what the more experience traders have to say

Have a great day!
Frank

(Sorry for grammer/spell mistakes!)
  • #2
  • Feb 25, 2016 3:26pm Feb 25, 2016 3:26pm
Quoting Ffl
Hi everyone! I wanted to start my first thread about the question that i'm facing right now, The question after how long do you think a ''system'' is worth calling a system. How long backtesting do you consider necessarily before you decide if the system WORKS To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for...
1 week to 1 month is enough
practising coitus interruptus
  • #3
  • Feb 25, 2016 3:52pm Feb 25, 2016 3:52pm
It depends on market. For example, last year EURUSD was downtrending and all trend system preformed well. But when the trend ended, they all collapsed. So you'd better chose a period with a trend and without it and test your system on this period.
  • #4
  • Feb 25, 2016 6:12pm Feb 25, 2016 6:12pm
Quoting Ffl
How long backtesting do you consider necessarily before you decide if the system WORKS
Only forward results (LIVE account) worth to be analysed ... Backtest and Demo are useless for profitability checking as they are not subject to real market conditions.
MT4/MT5 EAs/Indicators/Alerts coder
  • #5
  • Edited 9:03pm Feb 25, 2016 6:48pm | Edited 9:03pm
Quoting Ffl
Hi everyone! I wanted to start my first thread about the question that i'm facing right now, The question after how long do you think a ''system'' is worth calling a system. How long backtesting do you consider necessarily before you decide if the system WORKS To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for...
If you are a systematic trader or use statistics as a basis to assess the robustness of your strategy, the time horizon for back-testing needs to be as long as possible. The greater the sample size and the greater variety of market conditions, the more likely the resulting performance metrics have valid statistical meaning. Furthermore, the statistics produced provide valuable information that can be used to determine, based on past historic conditions, how well your current system is performing, what to expect during particular market conditions and whether or not your system is 'broke' or just entering a typical drawdown phase.

As a general rule for those that have a statistical bent, 10-15 years of history on D1 with >=2000 trades is probably sufficient (but better if longer) to at least provide a greater degree of confidence in your strategy, otherwise it is likely that you are simply travelling blind and have no indication whether or not your strategy possesses a real edge or is simply a victim of curve fitting. An analogy is this. If you decided to pilot an aircraft prototype in uncharted locations, would you prefer to rely on a test of say 100 flights in that uncharted region over the last few months or one of say 2000 flights over say the last 10 years over that same region. How do you ensure you have tested the system during abnormal weather conditions or storm seasons. If you are risk averse, you are likely to opt for the latter. This also usually is directly proportional to the size of your trading account (or how much of your livelihood is invested in the venture).

PS What FerruFx says in the prior post is also very valid. To overcome this pitfall you can do the following things if you are a systematic trader:
1. Ensure a contingency for slippage and real spread is embedded in your backtest. You can adopt a fairly realistic slippage factor in your back-test by allocating a % of slippage to Average True range (ATR) which mimics reality in those times where ATR is extreme and slippage is large. To play conservatively, you can double your spread as well during the back-test phase to ensure you conservatively address this impact.
2. Ensure you preserve an out of sample component of your available data to test results arising from your back-tested system. For example, use the last 15 years for your back-test horizon and then use then next year as your out of sample test phase. If all goes well and the walk forward test meets or exceeds back-test expectations, then you potentially have a robust system on your hands a year from now. A rolling window of out of sample testing is the way to go as each month that progresses, you roll forward the out of sample testing phase to ensure it meets or exceeds back-test expectations on a rolling basis. This way you can progressively embed new market conditions that were not experienced historically in your testing.
3. A systematic trader who goes live with their system continues to run their back-tested systems in conjunction with their live trading to compare and contrast performance metrics.
  • #6
  • Feb 26, 2016 1:36am Feb 26, 2016 1:36am
Quoting Ffl
Hi everyone! I wanted to start my first thread about the question that i'm facing right now, The question after how long do you think a ''system'' is worth calling a system. How long backtesting do you consider necessarily before you decide if the system WORKS To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for...
Back testing will only give you an idea of how your system "Should" work.
Only live trading will give you the real answer to your question. You need to live trade at least 4 different pairs to give you a perspective of success under live conditions. IE: stress, How you handle DDs, your entry & exit quality etc.
If you are still profitable in 6 months, you are on your way to being a successful trader.... Beware of the . he lurketh at every turn.
Look Left Trade Right
  • #7
  • Feb 26, 2016 4:12am Feb 26, 2016 4:12am
Quoting Copernicus
{quote} If you are a systematic trader or use statistics as a basis to assess the robustness of your strategy, the time horizon for back-testing needs to be as long as possible. The greater the sample size and the greater variety of market conditions, the more likely the resulting performance metrics have valid statistical meaning. Furthermore, the statistics produced provide valuable information that can be used to determine, based on past historic conditions, how well your current system is performing, what to expect during particular market conditions...
Hi copernicus and Ferrufx, thanks for this big explanation!
To first start about the spreads, spreads are not a problem in this case, if i close a trade (with profit) i always write the total amount you made - the spread. In the first tests did i already remove the spread problem (if i'm right) because i didn't wanted to really to much on how big the spread on that moment was!

So after reading this i will certainly test longer!

To look at how the system handles on other market conditions (and pairs), did i for example already downloaded the offline chart with the rules i use, which did look good too for now so thats my first indication that it (maby ofcourse!!!) will work on other conditions than this month.

Do you think making an ea is a good idea to make the test even better?

Frank
  • #8
  • Feb 26, 2016 4:15am Feb 26, 2016 4:15am
Quoting FerruFx
{quote} Only forward results (LIVE account) worth to be analysed ... Backtest and Demo are useless for profitability checking as they are not subject to real market conditions.
Hi Ferrufx!

I was wondering what is the biggest diffrents except for the spreads between demo and real?

Frank
  • #9
  • Feb 26, 2016 4:18am Feb 26, 2016 4:18am
Quoting Basill
{quote} Back testing will only give you an idea of how your system "Should" work. Only live trading will give you the real answer to your question. You need to live trade at least 4 different pairs to give you a perspective of success under live conditions. IE: stress, How you handle DDs, your entry & exit quality etc. If you are still profitable in 6 months, you are on your way to being a successful trader.... Beware of the . he lurketh at every turn.
Hi basil! thanks for your reaction

Starting with the entry and exit, those rules are already written on paper because i readed everywere that you needed an entry and exit plan to ensure your (potential) profit or lose was safe!
I always trade more pairs to create more market opportunities because in this case it doesn't matter which pair you choose

The biggest conclusion i could make from everybodys reactions is that you have to trade longer

Frank
  • #10
  • Feb 26, 2016 5:28am Feb 26, 2016 5:28am
Quoting Ffl
{quote} Hi Ferrufx! I was wondering what is the biggest diffrents except for the spreads between demo and real? Frank
Latency is definitively a big concern ... with the spread.

While in backtest all your orders are executed at the exact price you defined ... in real live market conditions, your orders will be filled at the available price assuming there's liquidity. You have no guarantee where your orders will be filled.

Hope my english wording explain it fine ...
MT4/MT5 EAs/Indicators/Alerts coder
  • #11
  • Feb 26, 2016 5:54am Feb 26, 2016 5:54am
Quoting FerruFx
{quote} Only forward results (LIVE account) worth to be analysed ... Backtest and Demo are useless for profitability checking as they are not subject to real market conditions.
Good to know your outlook about this one ... I was thinking of these earlier as I wanted to help some other great works that might have high probability of winning percentage.

Instead of demo, live account on a .01 leverage and careful calculations of whatever strategy you wanted to play on.
Since Frank Sinatra sings in his own way, my chart sing... I did it, my way
  • #12
  • Edited 6:17am Feb 26, 2016 6:07am | Edited 6:17am
Quoting Ffl
{quote} Hi copernicus and Ferrufx, thanks for this big explanation! To first start about the spreads, spreads are not a problem in this case, if i close a trade (with profit) i always write the total amount you made - the spread. In the first tests did i already remove the spread problem (if i'm right) because i didn't wanted to really to much on how big the spread on that moment was! So after reading this i will certainly test longer! To look at how the system handles on other market conditions (and pairs), did i for example already downloaded...
Hi Frank

Spread (being the difference between the bid and ask on entry and exit) is a variable beast and often during times of high volatility (and also low volatility) and depending on your broker can considerably widen. For example, this phenomenon is often associated with news events or in those very low liquidity times (often when you are away from the screen). The reality is that when trading live, as FerruFX has pointed out, these spreads (which are a transactional cost) can considerably differ from the back-test environment. This can add up, over the long term to reflect a considerable difference against your backtest. The same can be said for SWAP (interest holding costs) particularly if your strategy holds positions for long periods of time. Also dependent on available liquidity you can suffer at times considerable slippage (being the price you want to execute at versus the actual price you get).

Now if you have a handle on these transactional costs, you can build sufficient contingencies in your back-test to deal with them. You might not believe it but approximately 1/3 of the gross profit I generate from trading a diversified portfolio is eaten by these frictional costs so you not only have to have a strategy that can deliver an edge in the long term, but that edge must be sufficient to overcome these frictional costs of trading. It is a tough game we play on a very uneven playing field.

While it cops a lot of flack, personally I can't do without extensive back-testing. It not only gives you confidence in your strategy but it allows you to play around with scenario's to look at worst case situations which inevitably you face when you go live. As you develop confidence in your testing and examine the results against your live trading efforts, you can start loading up your back-test with the realities of live trading. Once this is achieved, you then have an excellent road-map to use to compare and contrast against your live trading performance.

EA's are a great start towards systematic trading but you must be aware that unlike discretionary trading, an EA and it's rigid logic may perform well in a narrow set of market conditions and then you may find that when conditions change, the EA blows your trading account. This is a common feature of EA's that can be overcome by systematic trading but it takes a bit of effort to get there. It is rare that a systematic trader employs a single system. They typically have a range of systems that work in different market conditions but are sufficiently robust in isolation to not also blow your account if the market conditions are not great. Furthermore, systematic trading usually is applied across a diversified portfolio of instruments across asset classes and not simply restricted to forex which can be a very highly correlated market.

Anyway, here are a few excellent reads on back-testing if you are interested in delving into this in greater detail.
1. Systematic Trading by Robert Carver;
2. Building Winning Algorithmic Trading Systems by Kevin Davey;
3. Trading Systems by Urban Jaekle and Emilio Tomasini

Regards

C
  • #13
  • Feb 26, 2016 6:16am Feb 26, 2016 6:16am
Quoting Ffl
How long backtesting do you consider necessarily before you decide if the system WORKS To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for now. Do you consider that the system is working or just lucky month??? Curious what the more experience traders have to say
Building a trading "method" that one trusts is such a personal endeavour that it's quite difficult to answer your questions objectively. You've been working "for months" on your own way of trading. Is that two months, six months, eighteen months? I can tell you that it took me about 4 years solid work before I was able to begin translating what I was seeing on a price chart into a trading method that I began to trust. And that was before I even began to back-test it.

You've been back-testing for one month. Does this mean you've sat for one month testing offline chart data (if so, how many weeks/months/years of chart data have you tested?) or does it mean you've tested one month's worth of chart data?

You've commented that your results are in your opinion a "good result for now", and then post-fixed this with a question about whether or not traders think your system is working or it's just been a "lucky month?" It's another difficult question to address because traders don't know your expectations or "system", or in fact, the number of trades you've taken while testing.

I think the advice you've already received is good advice and I'd reiterate; back-testing is a superb and time-efficient way of learning how the market moves and also building a method to trade that suits you. But there are certain things that back-testing can't (and never will in my view) address - trader psychology for example. Only facing the live market with real money at stake can we hone this particular aspect of our craft. And preparation for that can take months, years even.
  • #14
  • Feb 26, 2016 7:07am Feb 26, 2016 7:07am
Quoting simond2002
{quote} Building a trading "method" that one trusts is such a personal endeavour that it's quite difficult to answer your questions objectively. You've been working "for months" on your own way of trading. Is that two months, six months, eighteen months? I can tell you that it took me about 4 years solid work before I was able to begin translating what I was seeing on a price chart into a trading method that I began to trust. And that was before I even began to back-test it. You've been back-testing for one month. Does this mean you've sat for one...
Spot on
  • #15
  • Feb 26, 2016 7:41am Feb 26, 2016 7:41am
Hello Ffl,

Regarding the process of backtesting, you can't go wrong with what Copernicus has said.
Quoting Ffl
How long backtesting do you consider necessarily before you decide if the system WORKS
All systems work all the time in producing an intended result.

Consider a basic trend following system; it works well in granting profitable market exposure during prolonged trends. It also works just as well in cutting losses during directionless and choppy periods. The former results in positive gains to equity while the latter often results in having to take losses in order to preserve capital. At all times, the system works as designed. Most people have difficulties in using such a system because natural byproducts of the system are the emotional experiences that accompany the swings in account equity.

In short, you will know that a system 'works' when you become perfectly financially and emotionally compatible with the results produced.
  • #16
  • Feb 26, 2016 7:50am Feb 26, 2016 7:50am
Quoting Atokys
All systems work all the time in producing an intended result. Consider a basic trend following system; it works well in granting profitable market exposure during prolonged trends. It also works just as well in cutting losses during directionless and choppy periods. The former results in positive gains to equity while the latter often results in having to take losses in order to preserve capital. At all times, the system works as designed. Most people have difficulties in using such a system because natural byproducts of the system are the emotional...
That goes to my pin-up board. Thanks A :-)
  • #17
  • Feb 26, 2016 8:58am Feb 26, 2016 8:58am
Hi everyone!

First of all thanks for so much reactions!, It really helps to get a better view on backtesting and how REAL trading is like
(sorry if it takes some time to understand from my side)

The 4 big problems i could get out of the reactions where
1. Latency
2. Spreads
3. Transaction cost (holding a trade and taxes?)
4. Psychology

Starting with
1 latency, Thats indeed a thing i have not faced yet and did not think about untill now, Learning to deal with it is (like a lot of you said) by just starting a SMALL live account and thats i think the best way to learn so thats gonne be noted

2. Spreads i like Lantency not a thing i had experienced yet, if i'm right just subtracting it from the profit does not count? Trading news will never be my plan because it seems to risky and i try to avoid those times so that one i will hopefully not face wide spreads. but again reading all your comments, starting small is the best way to learn handle it.

3. Transaction cost again a point i don't know if i will face them, my trades are normally not longer open than 1 hour or less so if i'm right holding cost (swap?) will not be added, The only good part for me in the Netherlands is that you don't have to pay taxes on your profits , and that you won't have to pay ANY taxes untill you poses atleast 25.000 euro (which will take a whille hahaha )

4. The last but most important in my opinion psychology, Because everywere i came did this topic soon of late start rolling, so i'm already looking on how does a human work, what are the normal things we do and what is the MOST important to keep an eye on, because every human reacts the same on profits and loses was it to me already clear that it will be really important to know when to STOP, so therefore the book trading in the zone which in my eyes is a good start in the world of trading psychology .

Copernicus thanks for the book links! will give it a look

And to react on Simond, Testing is in my eyes making trades on the real days, so all trades i have made this month are in movements that happend between 1 february 2016 and 26 february 2016 , sometimes i skip a day because i know that i will not work that day. The trades i take variate sometimes 2 sometimes 4 or 5 but no more than 2 trades on the same time to protect me from big risks. On the side note did i test on offline chart data with the same rules which are 6 monhts worth of data
I hope this makes it a bit clearder?

Hope i explained a bit better! and the biggest tip everyone gives is to start live with a really small account so that sounds like a really good plan

Frank
  • #18
  • Feb 26, 2016 9:21am Feb 26, 2016 9:21am
Cheers Frank.

All the best :-)

C
  • #19
  • Feb 26, 2016 9:36am Feb 26, 2016 9:36am
Quoting Ffl
Hi everyone! I wanted to start my first thread about the question that i'm facing right now, The question after how long do you think a ''system'' is worth calling a system. How long backtesting do you consider necessarily before you decide if the system WORKS To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for...
In my years of experience I believe for a correct evaluation of a system in back testing, one must follow these rules:

1. Always use 3 different years for back testing. 1 year of downtrend, 1 year of uptrend, 1 year of sideways trend.
2. Check how the system copes with all these conditions.
3. Never use a trailing stop for back testing. MT4 does not process real ticks, so the trailing stops are always simulated, most of the time giving artificially positive results. The only trailing stop you can use would be large trailing stop, like 50 to 100 pips.
4. Only use hard stops or structural stops (Fractals, swing high, swing low etc..)
5. Never make 2 entry trades on the same bar.
6. never start with money management initially. If you have positive return on flat lot back testing, than you should start thinking about money management.
7. After the back tests are done and the parameters are set, always use live forward test for a period of time (at least 2 weeks), than go back and back test the same period and compare the results. If the results are identical or very close than that means you have a robust EA. If the results are totally different than check what logic in the EA could be causing this difference, and change that. Still no good? Throw that EA to the garbage....

I hope this helps a bit...
Good luck..
  • #20
  • Feb 26, 2016 10:24am Feb 26, 2016 10:24am
Quoting Ffl
4. The last but most important in my opinion psychology, Because everywere i came did this topic soon of late start rolling, so i'm already looking on how does a human work, what are the normal things we do and what is the MOST important to keep an eye on, because every human reacts the same on profits and loses was it to me already clear that it will be really important to know when to STOP, so therefore the book trading in the zone which in my eyes is a good start in the world of trading psychology . Copernicus thanks for the book links! will...
Frank,

In my trading place of work, I have a trading pin-board, which may even be similar to the one Copernicus has hanging on the wall of his trading place of work Pinned to my board is a quote from Mark Douglas that along my own journey has become what I consider to be one of the most, if not the most important trading quotes of all. In the very first pages of Trading In The Zone, Mark Douglas says,

"I assume that you already have your own system, your own edge."

Personally, my own debt of gratitude to Mark Douglas lies in that sentence because when I first read it I was in a place that "felt" like trading hell. But in reading those words I was able to concede that my own trading issue wasn't about psychology at all, it was simply down to the fact that I didn't have my own system, my own edge. In that moment it was as though the entire weight of the world lifted. I "binned" everything that I'd learned up until that point and went back to square one. My advice to anyone regarding trading psychology would be to not confuse the ramifications of some lacking in their trading method with those of a trading psychology matter. The two things can "feel" very much the same, but they are completely different beasts.

Your method is your method and I'm sure you know the answers on that front. As far as your stats go, I'm not a statistician but I think it's important to know the sample size that a statistician would consider large enough that it has statistical meaning.
0 traders viewing now