backtesting, The past... | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Backtesting programs 20 replies

Past news indicator for backtesting 4 replies

Best backtesting Period 3 replies

Custom backtesting code 9 replies

Backtesting vs. Thought Process 10 replies

backtesting, The past...

  • #1
  • First Post: Sep 25, 2006 7:59pm Sep 25, 2006 7:59pm
Ok, Im looking for some veteran backtesters and system developers here. I have backtested my system all the way back to January of 2004, over 2 1/2 years of data. Over this time, my system is outstanding. With some rescently acquired data, from January 2001, my system is quite different. I wouldn't say it's trash, but it sure doesn't perform the same (more drawdown).

My question is: to what extent do you rely on the past? Do you optimize a system to perform the best considering 2 years data, 5 years, 10 years? And by this, I dont mean optimize based on the past meaning curve fitting; I hope you know what I mean here. How far back is far enough?
  • #2
  • Sep 25, 2006 8:12pm Sep 25, 2006 8:12pm
Quoting twinchell
...How far back is far enough?
As far back as you can go.

...but in all seriousness, you are really just trying to learn where your system may falll over and have holes in it.. i.e. "When volatility month to month falls over an extended period my system would have to adapt" etc

That period you mention where there is a drawdown could happen again in the future. Maybe even the moment you start live trading it!
At least being aware that there was a period where the system had a hard time and it kept losses to a minimum is good to know.

I know for sure that if I had an EA that was unbeleivably stable and consistently profitable, I'd be searching high and low for data as far back as possible. Covering all kinds of market periods.

Bundy's status today: "Waiting..."
  • #3
  • Sep 26, 2006 6:01am Sep 26, 2006 6:01am
This is a very crucial topic, and it should be discussed at length, as evaluating the positive mathematical expectation of a system, IS what decides if it's worth pursuing or not.

It should be a statistically significant result.
I'm not a statistician (yet ) and this question should really be answered by someone with knowledge in probability theory/statistics, but from my own practical experience evaluating the net result of about 1000 buy signals gives a pretty probable long-time expectation (I actually like to go higher than that, about 10000 signals).
If you were to check minute data, and you would get on average one signal per day, that would mean a 1000 days minute data = 2.7 years = 1440000 minutes.

Then there are different conciderations:
1. Does it work on different pairs?
2a Does it work in different market conditions?
2b Does it work across different market conditions?
3. What is the volatility of the system?
4. What is the win/loss ratio?'
5. What is the frequency of the signals?

Ideally the answers should be:

1. Yes, it should work on atleast 3+ pairs that share similar characteristics (volume, spread etc - again practical experience, there ought to be some statistical basis for this).
2. Yes, it should work on all market conditions, or be able to avoid signaling in unfavorable market conditions (this needs not be a feature of the system itself, but can be determined by some other means).
3. This should be as low as possible, because this would make for easier money managment (i.e. you can "risk" more of your account) and consistant profits.
4. This should be as high as possible, but if it's low, the system can still be very profitable if the frequency is high.
5. If you autotrade, this should be as high as possible, while if you trade manually, this should also be managable (depending on your trading style).

There is quite a good article on points 3 and 4:
http://www.forexfactory.com/forexfor...n%2Floss+ratio

Thre also is a good article on automated system development, covering forward testing etc (can't find it now).
One thing I would like to add to that article is that you should test in different timeframes also, i.e. test random slices of the data of an arbitrary length. The less signals the systam gives, and the more volotile the system is, the more important this becomes. The system might work swimmingly well over the span of 10 years, but one year can show consistant losses. That is a significant consideration, money managment wise.
  • #4
  • Sep 26, 2006 9:16am Sep 26, 2006 9:16am
Thanks for the great info, keep it coming.
  • #5
  • Sep 27, 2006 5:44am Sep 27, 2006 5:44am
Hi,
Quoting twinchell
Ok, Im looking for some veteran backtesters and system developers here. I have backtested my system all the way back to January of 2004, over 2 1/2 years of data. Over this time, my system is outstanding. With some rescently acquired data, from January 2001, my system is quite different. I wouldn't say it's trash, but it sure doesn't perform the same (more drawdown).

My question is: to what extent do you rely on the past? Do you optimize a system to perform the best considering 2 years data, 5 years, 10 years? And by this, I dont mean optimize based on the past meaning curve fitting; I hope you know what I mean here. How far back is far enough?
I think it depends on what you are trying to measure.

If it performs differently, take a step back and see if there are any obvious reasons why it performs differently. Did interest rates change? Where there policy changes? Did the market change? Are there any identifiable changes which makes the system perform differently over different periods? Also general systems applied to all currencies perform differently to systems that work for specific currencies.

Also look at the number of trades. Most veteran backtesters take results with less than 100 trades with a pinch of salt and some only rely on results with 300+ trades.

Best regards
Alan
  • #6
  • Sep 27, 2006 8:22am Sep 27, 2006 8:22am
Well, going through 2004 I have many more than 300 trades. The main reason for going back even further was not to tweak my system of trading, but the money management behind that system. It has, however, come to my attention, thanks to a friend that knows my system, that the market has changed drastically within the last five years for the method that I trade, particually before 2004.

Thanks.

Quoting firehorse
Hi,
I think it depends on what you are trying to measure.

If it performs differently, take a step back and see if there are any obvious reasons why it performs differently. Did interest rates change? Where there policy changes? Did the market change? Are there any identifiable changes which makes the system perform differently over different periods? Also general systems applied to all currencies perform differently to systems that work for specific currencies.

Also look at the number of trades. Most veteran backtesters take results with less than 100 trades with a pinch of salt and some only rely on results with 300+ trades.

Best regards
Alan
  • #7
  • Oct 1, 2006 4:59am Oct 1, 2006 4:59am
My experience is that 4 years of historical data seems to be pretty representative of most mkt conditions. For my own systems I tend not to see any larger drawdowns going back any further than 4 years however like you I often see some of the largest drawdowns in the first couple of years of a 4 year dataset. As one of the other responders to this thread commented - I don't feel that this means that you need to discard your systems it just means that you are now aware of the kind of drawdowns you may experience. You should only discard your system(s) if the drawdowns you have seen in older data are beyond what you would consider acceptable. I can only comment on my own systems, however it is my experience that drawdowns tend to be normally distributed and that it can take about 4 years for all of the RH tail in the distribution to be witnessed. A good example of this is happening to me right now - a gbpchf model I'm trading within a broader portfolio had a 6% drawdown in feb 03 and has had only relatively small drawdowns up until now. The current drawdown started in July of this year and has now reached about 5%. Knowing that a 6% drawdown has been seen in the past at this stage I am not too concerned. Although ask me in a month's time !!!

One word of caution about data between Sep 11 and around Dec 01. The FX mkts behaved bizzarely after 9/11 and it took a while for them to get back to 'normal'.
  • #8
  • Oct 2, 2006 7:12am Oct 2, 2006 7:12am
Quoting firehorse
Also look at the number of trades. Most veteran backtesters take results with less than 100 trades with a pinch of salt and some only rely on results with 300+ trades.
Again - this is an interesting question - how many trades would you need in a timeseries as noisy as forex to be able to forecast future trades?
I'm talking from a strictly statistical/forecasting point of view here.
  • #9
  • Oct 2, 2006 4:59pm Oct 2, 2006 4:59pm
Hi Quathar,
Quoting quathar
Again - this is an interesting question - how many trades would you need in a timeseries as noisy as forex to be able to forecast future trades?
I'm talking from a strictly statistical/forecasting point of view here.
It is indeed an interesting question. I think that would depend on the consistency of your backtesting results. If the results were very consistent then you might be happy to trade the system with a lower number of backtested trades. If the results were inconsistent then you might want a higher number of backtested trades to determine if the system was reliable enough for your purposes. e.g. Sharpe ratio might be one way to go. Or as I've recently learnt, a Monte Carlo analysis might be another way to go.

Best regards
Alan
  • #10
  • Oct 3, 2006 6:39am Oct 3, 2006 6:39am
Quoting firehorse
Hi Quathar,
It is indeed an interesting question. I think that would depend on the consistency of your backtesting results. If the results were very consistent then you might be happy to trade the system with a lower number of backtested trades. If the results were inconsistent then you might want a higher number of backtested trades to determine if the system was reliable enough for your purposes. e.g. Sharpe ratio might be one way to go. Or as I've recently learnt, a Monte Carlo analysis might be another way to go.

Best regards
Alan
Hey Firehorse

Hmm...I don't really understand what you mean by consistency here. Could you clarify?
How can a result with a lower number of backtested trades be more consistant than a result with a higher number, and consistent with what?
Real/future results?
In that case, isn't consistancy a prerequisite?
The only way I see a system to be inconsistant is if you induced a programmatical error when you tried to backtest it.
  • #11
  • Oct 3, 2006 7:20am Oct 3, 2006 7:20am
More is not always better. Markets do change over time and if you fit your system to an excessive amount of data you simply "dumb" it down to fit all markets. You are concerned about now, not 2004. If you can generate 300+ trades I have to assume that it is an intraday system.

One rule of thumb is to test 30 trades per input that you intend to optimize. There is no perfect answer and this subject is a topic of great debate. In my opinion if you want your strategy to perform in today's market then it needs to perform well now, not in 2004. You watch it everyday and use money management. When it starts failing you re-optimize it and bring it back to life.

Your strategy is similar to a finely tuned race car. It is set up differently for every racing venue. You don't just find a mediocre setting and hope it lasts all season.
  • #12
  • Oct 3, 2006 4:23pm Oct 3, 2006 4:23pm
Following up on Philmcgrew's point - my experience is that parameter optimisation on the fly also has its issues. The primary one being knowing when to decide to re-optimise. The obvious thought is when my parameter settings appear to be going off. Firstly when do you know they are going off - is this just a temporary blip or is this something more permanent ? Or conversely do I re-optimise even when things are winning ? Either way your strategy for deciding when to re-optimise and what your criteria are for accepting a re-optimised parameter setting, need to be modelled and tested against a large historical dataset in order to know whether your re-optimising strategy is going to work in the long term. Of course by doing so you have now introduced another set of parameters in to your system that also need to be optimised ! If however you decide to leave the decision as to when to re-optimise to the human, you are introducing a non-deterministic process in to your system which is a) difficult to model and b) introduces a psychological and cognitive dimension to your system trading that I for one like to avoid. If you find that your re-optimising strategy works fine now but not back in 2004 you are back to square one and surely can't have any confidence that it will keep working when the mkt mode changes.

If you decide to re-optimise when you experience an x percent drawdown, then within your system there is a guaranteed drawdown but no guaranteed profit. The success of your system will then be determined by the average multiple of x profit you can book, before the x percent drawdown occurs. Now of course the quality of the underlying model will be different for all of us, but its been my experience that my particular parameters have not been stable enough in the short term to find a re-optimising strategy capable of beating a less optimal ship that is capable of sailing in all weathers. The additional hassle of re-optimsing rather than having a set and forget strategy is also something worth bearing in mind.

I am very grateful for this discussion, so pls accept my arguments above as nothing more than a stimulus for debate.
  • #13
  • Oct 3, 2006 4:58pm Oct 3, 2006 4:58pm
Quoting philmcgrew
More is not always better. Markets do change over time and if you fit your system to an excessive amount of data you simply "dumb" it down to fit all markets. You are concerned about now, not 2004. If you can generate 300+ trades I have to assume that it is an intraday system.

One rule of thumb is to test 30 trades per input that you intend to optimize. There is no perfect answer and this subject is a topic of great debate. In my opinion if you want your strategy to perform in today's market then it needs to perform well now, not in 2004. You watch it everyday and use money management. When it starts failing you re-optimize it and bring it back to life.

Your strategy is similar to a finely tuned race car. It is set up differently for every racing venue. You don't just find a mediocre setting and hope it lasts all season.
This might seem intuitively correct, but it flies in the face of all statistical/emprical testing. How would you know that your system is correct if it could as well be a random occurence that it "worked"? Thats what we are talking about here, separating actual patterns from random data. Too few trades, and you WILL NOT be able to tell the difference. Now the question is how many are needed, and how you would calculate it.

Furthermore, I would gladly take a "medioker" system that is statistically proven over an "optimized" system that is not. Who wouldn't?
When you tune a race car, you have a lot of premises that are all ready known, so it's not a good analogy...
  • #14
  • Oct 3, 2006 5:35pm Oct 3, 2006 5:35pm
Hi,
Quoting quathar
Hey Firehorse

Hmm...I don't really understand what you mean by consistency here. Could you clarify?
How can a result with a lower number of backtested trades be more consistant than a result with a higher number, and consistent with what?
Real/future results?
In that case, isn't consistancy a prerequisite?
The only way I see a system to be inconsistant is if you induced a programmatical error when you tried to backtest it.
I vaguely remember something about confidence levels from school maths and that was one of my ideas.

One consistency is between different parts of your data set. Does optimizing one half of your data provide good parameters for the second half?

Another consistency is if you change the parameters that the results change smoothly. If the results jumps about then I would call it inconsistent. That may or may not change with more trades in the backtest.

But as Phil has pointed out, by the time you have lot of trades in your backtest, the market may have changed thus invalidating your backtest.

Just my opinion.

Best regards
Alan
  • #15
  • Oct 3, 2006 6:52pm Oct 3, 2006 6:52pm
Quoting quathar
This might seem intuitively correct, but it flies in the face of all statistical/emprical testing. How would you know that your system is correct if it could as well be a random occurence that it "worked"? Thats what we are talking about here, separating actual patterns from random data. Too few trades, and you WILL NOT be able to tell the difference. Now the question is how many are needed, and how you would calculate it.

Furthermore, I would gladly take a "medioker" system that is statistically proven over an "optimized" system that is not. Who wouldn't?
When you tune a race car, you have a lot of premises that are all ready known, so it's not a good analogy...
You grossly oversimplify the subject of optimization. Entire books have been written on the subject of development and testing. Optimization is like fire. You treat it well and respect it, it will warm your house and cook your food. You misuse it and it will burn your house down. One way to avoid random data is to subject your trades to a Monte Carlo analysis. One of many ways to prove that your system will walk forward is to reserve some data as out of sample and then test your parameters on unseen data. Another is to test in real time market conditions before putting the system online. After it is online you can monitor your system. There are a variety of red flags to alert a trader that your system is not performing well. One last idea: why not just reoptimize at a given periodicity to keep it tuned to the market? I'm sorry my race car analogy dissapointed you. I thought it was pretty clever.
  • #16
  • Oct 4, 2006 6:23pm Oct 4, 2006 6:23pm
Quoting philmcgrew
You grossly oversimplify the subject of optimization. Entire books have been written on the subject of development and testing. Optimization is like fire. You treat it well and respect it, it will warm your house and cook your food. You misuse it and it will burn your house down. One way to avoid random data is to subject your trades to a Monte Carlo analysis. One of many ways to prove that your system will walk forward is to reserve some data as out of sample and then test your parameters on unseen data. Another is to test in real time market conditions before putting the system online. After it is online you can monitor your system. There are a variety of red flags to alert a trader that your system is not performing well. One last idea: why not just reoptimize at a given periodicity to keep it tuned to the market? I'm sorry my race car analogy dissapointed you. I thought it was pretty clever.
Heh, I wasn't dissapointed, it was poignant enough.. just felt it sort of missed the mark.
Hm, well forward testing is a given, but it still can be a matter of "luck" if it works, if the forward testing data happend to reflect favorable market conditions (which wouldn't be the norm).
I'm not saying that optimization is useless, quite the contrary.
However, when would you start optimizing? When do you know that your system is probable enough to trade, forward testing and all?
That's what I'm aiming at. There has to be a way to calculate the probability of success. Monte Carlo would probably be included in that, i.e. you could calculate the probability distribution. But wouldn't that have to be done after you have a substantial timeseries to check? Which brings us back to question one, how much data is needed? I don't know, we might be talking about the same thing... or I might be talking in circles... a bit tired right now.
  • #17
  • Oct 4, 2006 6:34pm Oct 4, 2006 6:34pm
Quoting firehorse
Hi,
I vaguely remember something about confidence levels from school maths and that was one of my ideas.

One consistency is between different parts of your data set. Does optimizing one half of your data provide good parameters for the second half?

Another consistency is if you change the parameters that the results change smoothly. If the results jumps about then I would call it inconsistent. That may or may not change with more trades in the backtest.

But as Phil has pointed out, by the time you have lot of trades in your backtest, the market may have changed thus invalidating your backtest.

Just my opinion.

Best regards
Alan
Ok. Good points. The smoothness is what I always look for (I would however like to see it after removing noise from the data, but that's a future project). Also working with several timeslices is imperative.
There is always the possibility of a market change, but systems that are overly sensitive to market conditions, for me personally, are a waste.
Also, wouldn't the changes to the market have to be quite drastic if they would totally invalidate a well tested system designed to work in all to-this-date market conditions?
But at any rate, ideally you would have (perhaps by a monte carlo simulation) parameters that the data should move within, to indicate a valid system...
  • #18
  • Oct 4, 2006 6:37pm Oct 4, 2006 6:37pm
I love the race car analogy. (Phil already knows that though).

The other part of the analogy is this: Giving a successful system/method to someone who doesn't understand risk/money management is like giving a 16 year old a Ferrari.....eventually they will wrap it around a tree.
Secret Identity: Bill Sullivan
  • #19
  • Oct 5, 2006 6:36am Oct 5, 2006 6:36am
Quoting philmcgrew
You grossly oversimplify the subject of optimization. Entire books have been written on the subject of development and testing. Optimization is like fire. You treat it well and respect it, it will warm your house and cook your food. You misuse it and it will burn your house down. One way to avoid random data is to subject your trades to a Monte Carlo analysis. One of many ways to prove that your system will walk forward is to reserve some data as out of sample and then test your parameters on unseen data. Another is to test in real time market conditions before putting the system online. After it is online you can monitor your system. There are a variety of red flags to alert a trader that your system is not performing well. One last idea: why not just reoptimize at a given periodicity to keep it tuned to the market? I'm sorry my race car analogy dissapointed you. I thought it was pretty clever.

This discussion is really about understanding how the parameters of your system behave over time. If you try plotting the profit/loss ratio (or some other indicator of model performance that is reasonably bounded such as (sum of profits)/(sum of profits+abs sum of losses) across your complete parameter space, over the life of a dataset one can start to appreciate the stability (or for many systems the lack of) stability in parameter space. If you don't have a data visualisation package capable of animating over time you can try plotting your full parameter space at discrete time intervals as a pseudo animation. A visualisation approach is more difficult if one has more than 2 system parameters. To overcome this you can use n-1 pair wise comparisons in 2D space to cover an n-dimensional parameter space using basic tools such as Excel. By periodically re-optimising parameters, in essence I believe you are trying to approximate the mathematical function that defines the trajectory of the behaviour of system parameters over time. The mathematical equation that defines your parameter trajectories will be different for each system and it is this property that will dictate how effective any re-optimising strategy will be. If the trajectory of your system is linear then re-optimising should work reasonably well however if it is non-linear then there will be less continuity of performance from one time period to the next, and hence re-optimising based on the recent past will fail to capture the inherent non-linearity within the system.
  • #20
  • Oct 6, 2006 6:20pm Oct 6, 2006 6:20pm
Quoting richp
This discussion is really about understanding how the parameters of your system behave over time. If you try plotting the profit/loss ratio (or some other indicator of model performance that is reasonably bounded such as (sum of profits)/(sum of profits+abs sum of losses) across your complete parameter space, over the life of a dataset one can start to appreciate the stability (or for many systems the lack of) stability in parameter space. If you don't have a data visualisation package capable of animating over time you can try plotting your full parameter space at discrete time intervals as a pseudo animation. A visualisation approach is more difficult if one has more than 2 system parameters. To overcome this you can use n-1 pair wise comparisons in 2D space to cover an n-dimensional parameter space using basic tools such as Excel. By periodically re-optimising parameters, in essence I believe you are trying to approximate the mathematical function that defines the trajectory of the behaviour of system parameters over time. The mathematical equation that defines your parameter trajectories will be different for each system and it is this property that will dictate how effective any re-optimising strategy will be. If the trajectory of your system is linear then re-optimising should work reasonably well however if it is non-linear then there will be less continuity of performance from one time period to the next, and hence re-optimising based on the recent past will fail to capture the inherent non-linearity within the system.
Ok, now it's getting interesting.
I have for instance < 10 parameters (a backtest in java), and although I have done some "manual genetic programming" to roughly optimize the parameters, full parameter space calculations takes a lot of processing time (I'm testing with a 2 million minute time series).
I don't really get you would accomplish it in excel though...
Also could you give examples on linear, and non-linear trajectories?
0 traders viewing now