Backtesting vs. Thought Process

I came across this thought provoking quote while surfing trading sites on the internet.

Usually back testing is not a workable thing for many reasons. The manner of your personal mental processes overrides any work that can be done on back testing whether the approach is canned or discressionary.

Here are some broad questions I would like your opinion on. In the context of system development:

1) Do you agree or disagree with this statement?

2) How likely is it that a system that backtests well will perform poorly?

3) How likely is it that a system designed according to correct personal mental processes will perform poorly?

4) What is the proper role of backtesting in system development?

5) What is the proper role of personal mental processes in system development?

6) What would be an example of successful personal mental processes?

Feel free to answer any, all, or none. Any thoughts on the general topic are welcome. Thanks in advance for your input!

James

great questions! the answers could fill many books!


1) Do you agree or disagree with this statement?

im not sure i understand the statement. what is meant by your personal mental processes overrides any work ?

2) How likely is it that a system that backtests well will perform poorly?

that depends on the process by which the backtest was done. if you just banged on your keyboard until you came up with a good backtest results, the system will probably fail in the future. if, instead, you wrote down your idea first, and the idea is rooted in some market function or idiosyncrasy that you have observed, then you went and backtest the idea and it looked good without then it has a MUCH better chance of working in the future. it is very important that you know WHY the system works. there should be a reason, and that reason should be understood BEFORE you go to the backtesting. you can fall into a trap of coming up with reasons why the system works AFTER youve built it, but its likely you will just be making up reasons to justify the system.

i have had many many many systems fail in real time trading after i thought they would do great. once you go through this process a few times, you start to get a feel for the dynamic at work. my advice is just build something and start trading it. and tracking the results. you get a feel for it afer a while, but you have to do some failing before you understand it.


3) How likely is it that a system designed according to correct personal mental processes will perform poorly?

i would say the likelihood is directly related to the experience of the trader.


4) What is the proper role of backtesting in system development?

first, to test your ideas (NOT to generate ideas) and see if they are legit. second, to get an understanding of your systems characteristics so you know how to do your position sizing.

Merlin,

You said: "im not sure i understand the statement. what is meant by your personal mental processes overrides any work ?"

Here is how I understand it (which may or may not be what the original author of that quote intended): The logic, common sense, and insight that goes into designing/refining a system is far more important than the backtesting process or results.

At issue is how you generate and refine your system. On the one hand, you have the thought processes. This is how you come up with ideas for making or improving a system. This is the “art” side of system design. Assumably right brained folks have an advantage here. This includes how you go about brainstorming, and is heavily tempered by experience and personal biases (try as we may, we can never truly eliminate those).

On the other hand you have backtesting. This is the “science” side of system design. Here is where the quants make their money, and the left brained among us thrive. The enemy here is curve fitting, but well conceived and executed backtesting can assumably keep that monster at bay. If backtesting is properly done, then it is assumably a valid process for refining your system. Or is it? The underlying premise of backtesting is that what worked in the past has a good chance of working in the future. Backtesting is as important to system design as that basic premise is valid.

The author of that quote is basically saying that well conceived / well thought out systems are the ones that succeed. How well tested a system is or how much profit it produced last year is far less important, perhaps even irrelevant.

Those are my speculations anyway, as to what that quote means.

I find it an intriguing concept, because I generally tend to over-quantify and over-analyze things by nature. When I start backtesting an idea in Amibroker, more often than not I’ll look up a few hours later at some complex beast of a system that bears little resemblance to my original concept. I also think I might be spending too much time trying to figure out backtesting code and not enough time staring at charts. If the keys to the kingdom lie at the other end of the spectrum, then perhaps I need to adjust my thinking. But how to go about doing that?

All feedback welcome,

James

James

I think we might enter a field here where (same as trading) different things work for different people in different ways.

Here is a link to a site on trading psychology by a Dr Brett N. Steenbarger. Just go to articles, there are some interesting ones - http://www.brettsteenbarger.com/ .

The next link might look a bit off the track (pun?) bit it makes for interesting reading anyway. Look for the manual download -http://www.emofree.com/default.htm .

If your thinking goes in the wrong direction after reading this don’t blame me

Hi Marejp,

Thanks for the links! Trading psychology may indeed be a good tool for changing thought processes. I've got a copy of Trading in the Zone by Mark Douglas, which is about trading psychology. Overall seems to be an outstanding book - lots of good tips. Although I fundamentally disagree with a couple of his premises.

Of course, the first step would be figuring out what I want my thought processes to be. Once I have a strong opinion about where I need to go, the tools will most likely fall into place.

Dr. Steenbarger's site looks interesting. I've bookmarked that one and will check it out in greater detail as I have time.

As for the other site with EFT - from scanning I suspect that I personally have too many disagreements with the main premises to get much useful there.

Thanks again.

James

Here is how I understand it (which may or may not be what the original author of that quote intended): The logic, common sense, and insight that goes into designing/refining a system is far more important than the backtesting process or results.


ok, gotcha. in this case, i agree completely with the author. and the point he is making is quite insightful.

At issue is how you generate and refine your system. On the one hand, you have the thought processes. This is how you come up with ideas for making or improving a system. This is the “art” side of system design. Assumably right brained folks have an advantage here. This includes how you go about brainstorming, and is heavily tempered by experience and personal biases (try as we may, we can never truly eliminate those).


there is a lot of art that goes into system development and backtesting (and this is what i think the author of the quote is referring to). for instance, my development process is filled with brainstorming, affected by experience, and affected by personal bias.

On the other hand you have backtesting. This is the “science” side of system design. Here is where the quants make their money, and the left brained among us thrive. The enemy here is curve fitting, but well conceived and executed backtesting can assumably keep that monster at bay. If backtesting is properly done, then it is assumably a valid process for refining your system. Or is it? The underlying premise of backtesting is that what worked in the past has a good chance of working in the future. Backtesting is as important to system design as that basic premise is valid.
.

even the quants such as myself are heavily reliant on their experience and creativity. as ive said many times, the process by which you produce your backtest result is far more important than the backtest itself. and that process can be quite complex, and it will be very personalized. my process is complex and has evolved over the years. the person who was most influential in my process is Perry Kaufman. his book "Smarter Trading" tells a convincing story about how your backtest results can be flawed.

I find it an intriguing concept, because I generally tend to over-quantify and over-analyze things by nature. When I start backtesting an idea in Amibroker, more often than not I’ll look up a few hours later at some complex beast of a system that bears little resemblance to my original concept. I also think I might be spending too much time trying to figure out backtesting code and not enough time staring at charts. If the keys to the kingdom lie at the other end of the spectrum, then perhaps I need to adjust my thinking. But how to go about doing that?


when you look up a few hours later and have a different system, this is a huge red flag that your development process is flawed. read Perry Kaufman, he goes into detail about this very issue. if i spend 20 hours on a research project, 10 of those hours will be spend outlining the system BEFORE i ever touch the code. you should treat each research assignment just like a scientist does, writing your hypothesis and setting out to prove or disprove it.


good luck with this stuff man. dont get frustarated if you feel lost or confused, this is hard material to tackle. after all these years im still confused

Thanks, Merlin!

As it happens, I'm reading "Smarter Trading" right now - about halfway through!

James

haha good stuff! make sure you get back to this thread when you are done reading it. i am interested to hear what you learned...

Usually back testing is not a workable thing for many reasons. The manner of your personal mental processes overrides any work that can be done on back testing whether the approach is canned or discressionaryWhy not a workable? Any trading is basing on some trade system (strategy). If you are haven't trade strategy and dealing by intuition only - it's bad for you. Our purpose - translate strategy to strict logic rules. It's inadmissible to evade from it when make a deals under "mental process" influence. If we have set of strict rules it's become possible to make back test. So we get an exact result of back test - the main thing - not evade from strict strategy rules when trade. There is no place for "mental process", especially when you in the open position.

red, what did you think of the book???

Hi Merlin,

Still finishing it ... too little time in the day, and there is plenty of substance in this one so want to give it the attention it merits. I've got some time off next week and should be able to polish it off.

My wife claims that it takes me forever to finish a book! Not true ... I just enjoy them at my own pace!

James