"curve fitting" or just improving??

Hello fellow forexers

I have what may be a silly question, but it seemed slightly in depth for the beginner section. Please feel free to move this if I'm wrong.

I'm working on an EA to perform a very simple string of trades that sometimes extends into the early morning hours. From visual backtesting and metatrader's (admittedly horrible) backtester it works like a charm, but now dealing with live data for the past couple months it has had some challenges that I need to fix (dealing with requotes etc) that appear to require me to change the actual strategy rather than simply fixing programming bugs.

So here's the question(s): at what point am I 'curve fitting', and at what point am I simply fixing unforseeable issues that have cropped up during live testing? Am I improving the strategy, or dooming the strategy to never work by being too focused on yesterday's data? And I suppose the most important question, is it even possible to 'curve fit' to live data, or is curve fitting just something people do to sell their new 'super wonder gonna-make-you-rich EA' on ebay with a killer backtest graph?

Many thanks in advance!!


Seavo

Some simple things you can do to make sure you're not curve fitting:

-- Try different markets. If you're testing it on just the EURUSD, see how well it performs on USDJPY, GBPUSD, etc.

-- Try different times. Use a block of data further in the past (say from Jan 2004 to Dec 31, 2005). Sometimes people optimize it for a block of data far in the past, then run the strategy forward from that point, without any changes, to see how it performs. For example, optimizing for 2004-2005, then running the strategy from Jan 01, 2006 forward, without changes, as if you started trading this on that date.

-- Make sure the slightest change in parameters doesn't cause a huge change in performance. Usually this is a sign of curve fitting. For example, say you changed a setting from 4.5 to 4.6, and performance jumped 25%. You should be able to change settings by +/- 10% without causing a dramatic change in performance.

If you are using Metatrader to do the optimization you can also look at the way the results cluster using the graph window. This allows you to see how the results vary as the params vary, big results in isolation are often unstable curve fit results.

Great point. I never thought of this, but it makes perfect sense. So, you would say that changing (for instance) an MA from 14 to 17 and getting crazy better performance is unlikely true whereas going from 14 to 28 is more likely an improvement rather than just a curve fit?

Also an excellent point. I remember my first "holy grail" EA that turned $10k into some huge 8 digit figure in a couple months. Oh, if only "90% modeling quality" was 90% true...

I guess my main concern is that I'm trying to find high probability setups, or at least setups that are profitable with good MM, and I'm worried that if the market changes its behavior the same probabilities won't remain the same (and will therefore need tweaking).

From what I'm understanding, curve fitting is something that you'll know when you're doing it by either plugging every combination of MAs into your strategy until you find the "perfect one", or by exploiting a weakness of the strategy tester to accomplish unrealistic results. Does that sound true?

Thanks again for your input. It has definitely been most helpful to me


Seavo

You can upload M1 data back to 2001 with this link...

http://www.forexfactory.com/showthread.php?t=22963

That will get you to 90% MQ.

Yep... a 28 period MA is significantly different than a 14 period MA, so you would expect a significant difference in performance. Going from 14 to 17, however, is much less of a change, and there shouldn't be a huge difference in performance.

It cracks me up how some people are so obsessed with Fibonacci numbers that they prefer to use, say a 21 period Bollinger Band over the standard 20 period Bollinger Band. Go ahead and plot those two on a chart and tell me if it really makes that much of a difference.