Optimization, statistics, and avoidance of curve fitting

I have been curious about this subject, and I'm interested in hearing some educated opinions on the matter:
Curve fitting is the major pitfall of optimization - anyone who has tried to move on to forward-testing knows this.

The way I try to avoid curve fitting is by optimizing on one length of time and back-testing on another. Some people call this a "forward walk." Of course, if you do this 10,000 times and pick the best result you are no better than the automatic optimizer
.
I try to find value ranges within my variables that, when tested with a "forward walk," produce a normal distribution (statistically) of +profit.

Another obvious practice to avoid curve fitting is to minimize the number of variables being optimized.

I am sure that much more sophisticated methods than this exist; what are they? What statistical tests / optimization practices are relevant to the avoidance of curve fitting?

This is a really good question, which will hopefully elicit some more knowledgeable replies. I personally have not had any success with any form of data mining, I've tried variable reduction, complexity penalization, varying the size of the training set, all sorts of things. At this point the only things I have had success with work across multiple markets with no optimization.

I actually try to curve-fit my EAs.

As a general rule of thumb, I optimize my EAs against the previous month or so, a relatively short period of time. I am betting on the assumption that the market will continue the overall macro trend for at least another week. When a market-shifting paradigm occurs, that's when I decrease the weight of my portfolio using these EAs.

I back this up with longer-term EAs with a smaller sigma to help reduce the volatility in my account.

Have you tested your theory by optimizing for one range of time (ex, June-July) and testing on the next range (July-August)?

I have seen this type of process fail repeatedly and have never seen it succeed; although technically for some EAs it might.

Here are a couple good articles that I found on the subject of this thread:


How Not to Fall into Optimization Traps?
Mathematics in Trading: How to Estimate Trade Results

Additionally, I think it is very important to avoid using any type of lot-scaling money management. If you have some sort of lot scaling or martingale in your EA then you will not be able to easily interpret any of the results as the bet size is not constant.

I think this is a very interesting discussion, and am surprised there hasn't been more replies from some of the EA & strategy developers here on FF. Surely others have come across this problem?

I too have spent a lot of time playing with the MT4 optimizer to get the best parameters for my EAs. So far all of these optimized strategies, that "appeared" very profitable, have fallen over flat once forward tested. As part of the optimization I performed, I always used large step sizes in parameters to prevent curve fitting to too high a degree.

I once wrote an EA that had its own inbuilt optimizer, running it on every bar of data to choose trading parameters. This EA could be tested in the MT4 strategy tester, and I used the MT4 optimizer to try to determine the best length of history for the EA to use with its inbuilt optimizer (I hope that all makes sense). This is what I learnt:

- An optimizer can make nearly any strategy look good on past history, if you try hard enough
- There was no optimization period that gave satisfactory results
- The past optimization parameters didn't seem to benefit future trades
- I tried a very simple on/off optimizer to reduce curve fitting to the minimum (i.e. don't trade if previous market conditions aren't favourable). This strategy tended to switch on when the strategy was not favourable (but the previous period would have been good) and switch off when the strategy would have been effective (but the previous period was bad).
- In general, often the strategy alone gave better results than the strategy with in-built optimizer.

As part of my experiments I did try a number of strategies, but eventually abandoned the whole project because it was all too much work and didn't seem to provide the results. I also considered building an EA with two optimizers; one optimizer to optimize the other optimizer, but decided it was all getting a bit silly and I was better off working on a better core strategy.

However, I'm sure that there are other better strategists out there who have had better results with using optimizers.

Cheers,

Michael

Thanks for your reply, PipMasterMik. Large step sizes are indeed a good idea. I have not yet tried auto-optimization but I had a feeling that it would have results similar to what you described.

Here is an idea that I've had for a while... unfortunately the programming might be a little over my head:

You know how the strategy tester displays that 2-dimensional "heat map" of the profit (or p.f, drawdown, etc)? Well imagine that instead of being 2 dimensional it was 3 dimensional; like little extruded cubes instead of flat squares.

http://furlender.com/Forex/graph.jpg

The third dimension would represent forward-walk success. The longer the third dimension is (the more extruded the cube), the longer those optimization parameters held up in a forward walk. Then you could pick the result which had the longest third dimension ,and was perched on the top of a normal distribution curve surrounded by other results with long third dimensions.

Interesting idea. The coding for this idea sounds possible but a bit complicated though.

One idea I used was to establish strategy performance sensitivity to a particular optimized parameter. As an example, move a stoploss by a few pips from the optimum and see how it affects the result; if the amount of profit swings wildly then you could say that this parameter has been too "curve fitted".

Ideally, a strategy would still be successful over a reasonable range of a parameter, with a gradual drop off in performance as the parameter moves further and further from the optimized value in either direction.

Cheers,

Michael

That is a very good idea as well!

Here is another idea that I've been playing around with. It seems logical to me:

Say you are evaluating the effectiveness of a filter. You get one optimization with 1324 trades and a profit factor of 1.67; another with 669 trades and a profit factor of 2.05. You want to know which optimization to use. For simplicity's sake lets assume that the R:R is 1:1.

Here is an equation I made to determine which result is better:

RESULT A
TL = 1324 (trades)
PL = 1.67 (profit factor)

RESULT B
Ts = 669 (trades)
Ps = 2.05 (profit factor)

Wn = (normalized win) = Ts/Ps
Ln = (normalized loss) = Ts/Ps^2

Wh = (hypothetical win) = Wn+TL-TS
Lh = (hypothetical loss) = Lh+TL -TS

Ph = (hypothetical profit factor) = Wh/Lh

Q = Quality of filtration = (Ps-PL)/(PL-Ph)

The higher the value Q, the fewer wins are filtered out in relation to losses.

The Q value for this instance is 0.81. Since it is less than 1, the trades that would be filtered out during the transition between Result A -> Result B by themselves would have a profit factor ABOVE 1. Therefore Result A is better.

It's nice to see a little more activity on this thread.

Interesting but I'm not sure I quite understand this idea. However, there are many ideas out there that I don't quite understand... Do you care to elaborate on the reasoning a little more? I think we may be using different definitions of Q.

I would have thought that it would be a good idea to determine the sensitivity to a single parameter value, which would result in a distribution of profit results (i.e. one profit result for each parameter value). The sharper the drop-off, the higher the filter selectivity (or Q, as often referred to in electronic/digital filter design). The higher the Q, the more highly-tuned the parameter is to the incoming data, or the more curve fitted it is (or more sensitive it is to losing profit when new incoming data arrives).

As the Q approaches 0, you find that your strategy would be profitable regardless of parameter value, which would indicate that their is a low dependence on features in the current data set, and hence you would expect similar results with new incoming data.

I hope this makes sense. I look forward to hearing a bit more about your ideas.

Yes this absolutely makes sense, and it is another competent evaluation of "Q."

I'll try to elaborate on my definition of "Q" :

Imagine you get two different optimization results as you increase the selectiveness (rule rigidness) of Filter X:

Result A:
200 trades, 1.7 pf

Result B:
100 trades, 2.0 pf



Pretend that all you have to go by is these two results and nothing else.
Which do you choose?

As the filter increases in selectiveness/rigidness from Result A -> Result B we eliminate 100 trades. The real question, that forms the basis for my equation, is : "what is the profit factor of the 100 trades that were eliminated from Result B but present in Result A?"

If the profit factor of those 100 eliminated trades is > 1.0 then the 100 eliminated trades were "net winners" and should not have been eliminated.

The formula I made determines the profit factor of Result B if those 100 eliminated trades were added back on, but with a profit factor of 1.0. I call this value the "hypothetical profit factor."

If the hypothetical profit factor of Result B is less than the profit factor of Result A, then the filter did a bad job, and you should pick Result A.

What I called the "Q value" is simply the hypothetical profit factor of Result B minus the profit factor of result A. (If the value is > 0, pick Result B, if value < 0, pick Result A)

I made a little program in C++ to automate this calculation. It is attached. If you do not want to run an .exe (for fear of viruses or whatever), the source code is included and you can compile it yourself.

I made a mistake the last time I wrote out this formula. I guess its been too long to edit that post, so here is a new post with the revised formula:

RESULT A
TL = 1324 (trades)
PL = 1.67 (profit factor)

RESULT B
Ts = 669 (trades)
Ps = 2.05 (profit factor)

Wn = (normalized win) = Ts/Ps
Ln = (normalized loss) = Ts/Ps^2

Wh = (hypothetical win) = Wn+(TL-TS)/2
Lh = (hypothetical loss) = Lh+(TL -TS)/2

Ph = (hypothetical profit factor) = Wh/Lh

Q = Quality of filtration = Ph-PL

Thanks for the detailed description, your high level description makes sense to me. It sounds like a good approach if your filter is only reducing the number of trades to become more selective, but does not impact on the trades that are not filtered.

I think there is a bit of an art to looking through the list of optimised parameters and trying to decide which is the best one to use.

One little thing that I also like to consider is the sample size; the more trades (assuming the trades are not all clumped together) the higher the statistical relevance of the analysis. A look at the backtested charts is necessary to determine if there were a couple of specific market events that drove the profit, creating a high risk strategy.

The other factor I always like to look at is the profit/drawdown ratio. If this is anywhere near 1 I see this as not a good strategy. This indicates that the risk and return are essentially equal, and if you have ended up ahead it is probably due to luck, not by your strategy having a statistical edge. If it is too high (say > 4) then I start to wonder whether it is over-optimised. The drawdown also provides a good guide for the expected future strategy performance, if in real use the strategy exceeds your expected drawdown it may be time to switch it off.

Another tip: Artificially remove the spread in your back-test data, so that Ask=Bid. Then, instead of closing trades with a take profit and stop loss, simply close your trades on the next bar's open (regardless of p/l).

This eliminates two variables and drills down to the core ability of the strategy to predict price change when the signal occurs .

I do this sometimes when making modifications to my core strategy. I also do this with "open" strategies in which operands ( >,=,>=,<=,&&,||) are optimizable external variables.


Here is an unrelated idea that I've been thinking about:

Say you have a strategy that generally works. As you optimize a filter for the strategy the profit factor increases to a point and then begins to decrease.

Instead of picking the inputs with the singular best stats it may be smarter to formulaically approximate the relationship between input -> profit factor and apply the resulting equation to your Lot sizes.

For example :
---
Profit Factor = Input*2 (linear relationship)

Previous Input = 1
Previous Lot size = 2

New Input = 2
New Lot Size = 4
---

By doing this you are purposely curve fitting the profit factor. BUT, and this is the key part, your profit on a "forward walk" should INCREASE. If it does, then all is good; if it does not, then the input->profit relationship is illusionary and there is no point optimizing it anyway.

To determine a close fit formula I recommend Matlab.

Interesting idea, I recall looking at this type of measurement to determine strategy success in the past. I've found that the market direction may not be be clear for sometimes quite a few bars, and I was often just testing against random market noise. Also, this method makes it hard to manage risks; a stoploss setting is necessary to protect the account, and this needs to be considered in conjunction with any directional prediction in order to determine system performance. Also, I think that the spread is crucial in determining strategy performance; a high spread in proportion to profit/stop sizes means that you need to gain a substantial edge in order to have a profitable strategy.

Since playing around with all of these experiments, my chosen direction was to use a 1:1 R:R with sl/tp settings that sit outside of normal volatility. Hence, the strategies are less susceptible to market noise and success/failure is dependent on correct prediction of reasonable sized market moves at >50% win rate.

in order to solve curve fitting problem,
i think that you have to test it on other symbols and equities,
so it perform good results as well.
my question.
on which symbols you must test your ea
before running it live?
EURUSD, majors, crosses, gold, sp500, oil?

I think this all depends on the type of EA it is, but generally I think it is only worthwhile to test on correlated pairs.

Example:
You optimize on EURUSD and check on GBPUSD. Or optimize on EURJPY and test on GBPJPY. Keep in mind the differences in spread though. A EURUSD strategy can prove unprofitable on GBPUSD simply because GBPUSD has higher spread. That does not speak for EURUSD profitability. Perhaps it may make sense to artificially change the GBPUSD spread for this test.

10X
sounds very smart idea.
i also have an idea to test ea's that were planned for trending market
and planned to work on 240M, or 1440M graphs
on SP500, GOLD, OIL
only on the long side.
cause stocks and commodities tend to go up.

How would you achieve artificially removing the spread in your back-test data?



so that Ask=Bid

Use this script. In its inputs set the spread to 0.

I know this is a bit late, but I like that idea... i'm definitely going to try it