Systematic trading

Now I understand. It seems like your selection method suffers from data-mining bias. So the issue has more to do with that than with what parameters you are using for the selection.

I think Foracy has some excellent points here. In my experience I've seen that very high profit/drawdown ratios often fall over flat as soon as you try them with out sample data. It looks too much like over-optimization or curve fitting, the results are just too good to be true. Having said that, profit/drawdown still seems to make the most logical sense in terms of understanding an investment (i.e. risk return).

I've also been thinking about how to select the best optimisation criteria. One idea I had was to break up the in-sample period into a number of smaller time periods (windows) and calculate the profit/drawdown for each window. Then add the results but heavily penalise the strategy ranking if any one (or more) windows do not meet a minimum profit/drawdown requirement. Hence, you should end up with the most consistent strategies, not necessarily the ones with the highest profit/drawdown.

Has anyone tried anything like this? Unfortunately I'm currently stuck with MT4 so I can't give it a whirl (in fact, I can't give anything a whirl, insample testing currently takes about 1 week for 1 year of data in the strategy tester...).

Unless you know how to avoid data-mining bias, it doesn't matter what you do. You will fool yourself in the end.

U r talking to him (-:

I tried this and on one specific strategy (which seems to perform in many ways to be fair) it had a very good result. Not so great other than that. My conclusion is that you are still comparing optimums with optimums and this is not the solution. (the only difference that the optimum continued for a little longer if the optimum was already reached it would drop)

I moved away from the above for now. I believe it is the optimisation of the drawReward at fault. It leads you to the local optimum and you want the next local optimum so you can at least earn your drawdown back (-:
But what to replace it with is the problem.

There are various ways, one (that I don't use) is randomizing the data in blocks of random size instead of single points of data.

Main point is that you don't randomize all data, you sometimes use sequential points and sometimes not. There are lots of different variations you can use.

OK, I understand. Thanks.

Here is something people don't post here ofter It's combined data of 18500 strategies that all made at least 1% per annum (cagr). Stratgegies are generated with my GP framework and operate on 4 currency pairs.

At this list, fitness is the same as MAR (cagr / max dd). Both lists (upper is in-sample and lower is out-sample result list) are sorted by out-sample fitness (MAR).

If you can read this, I'm sure this is more than interesting chart, as I said, these kinds of lists are not usually posted in public forums Each line is 927 strategies (the result set is just split to 20 slices and sorted), the numbers are averages. tc is trade count, win is 2 kinds of win percentages ppr is profit per risked percentage of account, dd numbers are statistical data from dd.

Feel free to discuss about this (or not).

18540 strats
0 ins(927) fit:0.75483 tc:1030.6 pf:2.05646 win:0.39078/0.07843 mar:0.75483 ppr:0.37814 cagr:0.04802 rar:0.04671 dd:0.07426/0.02187/0.01885/0.02636
1 ins(927) fit:0.77328 tc:1629.3 pf:1.66754 win:0.40146/0.12581 mar:0.77328 ppr:0.22795 cagr:0.05412 rar:0.05503 dd:0.07704/0.02177/0.01880/0.02350
2 ins(927) fit:0.81618 tc:1670.4 pf:1.64806 win:0.40255/0.13866 mar:0.81618 ppr:0.20622 cagr:0.05887 rar:0.05947 dd:0.07970/0.02187/0.01905/0.02342
3 ins(927) fit:0.82777 tc:1650.8 pf:1.59351 win:0.40325/0.13435 mar:0.82777 ppr:0.20770 cagr:0.06033 rar:0.06057 dd:0.07918/0.02221/0.01895/0.02297
4 ins(927) fit:0.86463 tc:1662.3 pf:1.58210 win:0.40264/0.13575 mar:0.86463 ppr:0.19897 cagr:0.06337 rar:0.06360 dd:0.07922/0.02167/0.01886/0.02250
5 ins(927) fit:0.90321 tc:1872.7 pf:1.56829 win:0.41039/0.14582 mar:0.90321 ppr:0.18349 cagr:0.06526 rar:0.06559 dd:0.07792/0.02112/0.01846/0.02182
6 ins(927) fit:0.97078 tc:1660.0 pf:1.59579 win:0.40847/0.13646 mar:0.97078 ppr:0.22136 cagr:0.07133 rar:0.07145 dd:0.07823/0.02141/0.01856/0.02207
7 ins(927) fit:0.98896 tc:1525.9 pf:1.72999 win:0.41453/0.12854 mar:0.98896 ppr:0.24327 cagr:0.06869 rar:0.06830 dd:0.07643/0.02089/0.01817/0.02177
8 ins(927) fit:1.02375 tc:1652.2 pf:1.64272 win:0.41293/0.13726 mar:1.02375 ppr:0.22796 cagr:0.07516 rar:0.07548 dd:0.07813/0.02055/0.01835/0.02200
9 ins(927) fit:1.05810 tc:1614.3 pf:1.61476 win:0.42109/0.14112 mar:1.05810 ppr:0.21629 cagr:0.07571 rar:0.07514 dd:0.07728/0.02054/0.01809/0.02140
10 ins(927) fit:1.08033 tc:1463.2 pf:1.64243 win:0.42817/0.13354 mar:1.08033 ppr:0.21280 cagr:0.07756 rar:0.07698 dd:0.07549/0.01968/0.01768/0.02077
11 ins(927) fit:1.09674 tc:1508.4 pf:1.64539 win:0.43529/0.13819 mar:1.09674 ppr:0.20389 cagr:0.07667 rar:0.07601 dd:0.07531/0.01972/0.01765/0.02071
12 ins(927) fit:1.14287 tc:1596.4 pf:1.62099 win:0.42550/0.14314 mar:1.14287 ppr:0.21392 cagr:0.08247 rar:0.08159 dd:0.07667/0.02012/0.01787/0.02117
13 ins(927) fit:1.13358 tc:1349.0 pf:1.70775 win:0.44090/0.12831 mar:1.13358 ppr:0.21978 cagr:0.07712 rar:0.07570 dd:0.07337/0.01919/0.01728/0.02073
14 ins(927) fit:1.17213 tc:1278.8 pf:1.69182 win:0.44364/0.12785 mar:1.17213 ppr:0.23841 cagr:0.07969 rar:0.07778 dd:0.07257/0.01893/0.01712/0.02076
15 ins(927) fit:1.22338 tc:1143.5 pf:1.69898 win:0.44927/0.12417 mar:1.22338 ppr:0.22157 cagr:0.08191 rar:0.07953 dd:0.07180/0.01839/0.01676/0.02032
16 ins(927) fit:1.21896 tc:1006.4 pf:1.76569 win:0.46877/0.11922 mar:1.21896 ppr:0.20752 cagr:0.07866 rar:0.07567 dd:0.06917/0.01750/0.01618/0.01901
17 ins(927) fit:1.28912 tc:901.1 pf:1.76399 win:0.47491/0.11266 mar:1.28912 ppr:0.21653 cagr:0.08360 rar:0.07973 dd:0.06788/0.01724/0.01599/0.01934
18 ins(927) fit:1.25454 tc:730.3 pf:1.77183 win:0.47613/0.10544 mar:1.25454 ppr:0.22823 cagr:0.08074 rar:0.07577 dd:0.06704/0.01696/0.01579/0.01918
19 ins(927) fit:1.34669 tc:597.5 pf:1.89740 win:0.49950/0.09312 mar:1.34669 ppr:0.27385 cagr:0.08685 rar:0.08052 dd:0.06419/0.01617/0.01529/0.01891

0 out(927) fit:-0.74754 tc:211.5 pf:0.38328 win:0.23041/0.06737 mar:-0.74754 ppr:-0.29001 cagr:-0.05697 rar:-0.05459 dd:0.07631/0.03923/0.02044/0.02292
1 out(927) fit:-0.65186 tc:327.1 pf:0.60092 win:0.31076/0.11413 mar:-0.65186 ppr:-0.12905 cagr:-0.05414 rar:-0.05543 dd:0.08302/0.04299/0.02256/0.02443
2 out(927) fit:-0.54127 tc:322.5 pf:0.70950 win:0.32701/0.12612 mar:-0.54127 ppr:-0.09238 cagr:-0.04277 rar:-0.04612 dd:0.07881/0.04055/0.02087/0.02249
3 out(927) fit:-0.43495 tc:315.2 pf:0.77983 win:0.33725/0.12446 mar:-0.43495 ppr:-0.05245 cagr:-0.03180 rar:-0.03506 dd:0.07313/0.03639/0.01909/0.02097
4 out(927) fit:-0.32497 tc:316.1 pf:0.84943 win:0.34500/0.12644 mar:-0.32497 ppr:-0.02944 cagr:-0.02210 rar:-0.02445 dd:0.06799/0.03244/0.01787/0.01972
5 out(927) fit:-0.21231 tc:360.5 pf:0.91044 win:0.36135/0.13876 mar:-0.21231 ppr:-0.00710 cagr:-0.01394 rar:-0.01685 dd:0.06579/0.02994/0.01748/0.01924
6 out(927) fit:-0.09216 tc:315.2 pf:0.96810 win:0.35941/0.12937 mar:-0.09216 ppr:0.02104 cagr:-0.00597 rar:-0.00716 dd:0.06443/0.02758/0.01751/0.01907
7 out(927) fit:0.03085 tc:290.3 pf:1.00095 win:0.37860/0.12255 mar:0.03085 ppr:0.07508 cagr:0.00238 rar:0.00143 dd:0.05807/0.02390/0.01604/0.01784
8 out(927) fit:0.16267 tc:316.5 pf:1.07877 win:0.37669/0.13317 mar:0.16267 ppr:0.06078 cagr:0.00955 rar:0.00742 dd:0.05875/0.02353/0.01625/0.01797
9 out(927) fit:0.30626 tc:302.3 pf:1.13440 win:0.38836/0.13667 mar:0.30626 ppr:0.07650 cagr:0.01736 rar:0.01478 dd:0.05688/0.02195/0.01581/0.01783
10 out(927) fit:0.45507 tc:277.0 pf:1.19499 win:0.39946/0.13223 mar:0.45507 ppr:0.09536 cagr:0.02441 rar:0.02195 dd:0.05349/0.01997/0.01478/0.01668
11 out(927) fit:0.62833 tc:286.7 pf:1.25025 win:0.40761/0.13678 mar:0.62833 ppr:0.10039 cagr:0.03187 rar:0.03154 dd:0.05063/0.01816/0.01388/0.01613
12 out(927) fit:0.82464 tc:304.7 pf:1.30292 win:0.41050/0.14398 mar:0.82464 ppr:0.12859 cagr:0.04160 rar:0.03973 dd:0.05054/0.01779/0.01384/0.01599
13 out(927) fit:1.06755 tc:255.0 pf:1.41192 win:0.42507/0.12841 mar:1.06755 ppr:0.15184 cagr:0.04743 rar:0.04677 dd:0.04448/0.01496/0.01217/0.01473
14 out(927) fit:1.33509 tc:244.3 pf:1.49081 win:0.43371/0.13048 mar:1.33509 ppr:0.18858 cagr:0.05774 rar:0.05652 dd:0.04324/0.01434/0.01178/0.01463
15 out(927) fit:1.66976 tc:219.7 pf:1.58730 win:0.44147/0.12828 mar:1.66976 ppr:0.20095 cagr:0.06861 rar:0.06750 dd:0.04114/0.01304/0.01107/0.01391
16 out(927) fit:2.09408 tc:194.4 pf:1.71196 win:0.46678/0.12511 mar:2.09408 ppr:0.19769 cagr:0.07546 rar:0.07179 dd:0.03608/0.01090/0.00954/0.01235
17 out(927) fit:2.66362 tc:172.3 pf:1.87243 win:0.48079/0.11936 mar:2.66362 ppr:0.23445 cagr:0.09063 rar:0.08554 dd:0.03412/0.01010/0.00909/0.01244
18 out(927) fit:3.59105 tc:144.3 pf:2.15069 win:0.49379/0.11539 mar:3.59105 ppr:0.29280 cagr:0.10877 rar:0.10097 dd:0.03054/0.00867/0.00811/0.01170
19 out(927) fit:7.17900 tc:119.0 pf:4.25150 win:0.54266/0.10464 mar:7.17900 ppr:0.47077 cagr:0.14749 rar:0.13487 dd:0.02471/0.00684/0.00664/0.01102

Mikkom

I am not sure how to interpretet these figures. Is this correct:

19 out(927) fit:7.17900 tc:119.0 pf:4.25150 win:0.54266/0.10464 mar:7.17900 ppr:0.47077 cagr:0.14749 rar:0.13487 dd:0.02471/0.00684/0.00664/0.01102

Investment amount: 10000

Profit=10000*14.7%=1470
drawDown=1470*2.4%= 35.28

Is this right?

I'm also not 100% sure what we are meant to be seeing in those numbers?

If I understood correctly, you are resorting each IS/OOS list of top 20 (ensemble?) fitness performers by fitness rank. That would scramble the dependency between Is/OOS no? Also, I'm confused by sorting both lists by OOS fitness. If one list is IS, wouldn't you just sort that by IS fitness?

1) Wouldn't it be more useful to sort one list (preferably IS) by rank, then simply post the corresponding oos results under that line?

2)Could you clarify if this interpretation is correct? You culled the top 18,500 IS strategies by requiring CAGR>=1%.

Then grouped the strategies (arbitrarily i.e. no order?) by dividing by 20. The resulting 927 strategies/group were then averaged to get aggregate results/group? I.e. strategy 1 has return r1,...strat 926 return r926, ... strat 927 return r927, so the resulting figures are just measured on the linear weighted results of the returns. Then each of 20 groups can represent a sample distribution of avg. results with corresponding average derived metrics, and all 20 averages can be used to generate a sampling dist.

3) If I did understand correctly that both lists are ranked & sorted by OOS fitness alone, then I would intuitively expect the OOS fitness mean and variance of each set should to be roughly equal (which they clearly are not), since both lists are comprised of the same set of results from the same pool (CAGR>=1%).

It makes more sense (to me at least) if the top list is ranked by IS fitness and bottom OOS fitness.. to which one could infer that even though the variance is widely different between both sampling sets and OOS avg. is lower, the OOS average is still positive and the bulk of the OOS fitness results are still net positive, which is good. Also by looking at it from the perspective of a sampling distribution, gives more statistical weight to the validation, IMO.

The other thing I would be cautious about is how much the net friction pulls down the mean of both sets, since there are a tremendous number of strategies being run in parallel.

http://i52.tinypic.com/29o0dv9.png

One other relevant bit of information would be the time period/MAR/CAGRs of the instruments themselves over the IS, OS, and combined IS/OS period and comparing to combined MAR/CAGR of average IS, OOS, & combined IS/OS strategy set.

CAGR summaries:
http://i55.tinypic.com/14alky1.png


Nevertheless, thanks for sharing. You are among a sparse handful of forum posters that I enjoy
reading.

Nope, this is simply the whole list split equally to 20 parts. In this example each line contains average values for 927 strategies.

In-sample is sorted in the same order as out-sample so it's possible to see correlation between out-sample fitness and in-sample values, that's what I'm trying to find - value(s) in in-sample data that can be used to estimate out-sample values to some degree.


In this run I have a filter that skips all strategies that have under 1% cagr.

Mikkom,

Could you please answer the above? How long is the out of sample period?

cagr is profit per annum (year), sample size is 01/2003 - 01/2010

max dd is max dd from closed peak balance in percentages.

Thats a very good result. I think that the most important thing here is that the in sample list is almost sorted which means that the fitness function is working very well (best performers in IS list are also best in OOS list). I would be very pleased if my fitness function was that good.

Is this results still with relatively short out of sample period (like a year or so) ?

out-sample is under 2 years (jan 2010-today). Note that I told what the fitness is, it's mar (cagr/max_dd)

So here you go - you figured out what I was trying to point. On my results, in-sample average MAR of certain range seems to correlate out-sample average MAR (of reasonable sample size) to some extent. This might be one of the most important things you read on this thread

Craig,

I alway enjoy reading your comments here on forex factory. What I am curious about is what type of systems do you use in relation to forex. Are they correlation based as this appears to be your niche on other markets?

I don't trade forex anymore, all my trading is in US stocks.
When I was trading forex it was basic RTM stuff.

how come not forex?

More markets, more opportunities.

I wanted to thank the originator and all of the contributors to this thread.

The information and insights were very helpful, and the various contributed links were excellent.

I hope the serious traders will continue to post their experience and findings.

The ATS journey is a very difficult path. Sharing tidbits and viewpoints makes that journey both easier and more pleasant. My new system arrives next week for this endeavor and I look forward to sharing in kind.

Best regards,
Merlin
Skype: merlinbrasil