The System Behind the System, by Merlin Jeffries

Originally published in Stocks & Commodities, February 2006 Issue.


The System Behind The System
by Merlin Jeffries, February 2006

When I was a novice system developer I would get excited by every marble-smooth equity curve I stumbled upon. Incredible backtest result in hand, I would start trading my systems with real money, only to find they were big loser in real-time. What happened?

What happened is that the systems I was finding, while excellent at predicting the past, were deficient when it came to predicting the future. This variety of trading system was almost impossible for me to distinguish from the future-predicting kind; until, that is, I started religiously following a process for developing my systems.


The Means Justify the End

In the world of system development the means justify the end. One can not judge a trading system based upon a backtest result (the end); one must instead consider how the backtest result was achieved (the means). The means, or the system behind the system, is a strategically designed process for developing trading systems, and it’s a trader’s main defense against being fooled into putting real money behind a hindsight based system.


The System Behind the System

Here I will present my system, or process, for developing mechanical trading systems. Hopefully it will give the reader a few ideas for developing their own.

First comes the often undervalued administration work. For each system, I create a “project” to organize my efforts. The project gets a name and identification tag. This comes in handy when I want to look back on my past research.

For each project, I move through the same seven steps:

[SEE ATTACHED IMAGE]

Step 1 – Conceptualize

The first step is to write down the concept and initial rules for the system. For example, if I am researching a simple breakout system, I may write:

concept:
A break of the 20-day high or low signifies the start of a trend.

rules:
ENTRY LONG: buy when price breaks the highest high of 20 days
ENTRY SHORT: sell when price breaks the lowest low of 20 days
EXIT 1: stoploss
EXIT 2: profit target
EXIT 3: max hold period

Next I will identify my system’s degrees of freedom (DOFs). This will come in handy later. In this example, there are three DOFs to declare:

DOFs:
1. stoploss size
2. profit target size
3. max hold period days

I could also consider the “20 day” breakout point to be a DOF. However, in this example I will assume that 20 days is “hard coded” and can not be changed during the project. If I may want to optimize or change any certain aspect of the system over the life of the project, I will declare it as a DOF.


Step 2 – Gather Data

Here I gather the market data that will be used to code and test the system. Following the example, I may decide that my total data pool will include the following:

EURUSD / 05.1998 – 08.2005
USDJPY / 07.1999 – 08.2005
EURGBP / 05.1998 – 08.2005
AUSUSD / 05.2000 – 08.2005

I like to divide my total data pool into four categories:

1. Build Data
This is the data I will use to program/code the system. I don’t need much data for this, just enough to ensure that my code is executing trades as designed.

2. Test Data
I use this data to find my system’s most robust DOFs through chart study and optimization.

3. Walkforward Data
This data will be used to determine if my system can predict the future.

4. Unseen Data
This data offers one final, “honest” test of the system’s ability to predict the future.

How a trader divides their data is more art than science. General guidelines I use are 5% Build Data, 40% Test Data, 40% Walkforward Data, and 15% Unseen Data. I also like to spread a specific symbol (i.e. EURUSD) among at least three data categories.


Step 3 – Code
(data used: Build Data)

Next I write the programming code for the system, and I use the Build Data to check that the code is executing trades as intend.

More times than I’d like to admit there was a flaw in my early code that squandered my entire project. I’ve learned it’s better to take an hour closely inspecting trades than to waste two days researching and testing the wrong system.


Step 4 – Refine
(data used: Test Data)

Using the Test Data, I optimize my DOFs until I find a set that show desirable results. I don’t worry much about the system becoming over-optimized, because if that happens it will not survive the next three steps.

Some software applications, like TradeStation, allow traders to run automated optimizations for the DOFs. I like to lightly use this feature to get a feel for the range of DOFs that will be profitable. If I try 50 variables for each of my DOFs, and 40 yield positive results, this is a good sign that the system is robust. Traders may want to instead manually plug in variables for the DOFs until they stumble across a set that seems to work. I like to take this approach as well.

The main purpose of this step is to study the system on the Test Data and choose the set of DOFs that are the most robust. Continuing the example, suppose I choose the following set of variables:

DOFs:
1. stoploss size – 30 ticks
2. profit target size – 40 ticks
3. max hold days – 22 days


Step 5 – Test
(data used: Test Data)

This step is simple. Using the DOFs from Step 4, I run a backtest on the Test Data, then save the backtest report for reference. I then carefully study the backtest report to determine if results are good enough to be traded in real-time:

> If I would not be willing to trade the system, I loop back to Step 4 to re-refine the system.

> If I am sure I would trade the system in real-time, based on the backtest results, then it’s on to the next step.


Step 6 – Walkforward
(data used: Walkforward Data)

This is where I find out if the system has promise. Without changing the DOFs used in Step 5, I run a backtest on the Walkforward Data:

> If the resulting backtest result is not profitable, or does not have tradable attributes, I will loop back to Step 4 and come up with another set of DOFs (re-refine).

> If the resulting backtest looks tradable, I cross my fingers and move to the final step.


Step 7 – Unseen Data
(data used: Unseen Data)

The final step is to backtest the system on the Unseen Data. I like to call this step “the moment of truth” because it’s quick yet awfully significant.

> If the backtest result is negative the project is over. All the data has been spoiled and any further efforts will be tainted with hindsight. When I reach this point I usually let out a sigh and mumble something about how the life of a quant isn’t all it’s cracked up to be. Unfortunately this is the way most of my projects end.

> If the backtest result is positive, a group of angels fly in my office and circle my monitor singing “hallelujah!” By passing this step, there is a good chance that the system has the ability to predict the future, and it will most likely end up in my portfolio.

The Importance of Unseen Data

“Unseen” data is nothing more than a second set of Walkforward Data. I added this aspect to my process after becoming frustrated with the notion of having only “one shot” at getting the DOFs on the Walkforward Data right before having to abandon the system.

The Unseen Data gives considerable flexibility to my process. I’m able to refine, test, and walk the system forward a number of times without contaminating all of my data. I am always left with one final “honest” test of the system.

It’s worth noting that many of my projects never reach Step 7. It’s perfectly normal to loop through Step 4-6 without ever having a tradable result on the Walkforward Data. After five loops through Step 4-6, the Walkforward Data can become extremely contaminated with hindsight, so much so that I can no longer trust the result obtained from the data. When I reach this point I will usually end the project, never officially making it to Step 7 (although I may still run the system on the Unseen Data for kicks).


It’s a Hardnock Life

The hard-to-endure truth is that I may do 30 projects and never find a good result on the Unseen Data. But this only gives me confidence in my process. After all, finding a robust system is truly a rare event for even the smartest system developers; I would have to question any process that allowed me to trade every system I code.

The key to a great development process is that it keeps hindsight out of the final judgment call of whether or not to trade the system in real-time. The better the trader’s process, the more failures the trader will endure. This is just one more tragic condition the trader’s must burden. The brighter side of having a good development process is that the trader knows that when they go to trade a system they have a good chance of making money, and that should trump everything.


FF;_______________

About the Author:

Merlin Jeffries is a professional trader and a moderator at Forex Factory. He can be contacted at support@forexfactory.com.

Nice article Merlin! A few comments and questions if I may:

The system development process you've outlined resembles steps of a software development methodology - particularly the spiral methodology with its iterative and incremental steps. The most popular at the time when I was in IT was called "The Rational Unified Process" for objected oriented design and development.

Creating the test data after design and before coding was a step that most projects I worked on overlooked. That is something it looks like you 've sorted out beforehand which is good. For build data it woud be called "program or unit testing", for parameter searches and optimising you would have system testing. Other software testing would come under load testing, regresssion testing, exception testing and so on. With the latter being handled by a dedicated QA team (if you were lucky to have one).

Anyway onto the questions:

1. Do you take the dataset for the pair and just split them in the proportions indicated? So for example for the EURUSD you have 7 years of data split:

5% Build Data, 40% Test Data, 40% Walkforward Data, and 15% Unseen Data

If you were to vary the %ages, which ones do you normally tweak and why?

Does each pair have it's own test data signature? For example would a low volatility pair like EURGBP perhaps require less walkforward data then a high volatility pair like GBPUSD?

2. Degrees of freedom (DOF) sound like system parameters that you would find in functional or non-functional specification documents in IT projects. With fixed paramteters being similar to core or crticial parameters set perhaps by the system architecture or design.

In other words, some you can tweak the hell out of while others are "set in stone" as it were.

3. Would you normally split the data sequentially so that the the build data comes earliest and the unseen data last? So for example for the EURUSD the build data is the 5% from May 1998 and the 15% unseen data would finish Aug 2005? Or do you "slice & dice" the data at times?

I could keep on asking but I'll start with these three for now...

Congratulations Merlin. Very impressive. When is the book coming out? I got to get a copy.

Dude, I've never looked at it that way! Always have to read outside of a dog.. I mean outside of a box..

I guess that explains why I think it somehow seems very familiar but I can't pinpoint it..
Now I have more reason to pay more attention to the stages of software system development..

sweet..

man i didnt even realize there were replies to this!!! ok, here is my belated reply...

im from a software development background so i have probably been influence by this subconsciously

yes it is important to divvy up your data before you start. its all part of keeping hindsight out of your project. hindsight is the enemy of the system developer, you must fight a methodical battle to have any chance of beating mr. hindsight.


i look at how much data i have, the pretty randomly split it up. the percentages are more important that what goes where. one thing, i do try to save the most recent data for the walk foward data. this is just a habit i guess, i like to see the most recent data be the final test for the system to pass.

Quoting Isotonic

Disliked

2. Degrees of freedom (DOF) sound like system parameters that you would find in functional or non-functional specification documents in IT projects. With fixed paramteters being similar to core or crticial parameters set perhaps by the system architecture or design.

In other words, some you can tweak the hell out of while others are "set in stone" as it were.

Ignored

you got it! if you make everything a DOF, you will spend the rest of your life testing the system. some things have to be locked in stone. the decision on what to lock and what to make a DOF needs to be done at the START of the project. if you find yourself unlocking locked parameters, you know hindsight is winning the battle.

Quoting Isotonic

Disliked

3. Would you normally split the data sequentially so that the the build data comes earliest and the unseen data last? So for example for the EURUSD the build data is the 5% from May 1998 and the 15% unseen data would finish Aug 2005? Or do you "slice & dice" the data at times?

Ignored

i dont slice and dice just because it makes the project more complex. i just chunk it out.

good quesitons here iso, sorry i am so late in answering!!!

better late than never!

Is Merlin the Merlin Jeffries as listed in this article? If so, do you have anymore articles posted elsewhere?

Thanks a lot for replying back!

yes its me!

this is the only article i have written, so i was excited to get it published in Stocks and Commodities!

maybe someday i will write more, but really i am terrible at conveying my thoughts or teaching others. i only wrote this article because i felt like there was a big need for it. no one is out here telling the novice system developers how the real game is played. they have to learn the hard way, just like i did. i thought this article would fill a rare void in the the abundantly available trading liturature.

Hey Merlin,

You are obviously a Legend and I have learned so much here, pity after my losses, Thank you very much. But on a personal note it has been bugging me to find out if you look like the picture in your Avatar?

Hi Merlin. I read this article when you posted it last February, very thought provoking, and have attempted at incorporating the process into my model design.

And just for the record, i never had any problems understanding what you were trying to "convey"...some of it was a bit on the wild side (grin) but i was always able to follow your thought process.

You really should write more Merlin! (besides your zillions of posts here at the factory).

Cheers,
Thom

Hi Merlin,
very cool article! congratulations!

I just have a question about how many trades per year you consider statisticly significant.

For example if your system executes 30 trades in a year, I my opinion it is not very reliable result. However if it executes 2000 trades it should be more statisticaly significant, right?

What is the ratio Number of Trades per Year that you feel is a good sample size? Could you give some examples of the trade numbers and time frames of successful systems you developed?

Thank you,
Robert

{link removed, see rule 2}

Hi, At last i found someone who can do it. I did machine learning for image pattern recognition and bioinformatics. i should say that is the basic for "intelligent" system. the more important is what kind of "parameters" should you input and how "smart" is your system is. The smartness of your system really depends on the developer. Alot of people use system with neural network only to find it useless. I think user must be able to understand the decision mechanism behind the system.

what applies to automated systems can also apply to discretionary systems. however you also need to consider for humans - psychological biases that affect your design and testing

here is a link represents the sample required
http://www.tradejuice.com/system-tra...ing-system.htm

Hey
thanks much!

1 question
do you test your system with other symbols as well
do you try it on stocks???

thanks

good read