How to find and develop your system correctly

Dear All,

I have been lurking around FF for quite sometime and after a while on here you start to see the same pattern repeating itself within the forums.

Generally someone sees a setup on the chart, they see a few instances of it decide that its profitable and hey presto we have a new system and a new hot thread on this forum. Please bear in mind that this post is not criticising anybody for doing this but moreover to help new traders discover the right way of doing things.

Lots of my time over the years has been spent reading thousands of posts to find anything of any kind of usefulness to put towards my quest of being profitable in the markets. Out of all these hours I have found many contributors who have given really important gems of information away. Looking at the way this forum works it is very easy for these gems to be lost or forgotten about and which is why I have decided to republish just a few of these to try to put you guys on the right track. Here follows are some of what I consider to be the most important and correct ways of designing a trading system. If any of you guys feel that you have anything to contribute that would be worthy of this thread please PM me so I can check that it keeps on topic.

I am in no way trying to take any credit for the following posts and if any of these posters are still around here then I would like to thank them all for contributing such wealth to my learning.

The System Behind The System
by Merlin Jeffries, February 2006

When I was a novice system developer I would get excited by every marble-smooth equity curve I stumbled upon. Incredible back test result in hand, I would start trading my systems with real money, only to find they were big loser in real-time. What happened?

What happened is that the systems I was finding, while excellent at predicting the past, were deficient when it came to predicting the future. This variety of trading system was almost impossible for me to distinguish from the future-predicting kind; until, that is, I started religiously following a process for developing my systems.

The Means Justify the End

In the world of system development the means justify the end. One cannot judge a trading system based upon a backtest result (the end); one must instead consider how the backtest result was achieved (the means). The means, or the system behind the system, is a strategically designed process for developing trading systems, and it’s a trader’s main defence against being fooled into putting real money behind a hindsight based system.

The System Behind the System

Here I will present my system, or process, for developing mechanical trading systems. Hopefully it will give the reader a few ideas for developing their own.

First comes the often undervalued administration work. For each system, I create a “project” to organize my efforts. The project gets a name and identification tag. This comes in handy when I want to look back on my past research.

For each project, I move through the same seven steps:

Step 1 – Conceptualise

The first step is to write down the concept and initial rules for the system. For example, if I am researching a simple breakout system, I may write:

concept:
A break of the 20-day high or low signifies the start of a trend.

Rules:
ENTRY LONG: buy when price breaks the highest high of 20 days
ENTRY SHORT: sell when price breaks the lowest low of 20 days
EXIT 1: stoploss
EXIT 2: profit target
EXIT 3: max hold period

Next I will identify my system’s degrees of freedom (DOFs). This will come in handy later. In this example, there are three DOFs to declare:

DOFs:
1. Stop loss size
2. Profit target size
3. Max hold period days

I could also consider the “20 day” breakout point to be a DOF. However, in this example I will assume that 20 days is “hard coded” and cannot be changed during the project. If I may want to optimize or change any certain aspect of the system over the life of the project, I will declare it as a DOF.

Step 2 – Gather Data

Here I gather the market data that will be used to code and test the system. Following the example, I may decide that my total data pool will include the following:

EURUSD / 05.1998 – 08.2005
USDJPY / 07.1999 – 08.2005
EURGBP / 05.1998 – 08.2005
AUSUSD / 05.2000 – 08.2005

I like to divide my total data pool into four categories:

1. Build Data
This is the data I will use to program/code the system. I don’t need much data for this, just enough to ensure that my code is executing trades as designed.

2. Test Data
I use this data to find my system’s most robust DOFs through chart study and optimisation.

3. Walk forward Data
This data will be used to determine if my system can predict the future.

4. Unseen Data
This data offers one final, “honest” test of the system’s ability to predict the future.

How a trader divides their data is more art than science. General guidelines I use are 5% Build Data, 40% Test Data, 40% Walk forward Data, and 15% Unseen Data. I also like to spread a specific symbol (i.e. EURUSD) among at least three data categories.

Step 3 – Code
(data used: Build Data)

Next I write the programming code for the system, and I use the Build Data to check that the code is executing trades as intend.

More times than I’d like to admit there was a flaw in my early code that squandered my entire project. I’ve learned it’s better to take an hour closely inspecting trades than to waste two days researching and testing the wrong system.


Step 4 – Refine
(data used: Test Data)

Using the Test Data, I optimize my DOFs until I find a set that show desirable results. I don’t worry much about the system becoming over-optimized, because if that happens it will not survive the next three steps.

Some software applications, like TradeStation, allow traders to run automated optimizations for the DOFs. I like to lightly use this feature to get a feel for the range of DOFs that will be profitable. If I try 50 variables for each of my DOFs, and 40 yield positive results, this is a good sign that the system is robust. Traders may want to instead manually plug in variables for the DOFs until they stumble across a set that seems to work. I like to take this approach as well.

The main purpose of this step is to study the system on the Test Data and choose the set of DOFs that are the most robust. Continuing the example, suppose I choose the following set of variables:

DOFs:
1. Stop loss size – 30 ticks
2. Profit target size – 40 ticks
3. Max hold days – 22 days

Step 5 – Test
(data used: Test Data)

This step is simple. Using the DOFs from Step 4, I run a backtest on the Test Data, then save the backtest report for reference. I then carefully study the backtest report to determine if results are good enough to be traded in real-time:

> If I would not be willing to trade the system, I loop back to Step 4 to re-refine the system.

> If I am sure I would trade the system in real-time, based on the backtest results, then it’s on to the next step.

Step 6 – Walkforward
(data used: Walkforward Data)

This is where I find out if the system has promise. Without changing the DOFs used in Step 5, I run a backtest on the Walkforward Data:

> If the resulting backtest result is not profitable, or does not have tradable attributes, I will loop back to Step 4 and come up with another set of DOFs (re-refine).

> If the resulting backtest looks tradable, I cross my fingers and move to the final step.


Step 7 – Unseen Data
(data used: Unseen Data)

The final step is to backtest the system on the Unseen Data. I like to call this step “the moment of truth” because it’s quick yet awfully significant.

> If the backtest result is negative the project is over. All the data has been spoiled and any further efforts will be tainted with hindsight. When I reach this point I usually let out a sigh and mumble something about how the life of a quant isn’t all it’s cracked up to be. Unfortunately this is the way most of my projects end.

> If the backtest result is positive, a group of angels fly in my office and circle my monitor singing “hallelujah!” By passing this step, there is a good chance that the system has the ability to predict the future, and it will most likely end up in my portfolio.

The Importance of Unseen Data

“Unseen” data is nothing more than a second set of Walkforward Data. I added this aspect to my process after becoming frustrated with the notion of having only “one shot” at getting the DOFs on the Walkforward Data right before having to abandon the system.

The Unseen Data gives considerable flexibility to my process. I’m able to refine, test, and walk the system forward a number of times without contaminating all of my data. I am always left with one final “honest” test of the system.

It’s worth noting that many of my projects never reach Step 7. It’s perfectly normal to loop through Step 4-6 without ever having a tradable result on the Walkforward Data. After five loops through Step 4-6, the Walkforward Data can become extremely contaminated with hindsight, so much so that I can no longer trust the result obtained from the data. When I reach this point I will usually end the project, never officially making it to Step 7 (although I may still run the system on the Unseen Data for kicks).


It’s a Hardnock Life

The hard-to-endure truth is that I may do 30 projects and never find a good result on the Unseen Data. But this only gives me confidence in my process. After all, finding a robust system is truly a rare event for even the smartest system developers; I would have to question any process that allowed me to trade every system I code.

The key to a great development process is that it keeps hindsight out of the final judgment call of whether or not to trade the system in real-time. The better the trader’s process, the more failures the trader will endure. This is just one more tragic condition the trader’s must burden. The brighter side of having a good development process is that the trader knows that when they go to trade a system they have a good chance of making money, and that should trump everything.

http://www.forexfactory.com/showthre...20#post2212020

Topherhk88

I rarely post on the FF, but I have seen a lot of these posts recently - questioning (or, in some cases, audaciously claiming) that it is impossible to trade successfully in the long term, or that any long-term prospect of success is weighted far more towards luck than a sound trading practice. Let me address this as thoroughly as possible, hopefully in a somewhat logical order.

The time-frames that are you trading on (1m, 5m, etc) will undoubtedly be exposed to a significant amount of noise - news releases or other liquidity spikes will be much more likely to stop you out if you hold tight-stopped, short-term positions. I am NOT saying that is it impossible to trade smaller time frames, but just for someone that is worried (and seemingly overwhelmed) by the numerous factors that effect the market, I think larger timeframes would be a better bet.

I will now run through the process that I personally use to construct trading systems. Yes, I only ever trade a system with a strong statistical basis, since trading anything other than that would be impossible for me to put any long-standing faith in, and subsequently impossible to mentally stick with in times of greater drawdown. Even an EA that I hadn't PERSONALLY tested and judged to be statistically significant in its expectation would be very difficult to stick with during losing streaks. But I digress...

The first step of the process is to identify variables that you believe create a slight statistical edge - where given 1000s of incidents of that variable occurring, you would expect the chance of "something" occurring slightly more 50/50 (the greater the ratio, the greater the edge of that particular variable). Once you have that variable, you look for other variables that can be used in conjunction to further increase your statistical outcome through filtering out some of the initial trades. Eventually, you have an entry condition that could be expressed as an if statement: if x=3 and y=5 and z=10 then you place a buy stop/sell stop at a certain predefined point (the numbers and variables used are completely arbitrary, so don't pm me asking how I defined z...). On that note, to truly create a system that you have any statistical confidence in, everything - every single possible action that you take entering, existing or modifying a position - has to be 100% defined.

So at this point, what do you have? You essentially only have an entry condition where you are confident that when x,y and z align in a certain way the outcome over many instances should be within some statistical expectation. Does this mean you have a winning system and can conquer the world? If only it were so easy...

At this point - which honestly is the point that many traders stop at, if they even define things so precisely at all - you simply know when to enter the market. That’s it. You have no concept of money management, stop placement (fixed pip or varying based on recent volatility), or exit conditions. How do you go about defining those? In much the same way as you defined the entry. By looking at 1000s of occurrences where your system produced an entry, and then meticulously testing variations of all of the above. Sounds pretty tedious right? Well, it is, but it is double for those with patience and reasonable logical faculties.

Let me give you an example. (Disclaimer: this is by no means an actual system - I am literally making this up right now with no basis in anything. I will sell it to you though…) Say you have defined your timeframe as the daily charts. The entry condition that you have defined is that when the RSI 96 is above 50 AND when you have an inside bar AND when the price breaks the inside bar by at least 15 pips (through the use of a buy). Your stop lose is based on some consistent measure of recent volatility (ATR, etc), and you are using fixed fractional position sizing for your MM. Your exit is defined as whenever the move reaches 1.5R in profit. You also mechanically cut loses depending on other x,y,z factors that you observe at the end of a given period (at the end of the day, in this case). All trading actions are taken at the end of the period, and only at the end of the period, to remain 100% statistically consistent.

So, now you have a system that is completely defined, but does it work? The only way to answer that (aside from live trading, of course), is to first backtest it over 1000s of occurrences. Another point is that in a statistically based system, you have to take EVERY SINGLE INCIDENCE of a trade that aligns with your variables, making it more difficult to do on lower time frames without an EA unless you want to watch the monitor at the end of every period like a crazy insomniac. Backtesting over a large sample size with 100% defined trading rules will allow you to see how your system would have performed over a certain time period.

To make it clear, when I say backtest, I don't mean what I think many in this forum do: looking at their chart over a couple of years (or months - or even weeks...) and gleefully counting in their head "winner, winner, winner, winner, loser, winner, winner, loser, winner", while half-focusing on the finishings on next year's 200 foot mega yacht. The testing should be nothing less than scientific. You are a market scientist, trying to create a market theory, so every stage of the process needs to be defined and testable. You will know that you are doing this right when you aren't thinking of all the money you are going to make but instead simply recording, in thorough detail, all of the data that you are going to need to properly formulate a compelling theory; as a side benefit, you will also limit testing bias, since you will no longer TRYING to prove anything, but simply testing to see if it could be a tenable theory.

Lets say you do this, and you find that your system over 5000 consecutive trades would have produced a win rate of 55% and a average win/average lose ratio, as a percentage of R, of 1.3/1. As a brief aside, for those that haven't done this before, constructing a system is always a balancing of two often-competing factors: win rate and ave win/ave lose ratio. Generally they are inversely correlated to a degree. A trend following system, where you typically have high R multiple winners will typically have a high ave win/ave lose ratio, but a lower win rate (often below 50%). A scalping system, on the other hand, where you are quicker to take profits, will typically have a higher win rate, but a lower ave win/ave lose ratio, since, well, you are quicker to take profits and therefore don't capitalize on those large R moves. It is the balance of these two factors that creates your expectancy - and, if you have developed a good edge, hopefully a positive one.

So are we done yet? Haha - not quite. Those two factors are only a small measure of a system's success. Ultimately you should focus on the biggest factor that will determine your system's biggest success: its annual return/worst annual drawdown. And the ratio itself is only part of the battle. Sure, you can have a 50/1 ratio, but if you have a max drawdown of 60% than the ratio is somewhat misleading, even with that 3000% return. You can of course decrease the drawdown by decreasing your position risk, but that will also exponentially decrease the ratio due to less compounding on the upside. Ultimately, you are looking for a decent ratio - but more importantly, a reasonable drawdown that is psychologically palatable for you, and which your system can reasonably recover from.

After that you have that, for an extra measure of confidence, you should perform a monte carlo test on your 1000s of results. You want to make sure that 1,000,000s of permutations of the results are consistent, and that the results that you obtained were not just the product of 'lucky' historical sequencing.

Finally, once you have all of that, then you are finally ready to...forward test. How glamorous, huh? So, you forward test the system over a couple of months, and if it clearly performs within statistical expectation, then, and only then, are you ready to go live.

Some other notes on system development:

1. Make sure that your sample size is suitably large. Constructing a system over 50-100 or so trades is complete foolishness, and 'curve-fitting', in the negative sense of the word (with the arduous strategy above being in the more positive sense). You are a scientist, and you need enough results to formulate a plausible market theory.

2. Make sure that EVERYTHING is measurable and defined. If not, then your system will have elements of inconsistency. Wishful thinking prays on these little pockets of testing-inconsistency, and can significantly distort your statistical expectation if you allow it to leak into a system that is crafted with anything less than 100% precision.

3. Make sure that your testing period includes diverse markets, and preferably functions over numerous market conditions and many years. Even better if it broadly functions across different currencies, but this is not essential. Essentially, you want to limit the myopic curve-fitting experience, where your system is highly calibrated to a rigid set of market parameters. If for instance, your system works great in 2008, make sure that is works well in 200x, since 2008 is not necessarily broadly representative.

4. Always remember that there will be testing inaccuracies. Be it that certain positions were different/entered/not entered based on spread changes, charting errors (hopefully not if you choose a good feed) and the tendency for human positive bias (mitigated by 100% rigid testing parameters). Generally, a shorter time-frame system will be affected more my the spread change issue than a longer-term one.

Whew! So there you have it: a system that you can put a fair degree of faith in moving forward. And that, ladies and gentlemen, is about all you can ever hope for in trading. Why? Well, to finally address the initial question in the thread, because you can never, ever, ever get away from the fact that the market is uncertain, and that at any moment in the future it can behave completely differently from any way that it has behaved before. So is that luck then? Only if you define luck as the market consistently behaving based on robust expectation. At that point it just becomes a question of semantics.

One thing that certainly isn't luck, though, are the long-term track records of the successful few; that type of legacy is the product of systematically superimposing robust constraints on the markets. But what about when the systems stop working - does that not then negate the proceeding accomplishments and make it all a lucky ride? Not in the slightest. For one, the more robust and broadly functioning the system, the less likelihood that it will stop working - which, of course, is a design feature that has nothing to do with luck.

Sure, Chris, but even THOSE can stop working too - isn't that luck? Well, yes, in that very rigid sense I suppose that you could define this inextricable luck component as the "duration of time that a particular robust system functions before it no longer functions". I will concede that limited point, but again, I more than accept that as inevitable uncertainty, that you will necessary have in any endeavor that you peruse. You could start an orange juice company that does great year after year for 20 years until the FDA comes out and says that orange juice knocks 10 years off the average life, and the market completely shrivels up. I never look at any system I develop as a sure thing, or something where I can simply press play, go into a 50-year coma, and wake up the richest man in the world. Again, this is where long-term success should not be confused with luck. There are always exogenous uncontrollables (black swans, as it were), but the part that isn't lucky is having a feedback mechanism to recognize when one has happened, and having a plan in place for when it does.

To bring it back to the example, say the system that you designed had a maximum drawdown, over a 15-year period, of 20%. With monte carlo analysis, over millions of permutations based on 1000s of initial observations, you conclude that you have a 95% confidence of never dropping below a 25% drawdown. Does this mean that it will never happen - is this the certainty that you were looking for? No way. It very well may happen; in fact, I would postulate that given enough time it will happen, since, after all, it is only a 95% confidence. Even a 100% confidence would still be based on only x thousands of occurrences, never escaping the uncertainty of what may come.

What you do know, however, is that if you drop below 25% drawdown then you are starting to venture outside of statistical expectation. So you make a fail safe. You write it on your wall and you stick to it no matter what. If the system ever breaches x% drawdown, then I will stop trading until I am again assured that it is operating within expectation, and that the drawdown was merely a standard deviation event; if, however, it doesn't resume its normal functioning - if it doesn't recover from the drawdown based on a reasonable logarithmic expectation - then it is time to go back to the drawing board, figure out what changed, modify your variables accordingly, and strive to develop a version of your system that incorporates all of your previous 1000s of occurrences AND the new ones that had previously thrown it off. Curve-fitting? Sure, but again over many, many occurrences so that it is broadly functioning. The more robust your model in the first place, the fewer changes that you will likely have to make, and it will be easier to integrate them without a complete overhaul.

Well, that’s about it from me for today. I hope this was helpful. I obviously know that it can appear overwhelming, but this is simply the reality that has allowed me to be consistently successful. Stop looking for certainty, stop worrying about luck and designing something that will prove to be infinitely viable. Become a market scientist, create a viable market theory, play it out until it stops working (hopefully, if it is exceedingly robust, you will be dead long before that happens), and then have a feedback mechanism ready so that you can roll with the punches, make your adjustments, and continue moving forward. And specifically for the initial poster, don’t worry so much about all the millions of factors that move the market. Stop trying to know and control everything - you can’t and you don’t need to.

Best,

Chris

This one is from a collection of notes taken from Acrarys posts over at EliteTrader forum.


1) Be DUM



"For every system I develop I use DUM.



D - Define



All systems are based on finding and pulling a fundamental truth about the market. Define what fundamental truth you'll be going after. Ex. All markets have a tendency to trend beyond random. Now you've got the definition that most technical-based hedge funds are derived from.



U - Understand



Determine the conditions under which the defined truth tends to occur. In the case of a trend tendency it could be when does the trend tendency begin beyond random? This will lead you to how do I measure a trend? Since trends can occur randomly, how do I determine if a trend is beyond a confidence level of randomness? Does the trending tendency beyond random exhibit the same degree of persistence beyond one year? two years? 5 years? If not, is there some point at which the persistence beyond random occurs every year? If so, does it also persist at the same frequency for 5, 10, 50 different markets? If so, you've discovered a fundamental truth and you now understand what you need to know about the behavior.



M - Mine



Once you understand the conditions under which the behavior occurs, you write the code necessary to map the understanding of the behavior. Is the code going to be all inclusive of many markets? or try to just go after the best of the best? Once mapped it's a mechanical process to determine how well it maps against the behavior. After you're satisfied you've developed a satisfactory method for mining the behavior, you can do an edge test to see if it happens beyond random. If not, use Monte Carlo sims to determine confidence levels for trading the method. Determine at what confidence level you'll stop trading. Examine the drawdown versus the profit. Is it worth risking any money on this? If so, allocate money using a money management scheme.



After you're done with this, you'll have your first system. Next, develop a complimentary system (non-correlated). Go through the same process for say a range bound system. Once you've gone through the mining stage, use the correlation test to weight the two systems. Apply the weights to the money management scheme and move on to your third system."





2) Target a behavior (Define)



"I don't look for a market for a system. Each system I've developed is targeting some behavior. If that behavior is present in multiple markets, then I could test it to see if my system captures the behavior better than random. If so, I'd just trade it on that market and check to make sure the behavior was persistent. For instance I have a volatility breakout model that I've used successfully in the SP market. I tested it against the DAX market and found the edge (ability to capture profits at better than random), was better in the DAX than the SP. I've been trading the DAX with it since then and it's done very well. Only thing I don't like is getting up in the middle of the night to trade.



Every model I've worked on has gone through the same process. Look at the behavior's present in a market, characterize them by creating a rule and checking the fit until all behavior's are noted. Then start looking to see if there is a component to the behavior that is non-random. If so, develop a system to mine it and create a way to monitor the behavior to ensure it's persistent over time. For example, one of the behavior's widely known is the trend day in the SP market. It can be identified just by visually inspecting a chart. I characterized it as a low/high within 10% of the low/high of the day and the close within 20% of the high/low of the day. With the definition I can see how many of these days have persisted over the years (averages about 25 days per-year). Then I can see if there is a way to identify these days in advance (realizing I'm going to also be capturing some false days as well)."



3) Basic data mining process (Mine)



"First find out what is going on in the market you want to trade in the timeframe you plan on holding a position. If you want to daytrade with one trade per-day then find out all the different ways the day has played out in the past. ex. trend day, two-way day, reversal day, etc.



Once you've done this you should have an idea of which type of day is most common and which is most profitable. Then define something which could be of value to trade one of the market types. An example might be in a reversal day to find out how often the market makes a low of the day in the first 15 min. of the session. If it happens often enough to be of interest then you go on to the next step.



Take every period for which the target is found and create a table of outputs with 1 for the target and 0 for non-targets. Then pre-process all the inputs into the target and convert them to binary inputs. (A common mistake is to take open, high,low, and close data -- analog and assume you can find relationships with the target). For ex. yesterday close > day before yesterday close. If found mark the input as a 1 if not present mark it as a 0. Do this for as many identifies as you can. This may present a hundred or more binary inputs leading to the target for each day of the data.



Then you'd pass the data into a backprop neural net and have it train on the data. (you'll need to set aside some data for out of sample testing). Once it's trained to hit at least 90% correctly test the NN on the out-of-sample data. If you hit at least 85% correctly then you can do one of two things. If you're a discretionary trader, setup the NN and preprocess the inputs every day and use the net to predict whether tomorrow has the target (in this example the low of the day is within 15 min. of the start of the session). If so use it to trade to the upside as long the net remains 85% correct. If you're a systems trader then go back to the net and look at the weights of the net to see which of the binary inputs were most important in hitting the target. Use the inputs to create a backtestable system based on the patterns. A system might be when xyz pattern exists then buy next bar above the lowest bar as long as the time is within the first 15 min. of the day. Set the stop to one tick below the low.

If the system tests profitable enough to be of interest then move on to the next step.



Next, take the trades and test them against random trades pulled from the same year (the edge test). Rank the trades versus random for each year of the backtest. If the trades score consistently above the 70th percentile then you can guess you've found a edge-based system. If not, then you have to assume you've found a temporal characteristic in the data that can be exploited for some period of time.



If it's edge based then all you need to do is adjust the trades for market volatility and apply a money management strategy. Check the trades on a periodic basis to ensure the edge continues and plan what to do with your next million. If it's not edge based you can still trade it but you need to setup a objective bailout method such as running a monte carlo sim and determining the bailout point to be say the 95% level of the predicted max drawdown point. Your trading would be more defensive using a non-edge based method as well. Maybe you'd split the trade size in half and have a 15 min. or 10% of daily range as a filter to adding the second position (letting the position prove itself) as long as the volatility was large enough to justify the scaled entry."



3) The Edge Test - What is it?



"The "Edge Test" is a concept. The concept is to separate luck from skill in determining the backtested results. How you do define the measurement is up to you. If I had a Cray supercomputer and had 50 trades to test for a year I could probably rank all the different combinations of 50 trades within the 250+ trading days. Since I don't have that kind of computing I have to sample the trades from the pool of potential trades. If I had the desire I'd build a minute by minute database and rank each trade against the exact entry and exit time for every day of the year. Then come up with a weighting scheme for the proportion of longs versus shorts. I'm sure there's many other ways to do the same tests. In the end it all comes down to "does the random selection of trades adequately represent the luck component that I'm measuring against?". If so, then it's worth keeping. If not, then try other ways of accomplishing the same goal.



When I've used the test I've disabled the stops and exit with profit strategies so that all I'm testing are my entry and exit at end of day versus random "luck" entry at open and close at end of day. The only purpose of the test is to determine if my entries are a result of luck or skill."



4) How to do the Edge Test



"To do the edge test you use a single method at a time.

First you backtest on the data you're using to develop the method. Then, when you're satisfied with the overall results you separate the trades by long and short by year.



It'll look something like this:



1996



Long +3.00 hold 1 day

Short -2.00 hold 2 days

Long -1.00 hold 2 days

Short +4.00 hold 1 day

etc.



Then you process the year of 1996 and pull out random individual trades with the same length of hold (being careful to avoid reuse of any one day).



Ex.

Long -2.00 hold 1 day

Short +1.00 hold 2 days

Long -2.00 hold 2 days

Short +1.00 hold 1 day

etc.



When you get done you total up results of the long and short trades for the random pass for the entire period.



ex.

Long -4.00

Short +2.00



You do this random pass thousands of times (Monte Carlo) and rank each each of the passes so that you have a distribution from 1% - 99% for both longs and shorts for each year of the tests.



Ex.



Long 1% -16.00

etc.

Long 99% +21.00



Then you compare the total you have for your tested trades versus the distribution to rank where your trades are as compared to the random trades. (do this for both longs and shorts for each year). If both longs and shorts rank 70% or better (20%+ better than random) then you might be looking at a edge.



Do the same test with out of sample data and shorten the time period to 3 months (so that you can view multiple forward time periods). If the numbers continue to be 70% or better on both longs and shorts then you probably are trading with a edge.

You do this test every 3 months after you start trading it to make sure the edge is not deteriorating. If it drops below 70% then stop trading it."



"I wrote a couple of programs. The first has a test trading system and writes a trade file with the entry date, length of hold in days and profit or loss in pts. with slippage included. The core code has to be changed for each idea that tested well in TS.



For the second program, I enter the year(s) of the test, the market being tested, and whether it is a test for longs or shorts.

I also have to enter the total profits generated from the original system (in pts.) for either longs or shorts. It then reads the trade file and selects the trades that are for the test period. It also reads a back-adjusted continuous file for the symbol and loads the data for the test year(s) in memory. Then it just does 5000 passes of random entries within the test period on the continuous data with a hold equal to each of the trade lengths.

A array is marked to signal that the period was already used to prevent trade overlap during each pass. After each pass through the trade list, it saves the net profit/loss in a array. After the 5000 tests are done, the array is sorted from lowest to highest profits. Then it goes down the list to find out where the profits from the system fit within the tests. Ex. If the system test made 240 pts. profit in the SP market, it would scan through the 5000 trade runs to find out how many it beat. If it beat 4000 of the trade runs, then the system achieved a result of 80% versus a expected random result of near 50%. I also print out the 50%, 90%, 95%, and 99% ranks to get an idea of the possibilities in the market. If the longs and shorts both test above 70% for 3-5 separate years, I start trading it. It doesn't mean it won't have drawdowns, it just means I'm trading a edge that's likely to have better results than random.



For daytrading, I only test a model to do the entries and exit at end of day on close (no stops or profit targets). This way, I've been able to find ideas that have an edge versus just mapping to the character of the market."



"When I've used the test I've disabled the stops and exit with profit strategies so that all I'm testing are my entry and exit at end of day versus random "luck" entry at open and close at end of day. The only purpose of the test is to determine if my entries are a result of luck or skill."





5) How to interpret the Edge Test



"All of the edge based models are developed based on a perceived structural anomaly. For the edge criteria if the longs or shorts drop below 70% of the random trades then it's no longer traded... I also suspend trading any model if it hits the 95% modeled drawdown or the 5% return level from the Monte-Carlo sims."

Cool thread.

Nothing beats hard work and discipline in the forex world.

No easy way.

Excellent.

Can you resume please?
KISSes

For what it's worth, I created a blog series on trading system creation from the ground up too. There's plenty of information there to dig through.

How to Create Your Own Trading System

Thanks TradeDream will take a look!

No worries mate. The notes you've posted in this thread are excellent, many developing traders could really be helped by this information.

Excellent thread, keep it up

Brilliant thread,

I appreciate when somebody like you shows people what is it about. Thumbs up