Lessons Learned during System Development | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Your and My Lessons Learned Losing and Gaining in Trading 5 replies

Lessons Learned Thread 727 replies

Cable Hypercross System Development 8 replies

Rookie Lessons Learned 5 replies

System Development Secrets 5 replies

Attachments: Lessons Learned during System Development
Exit Attachments

Lessons Learned during System Development

  • #1
  • First Post: Oct 4, 2005 12:52pm Oct 4, 2005 12:52pm
Hi,

I've been reading through the forum and somehow I'm in the process of developing my own system.. But not really doing actual building but more on the planning phase..

Anyways, just like the post title says, I want to know the lessons you learned during the development of your own system. What triggered this curiosity of mine was in one of merlin's posts on one of the threads:
http://www.forexfactory.com/forexfor...eferrerid=2899

Anyway, the thread tackles about the steps in becoming a (good?) trader.. And one merlin's step there was that he indicated that he realized that he was building systems the wrong way...

So basically, I want to know what these mistakes or wrong ways of developing a system so that I may avoid them during my development..

So hopefully, merlin will answer my question.. For others, please also post the lessons you learned during developing your own system.. So that persons who are planning of developing their own sys will be able to avoid such mistakes..

Again hoping for your kind replies..
Athan
  • #2
  • Oct 4, 2005 2:22pm Oct 4, 2005 2:22pm
Hi ,



I am fairly new in the Forex Trading field (I’ve traded stocks before on the TSX-Toronto stock exchange), and completely new to this forum.

I think though that trading system as such is something personal, tailor suited by each individual. Just like in life each individual is unique so is in trading.
“Know thyself” therefore applies here well.


For instance, I know that I am more of the visual person. My current career as graphic and web designer stems from years of study and experience in visual arts. I am also very found of dialectics, philosophy and psychology. Therefore the system I’m building and method I’m using for trading is visually-analytical. That’s one of the reasons I end up signing up with CMS-forex with their VT (Visual Trader) platform. Chart is basically all I need to trade. No formulas, no complex mathematical or geometrical calculations, virtually no indicators etc. I use support, resistance and general positioning of the chart as my guides. Then I study individual candlesticks (usually daily chart) to see where, when and to what degree the chart will most probably move.

I’m also using the current fundamentals (news, reports, commentaries, analysis) as a support for my observations and buying or selling moves.

I am still learning though, but do trade live with thoughts of this craft becoming my new career in the near future.



Cheers,

V.
  • #3
  • Oct 4, 2005 6:53pm Oct 4, 2005 6:53pm
I have been building systems for quite some time .. recently gone live. It has been quite a experience. I am maintaining a journal with screnshots here ..

http://www.elitetrader.com/vb/showth...threadid=54170


http://img222.imageshack.us/img222/6276/gbpchf6oh.jpg

This is a forward test of a BLACK box.

bottom line - it can be done.

good luck,
auto
  • #4
  • Oct 4, 2005 11:05pm Oct 4, 2005 11:05pm
Quoting auto
I have been building systems for quite some time .. recently gone live. It has been quite a experience. I am maintaining a journal with screnshots here .....
This is a forward test of a BLACK box.
bottom line - it can be done.
good luck,
auto
Auto, it's great that you've developed a system for yourself, but please don't end up spamming many thread's like you did on MT. Thats earned you a 5 day ban over there....
  • #5
  • Oct 5, 2005 12:38am Oct 5, 2005 12:38am
I think the biggest mistake beginners make when they first start out (and I did this) is to focus on indicators. I all honesty the focus should be on market behaviour after all its the market you are trading not the indicator. The indicator is a tool.

If you want to be a mechanic you study engines. What they do and how they do it. Once you have that knowledge down pat you can then ask yourself which tools would be helpful.

Same applies to the market. If you want to be a trader study market behaviour. Find out about its cycles and the whats and whys. After you know this only then should you pick the tools to than may help you trade it. You might be very surprised at how little you actually need.

Start with the Dow Theory its a great foundation.

Steve
I was here, here I was. Was I here? Yes I was!
  • #6
  • Oct 7, 2005 3:49am Oct 7, 2005 3:49am
Quoting SterlingTDR
I think the biggest mistake beginners make when they first start out (and I did this) is to focus on indicators. I all honesty the focus should be on market behaviour after all its the market you are trading not the indicator. The indicator is a tool.

If you want to be a mechanic you study engines. What they do and how they do it. Once you have that knowledge down pat you can then ask yourself which tools would be helpful.

Same applies to the market. If you want to be a trader study market behaviour. Find out about its cycles and the whats and whys. After you know this only then should you pick the tools to than may help you trade it. You might be very surprised at how little you actually need.

Start with the Dow Theory its a great foundation.

Steve
I quite agree. I was fortunate enough to skip over the indicator constipation stage, and so have never been enamored with them. I always laugh after someone posts the results of a good trade or two, and the very first question they get asked is "What indicator do you use?" To me, this screams "beginner", especially since it is frequently in the singular. That makes it seem the person is looking for the holy grail indicator. Don't get me wrong. There is nothing wrong with being a beginner. We all had to start there. Besides, if it wasn't for beginners, where would I get my money from?

(BTW, Steve, what indicator do you use? )

Dial
sxaxlxvxaxtxixoxnxbxyxgxrxaxcxexdxoxtxoxrxgx
  • #7
  • Oct 7, 2005 4:40am Oct 7, 2005 4:40am
Quoting diallist
(BTW, Steve, what indicator do you use? )

Dial
Actually Dial, The indicator I found to be the most reliable, on its own, is a coin flip. 50% accuracy, not too shabby.

I've noticed in some of the other forums that the threads which are most popular, in terms of views, are the ones in which somebody has bundled a combination of indicators together and offers it up to everyone. "do this and that when this indicator says this or that, but only if such and such is reading +/-30, between 7am and 9am."

Instinctively people will take the path of least resistance. When it comes to trading this path (usually indicator based) will typically yield good results in the beginning until it mysteriously stops "working". Everyone starts scratching their heads wondering why. More often than not if one checks out the PRICE chart, one can see that the market has entered another cycle. But most are too busy looking at their indicators (tools) to notice what really unravels the mystery ...the price.
I say this as someone who has walked this same path in the past. It can be terribly frustrating.

Learning price behaviour takes time but its worth it don't you think?

Now, what's for lunch?

Regards
Steve
I was here, here I was. Was I here? Yes I was!
  • #8
  • Oct 7, 2005 5:30am Oct 7, 2005 5:30am
Hi Steve, Diallist,

Though it's humiliating to admit it.. What you said actually hit the spot.. I mean, right now, I am studying on different indicators trying to know which one to use.. I guess that's why I never get satisfied why I should use a certian indicator.. Maybe what's more important is understanding why such an indicator works for a certain market belief (?)..

I guess what you're saying is to somehow come up with my own beliefs first, then use indicators as tools for those beliefs?

Anyway, I just remembered one of the indicators used by merlin is price. And to be able to use price, you have to know when a cetain commodity is over-priced or under-priced right? Which is discretionary in nature.. I guess I just have to somehow get a feel of the market, but as to when, I really don't know..

Tell me if I'm wrong or what I'm saying doesn'T make sense.. Coz right now, I just don't think I'm making any progress with my study with trading. Sorry if I kind of sound desperate.. But actually I'm really not, maybe just frustrated...

Currently, I'm just self-studying trading with no mentor or whatsoever.. just books, the internet, and of course, the most helpful community - FF..

Anyway, maybe what I'm blabbing here are just whines..

Anyway again, thanks for your replies.. if you have any advice, I would really appreciate it..

Athan
  • #9
  • Oct 7, 2005 5:33am Oct 7, 2005 5:33am
Haha... Posted my reply too late.. But anyway, it's good to know that I had the correct idea that it's more effective to look at price.. and not merely on indicators...

Thanks for confirming this Steve.
  • #10
  • Edited 10:57pm Oct 7, 2005 10:18pm | Edited 10:57pm
Quoting eastmaels
if you have any advice, I would really appreciate it..

Athan
I know what you're feeling eastmaels. Up until about four months ago I was in the same place you appear to be now. But finally, I've started to make some progress. Here's what I'm doing differently now from what I was doing four months ago. This fits my personality, it may not fit you, but you won't know until you try it on.

(1) Read James16's Chart thread and learn to take precision entries based on price action. But don't use price action alone. Make sure it is co-confirmed by fib retracements, pivot levels, support and resistance and trend lines. By making the most precise entries you can, you can then minimize your risk by using tighter technical stops.

(2) Don't trade just to be trading. Don't jump on the market just because it is moving. Have a clearly defined reason for entering the market. Practice having the patience to take only the "A" trades. I used to take 60 or 70 trades a month without making much headway. If I did get ahead, it wasn't long before I was drawn down back to my starting point. I was taking "B" trades and "C" trades. Big mistake. Now, I'm taking only "A" trades about 10 to 15 times a month and am getting more profit with smaller drawdowns.

(3) Move your stop to breakeven as soon as possible. This takes practice to know when the market has moved enough that it is unlikey to hit your stop, but if it does you've not lost anything and you can always get back into the trade on the next retracement.

(4) I used to trade 60, 30 and 15 minute charts. Some people can do this quite well, but I can't. Ever since I trade only 4 hour charts or higher (using smaller timeframes for entry) I have been more profitable. It is also a lot less stressful and time consuming as 4 hour charts allow you to be away from your computer for extended periods. I look for trades on the 4 hour chart based on price action along with one or more confirmations listed above. I tend to only take trades in the direction of the trend as seen on the weekly chart and I time my entries on the 15 min chart.

(5) Watch for price action to build on lower timeframes first and cascade it's way up the higher timeframes. For example, if you have a simple moving average and you are waiting for price to cross it on the four hour chart, begin by monitoring the 15, 30 & 60 min charts with the same sma. You'll usually (not always) see the crossover occur on the 15 first followed by the 30 then the 60 and finally the four hour. Kinda works like an early warning system. This will take practice to use effectively.

(6) Use consistent position sizing for every trade. I found out the hard way that it was a big mistake to risk 4% on a trade I thought was going to be a big winner and 1% on those trades I thought were only going to move a little. Murphy's law kicks in here and more often than not the 4% trades turned out to be the losers. So, even though I might end the month up in pips, I was down in equity because my losses exceeded my wins. I've changed two things. I now risk the same percentage of my capital on every trade whether I think it will move a lot or move a little. Second, I quit trying to guess which trades were going to move a lot or a little, because the truth is, you simply do not know and cannot know in advance. The best you can do is make an experienced based judgement on which trades have the highest probability. In other words, take only the "A" trades, and risk the same percent on each of them.

(7) Base the percent risk you want to take on your trades on the resultant true leverage. Since the size of your stops will vary, the range of the true leverage will vary as well. Four months ago my true leverage ranged from 20:1 to as high as 88:1. Insanity! Now, all my trades are 10:1 true leverage or less. Usually they are around 5:1 to 8:1. However, until you gain more skill and more consistency in your equity growth, I'd strongly recommend you keep the true leverage to 2:1 or less. When I decided to get my leverage under control, I started with 0.5:1 and moved up from there. To do this effectively though, you really need to have an account that lets you trade microlots (1K) or smaller like FX Solutions whom I use, or Oanda.


Hope this helps.

Dial
sxaxlxvxaxtxixoxnxbxyxgxrxaxcxexdxoxtxoxrxgx
  • #11
  • Oct 7, 2005 11:10pm Oct 7, 2005 11:10pm
Quoting SterlingTDR
Actually Dial, The indicator I found to be the most reliable, on its own, is a coin flip. 50% accuracy, not too shabby.

I've noticed in some of the other forums that the threads which are most popular, in terms of views, are the ones in which somebody has bundled a combination of indicators together and offers it up to everyone. "do this and that when this indicator says this or that, but only if such and such is reading +/-30, between 7am and 9am."

Instinctively people will take the path of least resistance. When it comes to trading this path (usually indicator based) will typically yield good results in the beginning until it mysteriously stops "working". Everyone starts scratching their heads wondering why. More often than not if one checks out the PRICE chart, one can see that the market has entered another cycle. But most are too busy looking at their indicators (tools) to notice what really unravels the mystery ...the price.
I say this as someone who has walked this same path in the past. It can be terribly frustrating.

Learning price behaviour takes time but its worth it don't you think?

Now, what's for lunch?

Regards
Steve
Yes! The coin flip indicator is great! I did find however, that a silver coin works best in trends, while copper works best in ranges. I also found it is much more effective if you can use a coin in the currency you are trading. I have a nice copper half-penny that I use for GBP/USD ranges.

The rest of your post is excellent and full of wisdom. You make some excellent points!

Whaddaya mean "what's for lunch?" ? I thought you were buying!
sxaxlxvxaxtxixoxnxbxyxgxrxaxcxexdxoxtxoxrxgx
  • #12
  • Oct 8, 2005 12:01am Oct 8, 2005 12:01am
Well, I do envy you for finding the system that's right for you.. For me, I'm still on that stage on finding a system that's right for me..

You also have some good trading rules, will adopt most of it, if not all..
Right now I'm making my own set of trading rules.. I've already came up with 4.. plus your advices.. will may be able to come up with 11... good math ey?

Thanks also for all those that replied..

P.S.
I'm sorry to butt in with your coin-flip indicator..
But I recently discovered that based on the value of the coin (a quarter vs. a penny) also somehow yields different results.. I really haven't forward tested this enough yet.. so use at your own risk..
  • #13
  • Edited Nov 22, 2005 4:57pm Oct 20, 2005 7:12pm | Edited Nov 22, 2005 4:57pm
man, the lessons learned are so numerous...

here are a few that come to mind...

1. Avoid Multicollinearity
this is where you use two (or more) indicators that tell you the same thing. for a complete definition, see this.

systems that have multiconllinearity are mostly over optimized, and the backtest results is nothing more than coincidence.

2. Follow a process for developing your system
i recently wrote an article for Stocks and Commodities magazine about "The System Behind the System" that addresses this very issue. if you just bang on your keyboard and hope to find a great backtest report, you are falling into a complex trap of hindsight.

3. Dont build systems that you cant trade exactly
what good is a great backtest report if you cant trade the system exactly as designed? some systems expect you to be on call 24 hours a day, ready to follow the signal in a split second. is this realistic? if the answer is no, you shouldnt attempt to trade the system because your human error will most definitely eat up your profits. when you make systems that uses pre-set orders and only requires you to be at your computer once or twice a day, its realistic to think that you can trade it percisely.
Relax and be happy.
  • #14
  • Oct 20, 2005 8:02pm Oct 20, 2005 8:02pm
Quoting merlin
man, the lessons learned are so numerous...

here are a few that come to mind...

1. Avoid Multicollinearity
this is where you use two (or more) indicators that tell you the same thing. for a complete definition, see this.

systems that have multiconllinearity are mostly over optimized, and the backtest results is nothing more than coincidence.

2. Follow a process for developing your system
i recently wrote an article for Stocks and Commodities magazine about "The System Behind the System" (see attached). if you just bang on your keyboard and hope to find a great backtest report, you are falling into a complex trap of hindsight.

3. Dont build systems that you cant trade exactly
what good is a great backtest report if you cant trade the system exactly as designed? some systems expect you to be on call 24 hours a day, ready to follow the signal in a split second. is this realistic? if the answer is no, you shouldnt attempt to trade the system because your human error will most definitely eat up your profits. when you make systems that uses pre-set orders and only requires you to be at your computer once or twice a day, its realistic to think that you can trade it percisely.
Merlin, great article and a very organized one. However I still can't understand 1 thing. The unseen data you mentioned?? Do you mean forward testing, i.e. putting the system into action for some time to see how will it perform and will the future performance be the same or very like the historic one??

I don't think this is what you mean by the "Unseen data", because you were using the term "Backtesting" there, so probably the "Unseen data" is just another form of historic data.

Does it mean that you break down the historic data to pieces and backtest the system on parts of data, part by part, in which the "Unseen data" is the last piece of data to backtest on??

Also an idea sparked in my mind while reading the article which I don't know how valid it is or will be.

Does the backtesting software or TS allow you to input future data manually. I know this won't be efficient specially that you will need to enter tick by tick data, but let's go with this till the end and then figure out the difficulties.

For example, let's say that we are testing a system which uses daily data. Today is October 20, 2005. I would like to enter manually the daily data for each day in the future till Dec 31, 2006. And then run the test to see how the system will perform. And then re-enter this future data once again, and re-test. Of course if the software can generate this data randomly, it would be better. In my opinion, this will help judge how such a system is robust relative to random data in the future. For example, a system which performs good in history, and poor for 5 sets of random data in the future is not robust at all.

I don't know if you got my idea or not?? Is it a worthy one, or is it just a non-sense glitch???

Thanks,

Nader
  • #15
  • Oct 20, 2005 8:11pm Oct 20, 2005 8:11pm
Man, this is some high class stuff! I'm richly fortunate to be among you guys. So many posts, so little time.
Because helping each other is a good thing.
  • #16
  • Oct 20, 2005 8:28pm Oct 20, 2005 8:28pm
Quoting merlin
man, the lessons learned are so numerous...

here are a few that come to mind...

1. Avoid Multicollinearity
this is where you use two (or more) indicators that tell you the same thing. for a complete definition, see this.

systems that have multiconllinearity are mostly over optimized, and the backtest results is nothing more than coincidence.

2. Follow a process for developing your system
i recently wrote an article for Stocks and Commodities magazine about "The System Behind the System" (see attached). if you just bang on your keyboard and hope to find a great backtest report, you are falling into a complex trap of hindsight.

3. Dont build systems that you cant trade exactly
what good is a great backtest report if you cant trade the system exactly as designed? some systems expect you to be on call 24 hours a day, ready to follow the signal in a split second. is this realistic? if the answer is no, you shouldnt attempt to trade the system because your human error will most definitely eat up your profits. when you make systems that uses pre-set orders and only requires you to be at your computer once or twice a day, its realistic to think that you can trade it percisely.
Thanks Merlin.. However, I think I won't be able to immediately use the steps in the article. But I know it will be a good reference in the future.. I still have a long way to go man.. Hopefully I can endure the process..
Thanks again..
  • #17
  • Oct 21, 2005 8:14pm Oct 21, 2005 8:14pm
Quoting narafa
Merlin, great article and a very organized one. However I still can't understand 1 thing. The unseen data you mentioned?? Do you mean forward testing, i.e. putting the system into action for some time to see how will it perform and will the future performance be the same or very like the historic one??

I don't think this is what you mean by the "Unseen data", because you were using the term "Backtesting" there, so probably the "Unseen data" is just another form of historic data.

Does it mean that you break down the historic data to pieces and backtest the system on parts of data, part by part, in which the "Unseen data" is the last piece of data to backtest on??

Also an idea sparked in my mind while reading the article which I don't know how valid it is or will be.

Does the backtesting software or TS allow you to input future data manually. I know this won't be efficient specially that you will need to enter tick by tick data, but let's go with this till the end and then figure out the difficulties.

For example, let's say that we are testing a system which uses daily data. Today is October 20, 2005. I would like to enter manually the daily data for each day in the future till Dec 31, 2006. And then run the test to see how the system will perform. And then re-enter this future data once again, and re-test. Of course if the software can generate this data randomly, it would be better. In my opinion, this will help judge how such a system is robust relative to random data in the future. For example, a system which performs good in history, and poor for 5 sets of random data in the future is not robust at all.

I don't know if you got my idea or not?? Is it a worthy one, or is it just a non-sense glitch???

Thanks,

Nader
yeah the unseen data is just a set of historical data. i guess the title i give it makes it a bit confusing. yes, unseen data is the last set of data. it is only good for one test. as soon as your system is tested on the unseen data, its "spoiled" and the project is over. thats why you dont use the unseen data until you get a good test on the walk forward data.

a few developers i know use the walk forward data a few times. while i agree with this practice, i think it is important to test your system on some completely foreign data to ensure that you are not letting hindsight creep into the process.

regarding your future data idea, i dont do this personally, but i know of developers that do use randomly generated data. i cant get past the theoretical point that market data is not randomly generated, and therefore has no bearing on how my system will perform in the future. i would take solace, however, in knowing that my system can even make money on random data. thats what i call robust!
Relax and be happy.
  • #18
  • Oct 21, 2005 10:30pm Oct 21, 2005 10:30pm
so, Merlin, to check that I've got what you call "unseen data" straight: you're suggesting backtesting an idea over a period of historical data but leaving a number of days/months (depending on the particular trading method) closer to today out of the inital testing so that this data can then be used to verify the validity of the trading method.

for example, if you come up with a method that generates a signal a few times a week, you could use data from 2000-2003 inclusive and then use 2004 and 2005 data bit by bit (to run a number of tests) to see how your method would have fared during that time.

This is the methodology I'm using and it certainly presents certain advantages. For one, I've found that setting indicator parameters too tightly gives great initial results but curve fits the historical data too much and the forward testing described above will quickly point that out before you put real money on the line.

When you talk about "foreign data", are you suggesting testing a method that works for the euro out on the yen (or even a different market altogether)? Each currency pair has its own characteristics and I found that some trading methods tend to lend themselves better to one pair than another. Maybe you can expand on this?

Nader, I think what you are referring to are Monte-Carlo simulations - a very interesting topic as well!
  • #19
  • Oct 22, 2005 10:12am Oct 22, 2005 10:12am
great article, you should of sent me a copy before i started my own project!

i like the way you break the data up into 4 units - will keep that in mind for further testing!

Quoting merlin
man, the lessons learned are so numerous...

here are a few that come to mind...

1. Avoid Multicollinearity
this is where you use two (or more) indicators that tell you the same thing. for a complete definition, see this.

systems that have multiconllinearity are mostly over optimized, and the backtest results is nothing more than coincidence.

2. Follow a process for developing your system
i recently wrote an article for Stocks and Commodities magazine about "The System Behind the System" (see attached). if you just bang on your keyboard and hope to find a great backtest report, you are falling into a complex trap of hindsight.

3. Dont build systems that you cant trade exactly
what good is a great backtest report if you cant trade the system exactly as designed? some systems expect you to be on call 24 hours a day, ready to follow the signal in a split second. is this realistic? if the answer is no, you shouldnt attempt to trade the system because your human error will most definitely eat up your profits. when you make systems that uses pre-set orders and only requires you to be at your computer once or twice a day, its realistic to think that you can trade it percisely.
  • #20
  • Oct 23, 2005 12:18am Oct 23, 2005 12:18am
Quoting bluemeasure
so, Merlin, to check that I've got what you call "unseen data" straight: you're suggesting backtesting an idea over a period of historical data but leaving a number of days/months (depending on the particular trading method) closer to today out of the inital testing so that this data can then be used to verify the validity of the trading method.
yeah you are understanding it correctly. however, it is not necessary to that the unseen data be closer to the current date. it can be any period you choose. and yes, it is used to verify the validity of the trading method, but only after it has already been validated what i mean by that is the unseen data is like the final check, in a long series of checks. the unseen data can not validate the system in and of itself.

Quoting bluemeasure
for example, if you come up with a method that generates a signal a few times a week, you could use data from 2000-2003 inclusive and then use 2004 and 2005 data bit by bit (to run a number of tests) to see how your method would have fared during that time.
you are referring more to the "walkfoward data" here. "unseen data" comes after the system is validated by walkfoward data.

Quoting bluemeasure
When you talk about "foreign data", are you suggesting testing a method that works for the euro out on the yen (or even a different market altogether)? Each currency pair has its own characteristics and I found that some trading methods tend to lend themselves better to one pair than another. Maybe you can expand on this?
by foreign data i meant data that the system has not seen already (ie already been backtested on). but you bring up a good point about a system working on multiple pairs. a robust system can work in many markets (ie EURUSD, USDJPY, bonds, gold, etc), but that doesnt mean a market specific system wont make you money.

Quoting bluemeasure
Nader, I think what you are referring to are Monte-Carlo simulations - a very interesting topic as well!
monte carlo has more to do with probabilities than it does the actual trading method itself. to run a montecarlo simulation, you would have to already have a validated trading method. the simulation will show you how the system expectations may vary, it makes no judgment on the expectations themselves. and yes, monte carlo simulations are a VERY interesting topic
Relax and be happy.
0 traders viewing now