Honing a strategy | Page 2 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

AMAZING Strategy!!! 3,266 replies

WRB/MC strategy:Coder needed to semi-automate my strategy 11 replies

strategy builder/tester for new scalping strategy 1 reply

Is opposite of losing strategy a winning strategy? 21 replies

Daily strategy report 1 reply

Attachments: Honing a strategy
Exit Attachments

Honing a strategy

  • #21
  • Oct 2, 2015 6:51am Oct 2, 2015 6:51am
Quoting Atokys
Hi Big, Don't take this the wrong way but it looks to me like you are simply curve fitting a strategy to very limited data. I don't know much about the strategy but based on what I can gather here it already seems too 'delicate' for actual market conditions with its excessive degrees of freedom.
I have never really understood what "curve fitting a strategy" means.

So you've been looking at the charts for a while and you notice that, quite often, when the Ash changes color you can make 5 pips.

You try it for a while and you see that you increase the probability of a win if you only trade LO (very good) and NYO (quite good).

You also notice that you get more winners if you only trade EURJPY, EURUSD, GBPJPY & GBPUSD....and if you avoid "danger levels"...and if you trade within the daily range...and you don't sell at 100 or but at 900.

And so on....Is that curve fitting?
  • #22
  • Oct 2, 2015 7:20am Oct 2, 2015 7:20am
Quoting BigOldFella
{quote} I have never really understood what "curve fitting a strategy" means. So you've been looking at the charts for a while and you notice that, quite often, when the Ash changes color you can make 5 pips. You try it for a while and you see that you increase the probability of a win if you only trade LO (very good) and NYO (quite good). You also notice that you get more winners if you only trade EURJPY, EURUSD, GBPJPY & GBPUSD....and if you avoid "danger levels"...and if you trade within the daily range...and you don't sell at 100 or but at 900....
Yes, thats curve fitting and those observations are quite unlikely to translate into a robust strategy if you perform a rigorous test.

When testing your strategy you have to try your best to disprove your assumptions. Its hard to accept but necessary.
  • #23
  • Oct 2, 2015 7:29am Oct 2, 2015 7:29am
Thing is that he did some forward testing and the results were good. Ultimetaly, it's forward testing that will tell you whether a strategy is good and so far, the results are impressive.
  • #24
  • Oct 2, 2015 7:31am Oct 2, 2015 7:31am
Hello,

Just to get back to the original question - it's hard to say if increasing the win ratio is the better option. Really depends on the type of strategy, but generally it's the way to go.

And agree w/ Atokys - testing should be the priority at this stage of strategy development - because that's where you are, you're still developing it.
  • #25
  • Oct 2, 2015 7:32am Oct 2, 2015 7:32am
Quoting BigOldFella
{quote} I have never really understood what "curve fitting a strategy" means. And so on....Is that curve fitting?
I can see Atokys's point. Curve fitting results from honing the assumptions of your strategy away from an original benchmark to maximise performance and most of us think that this just applies to back-testing. Importantly however, this feature is prevalent through adapting assumptions in live trading to respond to current market conditions. The problem is that like any experiment you need to test these assumptions continuously against the original benchmark (a control) to see how the current 'flavour' of your strategy adjusts over time.

There is nothing wrong with curve fitting, as this is an essential task required to optimise your strategy in accordance with the current market. We should all do that but where the trap lies is if we assume that these conditions will hold in perpetuity. We need some basis to monitor the efficacy of our assumptions and this is what many traders don't do.

Most traders find that when market conditions change their optimised strategies no longer work and leave them scratching their heads and at a loss what to do. Some unfortunately throw away very valuable strategies as they forget that the very assumptions that made the technique perform over a certain period simply need relaxing/adjusting.

It sounds like you probably have a great strategy but clearly with only 3 months live trading, it is probably too early to tell if your great performance can be maintained over the longer term duration. Something most traders don't do is to continue to test their strategies assumptions while live trading. You can do this by isolating all your assumptions as a checklist of variables which you then continue to probe with demo or paper trading efforts in tandem with your live trading. This way you treat it as an evolving work in progress.

For example, currently over the last 3 months your strategy is firing on all cylinders using stated assumptions that you have already identified. At periodic intervals (say 3 months) undertake a backtest over a defined time-frame (say the last 2 years ) using your complete investment universe to identify whether your current assumptions hold and then rank that complete hypothetical watchlist and compare their performance against those instruments in your live watchlist. See whether your current assumptions still hold. From that analysis you may be able to amend assumptions that are more reflective of current market conditions or alternatively discard some poor performers and add better performers from your hypothetical watchlist.

Such frequent interrogation and analysis will allow you to adapt your strategy over time to better respond to current market conditions and avoid the potential trap of getting into deep water without a paddle.

...anyway it sounds like you have a cracker of a strategy and congrats on your breakthrough :-)
  • #26
  • Oct 2, 2015 7:37am Oct 2, 2015 7:37am
Quoting 4xgmp
Thing is that he did some forward testing and the results were good. Ultimetaly, it's forward testing that will tell you whether a strategy is good and so far, the results are impressive.
Forward testing (data wise) is no different from back testing. A rigorous backtest is sufficient to satisfy concerns of strategy validity.

The idea that using only recent data to test is somehow superior to backtesting is a myth.
  • #27
  • Oct 2, 2015 7:43am Oct 2, 2015 7:43am
Quoting Atokys
{quote} Yes, thats curve fitting and those observations are quite unlikely to translate into a robust strategy if you perform a rigorous test. When testing your strategy you have to try your best to disprove your assumptions. Its hard to accept but necessary.
I come from a software engineering background where robust testing is a way of life.

It's harder but more important to prove a system doesn't fail rather than prove it works. Confirmation bias is bad bad thing.

The only way I can think of doing this is to carry on...but at what point do you come to the conclusion that it works?
  • #28
  • Oct 2, 2015 7:50am Oct 2, 2015 7:50am
Quoting Copernicus
{quote} I can see Atokys's point. Curve fitting results from honing the assumptions of your strategy away from an original benchmark to maximise performance and most of us think that this just applies to back-testing. Importantly however, this feature is prevalent through adapting assumptions in live trading to respond to current market conditions. The problem is that like any experiment you need to test these assumptions continuously against the original benchmark (a control) to see how the current 'flavour' of your strategy adjusts over time....
Hmm, what kind of framework could one use to optimize a strategy adaptively going forward?
  • #29
  • Oct 2, 2015 7:53am Oct 2, 2015 7:53am
Quoting BigOldFella
{quote} I come from a software engineering background where robust testing is a way of life. It's harder but more important to prove a system doesn't fail rather than prove it works. Confirmation bias is bad bad thing. The only way I can think of doing this is to carry on...but at what point do you come to the conclusion that it works?
I hate to be the bearer of bad news but you never can truly know. The only tool at your disposal is statistical validity which improves with extended sample sets. A more definitive measure is to trial your assumptions over a range of market conditions that seek to identify via back-test the complete spectrum of possible outcomes. What this is saying is that 2 years of back-testing is more valid than one year, 10 years of back-testing is more valid than 5 years and so on and so forth.

At least with 10 years of backtesting over a wide range of market conditions it is going to tell you how robust your strategy is and what assumptions work better in different market regimes.

The biggest problem we probably are going to face as traders is in determining whether your strategy is busted or not. By testing the strategy over a wide range of conditions however you will find out important information that can assist in determining whether it is broke or not. For example are you just going through an unfavourable drawdown period or is it indeed broke. Extended testing over a large sample set will give you information you can use to determine 'when to pull the plug' or not as opposed to sufferring the dissonance of dropping a strategy that subsequently shoots towards the moon.....a traders worst nightmare. :-(
  • #30
  • Oct 2, 2015 7:54am Oct 2, 2015 7:54am
Quoting Atokys
{quote} Hmm, what kind of framework could one use to optimize a strategy adaptively going forward?
*sniggers* I think you might know Mr CTA lover
  • #31
  • Oct 2, 2015 7:56am Oct 2, 2015 7:56am
Quoting BigOldFella
{quote} I come from a software engineering background where robust testing is a way of life. It's harder but more important to prove a system doesn't fail rather than prove it works. Confirmation bias is bad bad thing. The only way I can think of doing this is to carry on...but at what point do you come to the conclusion that it works?
So I encourage you to perform a backtest over as many markets as possible and over a long span of time (10 years minimum) and see how it performs. If the results satisfy your expectations then I guess it works.
  • #32
  • Oct 2, 2015 7:59am Oct 2, 2015 7:59am
Quoting Copernicus
{quote} *sniggers* I think you might know Mr CTA lover
But I really dont

I dont think forward data would give any unique insight that would prompt a system tweak.
  • #33
  • Oct 2, 2015 8:05am Oct 2, 2015 8:05am
Quoting Copernicus
{quote}...anyway it sounds like you have a cracker of a strategy and congrats on your breakthrough :-)
Thanks.

Exactly. Over the 6 months you can see how some pairs are profitable, and then are not, and then are. And some have been profitable for the whole period...and some have been unprofitable for the whole period. I suspect that, as Atoky says, 6 months is not a long enough time period...but with any strategy, do we ever know for sure that a massive draw down over a number of days isn't round the corner.

Having the results of over 10,000 trades (and the market conditions at the time of the trade was placed) allows me to do some very interesting "What If" analysis.

Using an example, if I'd only taken trades where the Ash on the H4 and D1 timeframes were the same colour as the Ash on the M30 then with win rate goes up slightly. So should I start using this in the live account?
  • #34
  • Oct 2, 2015 8:11am Oct 2, 2015 8:11am
Quoting Atokys
{quote} But I really dont I dont think forward data would give any unique insight that would prompt a system tweak.
Try and get your hands on 'Trading Systems: A New Approach to Systems development and portfolio optimisation' by Urban Jaeckle and Emilio Tomasini. They propose an excellent method of baseline back-testing and walk forward development which is a useful method of portfolio optimisation when you have limited capital. Effectively it recognises that unfortunately with limited capital you simply cannot diversify away to your hearts content so they propose a 'rational choice' method of selecting those instruments within a wide portfolio universe that offer the best recent performance. Too much back-testing however does not allow you the necessary period to evaluate system efficacy so you have to be strict in when you apply it using a sufficient sample size of live data to at least assist in statistically validating whether the system is broke or not.

The one true measure of system performance however is long term sustainability so what they do is compare base-line tests assuming assumptions are fixed over the period against 'stitched together' performance results associated with live results over the same period using adjusted variables.
  • #35
  • Oct 2, 2015 8:18am Oct 2, 2015 8:18am
Quoting Copernicus
{quote} Try and get your hands on 'Trading Systems: A New Approach to Systems development and portfolio optimisation' by Urban Jaeckle and Emilio Tomasini. They propose an excellent method of baseline back-testing and walk forward development which is a useful method of portfolio optimisation when you have limited capital. Effectively it recognises that unfortunately with limited capital you simply cannot diversify away to your hearts content so they propose a 'rational choice' method of selecting those instruments within a wide portfolio universe...
Well, I guess I'll have to look it up. I've never considered a rotating portfolio before.
Quoting BigOldFella
Having the results of over 10,000 trades (and the market conditions at the time of the trade was placed) allows me to do some very interesting "What If" analysis.
One more thing, the length of time tested is more important than the number of trades generated in that period. A 10 year backtest of a long-term system that yields 10 trades would give me much more confidence than a 1 year backtest that produced 1000 trades in a short-term system.
  • #36
  • Oct 2, 2015 8:19am Oct 2, 2015 8:19am
Quoting BigOldFella
{quote} Thanks. Exactly. Over the 6 months you can see how some pairs are profitable, and then are not, and then are. And some have been profitable for the whole period...and some have been unprofitable for the whole period. I suspect that, as Atoky says, 6 months is not a long enough time period...but with any strategy, do we ever know for sure that a massive draw down over a number of days isn't round the corner. Having the results of over 10,000 trades (and the market conditions at the time of the trade was placed) allows me to do some very interesting...
Yep, if it was me I would always select the best performer in terms of expectancy (as opposed to win-rate) as the choice for now but select a suitable timeframe where you periodically re-test assumptions. In fact Van Tharpe uses a particularly useful measure referred to as 'the system quality number' or SQN that also assesses the variability of returns in conjunction with expectancy. I have found that to be the most useful measure but you need a sufficiently large sample size to make any sense of it (eg. 100 trades) so that should determine the frequency of when you periodically test the system.
  • #37
  • Oct 2, 2015 8:24am Oct 2, 2015 8:24am
Quoting Atokys
{quote} Forward testing (data wise) is no different from back testing. A rigorous backtest is sufficient to satisfy concerns of strategy validity. The idea that using only recent data to test is somehow superior to back testing is a myth.
If you're talking about back testing using Metatrader then forward testing is very different from back testing.

There are some things that you can't back test and are vital to the success of the strategy: -

  1. the status of other pairs, for example if I'm buying GBPUSD and GBPJPY I don't want to be buying EURGBP even if the Ash changes colour
  2. spreads - time of day & volatility can change the spread dramatically
  3. relative currency strengths - if you want to check the base strength versus the quote strength

I'm not sure how you get round those problems if you want to back test...unless you know a way? That would be brilliant if you did?

  • #38
  • Oct 2, 2015 8:29am Oct 2, 2015 8:29am
Quoting Atokys
{quote} A 10 year backtest of a long-term system that yields 10 trades would give me much more confidence than a 1 year backtest that produced 1000 trades in a short-term system.
Yes, I can see your point here.

You want as wide a range in market conditions as possible and the only way to do that would over a large time period.
  • #39
  • Oct 2, 2015 8:31am Oct 2, 2015 8:31am
Quoting BigOldFella
{quote} If you're talking about back testing using Metatrader then forward testing is very different from back testing. There are some things that you can't back test and are vital to the success of the strategy: - the status of other pairs, for example if I'm buying GBPUSD and GBPJPY I don't want to be buying EURGBP even if the Ash changes colour spreads - time of day & volatility can change the spread dramatically relative currency strengths - if you want to check the base strength versus the quote strength I'm not sure how you get round those...
I use backtesting as a way of very bluntly determining a strategy's technical validity. You make a fair point about those issues, if they are a concern for you then forward testing is a solution.
  • #40
  • Oct 2, 2015 8:38am Oct 2, 2015 8:38am
Quoting Copernicus
{quote} In fact Van Tharpe uses a particularly useful measure referred to as 'the system quality number' or SQN that also assesses the variability of returns in conjunction with expectancy. I have found that to be the most useful measure but you need a sufficiently large sample size to make any sense of it (eg. 100 trades) so that should determine the frequency of when you periodically test the system.
See, this is where I feel I need more knowledge.

Currently I import the results into a database package and run my own set of queries to work out which pairs are best to trade tomorrow. It's a kind of moving average of returns. If they're one the way up I include them, if they're on the way down I exclude them.

But over the last 3 months I've only ever excluded once and excluded once.

Like Atokys says, it's not a long enough sample to draw any long term conclusions.
0 traders viewing now