What works, and what doesn't | Page 4 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

variety of resources what works and what doesn't 98 replies

EA works on one broker, doesn't on another 4 replies

If it works, it works... 3 replies

EA Works, Then Doesn't 1 reply

Why an EA works for some currancies and not others? 10 replies

Attachments: What works, and what doesn't
Exit Attachments

What works, and what doesn't

  • #61
  • Oct 5, 2015 6:15pm Oct 5, 2015 6:15pm
Hi FXEZ

This is fascinating.

Quoting FXEZ
{quote} 1) Trading systems developed and selected using sound statistical model building principles (not overfit)
Also, on your website you say: -
Quote
The more parameters that are tested in the finer granularity, the greater the chance that the in sample results will be over fit to only the in sample data and will not be generalizable to any other data.
I have the results (and certain market conditions of when the trade was placed) of over 10000 trades going back 6 months. If I'm right what you're saying is - the more filters I apply to the 10000 trades to get a profitable outcome the less chance of long term success?

So, for example, if I apply 10 filters I can get an average gain of around 100 pips/day. But if I only apply 7 of those filters I get an average gain of 30 pips/day. Are you saying that the 30 pips/day variant is more likely to succeed long term?
  • #62
  • Edited Oct 6, 2015 4:37am Oct 5, 2015 10:18pm | Edited Oct 6, 2015 4:37am
Quoting BigOldFella
Hi FXEZ This is fascinating. {quote} Also, on your website you say: - {quote} I have the results (and certain market conditions of when the trade was placed) of over 10000 trades going back 6 months. If I'm right what you're saying is - the more filters I apply to the 10000 trades to get a profitable outcome the less chance of long term success? So, for example, if I apply 10 filters I can get an average gain of around 100 pips/day. But if I only apply 7 of those filters I get an average gain of 30 pips/day. Are you saying that the 30 pips/day variant...
Those that use diversification as a method to 'punch through' market randomness tend to use exceptionally simple systems as a technique to tilt market bias in their favour. The process uses large sample data as a method to determine the direction of price bias giving the market maximum degrees of freedom in it's movement. To successfully diversify across a wide investment universe, it is essential that the rules are simple enough to be applicable to any liquid instrument or market condition. More rules or variables reduces the degrees of freedom through which price can move, hence can improve performance when market conditions are predictable but comes at a cost in that when market conditions change, a rigid system that does not respond to these changes can lead you into potentially nasty and extended drawdown periods.

In your question regarding the number of filters you use, you can really only test this assumption through extensive back-testing. The rules of simplicity we are talking about is more about the preference for simple rules (or fewer variables) in navigating through randomness and applied across a portfolio and while generally true, the ultimate arbiter of your decision should be statistical in nature.

The baseline aim of the simple strategy (or Stage 1 of system development) is to simply achieve a positive bias 'an 'alpha edge' that is robust and statistically significant. In terms of robust, what is being said here is that an adjustment or fine tuning of the variables will not materially affect the performance of the strategy. Significant material alterations in strategy performance from a simple tweaking of the variables is a major indicator of a 'curve fitted' solution that will not stand the test of time. This is an important point which allows FXEZ's third point on strategy adaption to be achieved down the track (more on this later).

To have a robust 'edge' in your baseline strategy means that your edge must be able to pay for the frictional costs of trading and then some over time, so when added to simply random performance you find you now have a strategy with a positive edge that delivers general positive performance and has been tested across a wide variety of liquid instruments and different market conditions. I actually believe that no single system can deliver positive performance across all market conditions and hence you possibly need multiple simple systems to achieve this that do not 'interfere' with the performance of the other, but rather supplement overall program performance. Examples being a very simply trend following or momentum strategy combined with a very simply mean reversion strategy where each strategy informs the other and trades are alternated between both.

Currently there is sufficient 'real performance data' available over history that suggests that simple trend following systems are all that's needed to achieve this edge, but in statistics, never say never as uncertainty is a given.....and I firmly believe that a simple reliance of trend following alone is not sufficient to statistically validate an edge. Hence diversification of simple systems reduces the risk associated with times when market conditions are unfavourable for extended periods. I note that FXEZ uses/or is exploring HFT as a mechanism to further squeeze positive bias out of longer term trading strategies (without negatively interfering with baseline system performance) whereas my focus currently is on simple but selectively applied mean reverting systems that assist in flattening the equity curve but not compromising the underlying baseline performance strategy.

Once you achieve a sufficient and statistically valid 'edge' using a simple system/s that give maximum room for price to breathe (within the noise), then you have the 'house edge' and then you can apply leverage using principles of diversification (Stage 2 of System Development) to 'add meat to the bones of your simple technique/s' of positive bias as opposed to methods that purely focus on 'optimising' for a particular instrument often at the expense of the portfolio. Diversification is used as a tool to shape the equity curve and make available more capital to trade than would otherwise be possible and hence have more ability to further diversify towards 'beta'. The alpha edge in the primordial strategy then augments beta to allow you then to prosper from market outperformance.

Now the third point that FXEZ is referring to of program adaption (Stage 3 of System Development) is a relatively recent addition to systematic trading but something that needs to be looked at closely as alpha generation can be further enhanced by altering the robust variables embedded in the simple strategies to adjust to market conditions and add a third layer of performance to the baseline strategy. This is also particularly important for those seeking diversification benefits but with limited capital where a degree of selectivity is required in the instruments chosen for diversification. Remember that a robust system possesses variables that through alteration do not drastically alter program performance, but they do possess sufficient flexibility to be altered to reflect variability of market conditions and further turbo charge performance without attracting a commensurate increase in risk exposure. This is achieved by baseline backtesting in conjunction with live trading walk-forward analysis that with a sufficient sample size can be validly used to adjust variables in reponse to current market conditions and squeeze extra performance from your program.
  • #63
  • Oct 6, 2015 3:25am Oct 6, 2015 3:25am
Quoting BigOldFella
Hi FXEZ This is fascinating.
Hi BigOldFella,

Aside from the awkwardness of my sentence that you quoted , I think the idea I was trying to get across can be illustrated by this example:

Let's say you wish to optimize a MA crossover system with two MAs with lengths from 1-200. If you parameterize the system by taking lengths 1-200 by 1s for both MAs, you get 200 * 200 = 40,000 possible "system" combinations, including duplicates. That is a really large number of possible strategies to evaluate, and just from two parameters! If you had a 3 or 4 MA system, the number would get intractable fast! Clearly there is little difference between say a 199 period SMA and a 200 period SMA: most of the info contained in the 200 SMA is contained in the 199 SMA. You could check the correlation of these two moving averages and see that they are very highly correlated. In R I just did this and got 99.7% correlation for the series I tested! Highly correlated predictors/features/moving average lengths tend to give you lots of the same thing and don't add much incremental predictive value. You want new information, not the same or very similar information over and over. So as a general principle, the best advice is to remove as many of the highly correlated predictors as you can. This will reduce your search space and reduce the chance to overfit.

If you parameterized your MA crossover from 20 - 200 by 20s you have 20 * 20 = 400 combinations including duplicates, or a much reduced space. So the idea is to use the correct granularity for your parameters. I think that's what I was trying to say. Possibly a more appropriate parameterization may be 5-20 by 5, 20-40 by 10, 40 - 100 by 20, 100 - 200 by 50 (11*11 = 121), but it's clearly more difficult to set that up than just going by 20.
Quote
So, for example, if I apply 10 filters I can get an average gain of around 100 pips/day. But if I only apply 7 of those filters I get an average gain of 30 pips/day. Are you saying that the 30 pips/day variant is more likely to succeed long term?
As to your specific question, if you include more filters you get closer to a perfect score because each filter removes losing trades. Filters may also remove winning trades. It is sort of like a neural network that memorizes the data and gets a near zero error rate the longer you run it, or the more complex you make it. What if you added 1,000,000 filters? 300 trillion? at some point you have 1 filter for each data point and could get a perfect score but at that point there is almost a zero chance of the system generalizing on future (unseen) data!

There are 3 main problems to deal with when maximizing some performance statistic in backtesting or optimization / system selection: 1) overfitting the historical data, 2) finding a "lucky" system that has no chance to generalize, and 3) lack of generalization. The first two are essentially statistical problems created by either the model building / optimization / model selection process or the data mining process. The third problem is endemic to trading system development and the unknowable future.

1) The chance of overfitting the historical data can be reduced by using longer training data periods, reducing the number of parameters, using simpler predictors and other variance reduction methods. Reducing complexity will tend to reduce performance on historical data, making the system seem less "good".

2) The chance of finding a "lucky" system by chance that blows up when it is run on new data can be reduced by running data mining bias tests which require a much higher performance to chop out systems that may just be lucky. This will also chop out some lower performing systems that may be good.

3) The problem of generalization is universal and difficult to solve. Some say it is not possible to solve. Rather than throw my hands up in the air and give up, I've put forward the hypothesis that by using an adaptive mechanism at the trading strategy level, that systems can be created that have a much higher chance to continue performing on new data individually and in aggregate. But of course there are no guarantees and any system may stop working at any point. Yogi Berra had it right, it's tough to make predictions, especially about the future.
  • #64
  • Oct 6, 2015 3:35am Oct 6, 2015 3:35am
Quoting Atokys
{quote} I agree that good systems have a dynamic component that shifts with market character. However, the idea of machine learning that you allude to is a flawed concept in trading system design. Having system rules and degrees of freedom that change all the time; one cannot confidently determine the system's robustness even if the backtest proved profitable.
Can you explain why it isn't possible to evaluate the robustness of a trading strategy when a strategy changes its rules over time and how the machine learning concept that I alluded to is flawed?
  • #65
  • Oct 6, 2015 3:39am Oct 6, 2015 3:39am
Quoting Copernicus
{quote} Quantitative Investing - Strategies to Exploit Stock market Anomalies for All Investors - Piard;
Following the Trend - Diversified Managed Futures Trading - Clenow.
I've been meaning to read the Clenow book and hadn't heard of the Piard book so thanks for the references.
  • #66
  • Oct 6, 2015 3:41am Oct 6, 2015 3:41am
What works: Waiting.. Trading on your terms not the markets

What doesn't work: Trading angry or happy

The market is one big emoticon.. with an emphasis on the "con"

But more helpful.. Real traders act when the time is right rather than when they would like the time to be right. And I guess it helps that they know what they are acting on and waiting for.
  • #67
  • Edited 4:02am Oct 6, 2015 3:46am | Edited 4:02am
Quoting Atokys
{quote} It can be literally anything. Even the term 'character' that we use to ascribe a quality to market activity is subjective and different people will perceive this shift in a different way from their personal trading metaphor. This doesn't only apply to individual markets but also to the correlations between markets.
I think the market's "character" is usually only noticed through the lens of our system/trading performance. When a system stops performing, has the market's character changed? Or has the system simply stopped working? What about other systems that continue working while our system stops? This quickly turns into a philosophical question. And I'm not a philosopher so I'm not sure how that debate resolves itself.

Quote
Markets are always changing

Or is the underlying nature of the market that its participants are always adapting and changing so in that respect the market's character always stays the same?
  • #68
  • Oct 6, 2015 3:58am Oct 6, 2015 3:58am
Quoting FXEZ
{quote}. When a system stops performing, has the market's character changed? Or has the system simply stopped working???
What's the difference?

The market is a system in itself that systematically destroys all systems. It finds the holes and fills them. So can you turn this philosophy into a system? Yes.. But inevitably it will be destroyed, unless your system accounts for this inevitability... So then is it a system you have? or a simple and rare understanding?
  • #69
  • Oct 6, 2015 4:02am Oct 6, 2015 4:02am
Thank you for the replies.

Two very comprehensive answers.
Quoting Copernicus
{quote} Those that use diversification as a method...
Quoting FXEZ
{quote}I think the idea I was trying to get across can be illustrated by this example: Let's say you wish to optimize a MA crossover system...
It will take me a while to digest all the detail...leave it with me.

Thanks again.
  • #70
  • Edited 4:37am Oct 6, 2015 4:26am | Edited 4:37am
Quoting Kong
{quote} What's the difference? The market is a system in itself that systematically destroys all systems. It finds the holes and fills them. So can you turn this philosophy into a system? Yes.. But inevitably it will be destroyed, unless your system accounts for this inevitability... So then is it a system you have? or a simple and rare understanding?
On a side note, that's why these market "scientists" will never get it. It takes an artist to see the lines, shapes and colours (not just technically).. A rigid genius is fresh meat.
  • #71
  • Oct 6, 2015 4:36am Oct 6, 2015 4:36am
Quoting FXEZ
{quote} I've been meaning to read the Clenow book and hadn't heard of the Piard book so thanks for the references.
There is an excellent chapter at the end of Clenow's book where he reverse engineers the performance of some of the notable performing CTA's and mimics historic returns from which he then defines the major assumptions adopted that give excellent insights into various methods of diversification. He really knows his stuff and gives the practitioner lot's of confidence in the systematic process. :-)
  • #72
  • Oct 6, 2015 5:09am Oct 6, 2015 5:09am
Quoting Kong
{quote} The market is a system in itself that systematically destroys all systems. It finds the holes and fills them. So can you turn this philosophy into a system? Yes.. But inevitably it will be destroyed, unless your system accounts for this inevitability... So then is it a system you have? or a simple and rare understanding?
Yes I agree that the adaptive nature of the market participants and thus the aggregate market causes the destruction of all systems at some times, but not all systems all the time. While some systems go down the tubes performance wise, other systems are on the rise, thus the use of diversification and adaptation. Most retail traders don't make use of diversification which has been dubbed as the only real "free lunch", but discretionary traders tend to be adaptive by nature.

Quote
What's the difference?
I suppose the difference is in knowing what is actually broken and thus what to fix, or where to profitably spend one's efforts. Let me give you an example. In 2013 I went live with some strategies that had been extensively tested and also looked good in forward test. The systems performed well for about 1 1/2 months then performance started to decline. I shut them off near the end of 2013 as performance came back to the starting equity. I (wrongly) concluded that the market had changed, and so spent most of 2014 developing new strategies. I didn't figure out until the end of 2014 that the problem was in my testing and execution framework that caused unreliable results. Had I known what was actually wrong, I could have spent the time much more profitably, addressing the actual problem.

If you don't know what's really broken, how can you fix it?
  • #73
  • Oct 6, 2015 5:27am Oct 6, 2015 5:27am
Quoting FXEZ
{quote} Yes I agree that the adaptive nature of the market participants and thus the aggregate market causes the destruction of all systems at some times, but not all systems all the time. While some systems go down the tubes performance wise, other systems are on the rise, thus the use of diversification and adaptation. Most retail traders don't make use of diversification which has been dubbed as the only real "free lunch", but discretionary traders tend to be adaptive by nature.
and on it goes
  • #74
  • Oct 6, 2015 5:33am Oct 6, 2015 5:33am
Quoting Copernicus
{quote} More rules or variables reduces the degrees of freedom through which price can move, hence can improve performance when market conditions are predictable but comes at a cost in that when market conditions change, a rigid system that does not respond to these changes can lead you into potentially nasty and extended drawdown periods. In your question regarding the number of filters you use, you can really only test this assumption through extensive back-testing. The rules of simplicity we are talking about is more about the preference for...
There's an additional thing that has to be kept in mind when statistics is applied to markets. As mentioned above the markets are emotional in nature, even with the advent of algorithms. Social statistics and combining them with a trading style that suits the personality of the trader are what can provide an edge.

Winning in trading isn't a one-size fits all solution. That's what many don't seem go grasp.
  • #75
  • Oct 6, 2015 8:28am Oct 6, 2015 8:28am
Quoting FXEZ
{quote} Can you explain why it isn't possible to evaluate the robustness of a trading strategy when a strategy changes its rules over time and how the machine learning concept that I alluded to is flawed?
It is flawed because it attempts to adaptively optimize something that defies all attempts at fine tuning. A profitable backtest result alone will not be able to suggest robustness because of this very nature of the system. However, if somehow the system demonstrates versatility in a walk-forward analysis across various markets then that would be quite something.
Quoting FXEZ
{quote} I think the market's "character" is usually only noticed through the lens of our system/trading performance.
I agree.
Quote
When a system stops performing, has the market's character changed? Or has the system simply stopped working? What about other systems that continue working while our system stops?
I can't answer this because I do not know what you mean by 'work'. Systems are primarily based on 'trends' which are subjective to our individual perception.

Tom Basso said that what gave him confidence in running his systems was manually poring over price data and scrutinizing how his systems performed under different conditions. If you have intimate understanding of your system then the question of whether a system has 'stop working' is meaningless.
Quote
Or is the underlying nature of the market that its participants are always adapting and changing so in that respect the market's character always stays the same?
Yes, markets have always been the same because they are always changing. By extension, the game of trading has also always been the same and a standardized approach can be robust if it incorporates and anticipates change.
Quoting Kong
The market is a system in itself that systematically destroys all systems.
Quoting FXEZ
{quote} Yes I agree that the adaptive nature of the market participants and thus the aggregate market causes the destruction of all systems at some times, but not all systems all the time. While some systems go down the tubes performance wise, other systems are on the rise, thus the use of diversification and adaptation.
All forms of trading, discretionary or mechanical, take place on some kind of system whether or not users are conscious of it. Profitability of trading systems seems to move in cycles. Periods where certain kinds of systems experience success will lead to their increased use, as markets shift from one character to another, the weaker hands and those with unsound practices are shaken out. Buying/selling pressure from such trading leaves the market on a systemic level and a period of relative calm sets the stage for the next profitable cycle. Imagine this playing out on the fractal manifold of price and you get the seeming chaos that defines markets. Longevity is the key to success.
  • #76
  • Oct 6, 2015 12:15pm Oct 6, 2015 12:15pm
There is too much philosophy on this thread. Can we please stick to trading and posting charts and ideas. Thank you.
  • #77
  • Oct 6, 2015 12:39pm Oct 6, 2015 12:39pm
Quoting Techanalyst
There is too much philosophy on this thread. Can we please stick to trading and posting charts and ideas. Thank you.
Thats the problem of many threads on FF.
Too much philosophy bullshit. And who post the most beautiful post!
  • #78
  • Oct 6, 2015 1:54pm Oct 6, 2015 1:54pm
Quoting Techanalyst
There is too much philosophy on this thread. Can we please stick to trading and posting charts and ideas. Thank you.
Quoting Good Lookin
{quote} Also, in addition to the forms of market structure (or liquidity, whatever floats your boat) trades you've already listed, consider Big Round Numbers in your analysis (1.1200, 1.1300, 116.00, 117.00, etc). I've read in a research paper a while back that somewhere around 25% to 40% of all market participants place their orders anywhere in a 20 pip range around BRNs. Often times their stops are also within that 20 pip range. I've used this info to determine some of my targets and entries with good success. I read this at least 5 years ago...
I added an actionable idea, i've been waiting for anyone else to do the same.
  • #79
  • Oct 6, 2015 1:55pm Oct 6, 2015 1:55pm
Quoting bloodpoodle
a thin quick run up leaves a gap in liquidty... nobody puts orders in the middle of huge candles.. So price falls right back down to the base of the big candle.
Should we try to fade the huge candles or to wait a retrace near the start of the candle and play in the same direction of the huge candle??

How to take action and when?
  • #80
  • Oct 6, 2015 2:45pm Oct 6, 2015 2:45pm
Quoting matheszabi
{quote} Hi David, I have 3 idea to write EA......
matheszabi,

I'm not 100% clear about what you're describing, but it sounds like it has discretionary aspects to it. I guess we both know that these would need to be resolved into strict mathematical rules before an EA could be coded.

David
0 traders viewing now