Systematic Portfolio Diversification - Data Mining Concept

Thanks for this mate....but I have a different project in mind which will be discussed in this thread and I would like to keep it focused as there is a fair bit to cover. I appreciate your candid comments regarding your past experience but each of us have our own preferred approaches and I would like to share my approach with others who may be interested and can collaborate.

Given that there are two Brown's on this thread I might have to amend nicknames. A Mr Brown for you maestro and a Brownie for mBrown :-)

systematic

Loozers stays Loozers

That's fine by me.

Methodological Outline

Before proceeding we need to be clear about what the approach described in this thread is all about as it will not be appealing to many traders who are:

1. searching for immediate returns;
2. have a strong opinion that markets are very predictable in nature and hence are over-leveraged to the hilt;
3. make quick subjective assessments of what is wrong or what is right applying simple illogical heuristics or faith statements to a complex problem;
4. believe that the markets are rigged when things don't work out as planned;
5. refuse to take personal responsibility for their actions;
6. have tiny dicks but huge egos to compensate;
7. are incapable of applying critical thinking; and
8. believe that they are right and that the market is wrong.

You know....about eleventy billion guys that inhabit this forum.....but usually only for a short time. *sniggers*.

All trading, no matter what form it takes is based on the principles of applying models to real market conditions. Whether discretionary, or systematic, a trader makes an informed (or ill informed) judgement about how a complex financial market behaves and then projects this hypothesis (model) of anticipated behaviour into a real market situation to test the underlying assumptions in order to hopefully achieve their desired future objectives. The models we apply are not the reality, they are only ever our best guesses or approximates of how the real market behaves.

Whether we like it or not, we are all applying the scientific method to our trading....some perhaps more rigorously than others. While achieving or not achieving financial objectives may be the tangible outcome of this process, as traders we are all on a mission to hopefully derive better models over time.

Given that there is no certainty in the future of any complex system (as by definition, a complex system is defined by a vast set of moving parts and more importantly in the interrelationships of these moving parts), common to all complex systems are emergent and non-stationery future conditions which are hopelessly intractable in nature due to the complexities involved in the 'state of the system' at any moment in time.

The state of the system today might be similar to the state of the system tomorrow, or the state of the system today may be vastly different to the state of the system tomorrow. Our models might be able to estimate within degrees of confidence an estimate of tomorrow, but the reality may be completely different.

The key principle lurking in this statement is one of non-stationarity. Complex markets possess non-stationery moving benchmarks. There are no 'Laws' in a complex market aside from the general laws of physics that are common to all complex systems. What drives market dynamics are the inter-related and emergent features of cause and effect that can and often do, change on a pinhead dependent on how the state of the system is defined at any given point in time.

This uncertainty in the future condition is the lifeblood of the trader and this degree of uncertainty is what allows a vast array of different models (approaches applied to navigating this market) to survive, whether that be models derived on principles of fundamental analysis, technical analysis, quantitative analysis etc etc etc or combinations thereof. If there was no uncertainty in the future, markets would be perfectly efficient with no arbitrage opportunities present.

As a trader, you can choose to either religiously follow one method or another, but for many other traders, they may elect to apply many different methods in their trading enterprise. There is no right or wrong approach. Certain models may be better than others some of the time or most of the time....but the bottom line is that the model you decide to apply needs to be based on a general principle or hypothesis about your opinion of how the market behaves. Having established a testable hypothesis, you then apply it to real market conditions and see whether it stacks up. If it does, then you can either continue applying the same model until it no longer works, you could try to improve your model by adjusting your hypothesis, you could park your model and look at it at a later time to see if it stacks up in the future or you could go back to the drawing board and develop a new hypothesis.

As scientific as this may sound....it is what we all are really doing. Some of us may be ardent in the belief that we have cracked the code to this market by adopting a particular model, however the reality of non-stationarity is that the market will always have the last laugh. It really pays to be humble in this game and not assume anything when it comes to navigating complexity.

So what this thread is about is the sharing of a particular approach to trading that attempts to address the non-stationary of market conditions and defines a solution that allows for a trader to survive in the long term in 'uncertain' market conditions while at the same time positioning themselves to occasionally reap out-sized rewards when conditions become highly volatile and there is a strong likelihood that participant behaviour is more coordinated and predictable than it otherwise might be.

This thread uses market data not as a means to project into an uncertain future, but rather as a process to ensure that each of the systems we deploy are at least capable of surviving the market conditions of the past (or as much of it as we can get). This is known historic market data and an aspect of the market condition that allows us to prepare ourselves for an uncertain future. We use the *known* past as a method to design a robust trading solution for the *unknown* future while at the same time leaving ourselves open to a possible windfall when market conditions are right.

The reason we use data mining in our approach is therefore for 'risk management purposes' as opposed to our estimate of how this data may or may not project into the future.

Cheers

C

PS Guys, be patient as we need to outline a context first before we can get into the details. I also want to ensure that we only maintain thread participants in this thread who are riding the same wave as me.

The Benefits of a Systematic versus Discretionary approach to 'Long Term Survival'

Discretionary traders certainly can capitalise on persistent market conditions where price action becomes more predictable operating around a temporarily stable equilibrium point....... and can significantly outperform systematic traders in the short term......but across a longer term array of different market conditions that are defined by non-stationery equilibria, very few discretionary traders survive.

Jarratt Davis had a small hiatus in the sun....yet I cannot find evidence of an extended run. Peter Brandt is another well know technical and discretionary trader who appears to have had a successful record.....however these guys are the exception rather than the rule.

The key to survival across different market regimes is adaptability. Discretionary traders tend to be specialists who excel in a particular trading style that responds to a particular market condition......however the downside to their specialty is that they rarely have an arsenal of alternative approaches that can be applied to different market conditions successfully.

This is where a systematic diversified portfolio that is already equipped with strategies configured for alternate market conditions come into their own. They are less proficient in 'making hay when the sun shines' for a single stable market condition and therefore under-perform the discretionary specialists during these times....but over an extended time horizon across a number of different market conditions.......like the tortoise and the hare.....they outlast and outlive their 'flash' discretionary cousins.

I have spent a lot of my time on IASG reviewing verified performance data to report on fund manager performance and have yet to find more than a very few discretionary traders that have stood the test of time maintaining performance metrics over 10 years plus.

The most successful breed of fund managers based on verifiable long term results are the CTA's which are a subset of the hedge fund segment but depend almost entirely on the repeated systematic application of diversified momentum/trend following strategies across their portfolio's. The likes of Dunn Capital, Winton, Dreiss Research, Transtrend, Fort Capital etc. etc. etc. having stunning long term metrics.

It is worthwhile spending some time examining their monthly performance results as frequently you see extended periods of losses. Trading on the right hand side of the chart is a different kettle of fish to the linear rising equity curves you see from these guys. Patience and resilience is required and long losing streaks to enjoy the long term benefits of what these guys deliver.

Cheers

C

The Reasons for a Long Term Approach to Data Mining as Opposed to A Shorter Term Approach - It's all in the Skew

The sample size is less important than the different array of market conditions encountered. For example in the High Frequency Trading environment, the sample size is huge (within a single market condition)....and strategies deployed in the very short term environment are very unlikely to still perform if conditions change (some exceptions here like order 'front running' bots). We use sample size more as a proxy to ensure we have a better chance of being exposed to alternating market conditions.

Certain classes of strategy (eg. divergent strategies) such as trend following and momentum methods 'rely' on non-stationery conditions and make their profits with market disruptions. Their performance results are a consequence of their style. They have Low Pwin% but relatively high R:R ratios....but also have a divergent risk feature being their positive skew that makes them robust. This characteristic signature creates the volatility in their equity curves. This doesn't mean they are 'worse strategies' or have 'higher risk exposure'. In fact they are far more long lasting than their convergent cousins and there are a swathe of white papers to demonstrate the efficacy and long lasting nature of these approaches.

The times these class of strategy shine are during volatile less stable market conditions. At other times they churn to simply keep their heads above water while waiting for 'crisis alpha' moments. Statistically these strategies are deployed to catch 'fat tailed' events on either side of the probability distribution.

The key to success in the divergent space is your ability to endure disappointment and continuously manage the downside risk. For most long term successful divergent strategies approximately 90% of trades or so are taken simply to keep the head above water...but the anomaly (say 10% of trades) were what led to the overall strong performance metrics of this solution.

The possible reason that alpha (arbitrage opportunities) persists for trend following and momentum is that the game is very hard and requires intense discipline and patience. You simply are diversified and follow price again and again and again with the same recipe....and let the market decide when it gives you a windfall.

The flip-side however to this argument is that convergent systems may have a hiatus in the sun over a short term interval.....but are very unlikely to last more than a few years duration. They rely on the leptokurtic peaks of the profit distribution for their bread and butter as opposed to the 'fat tails' like divergent traders.

Where divergent strategies come undone through death by a thousand cuts is when the fat tails are few and far between. On the contrary in a fat tailed environment, convergent strategies come undone very quickly.

Under a normal distribution, no strategy wins. Under a Non-Gaussian distribution you get two broad styles of trading strategy that can harvest arbitrage (the tails and the peak of the distribution).

These convergent styles of trading strategy work on the principle of reversion back to an equilibrium such as mean reversion styles of trading, counter trend trading, buying the dips and selling the tips, fundamental investing, grid trading and Martingale variants whereas divergent styles are forward looking and simply assume that conditions will change. Convergent styles are backward looking and rely on prediction, namely that price has taken an excursion away from a stationary equilibrium....but in the future will revert back to this 'known point'. The assumption typically used by this type of strategy is that the market becomes over-valued or undervalued and will revert. You will identify a convergent trader by virtue of some of the comments you will hear such as.....this trend is overextended, it's better to wait for a retracement in a trend before entry to get a better R:R etc etc etc. The statement is typically couched in predictive terms. This is great while it lasts but totally disruptive while it doesn't. A divergent trader however avoids any form of predictive statement and has no opinion on price other than 'the current price is right'. They are simply relying on the future price being at a different place to where it is now and hence they simply follow price like a shadow with tight trailing stops designed to get them out quickly when price changes direction and they do not attempt to project a future price position.

Now despite the fact that we might hear that a retail trader is a trend trader, does not necessarily imply that they are a divergent trader. In fact divergent traders are few and far between and tend to occupy the professional fund management space. The way you tend to sort out the who's who in the convergent or divergent space is based on the assumptions applied by each style of trading that are either predictive or non-predictive in nature.

The primary reason that we don't find many retail traders occupying the divergent space is that to survive long term as a non-predictive trend trader, you cannot afford to pin your hopes on a single instrument, timeframe or applied system and hence you cannot afford to have favorites and you need to be diversified which tends to imply that you need a greater capital base (though times are changing and not necessarily the case anymore). As a divergent trader you just never know when the moments of alpha will arrive. You operate in the land of the Law of Large Numbers as opposed to the land of 'preciseness' and you mercilessly manage your exits for risk mitigation and to preserve your capital base. You therefore need diversification as a means to improve your chances of a lucky strike to survive the brutal assault of the thousand cuts to your capital. For that reason, you will find that the divergent trader is always heavily diversified and also approaches this game from the perspective of the overall portfolio result as opposed to the individual return stream generated by a single component of the strategy.

The chart below represents the Trend Following Index which comprises the equal weighted index of 54 diversified fully systematic trend following FM's which apply divergent solutions in their trading enterprises You will see from the index performance that market conditions post GFC have changed where we now find we get extended periods of non-divergent behaviour (that these funds under-perform in) inter-dispersed with short sharp periods of strong volatility and momentum (aka trends) where trend following quickly recovers from drawdown and returns soar. Is trend following dead? Well when viewed from this height you can see it isn't.



The overall line of best fit generates a CAGR of 9.3%. The market conditions can in hind-site be attributed to central bank intervention during mean reverting periods, whereas the volatile market disruptions can be attributed to events such as Brexit, the GFC, oil shocks etc. The key to this diagram relates to how different strategies fair under different market conditions.

You can see from this aerial view (refer to chart above) that a single market condition such as a mean reverting environment can last for extended periods up to 10 years etc....however when conditions change, you get a massive disruption to participants where the prior equilibrium is disrupted and our mean reverting cousins become extinct.

If you only focus on the short term time horizon to derive your strategies, chances are you will skew your portfolio towards 'convergent styles' that bear far more intrinsic risk. They typically have high Pwin%, lower R:R but most importantly have negative skew which is a sure sign that they bear intrinsic risk far greater than what is revealed by their 'closed position' equity curves. If you had access to their floating equity curve....you would see this intrinsic risk in action.

Other strategic styles that have negative skew are Martingale and Grid Trading variants. The symptoms of this style of strategy is that their equity curves look glorious for a period of time....until they don't....and the account blow up comes without warning. You would think that you could turn off these strategies in time to save yourself from account blowup....but rarely is this the case as a degree of prediction is required to know when to turn them off.

Unfortunately when it comes to convergent strategies with negative skew (which work on the premise that market conditions are stable and price will revert back to a *known* equilibrium), the equity curve you experience during periods of exuberance when things work to plan is not the full story over extended periods where market conditions alter. That nice linear equity curve you are experiencing and the high win rate is a symptom of your strategy responding to the current market condition and this can last for extended periods. Sure it may be a great winner over these periods of stability, but you might be in for a rude shock when conditions change. The negative skew signature means that you frequently take small wins but occasionally take larger losses. When market conditions change you find that a small sequence of large losses is enough to blow your account. You will frequently hear the statement from a convergent trader that the strategy is a winner....(and then in small print) provided you can keep away from trending periods. That is because they are hiding the big picture. Mean reversion works until it doesn't and then it is usually game over unless you deploy risk mitigation methods that tends to interfere with those illusory straight equity curves and reflect the more realistic picture. Now this isn't to say we cannot deploy some convergent systems in a diversified portfolio as it is a means to improve cashflow reliability, but it certainly means that you cannot afford to overly bias your portfolio towards convergence.

For divergent strategies, the antithesis is true. Divergent systems are forward looking and do not assume that price will revert back to a historic point of significance. Rather their philosophy is couched in uncertainty itself and the philosophy assumes that the past is no reliable guide for the future. In fact they bet on the principle that price will not respect a past equilibrium but rather dislocate from the current market condition and reach for a new *unknown equilibrium*. As a result, divergent systems plan for the unexpected and work on the principle that most times you will be wrong in your future assumptions that market conditions will change....but while conditions remain constant you stridently manage your risk taking numerous small hits....until the dislocation happens where you ride the transition to the new market equilibrium until it lasts.......which hopefully pays for the past sequence of hits and quickly brings you out of drawdown. You will never know the exact destination of the new equilibrium but you open your system constraints to at least follow the price movement while it lasts.

So how much data is enough data? In terms of the philosophy of this thread, then there is never enough data. The reason for this statement is that we want to test if our strategies stack up in as many different market conditions that we can muster, as we are not concerned with how profitable we will be in the future. We simply want to ensure we have covered our bases and have a robust model that has survived the storms of the past and are more likely to survive into an uncertain future.

To harvest the non-predictable beneficial offerings of uncertainty you need to be able to participate in the unpredictable moment and that requires you to be present at the right time (when the market decides). If you focus on the profit side of the equation without paying heed to the risk side of the equation (which is something you can actually do something about) you may have a brief life of exuberance....but chances are over time that you will end up in traders heaven.

Unfortunately this philosophy therefore encounters a problem with over-inflated expectations. For a strategy to survive over the very long term, you need to prepare yourselves for a shock. To survive in the long term, a single strategy must endure horrific drawdowns when compared to the returns they deliver. Long term robust strategies typically have average annual returns half the size or less of their maximum drawdown. This ultimately caps the aspirations of traders seeking an immediate 'ticket to riches'.....but this is where so many pundits get it wrong. They send those strategies to the bin and system hop as a result, where in fact this was simply a realistic outcome resulting from the volatile signature of all stationery strategies.

So for example if you can endure 30% drawdowns then expect an average annual return of <15% per annum in the long haul. The relationship between Return and risk is important here as it is scalable with leverage (eg. position sizing). What this means is that double the position sizing and expect say a drawdown of 60% and return of <30%....but let's keep going into unrealistic land and assume we want a 50% average annual return. Sorry, you have just blown your account.

What this in reality land of uncertainty means (given that this is an average) is that you have to trade with small risk exposure to survive over the long term and this dictates your earnings potential but in conjunction with this premise is that to survive over long term trading horizons you need to expect being underwater for long periods of time. For those traders who have faced the long term, this symptom is one of the realities you need to face in your trading life. If you cannot accept this simple reality you are doomed as a long term trader.....however all is not lost provided you are prepared to diversify your systems, instruments and timeframes and this is where portfolio management comes to the rescue and will be a central feature of this thread that will be discussed.

Cheers

C

Hi C
Your posts are greatly amazing I am learning a lot from you.keep up the good work thanks

Cheers S :-)

When using data mining heavily in selecting the strategies, a bias comes in just due to large # of tests done. Explained in this paper and how to adjust for the # of test done when using data mining for generating signals: http://ssrn.com/abstract=2465675

another nice paper from the same author is https://papers.ssrn.com/sol3/papers....act_id=2308682 which tells you what to look for in a backtest to avoid pitfalls.

Your discussions are almost as good as many professional quant traders. Eventually everyone realizes that diversification is the only edge in the market

I think you will also like this blog (from ex MAN AHL PM) : https://qoppac.blogspot.com/

Many thanks STT. Heaps of useful content in the papers and the Blog. Very enlightening mate. I have some of Rob Carvers books and really like his mind....and his blunt sense of humour

All you guys doing is drawing hindsight maps of the trading gold mine, market does not work on hindsight

Model Assumptions and Hypothesis

This post will wrap up the background, hypothesis and core assumptions and then we can get started with laying down the processes used in this approach.

Our central assumption to the particular model is that given we are trading within a complex system (being these financial markets) where non-linearity and Power Laws apply and conditions are non-stationery, then the distribution laws we apply also need to be non-normal. In other words this model assumes that the history of different market conditions comprises periods of equilibrium (stability) randomly inter dispersed with periods of transition to different equilbria.

We are not assuming that this non-normality is a continuous feature at all times (as if it were then markets would always be in a state of transition), but rather that these disruptive transition events of different size and extent are irregular anomalous events in an otherwise efficient market context. In other words, at most times the market is efficient where a normal distribution fairly accurately describes a single market condition, but occasionally this semi-permanent stationery condition is punctuated by extreme events that reshuffle the participants of the system and the system's interrelated parts to create a new quasi steady state with a different behaviour based on principles of emergence. This is what adds the fat tails to an otherwise normal distribution of 'efficient markets'.

It is these 'unpredictable transition events' where this model makes it's money and the model assumes that these events under wide diversification occur with sufficient frequency to pay for all the random and cost drag effects from trading activities arising in more efficient stable times. Just remember a central premise of this approach that pitches it in a slightly different context to other Data Mining Models.

The key focus is on creating an unsinkable raft through the use of data mining methods. We avoid the temptation to make assumptions about the protected profitability of the return stream of each strategy as we globally worry about this at the portfolio level....but rather, if this return stream provided by a strategy serves to make the portfolio more robust in total by offering uncorrelated risk benefits (not return benefits) and it can be left on 24/7 while we do other stuff and be less concerned with future uncertainty that our competition.

Any returns generated by each strategy will simply be associated with the fact that we can outlast others and take advantage of less competitive 'extreme situations' where alpha lurks and delivers out sized returns to the survivors.

You may already be familiar with the symptom you frequently find in your P&L statements. When you drill down into them, you typically find that the monster returns and the monster losses can be captured by the top 5% and the worst 5% of trade outcomes, namely the outliers. The balance is where the churn occurs. Well we simply position our approach to protect our capital during the churn and make sure we are available to participate in these 'fat tailed' outliers of uncertain extent and duration.

Given this philosophy of a quasi static moving feast, we therefore anticipate that if we apply a divergent portfolio solution to this model, then there will be extended periods of grind where we will struggle to keep afloat and over extended periods of stability suffer building drawdowns. We therefore need to cap our adverse risk exposure to the long tailed distribution by applying prudently placed stops and trailing stops, but leave the positive side to the long tailed distribution as open as possible to occasionally take a windfall profit from disruptive market events. This in a nutshell therefore defines the broad configuration constraints of our data mining efforts where we will focus our attention.

A good test to this hypothesis will be to examine the history of portfolio returns of this solution against notable market crisis where this model should outperform whereas during 'normal' stationery market conditions, this model will suffer the same fate as many FM's in the divergent space. Namely a building drawdown profile and sad tales to the investment community.

If we find however that we outperform during more stable market conditions, then despite the outperformance, our model has failed its test and it is into the sin bin of failed models or we could assess the model to understand why it performed well despite our hypothesis and then potentially change our hypothesis to suit. If we find that we under perform irrespective of the market condition, then we have really stuffed up in our system design as it bears no resemblance to the over-riding market condition and this will be a signal that the problem is a design fault.

Now while proving that the model is valid is a noble academic exercise, our over-riding objective is to be successful over the long term and build our net wealth, but what shall we use to define our success? In this regard we need to look for industry benchmarks. .....so the ultimate arbiter to the success of the approach described in this thread will be how this particular model compares to an industry standard comprising the systematic divergent portfolio managers which I monitor currently.......so watch this space.

Anyway....enough of the jargon and the background, it's now time too roll up the sleeves and get into the workflow process of this approach.

You will find that the majority of our effort in this approach is based on initial work processes as opposed to that encountered by the discretionary trader. Once the hard yards are done you trust in your rigorous processes and simply keep away from the screens and the news. Your automated data mining processes are always occurring in the background (with the ticking sound of your PC having to work overtime), while you find time to appreciate other pursuits in life.

Victims of a long trading career tend to be people who begin to detest watching trading screens. They realise that the game is won over years and not weeks or months and as a result, their trading psychology is far less stressed than the average pundit. A losing streak of many months is taken in your stride as you know that you have covered your bases and any damage will be minimal and that one day.....when you wake up and check your balance, their is a windfall sitting there while the rest of the world is howling. No predictions of the future, no wasted hours of time ogling over trading screens, just a reliance in the science of the non-linearity of complex systems.

Stay tuned

C

PS If you are interested.....some useful background info vids that have relevance to the approach described in this thread.

I like Rob's style and content as well. Very accessible and nice writing style.

I also like Marco's work - he has built his quant firm (earlier inside Guggenheim's, now independant) from scratch. I have seen him in few conferences and he is also very easy to talk to. Very smart person and also someone who keep applying new techniques to markets (he is Erdo's #2 after all http://www.financial-math.org/blog/2...rs-in-finance/).

He has published extensively - check his papers @ http://www.quantresearch.info/. you will certainly find new ideas.

subscribed

always love to read new ideas

Many thanks STT

Since it is clear that you are trying to engage someone in a discussion, Yes, markets Do work on hindsight.

The market is not some mythical beast. It is the reflection of traders. This means that traders will remember historical prices.