Systematic Portfolio Diversification - Data Mining Concept

I once tried to set a screw with my hammer. I failed. Hammers don't work. A friend of mine told me to use a screwdriver instead. Man, you can't even put a simple nail with that! Screwdrivers are even worse crap. And please don't tell me about this saw thing, nobody reported he could set a screw or a nail with one. That's the definitive proof that all tools are useless!

(and brushes do repaint)

Nice work AD. How do you deal with extreme trending scenarios with your mean reverting strats?

Did you try the free stochastics or did you buy some price action courses from charlatans?

I dont know man. I know it is not imposible as the matter of fact and theory.
But it sounds crazy and almost impossible in practice.
LOL. Good saying Satraderma

The idea is to adjust the mean to the trend as you go; sooner or later it will catch up with the price

I also use a variant of PSAR which does the same job as above; the longer the position age; the shorter the exit trigger gets.

Holding period averages from 1m to 2 hours. All positions close at EOD regardless of their profitability.

Note on Forex Educators:
==================
since nobody asked me why would I check out forex educators. Eventhough most of educators don't actually trade; but they do have some good ideas.
I don't care if they trade them or not.

Kinda like mbrown; if I hear any idea from anyone; that sounds interesting; I will throw it onto my backtester; and get it massaged left and right; till I squeeze all the juice out of it. If none to be found; well; throw that piece of code away; so it doesn't clutter my database of EAs.

Pssst...you need to check out the indicator JJN-Fibo.

Is it a indicator renamed?

No. Google it and it will take you to the indy.

I must be missing the actual meaning of diversification.... if all the strategies are mean reverting, how can they be diversified? If mean reversion of "price" over to mean, wouldn't all indicator based mean reversion result in similar confluence on and around the same time/price/volume or whatever you use it to measure.
Not being combative Like I have in the past... just aligning my understanding of what true diversification should be.

Hi V

Good to have you here as you make the grey matter churn and you raise an excellent point. :-)

Diversification is a concept that is applied to divergence and not convergence.

Mean reversion is a convergent approach based on the principle of finding a stable solution that converges to a future predictive estimate and that the market condition is predictable (a nice stable signal with little perturbation aka noise). The assumption is that you know with a degree of certainty a future point around which price will oscillate. When it is currently far away from that estimated mean then it is predicted to converge to that known future point.

You therefore apply a mathematical model to oscillate around a mean like a Bollinger Band and wait for a price excursion away from that mean average and rely on repeatability for the short term that it corrects back to that central mean with probably overshoots so over time it oscillates around this mean ....again and again and again.....delivering the convergent signature of high win rate but relatively low Average Win to Average Loss ratio. You need to be lenient with any stop condition as timing the event is uncertain and you need to give room to breathe...but soldier on despite any rising unrealised loss....with the belief that in the short term it will follow your prediction to a known point where you get out with profit. That leniency is where intrinsic risk builds if it doesn't do what was predicted.

That nice steadily ascending equity curve is only representative if that market condition holds. As soon as that 'known' future point changes, then you go backwards and quickly as you automatically guarantee a string of heavy losses if you cannot identify the changed condition and adapt your mathematical model to capture it.

Intrinsic risk is held within a convergent model and the intrinsic risk lies in the statement...."what if market conditions change and are not like today?" Many of us will have all seen the movie "Margin Call"....well that firm (an alternate end to Lehman Brothers) held intrinsic risk as the bright rocket scientists in the firm developed mathematical models that concentrated risk within historical limits and provided that the markets respected those constraining limits applied by the model, then they would clean up.......but....and this is where the fat tails come in......what if the market did not respect those applied mathematical constraint limitations? Hey Kevin Spacey...it's time for you to do your thing.......we were wrong so get your guys to flog this shit away and transfer our intrinsic risk exposure of gigantic proportions to our client base. Pay them well for the day and then fire their god damn arses because we now recognise divergence is coming and our models don't work so well in that environment.

Sitting on the other side of Convergent Managers are the quiet Divergent Managers. They are the ones who predate on this ego centered tom foolery of predictive certainty. They are uglier in the boxing ring and less precise and coordinated and they have been hit with many small blows for sustained periods of time, but deliver king hits when the convergent boxer is forced to alter his recipe. There is less of this type of participant in the market but also a far less competitive marketplace when markets **** themselves periodically and this is why these non-predictive safety Sam guys who simply follow price last longer.

Most pundits apply a sense of deterministic causality to the future market condition and that they can predict that emergent future condition based on their understanding of how markets work....however despite this 'Talent' as L and Einstein would say, they are in for a rude shock if markets decide not to play this game. "God (aka the markets) doesn't play dice" is another statement Einstein would say. The markets behave like they behave but our models of how they behave are the things that are lacking.

Apply a successful mean reverting strategy (predictive model) in a stable market condition and you are very successful. There is a 'single' convergent narrow range of solutions to a stable market condition (ie wanders about a single mean estimator with little variance). As a result you get that nice straight progressing linear equity curve rising to the heavens where noise slows down your rapid ascent but the overall signal of the condition makes your bread and butter. Apply other mean reverting solutions (mathematical models) developed for different market conditions (around different means) to that single market condition, and they spectacularly fail (a nice straight equity curve to hell at a rapid descent).

In a single market condition, if you diversify by developing a range of successful different mean reverting solutions that all converge to that same future estimated mean, they are all just methods of converging to the same underlying mathematical model (effectively different symmetrical models of the same mathematics). They 'concentrate risk' around a known estimate.

Across a range of different market conditions typified by a range of different means and separated by market transitions (imagine a stepped market profile of different means separated by transitions) if you apply a narrow range of single mathematical solutions in a context where you need multiple mathematical solutions, you will fail spectacularly....hello LTCM :-)

Diversification works as a means to spread risk (divergent risk solutions to a non-predictable future outcome), whereas risk is increased if applied to a convergent approach (converges risk solutions to a predictable outcome). That's why convergence is typically a game not played by diversified managers. It is a game for the quants and engineers that like predictive certainty. They are specialists in a narrow range of market conditions.

So spreading your basket of eggs is a good recipe for achieving a range of different underlying mathematical models designed for a range of 'unknown' future conditions, but a bad recipe if that recipe simply comprises a single mathematical solution for a range of future 'unknown' conditions.

Convergence is backwards looking and predictive (history is the same and tomorrow will be like today) whereas divergence is forwards looking and non-predictive (history is not the same but it rhymes).

How frequently do markets **** themselves and move away from predictive certainty where outliers matter? Answer..... a few times but you cannot predict the whens? How frequently does the entire financial system **** itself based on the inter-relationships that exist between different asset classes. Answer..... a few times but you cannot predict the whens? A convergent player disregards these inconvenient truths of non-linear complexity and slogs on regardless as their equity curve accelerates and they become part time gods of predictive certainty. They back themselves up with a statement like "This system works I tells ya and I am giving you the golden goose (small print coming) but you just need to find a filter to turn it off when market conditions change". This entire meaning is pregnant with bias about an aspect of certainty....aka knowing how markets behave with precision.

Divergent guys are dumber than these guys and can't predict 'shit' but they use simple methods such as system design constraints and diversification to ensure the shit doesn't stick. :-)

Nick, I am confused when you speak of convergent diversification. It may be a matter of terminology. In my mind you are trying to go up the road of LTCM, this would mean you will make a lot of money for maybe quite a long time, but at some point, well.....you know the rest. I am trying to understand why you would not spread risk among many non-correlated asset classes. Is this incorrect? Would we not want our portfolio to be at least beta hedged to some degree?

Edit, okay I read again your thesis statement. You did forsee the LTCM issue. So, your main quest is to defeat this problem. The only way I can see to do it is be beta hedged and manage the portfolio in a weighted fashion that will move money towards performing sectors and asset classes and to decrease dead money. Easy to say, crazy hard to do. In theory we would need to find those laggards among leading classes, and know when to pop impending bubbles. Not an easy feat.

Hi J :-)

I think the riddle lies in your statement mate. They key is to not bias your portfolio to a single solution. When you make a statement "that I need to move money towards performing sectors and asset classes to decrease dead money"....that tends to imply a preference in the way you allocate towards a portfolio which is an implication that that your performance expectations are too weighted towards a convergent philosophy. For example the implication is that you need a degree of prediction involved in the way you 'best' allocate your capital or you have a preference for allocating capital towards performing sectors. I hope that makes sense mate.

I know that all this talk of divergence and convergence is probably confusing to many....but when you get into it, it really helps to define where your trading psychology is placed and what defines your view towards how markets behave. It takes a massive mind-flip for the biased convergent mind to come to grips with divergence. I have taken years to adjust my mental models for divergence as they were deeply embedded and ingrained in me. You will find that while you think you appreciate what divergence is about you will still be racked with convergent methods to solve the problem. It is incredibly hard to let go of predictive certainty as we are just not well equipped to handle it. We have heuristically evolved to the market condition we are used to and hence everything about us including our language is couched with predictive determinism. Even when we talk about our understanding of divergence we frequently find that lying in that discussion is a convergent mindset still lurking.

The term divergent is not necessarily synonymous with the trend trader. For example a trend trader that waits for retracements before entry with the conviction that price will go back into trend is a trend trader applying a convergent recipe for entries. There is a degree of prediction involved in their modus operandi.

The result is that they will be right most of the time and hence during more stable smooth transitions they will outperform their divergent trend trading buddies, but the times they are wrong and no retracement occurs will be missed opportunities that could be massive positive outliers.

Divergence is about being present to participate at all times for the unpredictable potential outlier. You cannot be in the market at all times as you will suffer death by a thousand cuts from stationery conditions but you can at least define non-normal conditions in your design as the point where you commence participating. Of course you are still wrong most of the time as you sacrifice a degree of certainty and immediate gratification for the possible divergent euphoria that uncertainty brings from time to time...all by it's very nature. We just ride the condition. These divergent conditions are massive disruptive anomalies where the Pareto Law applies and you find that the Divergent community get the 80% of returns lost by the convergent player. This more than compensates the battered crushed ego for the damage caused by the notion that we are simply not clever enough to predict the market with certainty.

Uncertainty is a consequence of a non-linear relationship that is exceedingly difficult to crack. We are good at linear relationships but crap at non-linear ones.

For example you might hear a statement or two that fat tails exist in every timeframe. Well while anomalous skewed data certainly does occur in all timeframes, the non-linear Power Laws compound in their effect towards the higher timeframes as the impact on non linear transitions scales up in magnitude from the small sample to the big sample. Convergent strategies have better 'scaled' bang for buck towards the lower timeframes as your trade frequency is far higher than larger timeframes and you get a far larger slice of the available convergent pie while conditions hold. The contrary is true for divergent strategies and the need for their emphasis to be directed towards lower trade frequency and longer timeframes. This allows for a better chance of obtaining a bigger slice of the fat-tailed pie.

If you think about participant behaviour and the total mix of participants in the financial system this makes sense.

An anomalous blip on the 1 minute timeframe will certainly impact the finite number of participants whose strategies are 'convergently' scaled for that timeframe but is unlikely to transit to the higher timeframes unless the non linear impact is so disruptive that it starts to exert it's impact across a greater number of participants who are 'convergently' scaled towards the higher timeframes. It starts as a seed and the disruption starts to expand from a small finite population to an ever growing participant base in a non linear multiplicative Power Law way unless that disruption can be quelled by either the market participants behaviour themselves or the regulators actions.

When it gets extreme it starts to exert its impact beyond the single market after commencing as a seed on the small timescale and rolls into a causal 'domino effect' as the 'convergent scurry' towards a defensive posture starts requiring the liquidation of other investments in other sectors that create different and new correlated relationships (a coordinated rhythm of participant behaviour to create the transition) that previously otherwise was not the case. Equities during normal bull market phases are relatively uncorrelated but during extreme events, what previously had low correlation now has high correlation and you get a sea of red on the exchange boards. There is a giant feedback loop going on (thanks to derivatives, leverage and playing with pretend money) while this occurs that magnifies the impact and turns the problem from a linear one to a non linear one and hey presto, that once stable market condition, or the entire stability of financial markets that were predicated on predictability turns on it's head.

Interesting. I see that you close all trades at the end of day. Does that mean you also stop opening trades past a certain time of the day?
Would the closure of all trades intraday be your way of capping risk?

V, you are welcome.

Let's not deviate from this thread's topic.

Diversification can occur at various areas; such as:
1- divergent/convergent (aka trend following, mean reversion)
2- timeframe
3- strategy (or edge)
4- parameters within a strategy
5- instrument
6- exit methods (same strategy, same entry parameters; but various exit techniques)
7- and many other areas

You may think that all mean reversion strategies are correlated; but actually they are not.

It all boils down to the correlation analysis of a backtest results.

In many cases; I have seen parameter diversification to produce strategies that are not correlated; and hence cancels each other out.

Here is a case study of a single strategy with 12 parameter diversification variations; and their correlation matrix.
The backtest is done from January 2011 to May 2018; using real tick data.

chart 1: cumulative equity curve (max drawdown is 8%, longest DD recovery is 45 days)
chart 2: individual curves for the 12 strategies
chart 3: correlation matrix.

Notice how most of the correlations are near the 0.0 axis. Some are highly correlated but most are not.

C,
Let's not deviate from this thread's topic.

My time frame of trading is intraday. I can apply my strategy to multi day; but I choose not to. There is higher success rate intraday than higher time frames (OP mentioned this few times earlier).

I would say that 99% of my positions are closed before EOD; using various exit techniques. Those that did not close; will be forced to close.

Regarding trading time of day; yes; I do have selective time. I usually trade from GMT 00:00 (opening of Japan market) till NY close.

I do not trade in between.

Nice info A. I would love to get my head around the context of your strategy a bit better but recognise you need to keep some info under the hood....but great food for thought

C,

the context is same what I indicated in few of answers earlier in this thread.

the mean reversion is on intraday timeframes; mostly 5m to 1m and even ticks.

the entry techniques are not really that special. they are publicly known by many (i.e. such as bollinger bands).

Checkout the thread by Alphaomega; "Simple Mean Reversion". that is one strategy that I trade a variation of it profitably for few years already

The secret is really in diversification, and also on how to avoid trendy days!

Cheers mate. I will check out Alpha's thread and have a good read :-)

PS I am luvin' this discussion at the moment mate. Not many want to enter into diversification. We are touching on some great issues here. Let's hope we can translate it into a workable solution with this project. It might have far wider application than I anticipate.

Hi,

We are some distance from the subject of reoptimization so this is a post for a pending discussion lest I forget. Basically, I'm really quite fed-up spending hours over the weekend re-optimizing certain strategies so I've been looking for a real-time auto-reoptimization solution. Fortunately, I've found one which I'll be trialing, I'll keep the thread updated whether it is worth the investment:

https://iticsoftware.com/en/mt4-autooptimizer.html

Another subject worth investing in is parametless system. it is in effect the same as walk forward optimization (wfo); but much more stable.

I found WFO not stable over time.

https://www.reddit.com/r/algotrading...ss_strategies/

http://epchan.blogspot.com/2008/05/p...odels.html?m=1

http://epchan.blogspot.com/2008/08/m...model.html?m=1