Systematic Portfolio Diversification - Data Mining Concept

Show us proof, so we can believe your beliefs .Show us statistical analysis of historic data performance versus actual results of automated programs. Have you any ?

L. You will note that this is a proof of concept. I will lay down all the details of the EA's we use over the next few years and by the end of the story we will have a real live demonstration that is validated by all that participate in this thread. At the end of this project you can then say to me....what a waste of time C?...if only you had listened to me in the first place....but unfortunately that is not how I operate. I learn through the school of hard knocks. How about leaving us be for a bit, or at least put your constructive and positive 2 cents into this project. I don't want this thread to run off the rails. I hope you appreciate that.

I am not aware of other data mining projects that purely work on the risk side of the equation on this forum and I think this subtle twist may be where the issues lie in that it avoids the tendency to over-fit results by utilizing risk-weighted metrics and portfolio scaling which appears to be a central dilemma to more traditional approaches to data mining from my personal understanding.

No, I do not have any proofs other than seat time. Historical prices are what support and resistance is about. When you get enough seat time you will see this as well.

Personally, I do not understand why trading is made so difficult. The act itself is not hard at all, it is the clicking of the mouse. What is hard is that traders want certainty in a random environment. This is not possible. If it were, there would be such a thing as a perfect trade, and we all know that cannot exist. No one would take the other side.

I have been reading Nick for many years and he very generously tells anyone that will hear that that diversification is one of the only edges any manager of money can ever have.

Ultimately, no one can show you proof of anything related to the market because every trader has their own cognitive biases.

Here is my contribution to the thread.

Cheers J. Luv'd the vid :-)

Another problem is that the green dots always redden at some point. Say you find the wiggling line the video shows. You test it by adding the green dots. You realize the model is terrible. You throw it away. Another model is found. You test it by checking with the green dots. You realize the model is better but still bad. You throw it away. Yet another model is found. You test it by checking with the green dots. You realize the model is pretty good... Do it a few more times and just by ramdom chance a model will outperform all the others at fitting both the red dots and the green dots... All the green dots are now in-sample (red).

Good to have you here Pip :-)

I am afraid by the end of this experiment; Loozers would have blown his many accounts; and will serve this thread from his heart.

Guys....will be back next week to continue. :-)

Cheers

C

Good points.

I have had my fair share of data mining, backtesting data mined systems, forward test them, walk forward test them, put them through Monte Carlo stress tests, etc. At the end of the day; automatically generated systems tends to fail miserably in live trading.

In my opinion; it is best to base the system building on human thinking heuristics; that way you have more faith in the system; and you narrow your focus to the edge generating ideas.

After an edge is determined; one can use various optimization techniques to improve upon it. Add various filters, try it on various markets, generate portfolio of diversified strategies, etc.

Other ideas I found useful, is to use tools such as Strategy Quant or AdaptiveBuilder to get some ideas from the data mined systems; and add human touch to it through R&D

Kind of Toby Crabel style.

best

Some of my best moments in trading have come from dumping raw data into excel and using the data analysis add on to clean up the data a bit. This way I know what to expect. I have an idea of what sustainable vs. unsustainable moves look like, so if I get into a screaming nearly vertical trend I will add at pre calculated points with lighter position size as the rapid trend progresses. WD Gann was on to something with the angles he used, he just went off rails when he got into the numerology and astrology stuff. Same will J. Welles Wilder. The RSI, ADX and PSAR was really thinking outside the box in the 70's, but then he became a mystic also and did all of that Delta stuff. It is seems that good old fashioned math is not really in style when it comes to this business.

Good points guys.

However what we are trying to do here might not be immediately obvious.

Where I am attempting to take us with this project is towards a collection of non-correlated divergent strategies (that are asymmetrically constrained) to prevent downside risk in a non-stationery market context and simply leave us open to potentially unlimited profit potential. It is not mandatory that divergent strats are used but if you use convergent strats they must be capped....which makes finding good ones a tough exercise. We are not looking for linear projected solutions into the future which is where curve fitting bias presents itself with the expectation that the projections are an extension of historical data. The reality of this market does not operate that way. (eg. I do not expect a trend following strategy to continue in a linear ascent as the trajectory is dependent on the market condition.).

One of the critical requirements of a data mining exercise looking for divergence is you need a wickedly big sample size that is far beyond the 200 recommended sample size of a Gaussian distribution. For example to observe the Pareto Principle at work (a common characteristic at work in complex systems) associated with Power Laws you will not detect it's presence with a restricted sample which will approach a Gaussian distribution. The impact of the fat tailed events while exceedingly few and far between but exceptionally powerful in their ability to shift any notion of an average or mean require extensive sample sizes to have a hope of including them in your estimate and deriving a far more realistic long term expectation of market behaviour. In the world of fat tails, the reality is that there is not such a concept as an 'average' which Taleb would refer to as "a bogus estimator (eg. use the wrong estimator and you get the wrong answer). You need extreme value theory which lies beyond the normal distribution and Central Limit Theorem. Another great quote from Taleb is that "a Power Law can be disguised by a Gaussian distribution in a small sample but a Gaussian distribution cannot be disguised as a Power Law"......in other words, sample size matters and the bigger the better.

So this exercise will be demanding as much data as we can get our hands on, which automatically reduces the ability to 'curve fit' arising from a random representation of the underlying data especially when we apply Monte Carlo sensitivity tests to the key variables (SPP). One of the problems however with large sample sizes comprising multiple market conditions is that variability in the equity curve is an omnipresent and a central feature. There are multiple moments arising in time around temporary basins of attraction as opposed to a perpetual oscillation around a single mean which would make the game easier and is the assumption of the convergent player.

It does not mean it is a bad strategy in the divergent space to have a volatile equity curve as this is a consequence of trading 'between' market transitions as opposed to the single market condition itself. It only is relevant if you are trading convergent styles which is a representation of what you achieve within a single market condition. In the divergent space, you do not measure success in the nature of the linear ascending equity curve at the strategy level....but you do this at the global portfolio level. Your success at the underlying strategy level is measured in terms of how long the strategy has survived in relation to it's overall risk-weighted return metrics whose signature is defined by the entire equity curve as opposed to a single statistic.

If I have a collection of strats to choose from, my selection decision is not made through such metrics as stagnation, profit factor, sharpe ratio, system quality number etc based on the individual system metrics as these all bias the result towards convergence....but rather I focus on how that system contributes in terms of risk weighting in a portfolio context. I compile the portfolio with puzzle pieces (each strategy's entire equity curve and not a single correlation number or metric) which assist in plugging the risk weaknesses across the entire time series of the entire portfolio. In the divergent space it is all about risk management where long term profitability and sustainability is just a symptomatic consequence.

So we are not looking for test results that possess a high profit factor or return result which I think is where the problem lies attributed to either a curve fit result or the failure to address that market conditions are adaptive in nature and not via a linear relationship. We are looking for a collection of volatile but robust divergent strategies that under a diversified portfolio possess different risk signatures that collectively smooth the volatility of returns of the portfolio by addressing stationery market conditions yet allow for abnormal returns due to each systems asymmetrically open ended nature when the market decides to dislocate from it's prior stationery state on a path to a new equilibrium.

There is no steadily rising equity curve as during periods of convergence you will flatline *hopefully with limited capital deterioration which is a major aim of this project*. Rather the curve is more representative of a staggered lift where during divergent conditions we leap. This curve should reflect 'lifts' when overall market conditions are volatile and diverging away from prior stable equilibria versus 'stagnation' when markets are converging towards historic equilibria.

There is an issue with diversifying into different markets or diversifying into different timeframes as correlations in these two methods are moving feasts. What is correlated this month may not be correlated next month. We have seen this in the last decade where traditional non-correlated assets such as commodities versus equities versus Forex wandered between an uncorrelated relationship 10 years ago to a correlated relationship in the last few years.

However system design is a far more preferable method for diversification as in applying a stationery design solution, the correlation relationship between systems in your portfolio remains permanent....however at the portfolio level itself, you need to keep an eye out in relation to the correlation between portfolio performance and the market condition. Some guys refer to this as 'market internals'...but what this is is simply a broad representation of the market condition that you might use as a filter or risk weighting mechanism in your portfolio solution.

Because we are using stationery solutions in a non-stationery market context, you need system diversification to obtain the required correlation profiles to collectively flatline the composite. This project seeks to apply this 'design' principle across markets and timeframes to apply overkill to 'capping risk' yet allow a sniff of market turbulence to take the portfolio off into the stratosphere.

Anyway, we will get into it next week as I have a bit to put together first.

Cheers

C

Cheers H :-)

Could you please space out your data mining to allow for patience , great currency traders trade only 4 to 5 times a year.

Bill Lipschutz"If most traders would learn to sit on their hands 50 per cent of the time, they would make a lot more money."

Psssst. LTCM (despite the nobel brains) was a convergent player (not a divergent one) evidenced by its tumultuous crash through it's exposure to predictive convergent arbitrage plays and Buffet despite his convergent fundamental value investing approach might disagree with you about diversification....but apart from that I hear you RR.

This is about divergent plays and this occurs during market transitions (when folks tend to scurry)...not all the time hence the ride will be a bit bumpy....but hopefully powerful *crossing fingers and toes*

.

Yep L....that will be part of the recipe :-)

How about Jim Simmons from Renaissance Capital; the hardcore quant who does not allow any trader to touch the buttons. All computers humping dumping trades in and out all day long.

He is the most successful most profitable hedge fund in the history of hedge funds.

If you want to trade discretionary; you should not use any software made by quants. You should get your prices by phone; or by walking to the exchange every day; and writing price in piece of papers. No calculators allowed. You must trade manually all the way.

you need to take a bus and be there in 1 hour; just to get the price; and back home in another hour. then drive again to place the order; and pay cash at the register; and get paper slip.

The summary of the above post; quant trading is not about eliminating humans; it's about automating routine stuff that you can get away with.

Some quants automate part of the system; and some genius ones automate most of the parts. But they need to watch the algos humping dumping; they won't leave them alone.

Hey C. Superb Stuff!
Work. Music.

Lekker music

Nice J. I hope all is well mate :-)