Systematic Portfolio Diversification - Data Mining Concept

This research thread takes us into the world of Data Mining and explores an approach that seeks to compile a portfolio of divergent strategies that are robust in terms of their ability to navigate a broad range of market conditions and have a track record that demonstrates a small edge.

The intent of this process is to put together a powerful risk-weighted portfolio of non-correlated strategies that will ride the storms of uncertainty without significantly eating into the capital base but are positioned to reap powerful returns when market conditions become extreme.

This thread is a sister thread to the thread titled 'Diversified Trend Trading Approach' and complements the portfolio principles described in that thread whilst applying them to a fully systematic method of application and uses data mining methods as a basis to ensure that the market data speaks for itself and that the propensity for curve fit results or subjective design considerations from human biases are kept to a bare minimum.

This thread is furthermore a research project and proof of concept. It is a living research project that I will be testing over approximately 2 years of live walk forward conditions.

Standard thread rules apply. I assume that participants are out of their nappies, have parked their egos in another thread, are grateful to the host of the thread who is freely sharing research and sharing ideas, have a sense of humor and are genuinely positive and pleasant contributors.

So looking forward to making a start and giving an outline of the project.

Stay Tuned

Cheers

C

Some Tools for the Project attached.

The current versions are Ver 1.4Attachment r so please upload if your version number is different.

1. Data Mining Database. It contains sample data just to give you an idea of the overall structure to the dbase and how to use it. Instructions on how to use this will be provided during the course of the thread.

It operates under MS Access 2013. It is freely supplied by me so I will only be making future amendments if they suit my personal purposes. In other words, this is not and never will be made to meet everyone's tastes and is simply used for the purposes of this project. Don't expect fixes or amendments in other words.

The database is used by me to house my research and compile data for further analytical treatment in MS Excel. The assumption is that it may be useful to some of you. Others will have their own solutions. Further MS Excel Models which are used in conjunction with this database to import MT4 and FX Blue data into the dbase and also to analytically crunch and compile treated portfolios will be arriving shortly.

Please note that EA's generated by me will be included in this database. We need to keep very organised and the inclusion of the EA in the database is mandatory as it links to associated portfolios, strategies and models......So if you want to use these EAs yourselves, then you will need a copy of MS Access 2013 or later.

2. Import Spreadsheet for DBase. This tool with sample data provided allows us to quickly capture info from either MT4 strategy tester (for backtest results) or FXBlue for Demo/Live trade results and convert it into a defined format for quick acceptance into the Dbase. Guys I am working with the following date format dd-mmm-yyyy so sorry if that doesn't suit. You will need to amend formulas yourselves in the spreadsheet and dbase if you want to use another format.

3. Portfolio Analytics This is a template version only with some sample data which we will be a major tool used for this exercise and utilises data exported from the Dbase. This spreadsheet template will need to expand as portfolio size escalates and when it gets large it sucks PC resources so I have had to provide a small version to start with as a full blown version is too large to provide as an attachment here. You will therefore need to expand it yourselves as your data mining venture and your DBase grows but I will show you all how to do this as part of this thread.

That's it for the moment guys. These are the 3 predominant tools we work with in our data mining project. Just keep your eye on the version number as there will be updates over time with new stuff.

I will also work with these tools as a fresh start so we can all keep track together.

Enjoy

C

I'm willing to contribute the core algos for this project if you're interested.

What do we mean about Robustness in terms of an overall Portfolio of Strategies?

Robustness in terms of a strategy's contribution to an overall portfolio of strategies is a reflection of the ability of a single strategy to survive a broad array of varying market conditions without detrimentally undermining the overall positive performance of the portfolio and its ability to also offer a positive contribution to the overall portfolio. In a nutshell, what we are looking for are strategies that offer a balanced risk-weighted contribution to the overall portfolio.

What I mean here is that for a portfolio to be robust it needs to:

1. Offer a general positive contribution to returns over it's lifetime, which is attributed to the ability to collectively derive a solution for the entire portfolio that has positive momentum at all times. We know that different strategies under different market conditions vary in terms of their over performance and under performance. We therefore need a solution that allows for a maintained overall positive contribution to the portfolio....and for this to occur, we need strategies that are uncorrelated in nature;

2. Ensure that no single strategy contributor offers undue poor performance on the overall portfolio at any particular time or during any particular market condition to unduly cause under-performance to the portfolio at any point in time. Ultimately given the adaptive nature of the market, you need to define conditions where you 'turn off' strategies that are unduly dragging down the performance of the overall portfolio. For this to occur we need to establish risk-weighted benchmarks that identify when to 'turn off' those strategies that no longer meet the minimum performance benchmarks of the overall portfolio.

If you think about this requirement closely, what this means is that each strategy contribution must:
1. Be uncorrelated with the balance of the portfolio. The more diversified your portfolio in terms of systems deployed, the greater ability of your portfolio to survive over the long term;
2. Offer a small edge to the overall portfolio. For this to occur, each strategy must stack up in their own right over an array of different market conditions and that the volatility of returns of the overall strategy must have a positive trajectory;
3. Not unduly influence the overall portfolio when in drawdown. For this to occur, your position sizing applied to any single strategy must be a reflection of it's overall risk-weighted contribution to the portfolio and there will be a time when you must call it a day for a particular contributor that no longer meets long term performance benchmarks.

So how do we go about selecting robust strategies for inclusion into the portfolio?

In essence we are looking for strategies that offer:
1. Excellent risk-weighted performance metrics over as much data as we can throw at the strategy....which is another way of saying that the strategy must have performed well over a very broad range of market conditions. For this data mining project we will be using data available from pre-GFC to current day.
2. Robust strategy parameters (Variables) that are not curve fit but rather demonstrate that the overall strategy itself (as opposed to the individual variables selected) are robust general purpose solutions to a broad range of market conditions. We will be using strident Monte Carlo testing of strategy parameters to sort the chaff out from the wheat.
3. Strong performance results in out of sample periods on their native platform leading up to their live implementation.

This is what we are looking for at the portfolio level in terms of the overall result of compiling individual strategies.



What you need to understand is that this fairly steady non-volatile curve is not derived from convergent strategies (eg. mean reversion, grid trading or martingale variants) but instead is comprised of an array of divergent strategies each of which have much more volatile return distributions. The secret sauce lies in how you compile these solutions to produce the desired end result.

Once this is achieved, you then need to risk-weight each solution with adjusted position sizing to achieve this.



It doesn't look much different to the raw portfolio does it?....but there lies the rub..........look what you can now achieve when you compound it.



So in a nutshell....this is what we are striving for. To achieve this you need patience and you need to really understand what risk-weighting is all about. This is where we leave the world of retail trading and enter the shady world of portfolio management.

Cheers

C

Why not Mr B. Thanks very much for the offer.

I have about 60 EA's currently defined for Pepperstone that are pretty powerful under a portfolio......but the more the merrier. Once this thread get's cracking we can start sharing EA's or tweaking them as we see fit and compiling our own discreet portfolio solutions from our own personal choices.

Cheers

C

Hi,

How many strategies do you intend to have in the portfolio?

You've just answered the question. I'll check back later for more information.

Mate....any size is possible provided you run multiple MT4 accounts and have multiple VPS. At the moment I am running about 60 strats on D1 and H4 on two MT4 accounts off a single VPS without execution problems and use FX Blue to consolidate as a portfolio. About 9 trades a day in terms of activity on these two accounts.

We have two ways of approaching this. We can progressively lift performance benchmarks and therefore keep strategies down to manageable levels of <100 or we go wide with vast diversification..which is a bit self-defeatist. I would prefer say <=100 strats.

Congrats on the new thread, very much looking forward to the discussions that will unfold. Cheers!

Good ideas; suscribed.

Does this portfolio include all type of strategies or just trend following?

Morning A

Given that the nature of generating strategies will be through data mining techniques over large sample sizes that extend across multiple different market conditions, the majority of strategies will be of a divergent nature, namely those with a Pwin of <50 and R:R >1.5 and positive skew.

It is not that these type of strategies have been selected by choice however, but rather that only those strategies that meet the required validation benchmarks are selected. In other words, those strategies that are more robust in nature and can address multiple market conditions are naturally divergent in nature (refer to table below as an example). So....provided the strategy can meet the methods of validation and robustness testing applied, then any strategy is a possible contended for the portfolio.

The table below comprises 57 data mined strategies over a long test range. Aside from the requirement to possess an initial stop, there were no other design restrictions applied to the data mining process aside from the need to meet aggressive risk-weighted performance results. This design specification therefore was wide open for the potential inclusions of any type of trading strategy. Note that the overall nature of results for the $10K portfolio has produced a Pwin of 40.2% with an average win of $137 and average loss of $66. Importantly have a look at the extreme positive skew of 3.7. There were a few marginally negatively skewed inclusions, but by far the greatest contribution was from positive skewed strategies. This just goes to demonstrate that despite the more volatile equity curves of divergent strategies (eg. momentum/trend following), they are far more robust than their mean reverting/ counter-trend cousins over the long term.




What this exercise seeks to demonstrate is that the strategies are simply a means to an end and a method to provide the building blocks for a portfolio.

Some of the key assumptions of this approach that gives a slight paradigm shift in the way we might typically think about trading are as follows:

1. The data is king as opposed to any strategy itself. What this leads to is the notion that in reality, strategies are disposable and you should not get attached to them. If the performance metrics are no longer being attained, then out goes the strategy with no hope of resurrection.

2. Given that the market is adaptive in nature, strategies that fail to meet ongoing benchmarks are dropped forever but new strategies are continuously mined and brought into the portfolio if they satisfy performance benchmarks and are found to be a good contributor to the overall portfolio. What this ensures is that your portfolio, like the market is always adaptive in nature and while lagging the market, responds dynamically to evolving market conditions.

3. That while the portfolio is adaptive, the test range applied to data mined inclusions always commences from the same point but progressively increases the data sample to reflect new and emerging market conditions. In other words, the robustness parameters applied to strategy selection are progressively increasing ensuring that more and more market conditions are included in the test range. What this does in apply a rising benchmark of robustness to the data mining process.

4. We do not focus on recent performance history and assume that more recent conditions are 'more likely' to be representative of the immediate current future. The assumption we apply to this exercise is that 'any market condition' lies just around the corner.

5. The entire process we apply here is one of 'risk management'. We have no control over future profits. What we are seeking here is to build an unsinkable ship that has overall positive momentum at all times and is left open to be in a position to turn black swan events into white swans. At the strategy level our emphasis is on risk management but at the global portfolio level when the hard yards have been done we ramp up our performance in 'relative safety' to get the best bang for buck we can with finite capital restrictions.

6. The market defines the order of returns that will be generated by the portfolio and not the nature of strategies included in the portfolio. The is a subtle and important assumption. What this assumption ensures is that you are not tempted to bias your portfolio (curve fit) to the most recent market condition by configuring your portfolio to only have the best performers of the current day in it's composition. If you look at the history of returns of any robust trading system that has navigated an array of different market conditions, the strong performance of the portfolio is determined by the market condition as opposed to recent market history. For example for a robust portfolio, you would expect that your performance arises from key market events such as the GFC as opposed to the everyday market churn of the current market condition (unless of course that the current market condition is extreme). If you find that your current strategy performance is amazing....while the current market condition is simply an average day.....then that is a red rag to a bull that your strategy is 'curve fitted'. More traders need to take heed of this nuance as it should be the market which determines your fate, not your sloppy strategy design. Ideally your strategy should float on the market condition as opposed to attempting to dictate terms to the market. Extreme examples of this are grid traders or Martingale Variants that refuse to take a loss....until it is usually to late. A key thing to watch for here is your unrealised equity curve against your realised equity curve. A significant disparity between the two signals that your strategy holds intrinsic risk and has the potential to have disastrous consequences over time.

The Data Mining Process to EA Development

We will be applying the following workflow process to EA creation. This is facilitated by data mining tools such as EA Studio, Strategy Quant or Adaptrade etc to generate algos supplemented with our own processes to hone in on possible strategy inclusions for the portfolio. While the tools above are methods to fast track EA development and allow us to continuously and quickly data mine for possible inclusions, it is not mandatory that we use them....so we can leave our portfolio open to other strategy inclusions that meet our required validation criteria.

I have spent a fair bit of time with EA studio and am happy with the outputs....however it certainly is not a mandatory requirement. The benefits of this solution is that you can quickly generate possible solutions without the need for programming and has a very good Monte Carlo analyser. The ability to generate thousands of strategies forces me away from having any preferred attachment to any particular strategy and therefore focusing on the performance metrics of the strategy as opposed to my personal attachment to them.

The proposed workflow adopts the following stages of testing and development.

1. Collection of test data and Ongoing Collection. We use our preferred brokers data plus data from another reliable broker source. In this instance I am using Pepperstone live data combined with Dukascopy tick by tick data. The reason for the dual data collection is that for robustness testing principles we have a cross check in place that ensures our data mined solutions are not victim to anomalous data from our preferred data source. The more valid and robust strategies work across platforms. Strategies whose performance results are specific to a single broker and not across others is frequently a sign of 'curve fitting'.

I use Dukascopy data so we can also test the robusteness of strategies developed by other EA generating methods on tick data. This is not to ensure that we get 'exactly' the same performance results as their is significant perturbation in data over lengthy timeframes....but more to ensure that under whatever platform we deploy our strategies, that overall performance continues to meet our defined benchmarks.

2. Define Strategy Parameters and Validation Measures - This is where we define the instruments (markets) we want to test, the timeframes tested on, the method of test used (eg. open prices or tick by tick etc.), indicators used, stop loss or take profit targets, performance benchmarks required etc. This phase of the exercise defines the universal constraints within which your data mining exercise will be conducted.

3. EA Generation. This is where based on conditions established in prior phases the data mining exercise commences and possible strategy solutions are generated ready for the Optimisation phase.

4. Analysis and Optimisation - If your strategy developer used random strategy generation then unfortunately a degree of optimisation will be required in addition to the preceding processes. While I am not a fan of optimisation, it unfortunately is a necessary process for random generators. For those strategies which have been derived from logic alone, but have passed the prior tests, then you can skip this optimisation phase. Unfortunately when optimisation is applied, you need to spend an extra effort in the next 'robustness test' step to undo the potential for curve fit results.

In addition to any optimisation conducted, this phase also includes the elimination of strategies that have exhibited weaknesses that has not been picked up by prior validation criteria. For example, you eyeball the equity curves of each generated solution and drill down into any possible anomalies that skew overall results.

5. Robustness Testing - This phase applies rigorous Monte Carlo testing across strategy variables to ensure that the resulting passed solution is unlikely to be curve fit and also is a method used to ensure the successful strategy is robust in nature and capable of performing well with data perturbation. We also test each resulting strategy across multiple data sources as a final validity check. In this case, I cross-verify performance metrics using Tick by Tick Data from Dukascopy.

7. Live Walk Forward Incubation Mode (Out of Sample - OOS) - For those strategies that have got through to this stage, this is where we run these strategies on your native platform under a demo environment for a period of 6 months to ensure that:
a) They are free of any possible execution error that could unduly influence your live portfolio; and
b) Performance results continue to meet existing performance benchmarks using OOS data.

8. Live Portfolio Application, Risk-weighted scaling and Performance monitoring - Now comes the live implementation where we start to apply some advanced portfolio management tricks to ramp up the performance metrics. All of our previous effort was placed on risk management....but now we get into the land of delivering very ambitious performance targets at the global portfolio level once all the hard stuff has been done.

Stay tuned guys as we will progressively drill into the detail of all this.

C

I love your post @Copernicus I really like what you share.

Subbed.

Some edges can only be learnt with experience, since developer is looking or specific edges, that make an overall edge, data mining has limited benefits on the performance in the future.This is all bollox , if market in the future behaves totally differently.Real edges are learnt by trading for real in the real market.

Reality check!Only mathematical models work in the future.

Most back testing is useless , if market behaves differently in the future. Most back testing is useless , if used for manual trading, humans have minds that behave differently in execution .Most manual traders have a profit gap, the difference between actual and back tested, 99 % of traders lose.

https://www.forexfactory.com/showthread.php?t=809399

https://www.forexfactory.com/showthr...0#post11457210

Markets don't work according to systems as you have just proved, so systems don't work, unless they are mathematical. Most back testing is useless , so systems results are useless, because psychology was not tested, in real trading psychology is involved in executing the system.

I am just going to bite my tongue and hope you either go away and infect another thread or are prepared to add substance to this thread. I am not force feeding anyone here. We are all big enough and ugly enough to take on board different ideas. If you have something to say on this thread then you need to back it up with objectivity. Not pithy links to subjective content that has no substance. That is for pre-school.

Hi C,

Please can you upload the spreadsheet template?

Hi

If you don't want my content , please delete it.

I have developed highly optimized & profitable expert advisors , only to find they fail in the future when market conditions change.I have finally succeeded in making profitable EAS. I was giving you my opinion from my experience and wasted years of all this back testing.Please carry on and delete my posts.

I have been through all this.

https://www.forexfactory.com/showthread.php?t=802780

Hello friend I need your Help can you ADD Option MM Automatic lot size and Trailing stop option in this EA I m uploading MQL File Please check.

Mr B....I might need to send this through to you by Dropbox as it is a bit of a beast in size @ 108MB and FF has a size limitation of 5.0MB for attachments. I will PM you and give you my contact details if you like. Just let me know.

As an example I did profitable EAS based on future market expectations, they worked this year because I included the following fundamentals into the EA behavior. If you used back tested behavior without these interest rates expectations , the ea would have under performed.

The point is EAS should be designed for future expectations.




All the buy signals on usd are valid, only sell signals are not valid , due to higher interest rates expectations of usd.

The following information was available on november 20, 2017
Professionals trade forex when there are expected rises or falls in interest rates, as a result of expected rises in interest rates, Eur/usd and gbp/usd have fallen in 2018.They set a target of 1.10 for eur usd, I don't know if this will make the target.

Goldman Sachs Predicts That the Fed Will Hike Interest Rates Four Times in 2018

Goldman Sachs said it expects a tight U.S. labor market and more normal inflation picture will lead the Federal Reserve to hike interest rates four times next year.

If you had listened to the big boys like Goldman Sachs and interest rates expectations, search for it google, for U S is expected to raise interest rates 4 times this year, and gone short on the weekly time frames on Eurusd and GBP/USD , you would have made over 2,000 pips on both of them. This expected raise in interest rates caused the dollar to get stronger, "interest rates expectations and how they drive dollar on weekly and monthly basis".This is not something new to fx trading, it has been going on since the 1970s, as far as I know. Interest rates expectations and economies change over months and years.This is how big currency traders trade.It worked 50 years ago and it worked in 2018.This is why dollar got stronger.The market does the same thing , over and over again.