Systematic Portfolio Diversification - Data Mining Concept

You are welcome. Example with 32 instances to max out 100% of my Threadripper 1950X (including HT):




This is using Lawlietfox (custom Firefox build) as it beats all the competition in terms of strategy generating speed in EA Studio (Chrome, Edge, etc.) because its a PGO build and has the best Javascript compiling / running performance in all the tests I´ve done with EA Studio (which are extensive as EA Studio runs 24/7 here, being a full time automated trader):

https://sourceforge.net/projects/law.../Release/61.x/

Mate, bottom line is that you have not shown you can trade.

All this theory fluff that you have posted and not one single trade so far...WHY????

I also expect that you will put up a TE to back your trading system.

Are you a salesman or a profitable trader? They are mutually exclusive.

Since you have not placed one single trade on your thread, you are a salesman

Riiiiight. The plot thickens (out of the cupboard the lurker springs with real intent). Thanks for popping in. Have a nice day......mate. :-)

.

Hehehe. Great to have you on board GT . Your experience with EA studio is really going to help. :-)

C'mon wake up. It's your shift again!!!



In previous posts we had a cursory glance at the MSAccess Dbase which is our repository for all our tailings. Now it's time to return to the Plant (EA Studio) and start to process our diggings.

In previous posts we discussed the development of logic which we embedded in core design principles that would lead us to potential productive ground (where the strategies respond to market divergence) and then set up the plant work processes to facilitate our search allowing for a huge number of randomly generated strategies to be mined within a constrained logic space) and validated to a very few that 'might' contain some precious gold.

So let's look at the output.



As you can see, this particular search generated 5738933 randomly generated solutions within this logic space of which 7756 solutions passed the first sampling test of validation.

These 7756 validated solutions then passed into the optimiser for a good ol' dose of optimisation including accompanying curve fitting.

Then were passed into the Monte Carlo process which had quite heavy rigorous settings using parameter randomisation and reduced our number of possibly valid candidates from 7756 samples to 839 samples. No guarantees....but the MC Test will have helped to concentrate any sample of gold lying within these tested solution and reduced the degree of curve fitting.

Then onto Multimarket validation where additional OOS data was thrown at the previously configured (data mined) solution and this further reduced the number of samples from 839 to 432. "What.....432 samples....are you kidding....I'm not testing each one of them"!....never fear guys we have further processes that are applied to automatically reduce the sample size based on principles of applied logic before we get to the dirty end of having to actually do some work.



Of the 432 samples the active rules embedded in EA Studio resolve correlated solutions and also eliminate 'similar applied strategy rules' to give a better chance that the generation of solutions are less correlated in nature.

This process reduces the sample size from 432 to 99 samples of the same single instrument and timeframe (actually it stops at 100 but if you keep the reactor going, better strategies replace worse strategies in accordance with your selected validation criteria. I hear the cries of desperation here. "C....you trade a diversified portfolio of instruments and timeframes. If this is the process for one instrument and one timeframe....we will be here for years?" Never fear, there are some shortcuts coming and remember that so far all this process has been handled by this wonderful plant. All you have had to do is set the valves and thermostats and turned on the ignition while it performs the slog as you watch new episodes of Survivor.

So now we have 99 strategies......that's still too large. So how to reduce this collection further?

You know how I told you that we would keep 2 years of the most recent OOS data free from data mining. Well in addition to providing a great method to test whether your strategy holds up after it has been mined is to let it operate on unseen data, the other major benefit is that it helps you to quickly eliminate strategies through simple eye-balling.

Given that a lot of the samples still contain curve fitting and many fall off the cliff when they are placed immediately into a live environment, this 2 year free period is critical to assess OOS performance.

So what we do is adjust the date range in the 'Tools-Data Horizon" module to include data to current day and then we refresh and recalculate the collection by hitting the buttons on the top left of the screen.



It highlights the un-sampled data region, automatically reduces those strategies that still do not meet your validation criteria (from 99 to 82) and then you quickly eyeball the remaining 82 strategies through the list for any anomalous clues or significantly deteriorating drawdowns.

Here is an example of two strategies that we eliminate with the eyeball for obvious reasons. There volatility signature is too steep even if we find that they recover in the future. We do not want any extreme volatility that could compromise the overall portfolio.

EA Studio also allows you to sort your entire collection by performance statistic. I only use return/drawdown as my preferred overall statistic of risk-weighting and as discussed previously pay little heed of other statistics in this divergent space I occupy. Eyeballing equity curves is therefore my preferred choice of making an initial cut.



Ok....so simple eyeballing has eliminated the strats from 82 to 42. Now it's time to take them back into the Monte Carlo world to assess their volatile risk signatures where we can refine the sample further.

This is where the process starts to slow down as we are now getting more serious and treat the refined tailings with a bit more respect.

See ya next post. Keep mining while I do some core sample testing with Monte Carlo.

C

Ok Guys....Core Sample Testing back in EA Studio has been performed while you all have been slogging away with the pick axes the results of the last dig are in.



So in the prior post we were left with 42 samples in the collection that were most likely to contain pay dirt. I have now undergone an extensive Monte Carlo treatment against this sample (that now includes a portion of OOS data) as discussed in the last post.

I have included below sample extracts of the visual qualitative method I use with the Monte Carlo technique to sort out what appears to have prospects and what doesn't.





Unfortunately this file is too large (16.9MB) to share as an attachment on this thread given the visual images contained in it...so you are on your own here guys. I have probably been spoon feeding too much anyway so far. The extract above will give you an idea anyway how I do this technique. It just continues for the 42 samples and a single fail result is enough to take the sample to the sin bin.

The core technique that is being applied here is eliminating those arrays with diverse splay. Remember that all of these 42 strategies perform during divergent conditions. We have already ensured that with our logic design build. Also this is confirmed by the concertina movement of the Monte Carlo array during discreet times where all equity curves of the array demonstrate correlated movement (previously discussed). What we want to do however is eliminate those samples (strategies) that create undue drawdown during non-divergent periods. The basis of our selection is that the MC array needs as little dispersal as we can obtain outside of divergent conditions.

During convergent conditions our return distribution of each strategy approaches a Gaussian distribution over a small sample or a narrow range of market conditions. The plot of your equity curve during these periods for divergent systems means nothing to the end strategy. What does mean something however is if there is a negative skew in the distribution itself during these periods. This reveals a design flaw in your divergent hypothesis.

Now when we apply a visual method of relative comparison between Monte Carlo results, the old visual cortex can be a powerful beast provided certain rules are laid down. Ensure that you are comparing apples for apples. In other words ensure that the array is configured for a singular instrument (market) and timeframe and also ensure that your array is measuring the same thing....(eg. randomisation of data history). Visual 'relative' comparisons of MC arrays are only effective when you are comparing apples with apples. Don't for example compare an array from Corn against an array from EURUSD. There is no direct relationship between the two different instruments in terms of any meaning that you can derive from the comparison.

The method deployed of visual comparison can be gleaned from the extract above. It is pretty self explanatory and at the risk of repeating myself...well I won't. :-). But I will give you a visual annotated example of what I mean to help you in your interpretation.



Anyway....enough interpretation.....You will note that we have reduced our sample size down to 8 valid samples from 42 using this technique. That's better I hear you all gasp. This is about as far as we can go with EA studio so in the next few posts we will be taking these samples off platform and using our own home grown tools to undertake further forensic sampling.

We throw out the rest of the dud samples in the tailing dump and leave that for the fossickers to deal with. We only worry about the economics of the find as opposed to the simple gleam of gold metal.

Now we also save our testing spreadsheet in our database in the attachments of the portfolio that this test relates to. Guys...the dbase is now reaching a size where we can no longer attach it to the beginning of this thread.

Where possible I will save relevant info in the associated posts so you can elect whether or not you want to use it or not. You already have a dbase for that purpose..but you will have to use it yourself. There is nothing like diving in the deep end with new systems. It brings the miner in you out of ya!!!!!

Cheers guys

Shift Over



C

PS Next few posts we will start getting into the additional analysis tools and methods to further refine the samples to detect the presence of gold in more quantifiable ways. We are slowly drilling down to the best samples according to this particular divergent approach.

Nice long term data-set you have there GT. I am envious. The longer the dataset across more market conditions, the less I have to crunch to beat the noise out of the solution.

OMG!!! You are totally organised!!! Great job.

Cheers Brownie. Without that database things would have started to get visually complex in this lab. My old method of connecting the dots was starting to get a bit messy. :-)



I hope people can get some good benefits from the dbase. It has wider application than this project and encourages you to re-use stuff that you have previously discarded in the 'past project' pile. So much good R&D goes to the land of lost ideas....but this at least provides a way to recover it later and perhaps forge new connections with it.

Everyone has there own preference in the info they retain...but at least this gives you 'max keep capacity' and retains the important ideas that might have better connections/relationships with future projects. You just gotta connect the dots a bit differently.

Focus on the Process not the Goal

Just as a brief interlude before we get to 'da lab' for more crunching, you will see that this approach is 'process ridden'. Well that is not a feature restricted to the hypothesis of 'divergence'. Data Mining brings you into the domain of 'enforced processing'. You need this to convert original mined samples (randomly generated solutions.... aka bedrock) into samples that have a good chance of displaying an edge.... (aka a good clue to where a concentrated economic deposit lies).

All we are doing is applying a well worn trail of logic of 'how to mine divergence' and forcing it through a fairly standard processing plant (albeit a plant assisted with lots of streamlined efficient and marvelous programmatic design aka 'Popov') that progressively treats the sample to strip away the noise. In this instance 'noise' means anything that isn't divergent. It might be market convergent features of the data set or 'curve fitted' features or other anomalous features that dilute the value of the final economic deposit.

In the trading world we frequently hear the statement, 'take care of the processes and this will take care of the objective' or the statement 'quality in gets quality out'. These are all aimed at the same underlying theme of ensuring that you apply processes in a logical systematic fashion to achieve a desired objective.

Data mining is all about systematic processing and letting the data speak for itself. It forces us away from the hidden 'bias' of forcing our own agenda into the equation.



As speculative traders, we are all fossickers..., but a focus on process turns us from fossickers into mining companies. You will note how I jokingly cross-refer to being in a 'working mine' as I step through this process. Well this is for three reasons.

1. To make what is in reality an arduous job of repetitive crunching a bit more fun (unfortunately that is the sting in the tail of data mining. It is an endless process of repetitive crunching where the novelty soon wears off. Fortunately some of the plant takes a significant burden of crunching off our shoulders;
2. To draw parallels to a different economic activity 'gold mining' versus 'quantitative trading' where you see the same underlying processes at work....eg. it's just the 'same shit for a different biz guys'; and
3. To hopefully attract the curious mind that is not necessarily skilled in quantitative statistics. Most people can understand the mining process...but start bandying about complex applied mathematical formulas and you hear a scurrying away of the curious mind to a more peaceful and 'familiar place'. I am very like that myself...and that's why I adopt less 'formulas' in my approach than visual methods (which with my limitations tend to work better). I totally respect the mathematician that understands when and where to correctly apply his statistical tools....but sometimes they come up wanting in this respect and they are the ones we need to watch out for as that can lead you down the garden path, where you are not only confounded by the formulas applied...but also the modus operandi and use of assumptions that can take us down the path towards an abstract world that departs from reality.

There are two different mindsets that might be reading this post right now. There are those mindsets that need instant gratification in looking at the 'Goal' first, and there are those that like the more in depth puzzle of appreciating the processes that are undertaken that lead towards an objective.

For the former mindset, they tend to be impatient and demand the results first. They want the 'grail' now and are unlikely to make the effort to understand it...."heh...don't know what's under the hood but this shit is fast!!!!". You also hear them in the investment community. They are the ones that tend to flip between different investment schemes on a simple path of picking the best performers. They also have minds that are full of useless factoids that are frequently incorrectly applied to a context as they have never taken an effort to really understand them. This is what a focus on process actually does. It forces the brain towards coherence as opposed to a jumble of mess where relational linkages between that mess are lost.

It is the relational linkages that matter. That is what creates the 'success'. Success is not the instant gratification that luck provides but a long winded process of putting together small wins in a coherent narrative that lead to amplified non-linear outcomes. You entire life will be confronted with choices. The way you plan for success as opposed to simply being in a lottery is to start small and progressively build better choices in your processes that magnifies the output in a non linear way.

What's wrong with picking something that that is proven to work first C?....I might hear you say....well if you know anything about the principle of survivorship bias....then you may see this as a fools errand. Simply pick from the best performers of the day and assume you will be successful in the long run and you will be deeply disappointed at the end of the day. This style of mindset is 'messy' with no appreciation of process or logic...and hence the end result tends to be a mess. Even though they do not appreciate process, everyone is actually a victim to it. Process unfortunately is the way this world works. You can either accept it and apply it...and thereby have a good chance of achieving a stated outcome....or you can reject it...and stay in your rudderless boat spinning your wheels backwards and getting nowhere.

To read this thread and keep with it requires an inquisitive mind that looks for ideas that might help them in their own logical processes. That is the value of this thread....not the lure of a promised outcome. Hell we have two years to find out whether we are right or wrong in the processes we undertake here. We need this time to make a better processing plant and achieve the benefits of compounding that non-linearity will bring along the way.

There will be lot's of checks and balances along the way to adjust our path as we go but keep the chain of processes consistent and logical. We will be making some of the processes applied more efficient as well. Well guys....sorry to disappoint but that is the same for every commercial enterprise. A Facebook, an Amazon or a 'Google' is an end result of a process that took years to achieve. It is the process that led to this outcome. It is the processes that are important in this game.

So if you are impatient and reading this....then that is a clue to what mindset you belong. The door is there for you to leave if this is unappealing. There is one thing that slows down a process such as this and that is having to stop the machinery to respond to an impatient mind that simply lacks the endurance to read and come to conclusions before they are even reached.

"History remembers the achievers not those with simply the trappings of success like 'lotsa money'. It is the narrative of long winded convoluted processes that led to that achievement that matters in the long run and why we remember them". Focusing on the processes is far more rewarding in the long run than 'show us you P&L!!'.

....anyway, start getting clean and dust off that mine dirt as we are about to enter 'da lab' and we don't want to contaminate those refined samples we have waiting for further treatment.

Cheers

C

Welcome lads....Please observe the following 3 rules while you are in the lab......don't contaminate the samples, don't contaminate the samples and don't contaminate the samples.....is that clear ?

Attached Image

From prior posts we have narrowed down our sample size to 8 possible candidates. They lie in the sample tray below but they are not the finished product. More work needs to be done....and unfortunately on your last digging exercise it's not looking good, but the testing so far suggests we at least may be able to find 1 ore 2 that have merit. That's just the way it is guys. Sometimes you hit pay dirt, and sometimes you just get some hints and clues.

These samples comprise data of both In sample and out of sample origin. We have ensured that our design logic has only been derived using the In-sample portion and have only used the OOS as a basis to narrow the list of candidates to those with potential. We will have further OOS as the successful samples will then be put in incubation mode in the demo environment going forward so we can gather stats as we progressively build the contents of the portfolio.

Portfolio application is left to the tail end of the process and we have lot's of further work to do before we dare to activate a portfolio live. At the moment we are just focused on individual return streams of the portfolio.

I will let you peruse the current treated sample with gloves on as we are soon about to hit them with further evaluative methods.

Stay tuned

C

PS You should be updating your database with these EA's and specifications and the state of your database will look something like this.



At the portfolio level (above):

1. the DataMiniSets comprise the setfiles used for generating the collection of strats on EURUSD D1;
2. The Collections comprise the prior 42 sample collection and the distilled 8 sample collection;
3. The 'Other attach" comprises the MC Visualisation Matrix we applied for comparative cross referencing across the 42 sample collection to reduce it to 8 (discussed previously).

At the strategy level (above):

1. the field 'EACode' should comprise your sample EA in mq4;
2. The Research and Design field should comprise your strategy specification attachments (given in this post); and
3. The result of your qualitative evaluation using the MC technique in terms of 1 being excellent and 2 being pass.

The INSample date ranges should be derived from the first and last date of trades undertaken while insample and where you design is created. We need this date range for further visualisation techniques we will soon be doing.

Mapping to Market Condition

Attached Image (click to enlarge)

Welcome guys to the mapping division of the lab. This is where we map those equity curves of the samples to the market to work out how robust they are. Don't be deceived by the supposed impact of slight variances to your results of your equity curves from using different broker sources and other such fancy tails you might have heard. They are all just methods of assigning blame to others in not being able to read the market condition.

There is inevitable variation and perturbation of every curve....however what we are here to demonstrate is that it is the variability that counts associated with market condition which matters. What we do in the lab is forensically examine our samples in the context of how they perform to market conditions both during in-sample and out of sample across multiple broker sources to eliminate the weak samples and discern the stronger ones that have potential under further testing.

You may have eagerly downloaded the prior samples thinking that they all would fly....however sorry to disappoint fella's. We are not there yet but we are on the path and I admire the spirit :-)

The first thing we do is take these samples to another broker source. We have so far tested the samples under FXOpen which contain both InSample and OutofSample components. Now we are taking them to Pepperstone and Dukascopy to put them under more rigorous tests.

I have loaded the equity curves and the data results into the database for both brokers and here are the results for the same timeframe range. Pepperstone is using Open Prices only as the strats have been designed to only operate of the open of the bar...however as a cross-check we use tick data with Dukascopy just to see if there are material differences.





From the simple stats above, most strats seem ok...certainly not impressive and you may be tempted to adopt them all as 'tested EA's' however you need to look closer at the statistics and you will note that some strats are similar in terms of performance and other strats are markedly different in overall performance metric. We have already looked at the limited impact of broker spread and also SWAP variations in terms of overall performance of these fairly short hold strategies (refer to average hold), so what's going on?

One of the clues lies in the open itself as both brokers use different GMT settings so the open will be different for the two broker sources.....so for a strategy to vary based on a timing difference that is broker specific as opposed to market specific....then that is a sign of a weak strategy solution. We want strategies that are more robust than that. Any other significant sign of material variation simply means the strategy is not robust. We actually do not need to understand the underlying reason.

So let's get into this mapping process and dig deeper into the robustness of each solution when applied to uncertain future conditions.

The first thing we do is obtain a weekly chart of the instrument concerned and then we use our database to define per strategy where the insample and out of sample periods can be mapped to the chart from our different broker sources. We use a weekly chart as differences between brokers smooth over longer timeframes and what we want to see is the market condition....not the broker differences.



In the chart above you can clearly see conditions of market divergence and periods of market convergence. What we want to see is that this is reflected in our equity curves.

We align each of our source equity curves with the market condition to see what is going on.

So here is FXOpen, Pepperstone and Dukascopy equity curves mapped to market.







Doing this alignment now allows us to interpret how this strategy performs overall and how it performs during particular forms of market condition.


If you look at the FXOpen map you will see periods of strong divergence represented by lifts in the equity curve at certain times. This makes sense when you map it to the market condition. You will also note that it has continued to perform out of sample (green area)...so you might be under the impression that this strategy meets your criteria for robustness....however not so fast Holmes!

Let's look at Pepperstone which includes earlier OOS data. These conditions were clearly divergent yet we should have had a rising equity curve during this period....however we didn't....*warning bells start firing off*. "Is this curve fitting I see?".....well no actually as supposedly we have already treated for curve fitting in prior treatment.....and also.....why do we perform well OOS in most recent conditions? Well the answer happens to be in the design logic itself which is constrained for a particular form of divergence....not a broad class of form of divergence.

Let's look at Pepperstone to confirm our assumption as that goes back OOS to 2000.....yep.......it stands out like ************. This strat simply didn't perform well during bullish divergent periods between 2001 and 2008. It has only been post GFC that this strat has performed......so in terms of a general all rounder......a big fail is put next to this strategies name.

Without going through all the heady details as you get the point now of this process, we find that of the beginning 8 sample strats that fed into this mapping process....we are now down to only 3 possible strats and only 1 really shows sign of hope. As opposed to putting this into words, I will put it into pictures for you to interpret on the next post.

Cheers

C

Mapping to Market Condition - The Results

34854289 - Fail


34856407 - Fail


34857662 - Weak Pass


34859174 - Fail


34860814 - Weak Pass


34863983- Fail


34865294- Fail


34866855- Solid Pass


Well that's as far as I can take it at the moment guys. I need to do further mining to build the collection of strats using this process. This process hasn't been concerned with the profitability of the strat and has been purely focused on risk mitigation.

We are left with one good sample and I have retained 2 weak ones that may be ditched in preference of more suitable samples if they arise. As these weak examples stand currently, they appear to perform in recent divergent conditions but simply managed to tread water without too much drawdown during pre GFC conditions. They might have a use in plugging portfolio weaknesses at a later date.

In terms of housekeeping with the database, I simply 'turn off the 5 failed samples' and only keep the 3 remaining samples active.



So now I go back to the mine and do this process again and again for different instruments and timeframes until I have built a sufficient number of robust strats where we can start looking at portfolio treatments and compilations.

At least these posts to-date provide a clue as to a visual process to interrogate your system performance. Pictures tend to convey much more meaning to me than dry statistics and provided you don't read into tea-leaves can provide useful info that often statistics simply fail to pick up.

I will be back in a few weeks hopefully with a robust collection we can work with.

In the meantime...we can discuss anything you like in relation to data mining.

Cheers guys

Shift is over.....it's beer o'clock :-)





C

Housekeeping



Guys I have made a few mods to the database and we are now at version 1.4 (refer to first post of the thread). I have also stripped out some of the large files so you can access most of the data detailed in this thread to this point in time.

Attached are the updated EA specs and sample collections (json) used to date for these using EA studio.

Have a great weekend.

Cheers

C

I looked at the code of the strategies. They are all built on the exact same skeleton. One entry trick. On exit trick. One position at a time. Same lot size. Fixed trailing stop. There is no way the generating process comes up with a solution that for example trails the stop with a SMA, or that adds to a position, either by average down or pyramiding. The initial SL is always the same, irregardless the volatility or the trendiness of the market. From the code it is clear that the indicators used, both for the entry and the exit, are supposed to capture all the state of the market but their parameters are fixed! OK the so called "solid pass" makes money in the past OOS. But this past is mainly a strong up trend. I doubt this code has any adaptatibility. We are very far from this: https://www.forexfactory.com/showthr...5#post11508125

Thanks Pip. To avoid jumping the gun...best if I get a portfolio up and running with a track record. Will report back much later and let you know how it goes.

Cheers

C

I use same trading method as described by you. It is reflected in my TE. But I have always done averaging down adverse trades up to 5 level before becoming bold enough to do pyramiding (fading). I would pray for range trading to prevail so that by level 5 I am at least be able to reduce drawdown by 30% to 40%. My TE lists all such trades.

Since last week I have changed the tack and have started using hedging and surprisingly find hedging helps rein in the drawdown. However, I am developing a sort of decision paralysis and unable to do the pyramiding game. Will practise it for a month to figure out how best to use hedging to control drawdown.

Ideally, I would like to establish some metrics linking hedging to drawdown control so that I can have a manual system in place.

I would appreciate if you can share your experience as to how to use hedging to drawdown and weave some kind of rough system to be followed manually.

Really interesting!

I could add this to my low stdev porfolio calculator.

Just stumbled upon this thread. It ended rather abruptly 3 years ago...

Would be nice to see the performance of the mined strategies in a live market over the last 3 years. Googling gives numerous blogs describing methodologies for mining strategies using the various commercial mining tools like the one described in this thread, but none of the blogs show actual results, live data or forward tests. I suspect the strategies generated by these commercial products are too simple (e.g. fixed TP, no SMA-based trailing stop, no scale-ins, etc.) to have a real "edge", and it doesn't yield a profitable portfolio.

Anyone willing to share results?