Systematic Portfolio Diversification - Data Mining Concept

I trade many parameterless strategies and they are more stable. However, I also trade my fair share of strategies with parameters. Look, optimisation can't turn crap into Gold because there is no such thing as feces Alchemy. If you have found a genuine edge, an edge that would be profitable unoptimised then intelligent optimisation is going to boost performance. The main stigma with optimisation comes from traders attempting feces Alchemy. Only strategies I'd be willing to trade unoptimised are optimised.

Ernest Chan: "Lately, I begin to understand what a trading model with no free parameter means. It doesn't mean that it does not contain any lookback period for calculating trends, or thresholds for entry or exit. I think that would be impossible."

As far as I'm concerned a parameterless strategy has no parameters. A tradng model with no free-parameters isn't the same as a parameterless strategy. Chan is wrong when he states he thinks it would be impossible: a simple system that buys at S2 and sells as R2 as long as the last three bars are white would be a parameterless system.

Unless I am missing something.

Some great ideas escaping into this thread guys. Please keep it going.

I am furiously developing some tools for the proposed logic I will be applying to my quest and will be sharing them on this thread so I can't really start the detail until these are complete....however there aren't too many threads centered around hitting the market with a swathe of strats designed with a particular logic in mind. ...so please keep the thread pulsing while I finalise these tools.

It is taking a bit longer than anticipated but I cannot really start with any detail until these are finalised.

This project seriously commenced as a mind fart following the strong desire to dive deep into full systematic application and I am not a programmer but need tools that generate these automatically based on the logic I provide to them, so this is a big learning exercise for me as I am a newbie to this landscape, however I am simply bringing what I know about portfolio treatment into this equation as I feel this may be a valuable overlay to anyone interested in this path, whether they be of any logic shape or colour/tic.

There are lots of advanced concepts being discussed by Brownie and Alphadude which I don't have my head firmly around, but hopefully with more discussion it will give me more clues on how to embed some of these principles into the project as no doubt we will bump into them during the course of this 'proof of concept exercise'.

So please be a bit patient and banter on about all things that may have application in this project.

Cheers guys

C

I'm new to this thread, but not to this subject, so I'll add my twopenneth :-)

In my view, considering a system (such as the one you describe here) as parameterless, is somewhat arbitrary. For example, you require the last 3 bars to be white. What about the last 4 bars, or 5 bars? That would be a parameter - albeit one you are choosing not to explore. Additionally, support / resistance levels S2 and R2 have to be specified in a concrete manner, which would imply more parameters.

(More to say on this, but will have to return later - I'm glad to participate in an interesting discussion on a subject which occupies my thoughts at the moment, so thanks guys!)

Matt

Systematic Portfolio Diversification means for me: Have retails a chance?
Diversification in following I reduce on FX only (not: where does the money go: Real Estates, Bonds, Interests, Loans, Carry Trades, abroad ...)

Motivation
The time where the postman brought us the newspaper with the market price is over.
The time where we have to pay extra for real market price via Internet is over.

IT included HFT will bring us ideally (only theoretically) balanced market. ----> vertical only. Market participants go up/down - IT leveled.
Then the market is largely uninteresting. This "ideal state" will not happen.
But what will the machines bring us, in the future, what today?
Market has changed during last ... 15 years. Have we now a chance to "predict" market? No doubt in modest extent.
However, the importance predictive of market participants will decline.
Market will be done or "manipulated" through very rich moneymakers via HFT Robots.
There may be restrictions through legislation but they will be insignificant.

Today - how to
HFT working now, big players too.
Nevertheless one can can go with good profit expectation discretionary (or who found an edge via EA...) into the market.
The theses/assumptions are
1. Market is moving.
up/down/again in opposite direction, sideway
slowly, quickly
in segments (box, waves, patterns, fibos, SR, seasonal, time frames, exchange times, correlation ...
2. Market usually does not move linearly
Scatters, waves
3. Trader has presumed goals in the market
ideal are assumed goals which occur concentrated /scattering).
4. Depending on the readiness for the holding period trader enters the market with an appropriate chance/risk expectation and tries to attack this goal.
5. So surprisingly little has happened with regard to strategy in the last 10 years.
6. However, a change of consciousness has meanwhile occurred.
Traders are increasingly realizing that they are exposed to spontaneous and significant market movements which go beyond the learned extent.
7. The willingness to accept grater losses (included no SL), hedging, using pending orders far away from now ... has increased.
Exaggerated: what ever happens I trade it.
8. What now
Use your tried-and-tested tactics, see 1.
If you trade in Positionmodus or Swing, do not choose large lot sizes, stay longer if required, use cost-saving trading (qualified broker selection, not so often in
and out, swapfriendly - eg EURUSD choose short if possible.

Not as a role Model, but as a practical example for Asymetrical Convergent and Divergent Approach to the Market, see
1. https://www.forexfactory.com/showthr...0#post11472500 Swing (small sequence compared to 3.)
2. https://www.forexfactory.com/showthr...0#post11477900 finished Swing box 1.
3. https://www.forexfactory.com/showthr...2#post11481572 Positions trading - transparent continued every day

Nice Post Issie :-)

The one thing we can count on is change. I think it was only the other day that the second last Blockbuster store closed down....who woulda thunk it? :-)

If the markets adapt, which is clear they do, then we have to be prepared to adapt with it. It is not all doom and gloom though provided you are prepared to meet the challenge of change and new technologies. The powerful tools that are available to us as retailers were not available previously, so we had better take up that challenge I feel..... but my gut tells me we need to take them up and fight 'Skynet' :-)

I still type with single fingers however so there are just some things that still stick from the past but I make do with what I can in a clunky sort of way and feel my way as I go into this change.

THe system rule is 3 bar not 4 or 5 bars otherwise I would have stated that. The 3 bars is a fixed rule. Not sure what you mean about S2 and R2 since PIVOT S/R has a fixed calculation.

For the sake of argument lets say the previous bar is a bull bar for long and a bear bar for short or even better to drive home the point: long & short entry conditions are perpetually TRUE thus this parameterless system is a simple sell limit at R2 and buy limit at S2.

Perhaps that system is not considered parametless because it has already choosed S2 (why not S1, or S3), and exit on R2 (why not R1 or R3).

Also the 3 bars; why 3 and not 4 or 5 or 6; or remove this rule alltogether.

The idea behind parametless system is to use a moving window (i.e. 1000 recent historical bars); to determine how the market behaved with pivot points (s1,s2,s3). You then run auto optimization once a day (at beginning of day). Some people run it on every bar; but that's an overkill. The idea behind adjusting the parameter values; is to adapt to different market condition. So once day is more than enough.

Let's try to turn the above strategy to parametless one:
1. Entry parameter on pivot level: On each bar (or on each start of trading day); the strategy makes stastical calculation on recent 1000 bars; and finds hout which level of pivots have the market bounced off more often then others (s1, or s2, or s3). If you find that S2 is the most bounced off level; then you choose it as the parameter value for entry; for today. After few days or weeks; you may find that s3 becomes the most important value; hence you adjust your strategy to this value automatically.

2. You can further complicate the calculation process; by finding out how many white bars have more bounce off from s2; using recent 1000 bars. choose that value as a parameter for today. If none found to be significant; then disable this trading rule alltogether for today.

3. Exit: same with the process above; out of the bounces in the statistical calculations found in (1) and (2); which level did the market hit the most after it bounced off; r1, r2 or r3?. Choose that value as a parameter for today.

Now you have a 100% parametless system.

In general; you do not have to get 100% parametless system; but you can start bit by bit; converting important parameters to parametless.

Another example of a parametless parameter being used by many (perhaps without them realizing it is parametless); is the use of ATR in setting stop losses, profit target, or risk allocation. ATR makes good sense because it adjusts the strategy automatically to various volatility conditions; and also various instruments.

Another *secret* of the quants; why most quants prefer to use Bollinger Bands?
Answer: I will leave this point up for discussion.

I realise that in your formulation the trade is a one-off fixed rule. However imagine another researcher trying a range of bar amounts, say from 1 to 10 - and finding that 3 bars produces a good result. Now he has discovered the identical rule to yours, but would consider his choice of 3 bars an optimised parameter - purely based on how he came to discover it.

I apologise if this distinction seems irrelevant, I don't think it is. Any indicator or system we might create is a point in a high dimensional space of models, which may be navigated.

Time for bed - BTW I hope my small contributions are taken in the spirit in which they are meant - one of light-minded debate.

Best Regards

Matt

Cheers bud, We'll continue this discussion later. I'm off to bed.

Cheers A. That was an excellent explanation mate and helps the brain adjust. Just what the Doctor ordered.

Can't sleep. This has got me thinking. First, I think you're describing systems with free parameters rather than parameterless. Second, so your description is essentially regular (each bar or daily) reoptimisations. So there are many traders who already autotrade systems with free-parameters/parameterless without realising it. In truth I do not know what is so revolutionary about this concept.

Looking forward C. Keep them coming!

Thanks Matt. I welcome your contribution.

It had we going as well.....and then I started to think about what makes good performance cars versus good all purpose cars and then it made sense. Both have system parameters which can be constant or variable in nature however at the entire car level (portfolio level) a robust car can navigate a broad variety of terrains but slowly, whereas a performance can only navigate a selected terrain very quickly. The systems that we deploy for a portfolio solution need a degree of variability, but what is important is how they inter-relate and combine at the portfolio level to allow for performance diversity or performance specificity.

A radiator has a specific purpose as a system but does not make a car and allows for a degree of variability...as does a carburetor etc. etc. etc but the combinations are part of a broader purposes...to make the all purpose car that can navigate uncertainty (aka market)...not to make the high performance car that can only operate on a smooth race-track unless we are sure that racetrack conditions are maintained....and then we can nail it.

In the context of data mining, we better make sure we simply do not make high performance cars or worse still just radiators by selecting systems all with the same performance criterion as that will not lead us to a solution that addresses market complexity and adaptability.

Which then took me on the day-dreaming path of ......Does the variability of the system and it's relationship with other systems compromise the objective or augment it?:-)

These sort of discussions really help adjust a 'singular' reductionist viewpoint to a systematic one. I continuously find I fall into the hole of reductionism where the particular model I am applying to explain a single observable feature starts to take on a life of it's own and becomes the reality, specifically when we talk of numbers and statistics where I tend to quickly lose the big picture of what is trying to be achieved in that we are seeking to find solutions (models) to apply to the reality.

Focus on the Design Specifications that Manage Risk and Leave the Profit Potential Open

Most backtesting is conducted in a manner that 'looks' for the straightest possible ascending smooth line in the context of the performance of a strategy against the market conditions thrown at it. We even go further and optimise at regular intervals to straighten the kinks. As a result, an emphasis is placed towards past conditions and an assumption is made that those conditions (or something close to those past conditions) are likely to persist into the future. Under this philosophy, an emphasis is based on biasing your strategy solution to more recent conditions as these conditions are more likely to extend into the near future.

The straight line which is a product of how a strategy responds to the market is either a derivative result attributed to faithfully capturing a past condition, or a random construction from data with no predictive potential or combinations thereof. A traditional focus is therefore placed on a rigorous method to distill the randomness from the past condition to treat the result for the propensity for curve fit results. We are then more confident that the treated solution can project in a linear way along the historic path to the future.

Where a divergent approach departs from this traditional method is that this approach simply uses the past to assess the suitability of the risk signature of the solution applied and ensures the design is open to any potential divergent event that may take a variety of different forms or paths. We are therefore very wary of straight lines from the past as if the straight line is not curve fit, then it suggests that past conditions have adopted a stable range of narrow conditions.

Under a divergent viewpoint we normally would not consider optimising, however with the randomly generated algorithm generators a 'loose' degree of optimisation is required to simply ensure that 'likely' divergent solutions are found in the resultant outputs of the collection. Once we have done it to obtain our initial suite of solutions for OOS, we don't optimise again. In this sense optimisation uses risk-weighted benchmarks on the nature of the curve produced such as return/drawdown ratio as a guide as opposed to say profit factor or simple return.

Divergent strategies simply do not fly in convergent conditions, so what we look for in our equity curves are return distributions that can be tied to the market condition of divergence. We therefore match the risk signature associated with the volatility of the equity curve to the market condition. We map the profile of the curve produced to the market condition and see whether it makes logical sense. Of course that curved volatile signature may also be curve fit, but it is our design statement that ensures that if it is, the adverse consequence of following a curve fit response is minimal. I think the key to this approach lies in that last sentence and why this approach possesses sound logic.

What we focus on in terms of divergence in our design considerations to limit adverse consequences (real or random) associated with future uncertainty is to deliberately close our strategy as tight as possible to limit adverse price movement. As a result, the design is very asymmetrically oriented to 'cut losses short' and let profits run. This 'tight' condition however needs to ride with future profitability to ensure that we do not leave too much profit on the table if the market conditions alters again to some unknown new condition. Now this is the typical weak spot and conundrum for most divergent traders. It is the exit condition where we need to place a large emphasis of our focus.

This trailing condition that is necessary to limit adverse risk places a narrow constraint on the possible future paths that each specific design solution can address. It limits our future profit potential in uncertainty.

So we then refer back to our data mining efforts and look at the curves produced for each strategy not as a method to project into the future but simply to ensure that the range of strats of divergent nature we apply do not all do exactly the same thing during a past adverse market condition. They all need to produce correlated bursts of profitability in divergent conditions, however it is during the 'lean times' of convergent market conditions where we need a swathe of different designs available to prevent an correlated impact of design features amounting to a significant adverse event for the entire portfolio.

We also allow for extended periods of OOS just before implementation as we do not possess any statement of conviction about recent market conditions. It is just data to us that possesses a risk signature embedded within it. We also test the design solution over different broker data and different market data to test for robustness as opposed to the squeezing the data lemon to extract every ounce of profitability from it.

So long as we can weather the OOS risk signature then that is a big tick for us in terms of it's ability to handle recent market conditions. If it is not profitable, then that conditions needs to have been a convergent one. If it is profitable, then that condition needs to be a divergent one. Initially I was planning to go with 6 months to 1 year OOS immediately prior to incubation for 6 months, but IMO I think a better test of divergence is hitting the solution with unknown data once the design has been specified and locked down during IS tests.....anyway this is digressing a bit.

We are looking for equity curve signatures that are correlated or uncorrelated when it counts. That is why our pursuit is not for the straight linear ascent response but rather a flatline during unfavourable conditions and an unknown blip of positive magnitude for any divergent market condition.

If we tie ourselves down to using a particular metric such as a profit factor, sharpe ratio, profit result, max drawdown etc. that summarises the equity curve as a single statistic which tends to be a feature of statistical sample data mining techniques designed for Gaussian (linear) conditions, then we may fail to define a solution that meets our logical criteria if the assumptions of that criteria are premised on non-linear conditions. Statistics only goes so far and sometimes is no replacement for alternative methods for describing wandering in non linear samples.

Below is just a single example of how we select from the myriad of solutions generated by the data mining platform. You will note that we do not care about straight lined linear ascending equity curves as they are the anomaly with a divergent mindset.



We place the chart of the market condition above the equity curve of the mined solution and then identify divergent zones (long and short) in the market and whether there is favorable positive correlation in the equity curve at these discreet phases. If the majority stack up, then the balance of probability lies in our favour that the equity curve is not curve fit....but it still might be...hence the application of design constraints in our design logic to cut losses short at all times. Note that we are not using complicated mathematics at all in this process but rather mapping the market condition by visual methods.

Now the risk area lies outside these defined zones and that's where we search for uncorrelated curves from different solutions to flatline the result on consolidation during these non-convergent phases of the market condition.

In sum total we then compile a portfolio that is strongly divergent in nature and robust in it's ability to weather the storm. You make hay while the sun shines during periods of divergence and then defend yourself to the ***ggery during all other more 'noisy' efficient and predictable conditions.

PPS. We need to note something here and remember it. We look for periods of positive correlation and negative correlation when we map to a market condition BUT in terms of a strategy's correlation to the portfolio equity curve....we are looking for uncorrelated overall strategy solutions. Yes, there will be positive correlation of all divergent strats at discreet phases in time with the portfolio itself when conditions are divergent, however the overall lower correlation of the entire result over the test period is a strong sign of a well constructed divergent portfolio that is rigorous.

Database is ready to go....at least in pilot form. I have included it on the first post. More tools coming that are necessary to streamline processes. You can use it or not....it's up to you.

PS...You are going to have to apply your nouse to how it works immediately. I will be giving detailed instruction later when we are ready to start...so please be patient.

This is not a please fix or amend this for my personal preferences arrangement. You can do this on your own personal time. This is simply to provide you with the tools I will be using to explain and test the logic applied in this thread.

Enjoy

C

Thanks for that explanation alphadude - so by parameterless, you mean no fixed, predetermined parameters, but rather ones set by a rolling, short term backtest? How are you fighting off overfitting, when using such a (relatively) short window?

Matt

the window can be of any size. your choice. my personal choice is 2 years of data. it will boil down to the nature of your system.

for swing trading; you would perhaps use 5 years or more. for very short term; 2 years or even 6 months is ok.

Lets consider one example; for an open range breakout system (see Toby Crabel book) you can use this parameless idea to determine when the daily high/low can occur most of the time. you can also determine the time of the open range; whether it is 30min, 1hour or 2 hours.

the benefit of parametless system are:
- sound edge. well understood mechanics of how/ why the system is behaving
- auto adaptivity based on scientific statistical analysis
- stable parameter values and makes sense
- adapts to different markets
- most benefit of all; it helps you the trader; do the due diligence properly. why would you accept an arbitrary parameter value or why follow the optimization values without understanding why. You will notice after practising parametless systems that you are becoming a better quant trader; precise market scientist. you will be able to see the markets under the hood. like an X ray machine. you will achieve nirvana of system design.


another good example is CrucialPoint idea of Gpbusd having a minimum range of 40 pips during London session. this info can help you decide your entry or exit techniques accordingly.

to make a system 100% parameless is not that easy. you need to be very good in statistics and programming. MT4 will be hard but it can be done.

Focus your attention to the main strategy edge. other parameters can be left fixed.

another trend following example; donchian breakout. how are you going to choose the Donchian period for entry and exit? better leave it to statistics.

and finally; the disadvantages of parametless systems:
- large amount of coding. it will consume you.
- takes longer to run a backtest; and also heavy on cpu in live trading
- hard to parameter diversify the system

that is why i personally dont have any 100% parameless systems; but i usually focus on the main edge. so ine or to parametless params are enough for a strategy

Hi Matt, regarding the risk of overfitting; it always exists in system design; however; parametless strategy has lesser risk of overfitting than optimization.

we have to understand there is a difference between backtest optimization and parametless optimization.

1. In backtest optimization: the fitness function is the PNL of the system (or any other risk/reward function of the whole system). This can produce overfitting results; as the combination of all parameters in the system; may by chance produce good result. this is a recipe for disaster.

2. Parametless optimization: it is done before we build the strategy. It is done in selecting a value; based on the market behavior. it has nothing to do with system profitabilility. It is a way to find a value for a parameter that *makes sense*. such as selecting the period of the moving average; or selecting the start time and end time of the open range breakout system. Different markets have different range breakout time range (JPY is different than GBP or EUR).

You can do the parametless optimization automatically OR manually during system design phase.

For non-programmers; you can achieve the same results during R&D. do your studies using excel or any tool you find easy to use; and determine the parameter value for the market you are studying. Then build your system from there.

If you want to master this way; read Toby Crabel book. The whole book is about finding those parametless values. He did many statisical studies around the open range breakout system.

attached is a statistical study of EUR/USD "Daily high OR low is completed by hour of day" vs "Daily high AND low are completed by hour of day"
Notice the result over the years are very similar; indicating the stability of this analysis finding.

Notice that by GMT 17:00, 70% of the time; both High and Low have already been completed. Depending on your system; this can help you make decision on when to enter; and when NOT to enter.
Each market have different statistics.

Alphadude, your post is a very interesting read. Please provide the title of Toby Crabel's book.

-----------------------------------------------------------------------------------------------------------

Got it: Day Trading With Short Term Price Patterns and Opening Range Breakout by Toby Crabel