Choppy market index: any good ideas? | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

VennD - Any Pairs - Any Style - Any Time 741 replies

Any indicator that tells when Choppy/Ranging starts/ends? 25 replies

Anyone got any good concentration tips/ideas/strategies 7 replies

choppy choppy choppy! 21 replies

Any Ideas on good charts, free too expensive 6 replies

Attachments: Choppy market index: any good ideas?
Exit Attachments

Choppy market index: any good ideas?

  • #41
  • Oct 10, 2017 3:14pm Oct 10, 2017 3:14pm
Quoting PipMeUp
Yes I'm still interested in a quantitative estimator of choppiness. Perhaps also in finding a better word for this because people always associate choppy and ranging market as being the same thing. I make a clear distinction between these two facets of the PA. A ranging market can be steady while a trend can be choppy. Admittedly the examples are far from perfect but they may give an idea. {image}
choppiness-ranging-trend if you wanna see it better, quit using candlestick, range bars or point and figure are way better
  • #42
  • Oct 11, 2017 4:32am Oct 11, 2017 4:32am
Quoting Nicholishen
{quote} I'm thinking perhaps a ZigZag based indicator that plots the percentage of swing overlap based on n swings (eg 100% is perfectly sideways and choppy) and the linear regression slope using the swing points as inputs.
The issue I found with this approach is that when the swings are big enough they no longer form a choppy condition. Look at the top right picture: the swings completly overlap without creating the choppiness. On the top-left image you can hardly define a single swing that make sense with respect to the (high) variance. I also tried the overlap of nearby candles with no better result.
No greed. No fear. Just maths.
  • #43
  • Oct 11, 2017 4:39am Oct 11, 2017 4:39am
Quoting alphaomega
{quote} Maybe we can get some useful ideas from the concept of Surface roughness. There are international standards for the calculation. I'm' pretty sure we can apply the formulas directly to market prices. You should also check the works of Benoit Mandelbrot. As you probably know he is the "father" of fractals. He's done extensive studies on market prices to develop some of his theories.
Perhaps roughness is indeed this better term I should use. Unfortunately I can't figure out how the Rx formulas can help. They use the values without taking their ordering into account. These two graphs are made of the same values. I simply put them in increasing order to build the red one. They clearly don't have the same roughness but all the formulas would give the same result because they are sums of powers of the Yi.
Attached Image


Regarding fractal I searched in direction of the Hurst exponent (https://en.wikipedia.org/wiki/Hurst_exponent). There are several estimators of it. The rescale range is the simplest and actually the best: it is robust to fat tail and outliers. It needs a lot of samples (50000) to give reliable result. The other methods need even more. That's obviously way to big.

I implemented the variation index based on the paper Dimension of the Minimal Cover and Local Analysis of Fractal Time Series by Dubovikov & Starchenko [2004]. I gives good results with small sample size (128 or 256) but the lag is important and it cannot localize short term choppy zones. Here are two examples of what if gives:
Attached Image (click to enlarge)
Click to Enlarge Name: EURUSD-D1-1.png Size: 21 KB

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSD-D1-2.png Size: 15 KB
No greed. No fear. Just maths.
  • #44
  • Oct 11, 2017 4:42am Oct 11, 2017 4:42am
Quoting metta87
{quote} choppiness-ranging-trend if you wanna see it better, quit using candlestick, range bars or point and figure are way better
I can only agree with you because that's exactly what I do.
I do it mainly for quantitative reasons: https://www.forexfactory.com/showthread.php?t=481691
CRB also break down the sharp moves
No greed. No fear. Just maths.
  • #45
  • Oct 11, 2017 9:11am Oct 11, 2017 9:11am
Quoting PipMeUp
{quote} I can only agree with you because that's exactly what I do. I do it mainly for quantitative reasons: https://www.forexfactory.com/showthread.php?t=481691 CRB also break down the sharp moves
Not sure if this helps you Pip but Bill Dreiss from Dreiss Research developed a choppiness index for his fund described in the attachment on this post and his fund's results do speak for themselves. :-)

Attached Image (click to enlarge)
Click to Enlarge Name: Capture.JPG Size: 90 KB
  • #46
  • Oct 11, 2017 10:27am Oct 11, 2017 10:27am
Happy to see you here C

The abstract says "E.W. Dreiss has developed the Choppiness Index using chaos principles to measure market trendiness". Choppiness is not the opposite of trendiness. Look at the chart in the last page of the article:
- The first rectangle marked Choppy is not choppy to me. It is a ranging market with well identified swings bouncing the box. Same for Jul-Aug '93. Almost a text book buy the support, sell the resistance. Yet the oscillator is high.
- The highest high in the middle is where I'd like to see the oscillator at its highs. The price goes nowhere, gaps up and down...
- The subsequent downtrend is quite choppy too yet the oscillator is decreasing.

I think the problem comes from the step three in the calculation. It (essentially) takes the width of a 14 day donchian channel. The problem is that when the market trends strongly this channel widens irregardless of the choppiness inside. I'm not telling this indicator can't be useful but it doesn't fit my bill.

If you draw your SDC I'd like to measure how messy it is inside independently of the slope of the channel is very steep or flat.
No greed. No fear. Just maths.
  • #47
  • Oct 11, 2017 3:43pm Oct 11, 2017 3:43pm
Have you settled on a definition for 'choppy/rough'? Is it a diminished rate-of-change in price movement, in both directions? Or maybe an increase in disorder (entropy or randomness) over a time period?

If trend can be defined as a higher rate-of-change in price in one direction over a period of time then a range could be defined as a lower (smaller) period between peaks and troughs, and possibly a lower (compressed) amplitude in price change. In that sense there isn't a 'real' difference between trends and ranges except that trends continue throughout the selected trading horizon (time frame), while ranges will reach local minima and maxima several times during the same horizon. In both cases order, and randomness are comparatively low, as price action can be defined, and, maybe, to a limited extent, anticipated.

So, choppiness has to be something that makes anticipation/definition difficult. Is it entropy? If so:

In the case of ranges/trends we could define a theoretical mean for a given time period for a trending condition, and a ranging condition. Then we could measure the divergence between the mean during 'choppy' periods.
  • #48
  • Oct 11, 2017 3:59pm Oct 11, 2017 3:59pm
Ok, I just saw your charts with the 'steady' and 'choppy' examples. Let's say that 'steady' ranges and trends fill up their (selected) time interval in a less uniform way - there is more discrepancy between each succeeding point (the movement in one direction) than in a choppy market.

In that case choppy periods are low discrepancy sequences. Sequences with a low discrepancy fill up the interval (0,1) in a more uniform way. That means that any portion of that time interval (subinterval) [a,b] contains approximately (b−a)×N of the points x1,…,xN, for each choice of N. Since candlesticks are basically a sampling of price data at selected points in time, you could use integration to get an approximate value.
  • #49
  • Edited 6:09pm Oct 11, 2017 5:56pm | Edited 6:09pm
Quoting clemmo17
Have you settled on a definition for 'choppy/rough'? Is it a diminished rate-of-change in price movement, in both directions? Or maybe an increase in disorder (entropy or randomness) over a time period? If trend can be defined as a higher rate-of-change in price in one direction over a period of time then a range could be defined as a lower (smaller) period between peaks and troughs, and possibly a lower (compressed) amplitude in price change. In that sense there isn't a 'real' difference between trends and ranges except that trends continue throughout...
Hi clemmo17. I love your thinking on this.

I know I am in deep water with you maths guys here as the economy of your explanations that strike right to the heart of the matter is quite unsettling to a babbler like me.

The way I think I understand you and Pip here is that you are looking for some method to discriminate between structured (ordered) versus non structured (random) sideways congestion and clemmo was applying principles of entropy as a means to delineate between these two states.

My take on my first read was that we could apply some form of efficiency ratio to discriminate signals that contributed to the causal structure of sideways congestion (eg. a structure existing between two horizontal constraints) versus simply noise that has no causative bearing on the structure apart from being the mortar that fills in the structure. For example Perry Kaufmann's signal to nose efficiency ratio seeks to delineate the signal versus the noise in trending conditions. I was thinking that we could apply this principle to sideways congestion in terms of structured form versus random form.

I therefore started to think about geometrical forms that might lead to causative structured form of sideways congestion such as support and resistance levels separated by vertical supports in the form of waves, fractals or euclidean geometry.....but then after reading clemmo's post...reckon that choosing a geometrical condition is probably a bit pointless as there are so many geometries that sideways congestion could take and it is a bit like finding a needle in a haystack. Furthermore, this type of structured form is not really applicable for the markets as that is not how structure is created (like natural systems). Structure in markets can be attributed to auto-corrrelation or repetitive behaviour separated by time intervals and this time interval can be arbitrary in duration.

But back to Perry Kaufmann's signal to noise ratio....can't we just apply the same principle that is engaged by him in measuring the signal that actively contributes to the vertical price displacement versus those price perturbations that add no value to that move....but we just turn it on its side and deal with horizontal trends?

I am probably in deep water here but just thought I would add my brain farts to the puzzle you are solving. :-) C
  • #50
  • Oct 11, 2017 8:00pm Oct 11, 2017 8:00pm
I hadn't heard about Kaufmann's ratio but this
Quote
It is calculated by dividing the price change over a period by the absolute sum of the price movements that occurred to achieve that change. The resulting ratio ranges between 0 and 1 with higher values representing a more efficient or trending market.
sounds very similar to what I was thinking. A choppy period is a less efficient market.
  • #51
  • Oct 12, 2017 4:37am Oct 12, 2017 4:37am
No clemmo17 I couldn't find a definition of this. It is still an ill-posed problem to me. The increase of entropy can be this definition. Again this is not about range vs trend. It is more about the energy/amplitude of the (not-so-)white noise component of the signal. Here is a graph to try and explain the thing. The purple line is the trend. I used a deterministic parabola. I added correlated noise. Between epoch 170 and 240 I also added white noise and a 2 standard deviation envelop in red. I think I'm trying to single out the amplitude of the white noise part (the width of the red envelop).

I don't understand your idea of discrepancy sequences.

Attached Image (click to enlarge)
Click to Enlarge Name: graph.png Size: 8 KB
No greed. No fear. Just maths.
  • #52
  • Oct 12, 2017 4:38am Oct 12, 2017 4:38am
Yes C some efficiency ratio is what I'd like to get. But one more time Perry Kaufmann conflates range and choppy. His indicator is the ratio between momentum and the length of price path. Therefore if momentum increases the oscilator will as well. I need it trend immune. Using Kaufmann after de-trending the data could be a start. This leads to Chande Momentum Oscillator. Despite its name it seems to be more trend-rejecting (it is essentially a bandpass filter). My problem seems to stem from the fact that white noise spans the whole frequency spectrum. So it overlaps the lowest bandwidth (the trend component) and the middle band (the correlated noise).
No greed. No fear. Just maths.
  • #53
  • Edited 6:19am Oct 12, 2017 5:53am | Edited 6:19am
Quoting PipMeUp
Yes C some efficiency ratio is what I'd like to get. But one more time Perry Kaufmann conflates range and choppy. His indicator is the ratio between momentum and the length of price path. Therefore if momentum increases the oscilator will as well. I need it trend immune. Using Kaufmann after de-trending the data could be a start. This leads to Chande Momentum Oscillator. Despite its name it seems to be more trend-rejecting (it is essentially a bandpass filter). My problem seems to stem from the fact that white noise spans the whole frequency spectrum....
Thanks Pip. It is a tough nut to crack but "the degree of correlation immunity" seems to be the key. I will have a good search around the journals and let you know if there might be some idea of relevance that could be applied.
  • #54
  • Oct 12, 2017 7:55am Oct 12, 2017 7:55am
Quote
I am probably in deep water here but just thought I would add my brain farts to the puzzle you are solving. :-) C
From what I read in your thread you're also trying to solve it under the name strategies diversification.

Let's pretend the market is always in uptrend. Buy-and-hold is the obvious strategy. => zero risk just wait

Let's pretend the market is always in some trend either up or down. A simple strategy is to alternate long and short. Copernicus is a happy man thanks to his DDT. Long period MA cross also does the job. The risk is limited to a wrong trend sign identification.

Let's pretend the market is always a random walk. The strategy is to not trade at all (riskless).

Let's pretend the market is a strongly autocorrelated noise... life belt picture for Cop' to not get drowned:
Attached Image (click to enlarge)
Click to Enlarge Name: autocorrelated_noise.png Size: 14 KB

As a side note this is a stationary process BUT IT IS NOT MEAN REVERTING despite it oscillates around the zero line forever.
To trade this you could go to the lower TF and trend trade or use a short period MA cross. Because it is stationary you can even average down.

Let's pretend the market is pure white noise:
Attached Image (click to enlarge)
Click to Enlarge Name: white_noise.png Size: 6 KB

No way to find a pair of MAs to cross... But it is stationary you can buy or sell anywhere and make money with a bit of average down. You can also grid trade.

Market is none of those but it can be seen as a mixture of all with the quantity of the ingredients changing all the time. Let's mix the uptrend and the white noise:
Attached Image (click to enlarge)
Click to Enlarge Name: noise_trend.png Size: 7 KB

Now buy and hold is no longer the smartest. It is more lucrative to buy in the lower part and take profit in the upper part then wait the price to randomly come back in the lower part. The sum of the swings are longer than the direct move. Going short is dangerous. You don't want to average down against the trend, do you?
Attached Image (click to enlarge)
Click to Enlarge Name: green_better_than_blue.png Size: 32 KB


Trend+autocorrelated noise? Buy the dip and forward grid (https://sites.google.com/site/prof7bit/snowball) until the swing fades.

To enter your trade you may widen your SL when the white noise part increases. You can also use this volatility by waiting the price to trigger a limit order with a much smaller SL. You will enter less often but the RR is twice or three times bigger...
Attached Image (click to enlarge)
Click to Enlarge Name: SL_matters.png Size: 20 KB
No greed. No fear. Just maths.
  • #55
  • Edited 8:52am Oct 12, 2017 8:34am | Edited 8:52am
Quoting PipMeUp
{quote} From what I read in your thread you're also trying to solve it under the name strategies diversification. Let's pretend the market is always in uptrend. Buy-and-hold is the obvious strategy. => zero risk just wait Let's pretend the market is always in some trend either up or down. A simple strategy is to alternate long and short. Copernicus is a happy man thanks to his DDT. Long period MA cross also does the job. The risk is limited to a wrong trend sign identification. Let's pretend the market is always a random walk. The strategy is to...
That's a fantastic post Pip....and thanks for adding the life-supports for me....as I am a visual person. Your breakdown of the different market conditions is exactly how I see it as well.....as in distinct phases of transition in market condition.

You are many steps ahead of me in dealing with what I refer to as sideways congestion. You have a more elegant and defined definition comprising both random and non random components.

I have my solutions with a diversified portfolio strategy when bias is evident under diversification by blending the return streams, but as I acknowledge and you have identified, I am exposed in relation to sideways congestion. Also there is a greater efficiency in relation to enhanced position sizing if I could discriminate these conditions with greater fidelity. My EDTT is quite efficient but could be more so as my SDC is really only attacking the trend definition with some breathing space for the other conditions you mention that are not specifically addressed.

Filters for market condition just don't seem to work given their lagging nature so I am at a roadblock with respect to a solution in sideways chop which does not compromise the other elements of the diversified strategy. If the market conditions continue under sideways congestion for extended periods....it is drawdown city for me despite the efficiencies I have built into the design to attempt to combat this.

Given the lagging nature of filters I believe in a rubbery round about way that the solution is found by destructive and constructive interference of return streams that are turned on 24/7.

At the moment I am working in zones where the potential for bias appear evident....but I strictly avoid the noise where possible.....and this is far easier than what you are proposing....while I say my strategies are 'turned on 24/7' the reality is that my avoidance of the noise means that in reality they are not.... ..but I can see where you are going with this. You don't mess around mate. You are going for the 'holy grail' in terms of a linear equity curve....but you are many steps ahead of me.

The thing about these type of problems is that often you need to just follow your current path and then something pops up where *hey presto* that solution you were looking for just miraculously appears.

I need to mull on it but having a different angle which you have provided here really helps mate....greatly appreciated. If something pops I will share and hopefully vice versa:-)

PS I love the snowball as you already know....but it was you that pointed out it's shortcomings to me.
  • #56
  • Oct 12, 2017 12:35pm Oct 12, 2017 12:35pm
Quoting PipMeUp
{quote} From what I read in your thread you're also trying to solve it under the name strategies diversification. Let's pretend the market is always in uptrend. Buy-and-hold is the obvious strategy. => zero risk just wait Let's pretend the market is always in some trend either up or down. A simple strategy is to alternate long and short. Copernicus is a happy man thanks to his DDT. Long period MA cross also does the job. The risk is limited to a wrong trend sign identification. Let's pretend the market is always a random walk. The strategy is to...

Very deep, thanks for the all directions heads-up....
I am gonna go with a little bit of conspiracy theory.... What if the markets are all about kill the retail money or dumb money???
The reason why I say this is there are hundreds maybe thousands of minds working on the subject (on the retail side) and no long term edge is found. Due to normal distribution (or hot hand theory) some periods one makes money, and some periods one loses. But on the long term all the retail traders end up losing.
Is this pure lack of knowledge or the markets are just simply after "dumb money"??
Just a question to be thought about...
  • #57
  • Oct 12, 2017 1:02pm Oct 12, 2017 1:02pm
Quoting GoldTheHun
{quote} and no long term edge is found.
how do you know that ?
  • #58
  • Oct 12, 2017 1:53pm Oct 12, 2017 1:53pm
I made a very quick (and dirty) experiment in "measuring" (very roughly) the white noise part... The band is supposed to widen when the market is more rough/choppy/chaotic and get tigher when the move is smoother... Not fantastic and laggy (of course) but it spots some interesting zones.
The oscillator plots the width of the band. The dots are here to show what I consider as a swing (to avoid ambiguity and/or give some definition of trend)
Attached Image (click to enlarge)
Click to Enlarge Name: just_a_quick_and_dirty_try.png Size: 38 KB


A bit better on the weekly chart (I don't know why)
Attached Image (click to enlarge)
Click to Enlarge Name: EURUSD-Weekly.png Size: 35 KB


@GoldTheHun: I think open markets were created to prevent speculation. Eating you alive is the markets' job.
No greed. No fear. Just maths.
  • #59
  • Edited 2:23pm Oct 12, 2017 2:03pm | Edited 2:23pm
Quoting metta87
{quote} how do you know that ?
It was just a thought, seeing so many people fail.

I have coded an indicator which collects traders real time transaction data from forexfactory and myfxbook. Trading opposite to the sentiment clearly creates an edge. I couldn't explain the reason for that. That is why I have come up with the idea. That is my edge for now, but unfortunately it is not based on my analysis but only based on exploiting the retail sentiment vulnerability... So I can not say it is my own talent.....

Watching this indicator is unreal. If you see the retail traders making a spike upward, the price will go down and vice versa. Although I am happy to see this which gives me a clear edge, it also makes me suspicious of the whole trading process....

By the way I only trade futures, no retail forex brokers, no cfds....
  • #60
  • Edited 4:19pm Oct 12, 2017 2:26pm | Edited 4:19pm
i measure the choppiness by looking at the candle color over all 28 currency pairs. 2 or more bullish/ bearish candles in a row let the indicator go down.
in this case the pair is not important the indi will show the same lines on all pairs. the colored lines are the 8 currencies and the black line is the whole market.

its interesting that timeframes from M30 and less are constantly choppy while H4 and more have "trend" and "range" periods.
you get the same results when you just measure the movement of one currency pair.

what people understand under a range market is a flat price channel over a long time.
i think this method is not suited to measure this kind of market behavior but i could be wrong, feel free to interpret the results.

this small research was a confirmation for me that my trading strategy works best on the lower time frames cause the market is more stable.
Attached Image(s) (click to enlarge)
Click to Enlarge Name: h4.PNG Size: 94 KB
Click to Enlarge Name: h1.PNG Size: 85 KB
Click to Enlarge Name: m30.PNG Size: 64 KB
0 traders viewing now