Statistical mean of the market [quant corner] | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Explore Real (Statistical) Market Edges 21 replies

The Birth of a Quant Trader 107 replies

Quant trading & QuantLib 6 replies

Quant Corner 12 replies

questions from a future quant 39 replies

Attachments: Statistical mean of the market [quant corner]
Exit Attachments

Statistical mean of the market [quant corner]

  • Poll

  • How much year-to-year growth does your strategy have?
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  • #1
  • First Post: Edited Oct 23, 2014 2:10am Sep 10, 2014 5:58am | Edited Oct 23, 2014 2:10am
I would like to dedicate this thread to the following questions:

 

  1. Does the market have a constant mean ? If yes how to approximate it ?
  2. If the mean is not constant, its variable as the market changes structure, how to approximate it given out sample size ?
  3. My research to find the best model that models the mean.


By the "mean" i mean the statistical mean ,not the median, which is the separator half of the distribution in case of the price the half line of the entire market which for example aproximately the (alltime highest highs + alltime lowest lows) /2

So since the market is ever changing, i fixed the sample size to 1440 M1 bars which is 1 day of data, since its probably a logical separator between samples because the volatility distribution is also separated by the low liquidity nights when the London SE and NY SE is closed.
So i chooses 1440 to be out sample size, perhaps it could have been 7200 aswell which is 1 week, because there is a volatility gap between Friday afternoon and Monday morning aswell.

I`ve tested various moving averages to find the mean but most of them werent responsive to the price.Here are the classical moving averages, i always used Median Price ,dont confuse it with market median, which is (High[position]+Low[position]) /2 because it reflects the market better than Close price.

So these averages have PERIOD 1440, M1 bars, MEDIAN PRICE ,SHIFT 0
And by colour are the following: Yellow=Smoothed Simple,Blue =Simple(Arithmetic) ,Purple=Exponential Weighted, Red=Linear Weighted

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM1QA.png Size: 31 KB


As you can see the linear weighted is the most responsive, and less lagging, and the smoothed is the worst,also the simple moving average is pretty far from a mean.

And then i saw that these "classical" moving averages are as far away from a statistical mean as the sky is from the ground, so i took my skills and coded my own moving average, and i`ve been experimenting with it.Set also to PERIOD 1440, and its orange colored, while the other MA's are the same color but set them dashed so that it becomes more visible:

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM1222.png Size: 34 KB

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM13333.png Size: 38 KB

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM14444.png Size: 40 KB

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM1sfsdf.png Size: 34 KB


As you can see its much more responsive to the price, and less lagging.Also i`m measuring the variance of the price around my moving average to try to gauge in the dispersion of price around it.
I suppose the less the variance is, the more accurately my MA measures the mean, the problem is that if i increase the magnitude of my MA's filter then the variance decreases almost to 0, and my MA becomes identical with a PERIOD 1 moving average.

So its almost like a period 1 moving average is the mean of the market, that is, that every tick is the current mean.However this is absurd because then the MA doesnt tell anything about the trend or the bias in the price.

So i would need a perfect balance between LAGG AND ACCURACY or in other words the mean which the price will hover around the most, which is not itself (not the last price).

=============££££££££££============$$$$$$$$$$$$$==============€€€€€€€€€€€€===============¥¥¥¥¥¥¥¥¥¥¥¥¥================

LATEST MODEL: Version 2.8 of my filter

  1. MY RESEARCH GOAL: 5,000% GROWTH (BASED ON BACKTEST)

"There's a sucker born every minute" - P.T. Barnum
  • #2
  • Edited 6:22am Sep 10, 2014 6:10am | Edited 6:22am
Take a look here i`ve set my magnitude to 1600x and now my MOVING AVERAGE is almost identical with a PERIOD 3 or PERIOD 4 MOVING AVERAGE, while its still set to PERIOD 1440, because the magnitude is exponentially decreasing as the price moves away from the last price.

As you can see the variance is converging towards 0, as my magnitude goes toward infinity

Attached Image (click to enlarge)
Click to Enlarge Name: adsasdasdadad.png Size: 32 KB


And if this is the true mean of the market then it doesnt help me to measure the trend or bias in any way.

So because of this, i must find a mean which is the optimal balance between the LAGG OF FILTERING & ACCURACY OF THE BIAS (TREND)

Anyone got any ideas?
"There's a sucker born every minute" - P.T. Barnum
  • #3
  • Sep 10, 2014 11:00am Sep 10, 2014 11:00am
You can't find an answer to this question because it is an ill-posed problem.

- If you need the lowest squared error with the price then the price itself is the trivial result.
- If you want it maximally smooth the linear regression is the solution: a line is flat, infinitely differentiable and all the derivatives are 0. You can't have smoother.
- If you want the price to "balance" around the line the median is the trivial result. Per construction you get as many samples above as below.
- If you want/accept to define a priori a tradeoff between the error and the smoothness, the Hodrick–Prescott filter is the optimal solution. Except you may also want the filter to not completely repaint at each and every sample

In any practical case you'll need to define a time scale. If you have 3 swings down, up and down do you consider they form an "oscillation" around your mean or do you consider they are 3 trends that the mean shall follow? This depends on the scale of the swings. So the question is: What do you consider a "good" mean?

This time scale will translate into a frequencies bandwidth. Whatever the bandwidth you select you can't find any causal filter which will cut off perfectly. There will be moves that you won't be able to classify "big swing" or "small trend".

After a lot of research I finally accepted that I had to use a predefined personal and totally not market data objective value for the bandwidth. I estimate this "target" mean by maximal likelyhood methods and get my Kalman filter/smoother you don't like.
No greed. No fear. Just maths.
  • #4
  • Sep 10, 2014 1:25pm Sep 10, 2014 1:25pm
Quoting PipMeUp
- If you need the lowest squared error with the price then the price itself is the trivial result.
- If you want it maximally smooth the linear regression is the solution: a line is flat, infinitely differentiable and all the derivatives are 0. You can't have smoother.
- If you want the price to "balance" around the line the median is the trivial result. Per construction you get as many samples above as below. - If you want/accept to define a priori a tradeoff between the error and the smoothness, the Hodrick–Prescott filter is the optimal solution....
Well i would like to estimate a volatility later on and compare it with the Garch[1,1] volatility to see which one is better so first i need a mean.

1) That would be nonsense, as this doesnt tell me anything about future events, in a time series like the market if i put on a trade at X index, what is the guarantee that X+1 index will behave as the mean suggested at X.If the trend is smooth then atleast i can estimate the likelyhood of it to continue.

2)The regression doesnt really adapt to market shape thus its impossible to measure SD on it, also it doesnt tell me anything about volatility, the variance can be distributed infinite number of ways around that line.

3) The HP looks pretty cool, what it the difference between that an the Kalman filter ?
I already got the C code for the Kalman but i would also test the HP filter too to see which one fits my goal better.

Quoting PipMeUp
\ If you have 3 swings down, up and down do you consider they form an "oscillation" around your mean or do you consider they are 3 trends that the mean shall follow?
A swing with amplitude less than 2*STOPLEVEL is pretty useless as its unpractical to trade it with retail accounts.

STOPLEVEL for me is 4 pips on E/U, so i would look for atleast 8 pip movements, if the swing is smaller than that then obviously it needs to be cut off as its only noise for me.

I guess there are parameters which which i can play with so i will configure them once i find my filter
"There's a sucker born every minute" - P.T. Barnum
  • #5
  • Sep 10, 2014 1:58pm Sep 10, 2014 1:58pm
I`ve found a HP filter in MQL4, but it looks like its repainting

Looks like i have to rebuild it myself, i hate repainting indicators, its better to lagg and to show the lagged bars than to repaint...

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM5.png Size: 37 KB
"There's a sucker born every minute" - P.T. Barnum
  • #6
  • Sep 10, 2014 3:53pm Sep 10, 2014 3:53pm
None of the examples was supposed to be useful. Of course HP filter "repaints". It is a GLOBAL estimator: it repaints the whole dataset. All the four examples were meant to show that the problem is ill posed. If you want to remove every movement smaller than 4 pips you can use range bars or renko bricks or grid bars. But this won't solve your problem.
No greed. No fear. Just maths.
  • #7
  • Sep 10, 2014 3:55pm Sep 10, 2014 3:55pm
Quoting Proximus
{quote} Well i would like to estimate a volatility later on and compare it with the Garch[1,1] volatility to see which one is better so first i need a mean.
I don't understand, why do you need the mean for GARCH?

Here is EUR/USD and it's daily GARCH. You can clearly see the extremely low vols of last year:

https://dl.dropboxusercontent.com/u/...rch_eurusd.png

GARCH is much better than ATR and the other stuff retail typically uses.

But to return to the original topic, I found out that instead of searching for the perfect moving average, so that I can look at it and see the current trend, it's much better to instead detect the current market regime and then apply a moving average only to that portion.

So we turn the things around. First we detect the current regime (up, down, flat), and only then we apply (if we want) a moving average.

This solves the jump problem, as can be seen in your picture, where the huge down move in a short time (jump) makes the moving average lag for a lot of time. If instead you detect a down regime, followed by a flat regime, you can recalibrate the moving average much more quickly and eliminate most of the lag.

Of course, detecting the current market regime is just another holy grail, like the perfect moving average, but I found out that it's a more successful approach.
  • #8
  • Sep 10, 2014 9:21pm Sep 10, 2014 9:21pm
Possibly my three favorite poster's (Adal, PipMeUp and Prox) all in the same thread pulling things to pieces again.

Thank you for your contributions! Love reading the stuff you post.
What's it all about? It's all about money.
  • #9
  • Sep 10, 2014 10:39pm Sep 10, 2014 10:39pm
Quoting Adal
{quote} I don't understand, why do you need the mean for GARCH? Here is EUR/USD and it's daily GARCH. You can clearly see the extremely low vols of last year: {image} GARCH is much better than ATR and the other stuff retail typically uses. But to return to the original topic, I found out that instead of searching for the perfect moving average, so that I can look at it and see the current trend, it's much better to instead detect the current market regime and then apply a moving average only to that portion. So we turn the things around. First we...
I dont need the mean for the GARCH but i need it for so that i would calculate standard deviations off it, and then compare that with the GARCH volatility.I want to do a volatility estimation as precise as possible.

But i also need the mean for separate reason, that is for trend/bias detection.
"There's a sucker born every minute" - P.T. Barnum
  • #10
  • Edited Sep 11, 2014 1:53am Sep 10, 2014 10:45pm | Edited Sep 11, 2014 1:53am
Quoting PipMeUp
None of the examples was supposed to be useful. Of course HP filter "repaints". It is a GLOBAL estimator: it repaints the whole dataset. All the four examples were meant to show that the problem is ill posed. If you want to remove every movement smaller than 4 pips you can use range bars or renko bricks or grid bars. But this won't solve your problem.
Yes i see now, i though it only repaints because it was coded wrongly but now i see that it works only with static data.

If it were a classical lagging algo, then repainting is only due to bad programming skills, because in normal cases you can always shift back the result LAGG bars on the data, and so it will not repaint, and the current bar will always show CURRENT BAR - LAGG data , this is the elegant way to deal with LAGG.

However yes, the HP filter changes dynamically.

Well , looks like i have to look into the Kalman Filter


A renko chart is flawed because it only looks at price, we need to consider time aswell ,because the acceleration in price cannot be ignored, because it causes excessive volatility.A renko chart in my opinion is not a very elegant method to filter the price ,and it filters too much of the useful information with it.
"There's a sucker born every minute" - P.T. Barnum
  • #11
  • Sep 11, 2014 4:04am Sep 11, 2014 4:04am
Quoting Proximus
Well , looks like i have to look into the Kalman Filter
Not necessarily. First you have to better formalize what you want to estimate.
No greed. No fear. Just maths.
  • #12
  • Sep 13, 2014 1:23am Sep 13, 2014 1:23am
Quoting PipMeUp
{quote} Not necessarily. First you have to better formalize what you want to estimate.
What if i try to curve fit my MA to become the mean of the normal distribution ?

That is if precisely:

68.268949213708589717046509126407584495582593345321 % Inside

And

31.731050786291410282953490873592415504417406654679 % Outside

1 standard deviation from the mean.If i can approximate this number for the normal distribution and curve fit my MA to have a 1 SD of this then my MA will normalize the price.What do you think?
"There's a sucker born every minute" - P.T. Barnum
  • #13
  • Sep 13, 2014 5:36am Sep 13, 2014 5:36am
The market return time series aren't normally distributed. You know that. You can't fit a MA so that the distribution of the errors (distance price-MA) is a gaussian.

Say you want to fit a line to a set of sample points. If the points are roughly aligned the error is small. If the points follow a parabola (turning point) the fit is very bad and the error is big. An inflection point (a little pullback) will also give a big error. If you regress on a smaller window the regression will fit the data better (smaller error). This is because a line can locally approximate any continuous function (Taylor). You fit before and after the turning points. So I had the idea of setting a target error and fitting a line on the last N close prices making N vary so that the error is the closest to the target error. Below is the result on H1 and a 10 pip range chart. The grey bands are the 2-3 std dev. The red lines are the 4th dev. You can see that the range bar are better fit. That's because they are more normally distributed (least-squares regression is a way to get the maximum likelihood for the Gaussian error model).
Attached Image (click to enlarge)
Click to Enlarge Name: AUDUSD-H1-linreg.png Size: 31 KB
Attached Image (click to enlarge)
Click to Enlarge Name: AUDUSD-RG10-linreg.png Size: 36 KB


If you wonder why I built this that because I wanted to see if I could enter in the 2-3 band with a SL on the 4th stddev when I have already decided on the direction from a high TF. In short I wanted to add in forming lower wicks in a up swing to have a tiny SL. It fails because when the move is strong the price never enters the 2-3 band.

EDIT: the color of the line is green when the linear regression has a positive slope and red otherwise.
EDIT 2: No you can't stop-and-reverse at the color change (I tried!)
No greed. No fear. Just maths.
  • #14
  • Sep 14, 2014 6:40am Sep 14, 2014 6:40am
I dont know a lot about statistics but i can make some observations and remarks on your posts Proximus. 2 things you might find usefull:

1. Use weekly instead of Daily (1440 min) because it is the only "real" close of the markets since the markets are 24-hour but from Monday to Friday.
2. If you really want to see the soul of the pair use tick data for statistical analysis as it is the only real data. Candlesticks are indicators too. 1 Minute candle is like all the price action in 1 minute like a moving average of some sort.

Thats all i can add guys hope it helped
Don't use both feet to test the depth of the river
  • #15
  • Sep 15, 2014 4:45am Sep 15, 2014 4:45am
Quoting PipMeUp
The market return time series aren't normally distributed. You know that. You can't fit a MA so that the distribution of the errors (distance price-MA) is a gaussian. Say you want to fit a line to a set of sample points. If the points are roughly aligned the error is small. If the points follow a parabola (turning point) the fit is very bad and the error is big. An inflection point (a little pullback) will also give a big error. If you regress on a smaller window the regression will fit the data better (smaller error). This is because a line can...
Well i tried it and the smallest error difference that i could do, so far, on 1.7 million bars was 0.555174% , which i guess is pretty large on that size of data.
Sample size is 1440 M1 bars, which is 1 day.The bands represent 2 standard deviations.

TEAL = Bollinger Band
PURPLE = My Band

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM1.png Size: 35 KB


Attached Image (click to enlarge)
Click to Enlarge Name: 2.png Size: 32 KB


Attached Image (click to enlarge)
Click to Enlarge Name: 3.png Size: 34 KB


As you can see, its much better estimated than the Bollinger Band, and it adapts much better to sudden price movements
"There's a sucker born every minute" - P.T. Barnum
  • #16
  • Sep 15, 2014 7:50am Sep 15, 2014 7:50am
Quoting Strawbale
I dont know a lot about statistics but i can make some observations and remarks on your posts Proximus. 2 things you might find usefull: 1. Use weekly instead of Daily (1440 min) because it is the only "real" close of the markets since the markets are 24-hour but from Monday to Friday. 2. If you really want to see the soul of the pair use tick data for statistical analysis as it is the only real data. Candlesticks are indicators too. 1 Minute candle is like all the price action in 1 minute like a moving average of some sort. Thats all i can add...
1) Thats correct, however thats true also for daily bars because there is a volatility difference between morning-evening and noon-afternoon market liquidity, and just for simplicity i used daily sample.

I will also test in on weekly data ,but so far i worked with daily.

2) Well i used median price instead of close price because it reflects the price better, using tick data would take too much time and there is not much difference between median and tick.
"There's a sucker born every minute" - P.T. Barnum
  • #17
  • Sep 16, 2014 10:45am Sep 16, 2014 10:45am
Quoting Proximus
{quote} 1) Thats correct, however thats true also for daily bars because there is a volatility difference between morning-evening and noon-afternoon market liquidity, and just for simplicity i used daily sample. I will also test in on weekly data ,but so far i worked with daily. 2) Well i used median price instead of close price because it reflects the price better, using tick data would take too much time and there is not much difference between median and tick.
ok expecting your results
Don't use both feet to test the depth of the river
  • #18
  • Sep 16, 2014 4:10pm Sep 16, 2014 4:10pm
Somewhat interesting?

http://www.forexfactory.com/#graph=50989
  • #19
  • Sep 16, 2014 4:30pm Sep 16, 2014 4:30pm
I`ve played a bit more with filters, and i came up with my custom filter which looks like this:

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDH41.png Size: 35 KB

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDH45.png Size: 42 KB


Its definitely more accurate and gives better signals than the HP filter, and its much more simple.

Also i constructed some deviations above it, and the distribution it returns looks like this (a very rough estimate ):

Attached Image (click to enlarge)
Click to Enlarge Name: asdas.png Size: 15 KB


Applied on the chart its like this:

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM12.png Size: 33 KB

Attached Image (click to enlarge)
Click to Enlarge Name: asdads.png Size: 28 KB

Attached Image (click to enlarge)
Click to Enlarge Name: art1.png Size: 31 KB


The purpose of it is to find a distribution which has a very very small tail, so that if the STOPLOSS is outside the "tail" then it has virtually a very small change to get hit.

So far this distribution has a very very tiny tail that everything outside deviation [3] has a probability of being hit of less than 0.0011375% which is considerably better than in the case of the normal distribution is 0.0026%, and also my bands are more contracted so from RR perspective its much more better aswell.
"There's a sucker born every minute" - P.T. Barnum
  • #20
  • Sep 16, 2014 6:08pm Sep 16, 2014 6:08pm
Quoting Proximus
I`ve played a bit more with filters, and i came up with my custom filter which looks like this:
Really? The moving averages (filters) in the first two pictures are clearly drawn by hand in Paint. You can see the pixels overlap, or the missing pixels where you weren't careful enough.

But if you insist that they are real, congrats, you'll be a billionaire very very soon. Just buy when the price is 15 pips below the average and sell when it's 15 pips above.
0 traders viewing now