Statistical mean of the market [quant corner] | Page 5 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Explore Real (Statistical) Market Edges 21 replies

The Birth of a Quant Trader 107 replies

Quant trading & QuantLib 6 replies

Quant Corner 12 replies

questions from a future quant 39 replies

Attachments: Statistical mean of the market [quant corner]
Exit Attachments

Statistical mean of the market [quant corner]

  • #81
  • Edited 6:18pm Oct 9, 2014 6:04pm | Edited 6:18pm
Ok this method stands so far, as i also introduced the filtering of the signal itself, to cutoff the small amplitudes which are only a noise, mostly we are talking about <2 pip noise, but it is still a noise and we dont want that.

I found that a filter like that can also cut off the price spikes and the minor fluctuations without causing unproportional lagg.It is proportionally useful to say like this.

Oh an also i will use the WINRATE as a metric too, so we will have 3 main metrics and 1 optional.

Why the winrate ? Well we got the expectancy so we really dont need the winrate, however the winrate plays the role when we are talking about consecutive losses. You see we only measured the intra-trade drawdown, but what if there are 4 consecutive losses, it is equivalent to 1 big intratrade DD, 4 smaller extra-trade DDs, yet they were not measured yet.

Although i dont measure directly the consecutive losses because that depends on luck, i could measure the maximum one, but it can also be derived from the winrate directly, so its ok to use only the winrate.

For example a winrate of 20%, will have a probability of 5 consecutive losses is about 96.875% under normally distributed trades (which normally are depending on the strategy) but roughly speaking we can assume it.

So obviously we want the highest Winrate possible to be able to use the highest lot size possible without assuming huge risk for the equity.

In case of a 20% winrate like above we need to have a very conservative lot size otherwise it would be a cowboy madness. Risk management is the first priority for any trader.

So my current GEN 2.6 FILTER HAS:

  1. 1.45 RFE
  2. 393 TRADES
  3. 22.3% WINRATE
  4. 1.47 PROFIT FACTOR

The equity curve looks like this:

Attached Image (click to enlarge)
Click to Enlarge Name: shhhhhhh.png Size: 18 KB


This is my HOLY GRAIL for the moment, I hope that my research will show me better ones later on.
"There's a sucker born every minute" - P.T. Barnum
  • #82
  • Oct 10, 2014 11:59am Oct 10, 2014 11:59am
Ok I`m bored as my PC is optimizing so lets do a little risk management calculation:

How much Risk % / trade should I take if my current winrate is 22.3% and have a risk of account wipe of less than 0.00001%?

Well the equation is simple:

We got a 0.223 winrate in and we need to have Y amount of consecutive trades, that is less than 0.00001%

So the formula is (in a normally distributed case)

(1-0.223)^y <0.00001 , therefore the answer to this is y>36 so lets take 37 for example

We will have at most 37 consecutive losing trades according to our risk limit to be probable, so we need to adjust the account to that which is: 100% / 37, which is 2.70%.

CONCLUSION:

Therefore we can never ever risk more than 2.7% of the equity on a single trade because than can jeopardize our account, if we were to have a 22.3% winrate.Now the winrate may vary because the swaps are not considered, but still it is a good guideline for the traders who risk 30% of their account /trade

Furthermore all depends on the robustness of the 0.00001% number, so even if we risk <2.7% there is still an unprobable chance that the account can have 38 consecutive losses and the money is gone, so how much does that 0.00001% number mean in terms of probability.

Well i`ve seached a bit and it turns out that the probability of us being in an plane crash is aproximately 0.00001%, so you have equal chance of suffering a plane crash than to lose the entire account if you risk <2.7%, so therefore it is not risky.
"There's a sucker born every minute" - P.T. Barnum
  • #83
  • Oct 11, 2014 7:43pm Oct 11, 2014 7:43pm
Ok here is the filter 2.7 version:

Attached Image (click to enlarge)
Click to Enlarge Name: shhhhhhh.png Size: 50 KB


This is the balance curve difference between the 2 versions

And this is the nominal difference:

  1. 1.65 RFE (+0.2)
  2. 397 TRADES (+5)
  3. 22.6% WINRATE (+0.3)
  4. 1.49 PROFIT FACTOR (+0.02)

Seems a little bit improved, well the balance chart is not showing much, but yes, it is hard to find a perfect holy grail, so keep searching. The filter gets better and better with each trial, so it is just a question of patience and good ideas.

"There's a sucker born every minute" - P.T. Barnum
  • #84
  • Oct 13, 2014 12:09pm Oct 13, 2014 12:09pm
Quoting PipMeUp
You should therefore naturally try with range bar charts.
Ok i`ve checked if the range bars were good enough, but looks like it is not:

The 10 pip constant range bar failed horribly on the long run, so i used 46 pip bars, because 46 pip is the average bar size on H4 timeframe to be similar.

So the best 46 pip bar results looks like this,everything else was rubbish:

Attached Image (click to enlarge)
Click to Enlarge Name: T1.png Size: 72 KB


Attached Image (click to enlarge)
Click to Enlarge Name: T2.png Size: 114 KB


Not very good. I`ll just stick to normal prices for now, maybe i should try logarithmic charts, i`ve heard that it matches stock index returns distribution, might be similar in FX to0.
"There's a sucker born every minute" - P.T. Barnum
  • #85
  • Oct 13, 2014 1:28pm Oct 13, 2014 1:28pm
You don't really explain what you're trying to do and how you do it so it's difficult to help or even give ideas. I thought you were using the 2nd moment of the distribution of the price return (variance) but it looks like not. You seem to focus on the mean exclusively. In this case of course the normality of the range bars won't help since the mean is the same for both the kind of distribution.

It is very surprising that the range of the bars make a so big difference in the results. Are you sure you don't overfit the dataset? How do you reject the null hypothesis that "the results are random"?
No greed. No fear. Just maths.
  • #86
  • Oct 13, 2014 1:42pm Oct 13, 2014 1:42pm
Quoting PipMeUp
You don't really explain what you're trying to do and how you do it so it's difficult to help or even give ideas. I thought you were using the 2nd moment of the distribution of the price return (variance) but it looks like not. You seem to focus on the mean exclusively. In this case of course the normality of the range bars won't help since the mean is the same for both the kind of distribution. It is very surprising that the range of the bars make a so big difference in the results. Are you sure you don't overfit the dataset? How do you reject...
I`m trying to find a mean which i can use like a moving average to generate BUY/SELL signals if the price crosses above or below it.

This is the only way i found to make sure the filter is accurate, as you can measure directly it's results on the balance, and it is fit for trading, spreads included.

The variance was used for another project, but I abandoned that one as i got nowhere withthat. Now i`m only focusing on the filter.

I am overfitting, and that is the point, if it is underfit then you have no way of guessing what parameters are optimal. Yes you can use clustering or you can find the best period of autocorrelation but it will be too random, as the market can change shape any time, and reshuffle all the parameters at an instant.
So my theory is this: if I overfit it, and the sample size is big, then if the results are good, it will be a durable strategy for long term.

And so far the previous filters are overfitted, and yet shown good results, because it adapts to the shape of the market very good.

So if i can fit my mean to the longterm shape of the market then I can also use it to forecast atleast that much time as much historical data it had.

I have 13 years of historical data, so i expect my strategy to work atleast another 13 years. While recalculating the parameters lets say yearly, to keep it updated.
"There's a sucker born every minute" - P.T. Barnum
  • #87
  • Edited 4:29pm Oct 13, 2014 3:21pm | Edited 4:29pm
I have to aggregate the DD , the EXPECTANCY and the number of trades so that it can be compared considering all the caracteristics of the particular filter:

So i invented the PROXI SCORE which is calculated as follows:

PROXI SCORE= EXPECTANCY / MAX INTRATRADE DD * NR OF TRADES

or

PROXI SCORE = RFE * NR OF TRADES

Now the Profit can be calculated as Expectancy * Trades, but that is the end profit and doesnt weight in the DD Risk / trade in each particular case. And the DD/ trade will be the biggest factor to look at when sizing the LOT, so we must consider the DD too so that after we can properly set our maximum risk.

So instead of purely looking at the profit, we look at the PROXI SCORE as that will decrease the profit's nominal value if the RISK TOOK is too much to hold for that particular earning.

So i will recalculate the PROXI SCORE for my earlier filters and lets compare them now.

EDIT:

Yes the current 2.7 filter is the best one from comparing the PROX SCORE:

  1. 655.7 PROXI SCORE (+85.3)
  2. 1.65 RFE (+0.2)
  3. 397 TRADES (+5)
  4. 22.6% WINRATE (+0.3)
  5. 1.49 PROFIT FACTOR (+0.02)

"There's a sucker born every minute" - P.T. Barnum
  • #88
  • Oct 13, 2014 6:59pm Oct 13, 2014 6:59pm
Not sure where you are going next but... I would have thought that you need to study two MA's together (could be more MA's but lets just stick with two for now).

What I am getting at is consider looking at the inter-relationship between two MA's. I know thats a pretty open statement as there is an infinite number of variables you could use, but a bit of common sense and you should be able to get ballpark variable values to use. I am not talking about a simple MA cross strategy, more I mean how each of the MA's are behaving in comparison to the other.

e.g.
If we just consider one aspect of the two MA's inter-relationship - How far apart are the MA's
Historically what is the maximum distance they have ever been from one another?
Historically what is the typical distance apart before they return? (i.e. is the market at an extreme point where price is statistically likely to collapse soon)
Historically what is the typical length of time that the two MA's stay within 5% of daily ATR? (i.e. what is the statistically likely length of time that pair ranges for)

The options are endless, above is just a few that comes to mind. (I have never studied the time element, I just thought of that then..)

When I refer to distance above I recommend you use a yard stick that is relative, e.g. % of ATR or similar. When using the ATR, measure it against a timeframe much higher than the timeframe of study. So if you are studying the H1, use the Daily ATR.

I know thats a different spin on what you have been trying to do, but that seems to be a more useful harvest of data than a simple Buy/Sell line.
I have only just started down this road myself, so I would be interested in your thoughts.
What's it all about? It's all about money.
  • #89
  • Oct 13, 2014 7:13pm Oct 13, 2014 7:13pm
Quoting PipMeUp
{quote} Because you can't measure them at the very moment they happen. The ATR lags like any other filter. If you substract the detected spike from the PA you'll actually add an opposite spike just after the original. This is exactly where I failed with my cycle detection and removal. I could never compensate the lag.
Hi PipMeUp,

If you dont mind just for the sake of interest would you expand on what you said above? You know your stuff and I am interested to understand what you mean a bit better. I dont get "If you substract the detected spike from the PA you'll actually add an opposite spike just after the original"

I personally dont see the need for a MA filter like this but think it could work if you did the following:
If we were creating an MA applied to the H1 chart, filter out all the bars that had a true range in excess of y% of Daily ATR.
ATR is an important relative measure but if you use a higher timeframe it shouldnt "lag."
Yes the chosen % is arbitrary but if a single H1 candle was to move 50% of the Daily ATR you surely could consider it a spike.

Again I am just curious, not trying to go anywhere with this.
Thanks
What's it all about? It's all about money.
  • #90
  • Oct 13, 2014 7:15pm Oct 13, 2014 7:15pm
Quoting Innate
Not sure where you are going next but... I would have thought that you need to study two MA's together (could be more MA's but lets just stick with two for now). What I am getting at is consider looking at the inter-relationship between two MA's. I know thats a pretty open statement as there is an infinite number of variables you could use, but a bit of common sense and you should be able to get ballpark variable values to use. I am not talking about a simple MA cross strategy, more I mean how each of the MA's are behaving in comparison to the other....
More than 2 MA's is nonsense, because if you say BUY = SMA(20)> SMA(50) & SMA(20)>SMA(100) you can just say BUY = SMA(20)>SMA(100) , so it is equivalent.
You could use more than 2 MA in the sense of filtering like BUY = SMA(20)> SMA(50) && EMA(372)> EMA(421) , but i think if the filter types are too different, the curve fitting becomes too rigid.

You see i`m curve fitting adaptively and have no interest in rigidity and non-adaptivity. So i think all the filters must be homogeneous of the same nature, but with different parameters.
Or atleast similar, in the sense that 1 can be a slower less adaptive signaling device, while the other is a quick adaptive and non lagging one, and the goal would be to find the perfect balance between these.

There are infinite variations of MA's but there are many papers studying MA's and their behaviour on historical stock & other markets. So its not simply a blind hunt.

So far i dont want to complicate my research with ATR and other indicators, i`m looking only at MA's now.
"There's a sucker born every minute" - P.T. Barnum
  • #91
  • Oct 13, 2014 7:48pm Oct 13, 2014 7:48pm
Quoting Proximus
{quote} More than 2 MA's is nonsense, because if you say BUY = SMA(20)> SMA(50) & SMA(20)>SMA(100) you can just say BUY = SMA(20)>SMA(100) , so it is equivalent. You could use more than 2 MA in the sense of filtering like BUY = SMA(20)> SMA(50) && EMA(372)> EMA(421) , but i think if the filter types are too different, the curve fitting becomes too rigid. You see i`m curve fitting adaptively and have no interest in rigidity and non-adaptivity. So i think all the filters must be homogeneous of the same nature, but with different parameters. Or atleast...
I agree with your comments but dont miss the wood for the trees. No need to use ATR, just use pips if you focus your study to a single pair. I wasnt suggesting you study anything more than MA's.
IMHO you need two MA's as you need a reference point to the other, it seems highly unlikely one can use a single MA to establish the mean of the market. Regardless of the MA type used, it is the inter relationship that yields useful data.
What's it all about? It's all about money.
  • #92
  • Oct 13, 2014 8:02pm Oct 13, 2014 8:02pm
Quoting Innate
{quote} I agree with your comments but dont miss the wood for the trees. No need to use ATR, just use pips if you focus your study to a single pair. I wasnt suggesting you study anything more than MA's. IMHO you need two MA's as you need a reference point to the other, it seems highly unlikely one can use a single MA to establish the mean of the market. Regardless of the MA type used, it is the inter relationship that yields useful data.
Fixed pips are also static and rigid, not very good tool if we want an adaptive proces.I already have a reference point, i use price.

Maybe i`ll complicate my MA later on with some oscillating formulas, but at the moment i want to find the best one in its purest form, then i will adjust it further.
"There's a sucker born every minute" - P.T. Barnum
  • #93
  • Oct 13, 2014 8:21pm Oct 13, 2014 8:21pm
Also the 2 moving average method, atleast for my current filter is not an option. It may be for other filters, but the current one can't handle it.

Attached Image (click to enlarge)
Click to Enlarge Name: 1X.png Size: 73 KB


Attached Image (click to enlarge)
Click to Enlarge Name: 2X.png Size: 57 KB


Attached Image (click to enlarge)
Click to Enlarge Name: 4x.png Size: 38 KB


This is the best result with 2 filters, and it's not good, so at the moment i`ll stick to 1 filter. It may seem like a nice swift filter on the chart, but the results speak for themselves, the profit is negative and every other stat is too.

So looks like 2 filter is too overfitting the data, without any flexibility, or atleast for my current filter, if the filter would be more adaptive, then perhaps it would not need 2 of it or even with 2 it would not loose much of its adaptability.
"There's a sucker born every minute" - P.T. Barnum
  • #94
  • Oct 14, 2014 4:57am Oct 14, 2014 4:57am
Quoting Innate
{quote} Hi PipMeUp, If you dont mind just for the sake of interest would you expand on what you said above? You know your stuff and I am interested to understand what you mean a bit better. I dont get "If you substract the detected spike from the PA you'll actually add an opposite spike just after the original" I personally dont see the need for a MA filter like this but think it could work if you did the following: If we were creating an MA applied to the H1 chart, filter out all the bars that had a true range in excess of y% of Daily ATR. ATR...
Attached Image (click to enlarge)
Click to Enlarge Name: bad_made_worse.png Size: 26 KB
No greed. No fear. Just maths.
  • #95
  • Oct 14, 2014 7:43am Oct 14, 2014 7:43am
Quoting PipMeUp
{quote} {image}
Thats only if the filter is configured wrong

Most of the time this happens:

Attached Image


The initial spike gets smoothed out, causing lagg yes, but the benefit of accuracy due to smoothing is bigger than he lag handicap caused by smoothing. Therefore it was worth filtering.

The optimal configuration is always a middle way between accuracy and lag.
"There's a sucker born every minute" - P.T. Barnum
  • #96
  • Oct 15, 2014 11:34am Oct 15, 2014 11:34am
To answer the question about if the market has a "mean": Well in practice it seems so to me.

Its EMA 24 on H1. Or EMA 48 on M30. Or EMA 96 on M15. Or... well you get it. I use that as a measurement for when the market sort of resets itself. But i am an intraday trader so this kind of thing is important to me. To someone who trades one trade a week or something like that, this kind of thing means little to nothing.

In my thinking i try not to think that price will ever go through this moving average. I think it works. I dont know if this is REAL or just something i use personally.
My signature is: "Classified".
  • #97
  • Edited 1:41pm Oct 15, 2014 1:05pm | Edited 1:41pm
Quoting yaed
To answer the question about if the market has a "mean": Well in practice it seems so to me. Its EMA 24 on H1. Or EMA 48 on M30. Or EMA 96 on M15. Or... well you get it. I use that as a measurement for when the market sort of resets itself. But i am an intraday trader so this kind of thing is important to me. To someone who trades one trade a week or something like that, this kind of thing means little to nothing. In my thinking i try not to think that price will ever go through this moving average. I think it works. I dont know if this is REAL...
Look, there can be as many correct means to the market as there are stars on the sky, many could work and be profitable while most of them would not be.

I am only looking for the best one, while EMA and SMA's can be used as a basic tool and could work, i`m looking for very advanced MA's.

The EMA,SMA,WMA, and other moving averages are very basic and primitive retail trader tools, while mine are very sophisticated and advanced ones.

This is how I would rank them:
The tools that detect the mean of the market, starting from the worst and most primitive and basic tools to the best most sophisticated tools.

  1. Diaper level: Trendlines (because they are too subjective)
  2. Nursery school: Simple MA, Cumulative MA, Geometric MA, etc.
  3. Kindergarten: Linear weighted MA, Sine weighted MA, Triangular weighted MA, etc.
  4. Elementary School: Exponential MA, and other ARIMA(p,d,q) type models
  5. High School: Unknown ???
  6. University Graduate Level: My current 2.7 filter
  7. University M.A. Level: Unknown ??? Probably some sort of basic Artificial Intelligence.
  8. University PH.D. Level: Unknown ??? Probably some very sophisticated Neural Network that Expert Hedgefunds use.
  9. Nobel Prize level: Unknown ??? A tool that can predict price so accurately that it can turn 100$ into 1 million $ in a week with minimal risk and not overleveraging. A true Holy Grail in every sense.

Obviously my goal is to reach that Holy Grail level, but very advanced quantitative trading requires a lot of mental work and a really fast PC and it's very very hard.

"There's a sucker born every minute" - P.T. Barnum
  • #98
  • Oct 15, 2014 2:04pm Oct 15, 2014 2:04pm
Quoting Proximus
Maybe i`ll complicate my MA later on with some oscillating formulas, but at the moment i want to find the best one in its purest form, then i will adjust it further.
Just take a look at the MACD formula to understand where the problem comes from.

MACD Line: (12-day EMA - 26-day EMA)
Signal Line: 9-day EMA of MACD Line
MACD Histogram: MACD Line - Signal Line

So its an oscillator, but what oscillates ? The exponential MA. So you see all derivative indicators or oscillators, that is how i call them because they are derived from MA's, are using moving averages.
The base of all oscillators are moving averages. So if the foundation shakes, the entire oscillator is worth nothing. So the problem is with the EMA.

We know that the EMA is not the best one, as my current filter is much better. The EMA is just a Elementary level indicator, from my indicator rankings.

So it doesnt matter how good is the oscillator, if the underlying filter, the EMA is bad.So first i just find the best MA out there, and then make an oscillator from it, if needed, who knows maybe its better just in its pure form. Nobody said that the oscillator based indicators are the most efficient.
"There's a sucker born every minute" - P.T. Barnum
  • #99
  • Oct 15, 2014 2:09pm Oct 15, 2014 2:09pm
Quoting Proximus
{quote} Look, there can be as many correct means to the market as there are stars on the sky, many could work and be profitable while most of them would not be. I am only looking for the best one, while EMA and SMA's can be used as a basic tool and could work, i`m looking for very advanced MA's. The EMA,SMA,WMA, and other moving averages are very basic and primitive retail trader tools, while mine are very sophisticated and advanced ones. This is how I would rank them: The tools that detect the mean of the market, starting from the worst and most...
Im thinking neural networks.
My signature is: "Classified".
  • #100
  • Oct 15, 2014 2:23pm Oct 15, 2014 2:23pm
Quoting yaed
{quote} Im thinking neural networks.
That will give you really big headaches, i just dont think its viable to research the NN field if you dont have like a team of PHD mathematicians and econometricists. The Hedge Fund named Renaissance Technologies has atleast 91 PHD's working on that field. What makes you think you can compete with their cumulative knowledge? Besides you would need a supercomputer to constantly test all those parameters for the NN.

I`m thinking more into a lower level artificial intelligence like genetic algorithms or intelligently enhanced filters with intelligent signal processing.

But as for now I really hope i can find something better with less sophisticated methods. You see not always the most complicated method is the best one.

But if not that will be probably the direction I will head, so i suggest that you forget about NN's as they are too complex, and focus more on lesser intelligent filters.
"There's a sucker born every minute" - P.T. Barnum
0 traders viewing now