Statistical mean of the market [quant corner] | Page 4 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Explore Real (Statistical) Market Edges 21 replies

The Birth of a Quant Trader 107 replies

Quant trading & QuantLib 6 replies

Quant Corner 12 replies

questions from a future quant 39 replies

Attachments: Statistical mean of the market [quant corner]
Exit Attachments

Statistical mean of the market [quant corner]

  • #61
  • Sep 30, 2014 3:51am Sep 30, 2014 3:51am
Sh!t this took longer than anticipated, but finally my platform is ready.I`ve tested a simple strategy and it almost showed identical results.

I`ve triple checked my code but no matter how the end balance was never identical in MT4 as in my platform.It is funny because i`ve calculated the pip size correctly, as i used USD base currency and EUR/USD, no swaps, so the conversion of the money was done properly, yet the end results differed by about 100-200 $.

Its funny because the MT4 is the one that is bugged, as i`ve set the optimization to the open price, and although i used median price, the damn strategy tester is always using the OPEN price which is damn annoying.

For statistical significance i only use median price and median based filter, to be homogeneous, not mixing CLOSE price with median moving average and whatnot.

Its annoying but thats it, and so this will anyway give more accurate results, and the median one has better stats if you think about it.

Attached Image (click to enlarge)
Click to Enlarge Name: GGG.png Size: 30 KB

Attached Image (click to enlarge)
Click to Enlarge Name: GGGGGGG.gif Size: 18 KB


You can see the balance curve in both pictures is identical, minus the offset that is caused by the different calculation method.

So finally i can now begin to work again on my filter
"There's a sucker born every minute" - P.T. Barnum
  • #62
  • Oct 1, 2014 12:55am Oct 1, 2014 12:55am
Ok basically these are the differences between MT4 and my Strategy tester:

Attached Image (click to enlarge)
Click to Enlarge Name: mytest.png Size: 118 KB


Attached Image (click to enlarge)
Click to Enlarge Name: mt4test.png Size: 19 KB


As you can see mine gives worst results, on the same strategy, however mine is the more plausible one, because i keep the homogenity between price types.

Also in case you wondered this is a simple 2 moving average strategy just for startup.

And the MT4 computes it in about 20 seconds, my tester in <1 second
"There's a sucker born every minute" - P.T. Barnum
  • #63
  • Oct 1, 2014 8:46pm Oct 1, 2014 8:46pm
Ok the problem is solved, the difference was only because of the swaps.Yes the swaps, although i work in offline mode, and my hist data was downloaded without swaps, MT4 still calculates swaps, so looks like the swaps are embedded in MT4, and it is correctly embedded, its not just the latest swaps used all across the data, but the historical swaps are assigned correctly.

So MT4, good job using historical swaps.

However i decided to not use swaps in my test, so i will test it without swaps, so there will always be a little miscalculation between my tester and MT4 tester.

Funny because i`ve already found a better method than the 2ND generation with an edge of 1.95, yet its without swaps, in MT4 it shows 1.45 because of the swaps, so although by using swaps it has a worse expectancy, without swaps it has a better one.

And its funny because the correct result is the 1.95, because i`m only testing its curvature to fit the market's curvature, i dont care about how the swaps will weight in, or atleast not yet.Ofcourse in real time trading swaps will be a factor too, but then i shall focus on positive swap trading , but the primary objective is to find the shape of the market.And for that swaps are only a obstacle.

The speed of my optimization is about 560 miliseconds in my tester VS 2 hours in MT4, hmm pretty much a time saver

Anyhow here is a screenshot of my new filter, not much difference can be seen with the eye compared to the previous one, but this one is better:

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDH4.png Size: 56 KB
"There's a sucker born every minute" - P.T. Barnum
  • #64
  • Oct 2, 2014 4:58am Oct 2, 2014 4:58am
Ok i still didnt found anything better than the generation 2.5 model.

Here is my ARMA model vs GENERATION 2.5, as you can clearly see my generation 2.5 model is not that adaptive, yet its more profitable and precise

Orange is ARMA, The GREEN/RED line is my GENERATION 2.5 filter.In case you wonder why the ARMA is not showing BUY/SELL signals it is because it has a much smaller edge than my own filter does.

Attached Image (click to enlarge)
Click to Enlarge Name: ARMA2.png Size: 49 KB


GEN 2.5 filter has expectancy of 1.95 while the ARMA has about ~1.15.I`ve drawn some linear regressions at the major swing points, and yes of course the ARMA is smoother and follows the mean price better, but at signal giving its a piece of trash.If you are already in the trade, then it looks wonderful that the trend is shown smooth, but if not, then you get many many fake signals, so its not that good if you actually look into it.

Attached Image (click to enlarge)
Click to Enlarge Name: ARMA.png Size: 64 KB


EDIT: the previous pictures are not at optimal settings, here is the ARMA vs GEN 2.5 both at optimal settings (ARMA= BLUE BUY /PINK SELL , GEN2.5 GREEN BUY / CRIMSON RED SELL)

Attached Image (click to enlarge)
Click to Enlarge Name: SASDASDASDASDAD.png Size: 59 KB


It can be seen that the ARMA gives late signals and more fake signals than my filter.
"There's a sucker born every minute" - P.T. Barnum
  • #65
  • Oct 2, 2014 7:19am Oct 2, 2014 7:19am
Quoting Innate
What are your thoughts on the Adaptive Moving Average?? This is mql5 but I am sure you could find a MT4 version. http://www.mql5.com/en/code/10
Ok i finally took a look at this one, although it was hard to program it because i had to redo the indicator so that it could be compatible with my code-set.

And very bad results, the best edge i got was ~0.3. And its no wonder as the MA sometimes goes flat around consolidation periods, its too smooth, and cannot adapt very well to sudden price surges, many HIGH/LOW's shoot out from the mean like a rocket, and the MA doesnt even move a pip in that direction.

Now i know that many of those are just noise, but a good filter adapts to the noise aswell, as it cannot distinguish the noise from a possible outbreak.Everything must be balancet perfectly

[LAGG & CERTAINTY] = [NONLAGG & UNCERTAINTY]

This is how balanced it should be, non 1 pip more to lagg, not 1 % less to probability

And this indicator is really biased for the nonlag & uncertainty part, the signals are swift but very uncertain.

Besides its only a mix of 3 indicators, its nothing better than MACD which is also a trio.Adding more indicators is not going to solve this problem.

You can design a filter which contains 2 indicators, then its smoothed by another 50, it wont be any more accurate

The problem is with the initial indicators themselves, not with the quantity of them, but with the quality , hope this makes sense to you.Good luck!
"There's a sucker born every minute" - P.T. Barnum
  • #66
  • Oct 5, 2014 6:18pm Oct 5, 2014 6:18pm
Quoting Proximus
{quote} Ok i finally took a look at this one, although it was hard to program it because i had to redo the indicator so that it could be compatible with my code-set. And very bad results, the best edge i got was ~0.3. And its no wonder as the MA sometimes goes flat around consolidation periods, its too smooth, and cannot adapt very well to sudden price surges, many HIGH/LOW's shoot out from the mean like a rocket, and the MA doesnt even move a pip in that direction. Now i know that many of those are just noise, but a good filter adapts to the noise...

Yes it makes sense. Thanks for your analysis!!

I must admit that I am a little lost to the entire purpose of discovering a new MA but this has been a very enjoyable read so far so please dont stop.
Good to understand the limitations of what indicators are able to tell you (& and when those limitations happen) but isnt this research leading to the conclusion that you need more than one MA?
What's it all about? It's all about money.
  • #67
  • Oct 6, 2014 1:19am Oct 6, 2014 1:19am
Quoting Innate
{quote} Yes it makes sense. Thanks for your analysis!! I must admit that I am a little lost to the entire purpose of discovering a new MA but this has been a very enjoyable read so far so please dont stop. Good to understand the limitations of what indicators are able to tell you (& and when those limitations happen) but isnt this research leading to the conclusion that you need more than one MA?
I`ve been thinking about using more than 1 filter, but not in the sense that you said in the previous posts.Stacking MA's into one another is not really a good idea, because either the lagg will be too big due to overfiltration, or the signals will be too weak due to underfiltration.

Instead i thought about keeping the current filter and using another filter to filter price, so it could become a 2 MA crossover type of method.

But i`m not doing this yet, there are a few more models to be tested, so at the moment i`m ticking out the remaining possibilities.
"There's a sucker born every minute" - P.T. Barnum
  • #68
  • Oct 6, 2014 10:15am Oct 6, 2014 10:15am
Ok this 1 filter approach is going nowhere, you are right Innate, we need more filters.

I`ve tried almost all types of models, which is enough to make me believe that 1 filter is not enough.I used ARMA,ARIMA,AFIRMA, AFIRMA+MA, ARI2MA,ARI3MA,ARI4MA (the 4rth integrated took me 6 hours to compute, fuck it ),etc etc, and yet none proved to be good enough filters to filter out the noise optimally.

You see its not even the whipsawy movements are the problems here, but rather those nasty sudden spikes, and "stophunts" which distort the signal, and sometimes make us exit the trade when its not even the case.Just because a quick stophunt happened doesnt mean that its a reversal signal isnt it?

Attached Image (click to enlarge)
Click to Enlarge Name: EURUSDM15.png Size: 32 KB


Its this kinds of movements that i`m angry about, because the trend is smooth and obvious ,yet your signals are distorted by these nasty moves.

 

  1. The second problem is that, after the filter is too complex, it can be always written as a simpler model, for example an ARI2MA with X,Y parameters is 99% indentical with a SMA [X], so its worthless to calculate the complex algo, when its almost identical with the simple one.There are exceptions, but basically the more complicated the filter becomes, there will be always a filter which will be indentical with it and less complicated.


So now i switch to 2 filter methods, basically 2 moving averages in layman terms.

"There's a sucker born every minute" - P.T. Barnum
  • #69
  • Oct 6, 2014 10:41am Oct 6, 2014 10:41am
@ Proximus
this may help for different approach
http://www.forexfactory.com/showthread.php?t=167720
My Best Trade is Next Trade!
  • #70
  • Oct 6, 2014 11:43am Oct 6, 2014 11:43am
Quoting MAM.Trader
@ Proximus this may help for different approach http://www.forexfactory.com/showthread.php?t=167720
Sorry but i`m too lazy to read trough that whole thread, is there any particular post that is interesting there to save me some time ?
"There's a sucker born every minute" - P.T. Barnum
  • #71
  • Oct 6, 2014 11:47am Oct 6, 2014 11:47am
trading not good for lazy
fyi

Quoting Proximus
{quote} Sorry but i`m too lazy to read trough that whole thread, is there any particular post that is interesting there to save me some time ?
My Best Trade is Next Trade!
  • #72
  • Oct 6, 2014 4:32pm Oct 6, 2014 4:32pm
Quoting MAM.Trader
trading not good for lazy fyi {quote}
I really dont think you can make that assertion about Proxi. If anyone has the energy to get stuff done its him, its just all about where you direct that energy. I had a look at the link you directed Proxi to and it is a thread I have skimmed through before. Quite frankly that thread creates more questions than answers. The thread basically throws around a number of ideas and ends up discussing Machine Learning topics. This is something quickly dismissed around here due to the inherent difficulties in that level of application.

So if you would be so kind as to not show Proxi your laziness, what in particular from the plethora of topics discussed in that thread should we include into this discussion to keep things advancing here?
Please no offence, I am genuinely interested.
What's it all about? It's all about money.
  • #73
  • Oct 6, 2014 4:39pm Oct 6, 2014 4:39pm
Quoting Proximus
Ok this 1 filter approach is going nowhere, you are right Innate, we need more filters. I`ve tried almost all types of models, which is enough to make me believe that 1 filter is not enough.I used ARMA,ARIMA,AFIRMA, AFIRMA+MA, ARI2MA,ARI3MA,ARI4MA (the 4rth integrated took me 6 hours to compute, fuck it ),etc etc, and yet none proved to be good enough filters to filter out the noise optimally. You see its not even the whipsawy movements are the problems here, but rather those nasty sudden spikes, and "stophunts" which distort the signal, and sometimes...
This might be a silly idea (it just popped in my head so here goes)...
but if the spikes are what is 'distorting' your signal why not remove them from the equation?
You could use ATR or some other form of volatility measure to ignore all bars over a certain size.

Not sure if this will actually be of any benefit but just trying to zero in on your perceived problem. (Yes I am aware of the issues this sort of MA would have on a lower time frame).
What's it all about? It's all about money.
  • #74
  • Oct 7, 2014 2:53am Oct 7, 2014 2:53am
Quoting Proximus
I've tried almost all types of models, which is enough to make me believe that 1 filter is not enough.I used ARMA,ARIMA,AFIRMA, AFIRMA+MA, ARI2MA,ARI3MA,ARI4MA (the 4rth integrated took me 6 hours to compute, fuck it ),etc etc
The differentiation is used to make the time series stationary. What is the point of a 3rd or 4th order differentiation?

I still can't understand why it takes hours to compute this. How many candles do you consider? Even with 10000 samples I can't imagine it takes more than half an hour with the naive regression approach.

Quoting Proximus
and yet none proved to be good enough filters to filter out the noise optimally.
Again you use the term optimally but you don't specify with respect to what.

Quoting Proximus
The second problem is that, after the filter is too complex, it can be always written as a simpler model, for example an ARI2MA with X,Y parameters is 99% indentical with a SMA [X], so its worthless to calculate the complex algo, when its almost identical with the simple one
The MA(q) part is a EMA. The AR(p) part is a FIR filter MA yet usually not a SMA. This is not a problem. It is the definition of the tool. Did you read Box and Jenkins book (I know it is a headache maker but...)

Quoting Innate
{quote} This might be a silly idea (it just popped in my head so here goes)... but if the spikes are what is 'distorting' your signal why not remove them from the equation? You could use ATR or some other form of volatility measure to ignore all bars over a certain size. Not sure if this will actually be of any benefit but just trying to zero in on your perceived problem. (Yes I am aware of the issues this sort of MA would have on a lower time frame).
Because you can't measure them at the very moment they happen. The ATR lags like any other filter. If you substract the detected spike from the PA you'll actually add an opposite spike just after the original. This is exactely where I failed with my cycle detection and removal. I could never compensate the lag.
No greed. No fear. Just maths.
  • #75
  • Oct 7, 2014 3:24am Oct 7, 2014 3:24am
Quoting Innate
{quote} I really dont think you can make that assertion about Proxi. If anyone has the energy to get stuff done its him, its just all about where you direct that energy. I had a look at the link you directed Proxi to and it is a thread I have skimmed through before. Quite frankly that thread creates more questions than answers. The thread basically throws around a number of ideas and ends up discussing Machine Learning topics. This is something quickly dismissed around here due to the inherent difficulties in that level of application. So if you...
Thanks, i wanted to say that, but then it would make me look selfish, so its better that you said it.

Quoting Innate
{quote} This might be a silly idea (it just popped in my head so here goes)... but if the spikes are what is 'distorting' your signal why not remove them from the equation? You could use ATR or some other form of volatility measure to ignore all bars over a certain size. Not sure if this will actually be of any benefit but just trying to zero in on your perceived problem. (Yes I am aware of the issues this sort of MA would have on a lower time frame).
The question is what size, does all candles above pip size X are useless, or only certain candles, of certain size, at certain points.
So its more relative than you think.The one thing that I learned about the FX markets more than anything that there are no absolute values here, everything is relative, so this is the source of its complexity.

If i could say certain that every 561 pip weekly bar will show a reversal, then it would be a holy grail by itself, but as soon as banks and HF would distort this value just to make sure retailers dont win on this, the edge is gone.So this is why there are no absolutes.
"There's a sucker born every minute" - P.T. Barnum
  • #76
  • Oct 7, 2014 3:38am Oct 7, 2014 3:38am
Quoting PipMeUp
{quote} The differentiation is used to make the time series stationary. What is the point of a 3rd or 4th order differentiation? I still can't understand why it takes hours to compute this. How many candles do you consider? Even with 10000 samples I can't imagine it takes more than half an hour with the naive regression approach.
I just tried them all to make sure I don't miss any viable alternative, sort of prove beyond questionable doubt that that particular approach is useless.
Well there are many iterations, i got almost 20k bars and there are atleast 6 recursive iterations packed into one another, so it takes time.

The 1-2 level integration doesnt take more than 15 minutes, but after the duration is increasing exponentially.

Quoting PipMeUp
{quote} Again you use the term optimally but you don't specify with respect to what. {quote} The MA(q) part is a EMA. The AR(p) part is a FIR filter MA yet usually not a SMA. This is not a problem. It is the definition of the tool. Did you read Box and Jenkins book (I know it is a headache maker but...)
In respect to maximum expectancy for the signals, minimum drawdown, and more signals to make sure the expectancy is estimated correctly.
EMA is considered an (ARIMA) (0,1,1) , but mine was more like (1,2,1) and yet it was almost similar to a simple moving average.I did not read it.
"There's a sucker born every minute" - P.T. Barnum
  • #77
  • Oct 7, 2014 5:04am Oct 7, 2014 5:04am
What takes time is the estimation of the orders p, d and q. If you already know you want a ARIMA(1,2,1) it should be done real time even if you use a Levenberg-Marquardt gradient descent to fit the model. The differentiation adds to the lag but not to the computation time as it is a O(1) algo. 20K bars seems a lot. Is there really a difference with using only1-2K bars?

The variance needs be constant and the noise Gaussian for ARMA (stationarity). You should therefore naturally try with range bar charts. Also this will mechanically break down your spikesand remove the need for non-linear optimization.

The pdf may be of interest. It says that the differentiation is a problem. Did you try a multi-time frame approach? My idea is that you could estimate an ARMA on a higher TF (1 sample out of N) and use it to estimate/forecast the trend. You substract the model from the lower TF data (i.e. making the data stationary by detrending) then apply the ARMA on the result.
Attached File(s)
File Type: pdf doc.cfm.pdf   1.2 MB | 621 downloads
No greed. No fear. Just maths.
  • #78
  • Oct 7, 2014 11:16am Oct 7, 2014 11:16am
Quoting PipMeUp
What takes time is the estimation of the orders p, d and q. If you already know you want a ARIMA(1,2,1) it should be done real time even if you use a Levenberg-Marquardt gradient descent to fit the model. The differentiation adds to the lag but not to the computation time as it is a O(1) algo. 20K bars seems a lot. Is there really a difference with using only1-2K bars? The variance needs be constant and the noise Gaussian for ARMA (stationarity). You should therefore naturally try with range bar charts. Also this will mechanically break down your...
I dont know ahead, that is why i tryed all possible outcomes to make me doubt the 1 filter methods.I didnt used noise operator in my filter because it gave too random results, which had an unpredictable effect on my filter's results.

Sort of like a butterfly effect, 1 little random variable can cause huge differences in the outcome.

I will try range bar charts if nothing else will work, but for now i have many more ideas to be tested.

Ok i will look into the paper, but i am now working on H4 timeframe.
"There's a sucker born every minute" - P.T. Barnum
  • #79
  • Edited 7:17am Oct 8, 2014 6:44am | Edited 7:17am
Ok i`ve found a fundamental error, in my methodology, which is: MA cross tehnique

How naive of me of using such a silly method (that most retailers use).My filters were designed to find a suitable mean to filter out the most noise possible leaving only the trend, yet i tryed to use it as if it were an MA to cross price over it.

This was the fundamental error that has slowed down my research, instead of using the filter itself to get signals, I used the price/filter cross, which was a really dumb idea.

Not even the 2 filter approach is good, that is the filtered price (to reduce spike) & main filter cross, because it still causes excessive lag.

So the benchmark remains, but i shall change the signal critera to use only the filter itself with a treshold level which will be derived from the filter itself and its curvature.

Now I will redesign my QUANT TESTER software to use this new criterias, and redo the main test again to see if the expectancy does grow or not

If it does then i will bang my head in the wall once, and proceed forward with the research.

================================================================================================================

I will also introduce a new metric to check the progress which i will call RISK-FREE EXPECTANCY, and shall be calculated as:

RISK-FREE EXPECTANCY = EXPECTANCY / MAXIMAL INTRATRADE DRAWDOWN

where the MAXIMAL INTRATRADE DRAWDOWN will be the maximal drawdown of the equity that can occur inside 1 trade.

So this will apply if there are multiple similar expectancies, yet the drawdown varies severely, let's take an example:

1) Expectancy 1.5 ,MDD % 1.5% of the EQUITY
2) Expectancy 1.4 , MDD % 1.1% of the EQUITY

So by only looking at the expectancy we could assume that 1) system is better than 2), however by checking its RISK-FREE EXPECTANCY, we find out that the DD is too unproportional, so 1) has 1 RFE, while 2) has 1.27 RFE, so 2) is actually better if we factor in the RISK

And every professional trader, that cares about it's equity must consider risk, so the RISK-FREE EXPECTANCY metric is very useful.

So we have 3 metrics at the moment as this new one will replace the EXPECTANCY and the MDD % so the current 3 metrics are:
- RISK-FREE EXPECTANCY = EXPECTANCY / MAXIMAL INTRATRADE DRAWDOWN
- NR OF TRADES
- PROFIT FACTOR (although this is optional)
"There's a sucker born every minute" - P.T. Barnum
  • #80
  • Oct 9, 2014 4:59am Oct 9, 2014 4:59am
Nope this method is not working, because the signals cannot be static, so not the curvature (measured in angles or whatever) nor the distance between the minimum and the curvature (in pips) will give good signals because they are both static.

Now obviously i could calculate the coefficients of all the signals but that would only work on the past data, and tells absolutely nothing about the future events, so the signal must be dynamic aswell.

So looks like I have to use the "silly" method that retailers use which is the crossover tehnique.

However it will be the 2 crossover method as the price itself is a really raw mess of data.
"There's a sucker born every minute" - P.T. Barnum
1 trader viewing now