Statistical mean of the market [quant corner]

“I have not failed. I've just found 10,000 ways that won't work.”
― Thomas A. Edison

Haha thanks thats inspirational.

Anyway if it would have worked, then i would have beaten 6 of the top 10 people currently in the FF 2014 yearly Leaderboard, which is quite a saying, of course those are live accounts and mine is not even a demo test just a backtest, but still it would have been similar as i counted in the spread.

Since my average yearly gain would have been: 9.48%,without lot scaling, which is not realistic because the lot needs to be scaled as the account grows, and my risk policy would have been to always have a constant % of account based risk, so my growth would have been much bigger. But in terms of pips its more accurate, my best filter would have gave me 7,672.7 pips in total or 948.03 yearly pips, wheareas only: Breza,doktorforex,kdxrider3 and julius3419 had outperformed my research.

Which is funny because although it is only a fictional if scenario, yet these 4 guys have very high succes in FX trading in live accounts, while probably the majority of the other traders are struggling to get breakeven or 1-2%/month if lucky which is interesting as there is certainly a gap between succes in FX trading, there is no "middle strata" here there are either constant losers/breakeven guys OR immensely succesful traders, there is just no middleground.

So this motivates be because there is an edge there, judging these numbers, as those guys who found it are immediately becoming very succesful, while the rest of them are just forever struggling.

You sound depressed today prox'. I think that the failure of your experiment is due to the fact that you try to predict the market. And also that you overfit your data...

Unlike you, I don't think the distribution of the market is changing from one distribution to another. I also don't think it is predictible. I would ask one question to these top 4 traders: "Do you predict the market or do you follow the market?".

A Bernouilli distribution has one parameter: the probability of the positive outcome. A Gaussian distribution has two parameters: the mean and the variance. A Cauchy: mode and scale. A Beta: alpha and beta, etc... I think that these parameters are changing. I think they are changing constantly and randomly. Your optimal EMA 25 is just the mean EMA over your dataset. Imagine a Gaussian which mean and variance change over time each driven by a separate random walk. A random walk is weak at visiting it's state space: it means it will stay more often than not close to where it is currently. This implies the Gaussian will look the same during a no so short period of time. A trend stays in place for some times. A range does as well.

You say the market can be seen as the sum of two distributions, one for the ranges and one for the trends. Let's call them R and T. You searched for (R+T) as one distribution. Now let's rewrite it w*R+(1-w)*T. When w is close to 0 market is trending. When w is close to 1 market is ranging. Now let's say w it driven by a random variable (bound between 0 and 1). What I've been trying to do thus far is to estimate all of those hyper-parameters. For the trend I'm quite satistied. What my filter finds in real time is pretty close to what a human finds in hindsight. But it fails in the ranging periods.

The picture clearly shows this in the period from March to April. Where yaed has drawn an horizontal line my filter finds a somewhat double top. I don't think it can be made better without knowning the future.

At first I thought a range was no more than an horizontal trend. I now think it is something else mixed with the trend. If you set w=0.5 above you get a volatile trend. Also it seems to exist two kinds of ranges: choppy (looks like nothing) and steady (looks like a sine wave). Each with its own distribution (steady=Arcsine and choppy=Gaussian). Or a mixture of both. Or simply a Beta(a,b), which can approximate a Gaussian pretty well and is the Arcsine for alpha=beta=1/2.

Well it sucks if 3 months of work is worth nothing. Anyway one thing i did found out and that is that the market is indeed a zero sum game, which previously i thought its not.

I dont think that that approach will work, because as you said the parameters will change randomly.

What i did is that i`ve measured the distribution of the probability itself, of switching between up and down candles from close to close, and at first look there is a little bit of advantage in shorting, precisely a 0.2057031732155 % higher probability of red than green candles, but not so fast, because although theere are more red candles (of EUR/USD i`m speaking) the size of the candles is higher for UP candles.

So what i did is that i multiplied the probability of each candle with its average size, so that is:
50.2057031732155 % *16.281656862502 pipettes / 100 = 8.17432031606921 of DOWNSIDE MOVEMENT
49.7942968267845 % * 16.4255877374142 pipettes /100 = 8.17900591351193 of UPSIDE MOVEMENT

Which if you substract should be 0, which is not, there is an offset of 0.00468559744272 in favor of up, which mean that the EUR/USD should move that amount down until it becomes 0 to hit the perfect equilibrium state. This could be used as a money flow indicator:
The probability of direction * the size of the movement in that direction = The money flow in that direction

And if its at extremes, or the offset is big then the market needs to balance itself and reverse.

This pretty much proves that the market is indeed a zero sum game, no new money is entered directly in the market (or atleast not directly, because although the market capitalization increases, it's not increased by the number of new participants new funds but rather by the increase of the money supply of that particular currency). So this is more an economic phenomena than a mathematical one.

The ownership of the money indeed can change, obviously the big banks rock at this, but the market itself doesnt increase in this way. And its also logical, the more i thought about it, because for each BUY trade you need a SELL trade, so no matter how much money i have in EUR, there must be an equivalent amount (relative to the exchange rate) in USD for me to be able to execute the trade.

So if i put 10.000 € in the market, the market's size doesnt increase, because the mere fact that i got the EUR already in my pocket, is already a participant in the market, not an active one, but a passive because you already use it for shopping or etc. While if i decide to switch that 10.000€ into 12,436.85$ then i shrunk the € market, but i`ve increased the USD market, yet the combined EUR/USD market remained the same size. It's logical.

While the EUR/USD combined market remains the same no matter what, the only way it can increase it through inflation, or the looseing of the money supply, because the purchasing power may vary relative to the GDP, once a CB decides to adjust the money supply to the GDP then the market increases. It's just basic economics if i think about it.

If both currencies are inflated in the same rate then the exchange rate doesnt change, but the market capitalization increases. If they are not then the market still grows, but in the favor of 1 currency vs another. As the FED printed a lot of dollars and devaluated the USD to pay off the debt, the other CB's have also had to devaluate it to keep the market in balance, so this is probably the main reason why the FX market has growth to 8 trillion, and not because Joe the average trader discovered FX trading from Google Ads...

So probably in the future i`ll be looking more into the economic mechanism behind the market, because it makes more sense than to guess the probabilities.

I have followed this thread as one of my ea's pivots (pun intended) on the MA so i was very interested in your results. Thank you for sharing.
I have previously studied the 'best' MA for my system and came to the similar conclusions as you have. ie the mean of the market keeps changing, making it ridiculously hard to pin the tail on the donkey.

I can not proclaim to have figured something out about the MA, but if I was going to consider further research into MA's I would focus on interrelationships.

Not sure where you want to take things, but I dont think you have failed at all. Its just not that easy. You were comparing your results with the top performers, I bet that they would be using a confluence of signal (unless they are naked traders) and your study into the MA is only one part of that confluence signal.

Anyway back to interrelationships. I had mentioned it to you before in the thread with regard to just comparing two MA's to one another.
Here:
Statistical mean of the market [quant corner] - Page 5 @ Forex Factory (just throwing ideas around, cant say they will be successful)

This is just one application of local interrelationships (just MA's), however the interrelationship that I have been studying is between the MA and the ATR. There was a week in Dec 2008 on EURUSD that my ea would really struggle with in testing. Its an outlier (extreme) event that throws the data set (you mentioned 2008 in your post above as well). One day I noticed that this very week was actually the week of the EURUSD's highest ever weekly ATR (something like 650 pips from memory) and I wondered if there was a link. I am aware that correlation does not mean causation, but you got to try something right .
It makes a bit of sense that they are linked. If the ATR is high that means the delta's between each bars close (if you are studying the close or median etc it will be the same concept) will be bigger than 'regular' market conditions. If I am not mistaken it is these extreme daily ranges that are skewing the data.

I have not got to the bottom of finding the link, I keep getting distracted looking at all the other pretty colours in the trading rainbow.
Initial study gives me the hunch that the interrelationship is polynomial. I need to do a lot more testing. I think the only way I will get a satisfactory answer is via a genetic algo.

Anyway mate, I have faith in you. Keep it up. The knowledge you have gained here will help you on your next quest.

The VOLUME of the FX market is 5 trillion (not 8), not the AMOUNT OF MONEY.

If I have 1 USD and you Prox have 1 EUR, and we swap them between us, we have just done 1 unit VOLUME of EUR/USD trading. But we can do this swap ten times in the same day. In that case, the VOLUME will be 10 units EUR/USD. Notice that we still have 1 USD and 1 EUR, yet the VOLUME is 10 times higher.

So the VOLUME of the FX market is related to the frequency of trading (money velocity), not to the quantity of money.

This is good. More VOLUME means more noise, which means more inefficiencies to trade and exploit. Noise is good, unlike what most retailers believe.

And yes, you should look into the economics behind the market, or more exactly at its structure and micro-structure. Then you'll go past trying naive methods like a simple EMA good over 15 years and get to the real edges.

Ok then, i confused the volume with the market cap. Anyway the volume only causes volatility, but the price itself is i think defined by the market cap of each separate currency.I`ve looked into fundamental analysis, and i tried to quantify it but with less sucess. I`ve quantified interest rates, money supply, GDP, inflation, unemployment, etc, and it didnt give me anything better than technical analysis. But nevertheless i will look into it again...

For example if i got 2 cows and you got 1 cow, then we assign a currency (which is just a representation of wealth) to it, my 2 cows will be worth 1 P and yours will be worth 1 A



Like this, this is how supply demand relates to the currencies. Just replace the P currency with Eur and the A currency with Usd and the cows with the 2 respective countries economy and there your got the EUR/USD market explained in a nutshell.

Not really.

As they say, the market can stay irrational longer that you can stay solvent.

Ie: your cow can die, but the currency may not drop and in fact it could increase because for example someone else who holds it launches a rumor that you have 10 more hidden cows.

Are you able to expand on that comment? I agree with you. Not asking for you to give away too much, just keen to learn any clues you might have on how to exploit it.

Thank you.

More noise means that the coastal length of the market is longer.

If EUR/USD would move in a strictly linear rate, with 0 retrace, from 1.28 to 1.25, you can make 300 pips. But if after each 100 pips it retraces 50, you can make 300 (original move) + (50 retrace + 50 resume trend) + (50 retrace + 50 resume trend) = 500 pips. The more noise, the more pips you can extract from the market.

Which explains why HFT is so profitable, because with a 2 pip ruler, the coastal length is reaaaaaally long - the move from 1.28 to 1.25 would probably have over 2000 pips when using a 2 pip ruler.

How does that help one? If you want to make 100 pips, you need to catch 30% of the move from 1.28 to 1.25 if you just do a single trade. But if you trade the 2000 pips coastal length, you need to catch just 5% of that to make your 100 pips. So you can snipe your entries better, and enter in much more favorable situations.

Thank you thats a really good answer.

hi PipMeUp, i have questions if you dont mind answering :

You mentioned about this :
The only thing I could forecast was a smoothed version of the RSI 3 bars ahead (using a AR(2) process). I prefer using ML to estimate the market. The trend for instance. But it doesn't forecast a change in trend.

What is ML ?
What is the setting for RSI ?
What is AR(2) process ?

hope that you dont mind explaining.
thanks Bro