Kalman filter and currencies strength

Lately I'm playing around with Recursive Bayesian estimators (http://en.wikipedia.org/wiki/Recursi...ian_estimation) for which the Kalman filter is a special case. I won't enter in the gory details; see Bayesian Forecasting and Dynamic Models by West and Harrison.

At first I was trying to estimate the trend and the volatility of the market... But that's not what this post is about. It is about why you'd better off be long on the Dragon today.

The market return is not Normally distributed but somewhere between Cauchy and Student distributed. Kalman filter performances degrade quickly for non-Normal innovations. That's why I use another Kalman filter which estimates the error of the first one and I feedback this information into the first model as a modulation of the allowed variance of the state. A kind of dynamic cursor "lag <-> smoothness".

This first model is a second order polynomial local estimation. I chose a 2nd order polynomial because it can approximate (Taylor) any smooth function like a sine wave for a market with a cyclic component (range or valatile trend) or an exponential trend (for index and stocks). Also the accelaration factor helps catching up with the price in case of a sudden move. The second filter just uses a constant model. I use H4 to gauge the daily trend.

Here is a screenshot of EUR/USD H4. The blue line is the ideal, but non-causal, low pass filter sinc (http://en.wikipedia.org/wiki/Sinc_filter) used with 41 samples. It gives you an idea of the lag. The filter is green when the trend is most likely up and red otherwise. The dash envelope is the 95% confidence interval of the mean price estimate. Below are the error between the price and the estimate (black) and this value filtered with the 2nd filter (red). It doesn't follow the error too much to not make the main filter over-react.

Now I have an estimate of the most likely trend. It is important to keep in mind that it is a likelihood not a certainty! I can use this information to define if EUR/USD is most probably up or down. OK I hear you, I can also look at the chart and see if it is raising or falling :-). You are right. But the idea I had is to repeat this with each of the majors, not only to know if they are up or down but how much they are up or down.

If GBP/USD is going down it means USD is stronger than Pound. If EUR/USD is down as well EUR is weaker than USD too. But which of GBP or EUR is the weakest? One cannot simply use the pip value of the trend because the value of a pip isn't the same for all the pairs. For this reason I use the percentage of variation. By dividing a price difference by a price you get a unit-less value. This way I get rid of the unit/scale and I can compare EUR/NZD and USD/JPY directly.

Because EUR/GBP is (EUR/USD)/(GBP/USD) it isn't necessary to filter all the pairs. The price estimate and the trend estimate of the majors are enough to linearize with a good enough accuracy. If you find GBP is weaker than EUR than you can order the three currencies: GBP < EUR < USD. It makes sense to search a good entry short on GBP/USD because it's where we have a good chance to catch a trend.

Now for each pair I order the currencies pair-wise, stronger -> weaker. Each of the currency is rewarded a score each time it is stronger than another. This score is the -estimated- percentage growth. At the end each currency has a score and the list of the currencies stronger than it. I can organize this in a graph. It is supposed to represent an estimate of the money flow.


Don't ask me for a custom indi as I can't program MQL. If you're interested in coding an indi based on this idea I can help you if time permits.

Open question: if there is a cycle in the graph, does it makes sense to basket trade the envolved currencies in a ring?

very interesting ,will look at the links you posted,presently i use the value changes for bias and holding trades and add ons,i havnt found anything that comes close to the information i get from these pairs.although i viewing the value change in pip value and overall percentage .havnt quite got it perfect in excell but still in progress.

you mention trading in a basket,this can be very rewarding.a good example of this type of trading was today,i predominantly trade cable which like all related pairs was being pushed down .the leading pair in todays change was cable.its value was 60 plus pips whilst the other doller crosses had not moved .the yen pairs had also stalled but at this point no sign of direction change.the strength in cable was the clue.i placed trades on 4 yen pairs ,the doller chf,doller cad and euro.including my initial cable long ive got close to 400 pips.

the correlations are dismissed a lot on this forum because ther not understood.the way you have described and researched it shows the way the correlations in regards to there relationships are very much there, offset by there pip value along with changing conditions within there own individual fundamentals.but those cycles are there and can be used

great stuff

dice

Interestingly my filter finds that GBP/USD is up while at the same time it computes that GBP is the weakest and USD is the strongest.
I'm wondering if it is a hint that the recent rally of GBP/USD is only a retracement to fade?

It appears that a currency can be strong either because it is receiving a lot from some other currencies or because it is receiving from all the others. Usually the 2 conditions match. Which is the strongest? USD with the biggest score or EUR receiving from all the others?
(the score is the sum of the trend slopes)
Also note there is a cycle JPY->AUD->USD.

Im doing something similar. Im playing with the indexes. 7 currencies / 21 currency pairs. Im marking them with some statistical recognition once a day. Giving every currency pair a sequance number of ("up day"=1 and "down day" =0).

It could be presented as a money flow. But its hard to say. Definately have some weaknesses. Definately its easier to trade but one needs to be disciplined while we are in the trend and when we have a correction. Hard stuff.

As I see it, It presents me which is the hottest currencies out there. Meaning in a negative and positive way. Top 2 in + and - Lower 2.

The problem with this systems is always the time lag of the move into correction, later into deeper corrected stage and eventually presentation of a new trend. Hard to programme this.

That is why is so important that one is definately is trying to figure it out daily what is going one. Im using 4 different speeds of indexes.

Strong trend ones are definately giving me a presentation of the money flow direction (Picture 1). I obey it and I try to play only this directional plays.

Last few weeks we are strongly lovin the GBP and EUR, while we are trying to get rid of CAD and JPY. Pretty simple stuff in a way, but if you are not trading monthly/weekly charts and one does not use stops higher than 300 pippers it is still not an easy task when to enter and how to call a corrections end.

Speed Trend chart (Picture 2) is where one could find some evidence where the play is going to end. At least suspect what is gonna occur in a correction. Very hard stuff to do as volatility at this extremes is going to get higher. Especially it fades the initial trend move which represents in one very strong contra trend daily candle.


Currency Pair (Picture 3) chart sees currencies GBP as one of the strongest and AUD one of the weakest. In last month after few strong weeks of higher rate we could see some pause in the rise. That is also seen on the daily chart, with ease. But as seen on the chart, we can see that already on 6th of december one could see that we are about to get in a correction/few days pause. Why?

AUDL(long) and AUDs(short) indexes crossed on 4th of december, Still in negative zone. Same goes for both GBP indexes. They crossed into weakness on 6th of december so we have a strong suggestion of a deeper correction. At least one should be now very carefull about buying the dips, definately one should in the last few days get rid of position one held days before. We could easily move on a HH daily, but should be limited.

Now I come to my questions. Do you still work on this issue?
Can you share what were your general findings concerning index trading.
Cycles? Did you have some results in finding promising factors to edge probabitity into your favour concerning "only retracement".

Or some findings..

Did you ever try to do a sequance recognition on the indexes. Like they do it at the "Ultimative thread" I think I saw you there.

In your own time,
Thanks
21vs7

I didn't investigate on the cycles. I found this could be interresing for basket traders.

The lag is a killer with this indicator. It often is. When it finds a pair is strongly trending the move is already made. But fading this move is deadly. So you need waiting for a retracement. Going multi-timeframe was what I found the logical next step. You did this as well. For instance A/N is currently in my top bearish list for daily. Of course nobody needs a Kalman filter to see A/N is bearish on daily... But it turned bullish on H4. I would translate this by: time for the bears to take profit. It is also bullish on H1. When it becomes bearish on H1 I would try to find a technical opportunity to short against H4 in order to enter the next swing on D1. Of course I will be late on H1 and certainly too early on H4. But it is better than a 300 pip SL on D1 given that I cannot know if this move isn't the first leg of a reversal...

Unfortunately it is very hard to objectively estimate the validity of this approach to estimate the money flow/currencies strength. Like with any indicator you cannot estimate its power outside the scope of a trading system using it. If you get poor results it is because of the indi, of the system, of the trader failing at using the system? If the results are good you don't know if it is because the indi is a good tool or if you just had luck to be in a favorable market condition for the system over the testing period...

This is exactly right. Im testing this from august 13 and Im getting very good results. Delighted. But still cant see if this were really favorable last four months or this could really work for an extended period.

What Im sure is, that with filtering entries out and finding good management control of the open positions , all that could provide some relatively good yield curve.

Going thru statistical work where Im mainly concentrating on %winnings by EA/6 by each rule. Finding the optimum R:R which is the best without any additional filtering. Thereafter, Im gonna try to find the balded part up there.

Im at the fourth phase atm, and Im thinking Ill already found some very interesting settings, which were positive thru out this range of testing. I killed a lot of bad seeds, but kept only few and Im still narrowing it down..takes time.

I cant backtest this system as it takes the whole market at once. I tried to do it in excel but would take to much time. Im workaholic, but that would take months to establish, then months...etc

Very interesting PipMeUp. May i know the result you get, is GBP/USD goes up at that time / not long after that ? This is a kind of finding the turning point, very nice.
I am sure this is using different calculation from the Recent Strength indicator by Hanover : http://www.forexfactory.com/showthread.php?t=163158. I am weak in math, may i know how you get the number for each currency ?

Thanks in advance.

I actually stopped using KF for this because the lag is too high. I changed for something which is also more simple (but still lags!). To know the strength of the daily I use hourly candles (oversampling).

To get rid of the unit of the prices to compare the various currencies, I use percentage of increase. The average percentage over a window of N recent prices is the N-th root of the product of the N percentages. It turns to be much simpler to use the logarithm of the prices percentage, divide by N and take the exponential. This turns to be a SMA of the log of the ratio of the prices.

The period of the SMA must be such that it covers exactely an integral number of each trading session. This way you don't compare asian session and US session. I use 4 days (96 hours).

Now you get the rate of change of the strength for one pair. To get the strength you need to integrate over the time. Unfortunately this integral won't converge. I solved the problem by truncating the integral. The integral is simplified by simply adding the last P result. Up to a constant this is again a SMA. I smooth/integrate over 336 samples (14 days).

If you substract 1 for the result you get a value which is positive when the pair is strong and negative when it is weak. Beware: this is not really a valid trend indicator. I multiply by 100 to get readable values. Any factor works.

So the final formula is a double SMA: SMA_336(100 * exp(SMA_96(log(close[t]/close[t-1])) - 1)):


After you can use the same pair wise comparison and build the graph:


This suggests that picking a bottom for G/U is dangerous but you can buy the GBP using G/A.

Perhaps better if I post a chart of G/A as well with the oscillator

Nice image PipMeUp, and it's very interesting how you come to that equation. Great work

I wonder if you have ever thought about using your KF measurements as part of an input for a machine learning system to get trading signals? Some of the systems I use take advantage of similar measurements of currency strength/weakness and have decent forecasting abilities. I generally do machine learning using R or some C based machine learning libraries (libs like waffles or shark). I see that you could potentially build a system here where the KF acts as an input and you can get predictions of trend evolution for a basket of currencies.

The idea was to select a basket of three "probably trending pairs" by pairing the three strongest and the three weakest currencies. The pairing is made by taking care of their correlations. Using this method I get weights to apply to the MM: they are the three pairs and the percentage on the screenshots. The pair made of the remaining two currencies is probably ranging and should perhaps be given to a mean reversion based system.

About using a KF to estimate the market trend I made one using range bar charts. Thus far it classifies 9 market trends: extremely up, very up, up, a little bit up, flat, ..., extremely down. They are color coded. From this I split the data for each class to get the probability of the distance of the price from the mean given the class:


The result is nicely linear and I can improve the estimation of the mean. That's the green line and the blue and red bands/enveloppes


My problem now is to use that to update the exposure of the system. Here I'm completely blocked for now.

However if I understand correctly what you are doing with this seems to be to get an estimate of the current trending state of the market, this might not be that useful since you're interested in the future and not in the state that has developed up until now (what good is it to know that the market is trending if when you enter a trade it will stop?). I wonder if you have attempted to use machine learning methods - using your KF values as inputs - to estimate what the future trending state of the market will be. What would be most useful would not be a measurement of the current trending state of the market but a probability estimate of what the future trending state of the market would be. I wonder if your KF based measurements have predictive ability regarding future market trending states. Since you are evaluating a complex array of market interactions it seems plausible that this might be the case. What do you think?

Forgive me if I am misunderstanding something, perhaps you're already getting future predictions and I am somewhat missing this information.

I never tried to predict the state beyond a polynomial extrapolation which has absolutely no predictive power. The future trending state will be the same as the current one plus a random drift: linearly increasing variance. The trend is the red line below the chart. The blue one is the rate of change of the trend. Both osciallte around their mean (=0). Note the scale unit is one standard deviation of their own values (orange=2 sigma). The trend is mean reverting yet I never tried to fit a O-U process. OK. One more line on the TODO :-)

Did you ever get the Ornstein-Uhlenbeck fit?

Yes I tried and the result was that there is no detectable mean reversion in the residuals. ARMA models also fail.

Nice thread!

Have you tried toying around with other Machine Learning algorithms? I have noticed the widespread use of xgboost and gbm within the machine learning community.

Edit: I have also read elsewhere on using the outputs from ARIMA as a feature as well. Yet to have tried it thou so Im not sure bout the effectiveness of that.