Old Dog's Taming of the Beast?

Hello everyone,

I am opening this thread to throw a few ideas I have into the forum and see if we can "make them fly"!

For those who haven't seen my posts, here is a potted background of who I am and why I am here....

Potted History

I have been trading for around 25 years and never bothered with public forums until a month or so ago someone pointed out the Abbonacci thread here on FF. I was somewhat intrigued and thought I would take a look. Sadly, I soon detected the unmistakeable whiff of Snake Oil and I decided to move on with my own ideas.

CanuckCT and several other experienced traders picked up on these ideas and CanuckCT kindly opened a thread to promote my system. This has gone very well, he and Squalou have written some excellent indicators and we are now on the verge of opening a chat room so we can take trades in real time together...

http://www.forexfactory.com/showthread.php?t=258762

While chatting to CanuckCT, he pointed me in the direction of "The Beast", an ingenious trading system based on Macman's Sixths indicator and coded into an EA by Steve Hopwood. Macman was interested in ways to optimise the system and I posted a few ideas in the original thread...

http://www.forexfactory.com/showthread.php?t=259284

Macman's approach was to diversify across a wide range of pairs so that hopefully when one pair is in DD, another is in profit. "TB" had a very good first couple of weeks and someone even mentioned the "G" word. (Like superstitious actors who only refer to a certain Shakespeare play as "The Scottish Play", I have been in the game too long to use the "G" word - I only refer to the "Cup of the Last Supper").

The obvious next step was then taken by someone who introduced the idea of using TB on a Basket of pairs. However, Steve rather took exception to this and abandoned the thread on the grounds that it had been hijacked by "...all sorts of weirdness...".

He opened a new thread on the original idea...

http://www.forexfactory.com/showthread.php?t=263253

However, in the last week or so, The Beast has bitten back!! Huge gains have turned into massive drawdowns and the punters are unhappy! It does appear, though, that Steve has had second thoughts about the basket idea as he has now opened several new thread on just this very topic...

While all this fun and games has been going on, I have been pottering away thinking about what started out as "the optimisation problem" but turned into a whole new philosophy (for me, at least). What I want to post here initially, is my first response to the vexed question of how to "tame The Beast".

(Interestingly, 7bit opened a thread recently on very similar lines while I was working on this...

http://www.forexfactory.com/showthread.php?t=262827

That is very comforting as it helps to reassure me that I am not completely crazy and that other minds are turning in a similar direction....)

Taming The Beast

The basket idea is very attractive, but the limitation of all the current ideas from T101 through to the recent Phantom 6 method, is that the pairs in the basket are traded with equal lots. This is, of course, necessary for the systems to work as all the schemes use hedge-able baskets where the base and quote currencies "cancel out" (like cancelling fractions when multiplied).

It is perfectly possible to calculate the weights of each pair to create a perfect hedge. It would be rather pointless, of course, as the basket would never move -- see the first chart below. This shows 3 pairs (EU x UC vs EC) with weights 0.717, 1.033 and -0.741 respectively, and as you can see, with the exception of the odd spike, the basket hardly ever moves outside the sum of the spreads. This is precisely as we would expect, otherwise arbitrage opportunities would arise that the Big Boys' computers would jump on in microseconds!

So in order for the basket to move enough to be tradeable, the weights must be anything but those of the perfect hedge ratio, and equal weights is as good as any.

But what is this basket?? All we really have here is just another synthetic currency that is just as prone to ranging, trending, whipsawing etc etc as any real pair. So what have we gained? Answer : Nothing!!

What I really want is to make the market do what I want, not what it wants...

The Cunning Plan

So... this lead me to what Blackadder calls a "cunning plan". In fact, this one is so low and cunning you could put legs on it and call it a weasel!

What happens of we do the exact opposite of the above? Instead of taking a hedged basket of pairs and trading them in weights that are not the perfect hedge ratio, what happens if we take a basket of pairs that do not form a hedge but demand a set of weights that makes them as close to a hedge as possible?

The result is shown below in the second picture. Here, I have used some linear algebra to compute a least-mean-squared fit of the pairs to a horizontal line. Because the basket is not hedged, obviously the fit cannot be perfect. But the magic occurs when we find set of weights such that the basket is cointegrated.

I don't want to go deeply into this topic as it is very technical (see the two easy-to-read papers attached). All I ask is that no-one on this thread mention the dreaded "C" word. Correlation is inapplicable to financial time series as they are not stationary!! Cointegration is applicable. What I am calculating is a weighed sum of non-stationary series that yields a series that IS stationary.

Now I am home and dry, because it means that the basket has some temporal stability. Some of the baskets I have found have had stable kernels for several years! OK, they might not remain cointegrated for ever, but we can recompute and retest as often as we like and tweak the lot sizes as appropriate.

In similar, vein, I can demand a basket that is maximally non-stationary!! Why would I do this? So I can create a synthetic currency that generally trends - see the third picture below.

So... if you have a killer system for trading ranging markets but which slays your account during trends (like TB), just trade the ranging basket. If you have a ripper trend trading system but which falls flat during ranging, trade the trending basket. We know with a high degree of statistical probability that the basket will not wander outside the bounds we have set.

Those who know about this stuff will immediately recognise this as Statistical Arbitrage (Stat Arb) which is the game the Big Boys play. While we must never forget the lesson of LTCM, I believe that if carefully handled, we should be "cautiously optimistic" about this idea.

I have tried this both on demo and live and am very encouraged by the results. I am currently implementing the entire Stat Arb engine in MT4, see final picture.

Note on the thread

I want to keep the thread free from "How do I load an indy" type newb questions, so am limiting the posters to 2+v. I hope that is OK. I may lift this restriction in future.

Also, I don't intend to post all my code as it is the result of many years of work (the linear algebra library is a thing of joy and beauty!). If you want to implement this stuff, please do a bit of homework and do it for yourself. The idea alone is enough to get you started.

EDIT: 7bit has just posted some useful code in "R" if you want to have a play. I don't use it myself (never heard of it!!!). I just DDE export from MT4 to Matlab. The regression/cointegration is just 2 lines of code in Matlab. Then DDE back to MT4.

Thanks and best regards,

Old Dog

Hello all,

For those who...

(a) noticed this thread exists,
(b) read the first post and
(c) understood it, http://cdn.forexfactory.com/images/s...m/big_grin.gif,

...you may have noticed the subtle but not insignificant problem of the lot sizing - sadly we cannot specify our lot sizes to 5 decimal places.

I have now modified the cointegral regression with some integer programming to force a maximum of 1 decimal place in the weights so the lot sizing now becomes practical.

This does rather damage the stationarity and the following basket is right on the edge of the T-statistic in the Augmented Dickey Fuller test. But I reckon it is close enough to trade as it has been stable for more than a year.

So our basket is:

GBPUSD -1.0 lot
EURUSD +0.4 lot
USDCHF -0.3 lot
AUDUSD +0.4 lot
USDCAD +1.0 lot

Obviously you can scale according to the size of your account. Reverse the signs for long and short.

The basket is shown running live in Metatrader in the attached chart. Since the (quasi-) stationary resultant series is (approximately) normally distributed we can legitimately talk about standard deviation (again, a meaningless concept for individual pairs). Here, the SD is 230 pips and 2SD is 460 pips.

I am taking secondary (smaller) trades at +/-1SD and primary (larger) trades at +/-2SD. Statistically, you have a 95% assurance that the basket will not wander beyond 2SD (460 pips) so you know your potential DD in advance. I am looking to close in the middle, so a very similar strategy to The Beast, but based (IMHO http://cdn.forexfactory.com/images/s...m/big_grin.gif) on sound theoretical foundations.

Best regards,

Old Dog

You can do fractional lot sizing on Oanda.

Could you please post a screenshot of the trending kernel, but starting from 24 Aug instead of 4 Oct? Or even better, both the trading and the ranging kernels, but from July. Because the screenshots could be just cherry picking.

I don't do "cherry picking", young man! I am a serious trader. Comments like that are likely to earn you a swift ban from my threads.

The charts in Post #1 cover 5500 hours which is the best part of a year in trading time. That more than covers the period you are talking about.

Old Dog

A) Yup
B) Yup
C) working on it

I am certainly with you. Your insights into trading and the subsequent discussions we have had have been "eye opening" to massively understate it!

I'm gonna keep working with the levels that we are familar with and hopefully catch up with you in the week!

ps.... congrats on your first thread!!!

Regards

Lee

So...my statistical background is, well, almost non-existent. I need a visual. Does it make sense to take the attached indicator and tweek it so that an offline chart for this specific basket and weighting can be created? Does that make any sense at all?

Cheers
CanuckCT

This makes perfect sense. Multiply each pair with its weight and add them together.

Don't forget to first divide each lot size for USD/XXX pairs by the current quote (and leave the XXX/USD as they are) to make each weight relative to the quoted prices in the chart again. (because the exact inverse thing must have been done after the regression to arrive at the lot sizes and you want to undo this again to plot the chart)

Edit: If the numbers are pip based spread betting lots (as old_dog pointed out) then they should already be usable for plotting the chart. But then they MUST be transformed to dollar based lots for TRADING on MT4 or Oanda.

Hi Canuck!

Good to see you here! You could adapt this indy very easily, but please bear several things in mind which are VERY important:

(1) This indy computes OHLC candles. This is largely meaningless for baskets as the HI and LO do not occur at the same time within the hour on all pairs. So the wicks are hugely exaggerated and misleading. I only compute using the close prices.

(2) As I pointed out on the "TB" thread, the indy is slightly broken in that it calculates the cross-rate pair weights wrong.

(3) Please remember that I trade though a SPREAD BET HOUSE. Aside from all winnings being tax-free in the UK, my lot sizes mean something quite different to yours. When I place "1 lot" I am betting £1/pip on the pair. If I place 0.1 lot I am betting 10 pence a pip. This is identical for all pairs. But with a forex broker, "lots" means just that. Every pair wins or loses a different $$ amount depending on the MODE_TICKSIZE, MODE_LOTSIZE and MODE_TICKVALUE. You will need to take this into account when you calculate your lot sizes. The sizes given in my charts are the raw weights for SPREAD BETTING ONLY! (EDIT: i.e. These are the naked, unmodified coefficients!)

Very best regards,

Old Dog

EDIT: I see Bernd has beaten me to it - his explanation is, of course, quite correct for the adjustments you need to make for a traditional broker.

It would be less confusing if you would additionally output the naked unmodified coefficients as they come from the regression model, so everybody could calculate his own lot sizes according to his type of broker.

(my above posting works only with xxx/usd and usd/xxx pairs, all others need to use MODE_TICKVLUE & friends to make it more generic).

(if the above lots are all relative to pips in the chart (spread betting) then they need not be modified for plotting the basket but MUST be transformed to real dollar based lots for MT4 brokers or Oanda.)

(The oanda profit calculator is of great help if you have managed to completely confuse yourself with the lot size calculation, I got my calculation wrong in the first two attempts, but the calcuator could finally convince me that my third attempt was finally correct)

So best to have the result as a line chart...I see that. I'll stumble along...the only thing worse than my statistical background, is my programming. I'll give it a whirl.
Cheers
CanuckCT

Hi. Are you willing to share your cointegration estimator? Are you doing real integer optimization? (!!)

Have you looked at out of sample performance or played around with any regularization?

I would prefer a looser but less over-fit basket which could be more stable.

Also, if one trades with FX lot-sizes and not spreadbet lot sizes, then the proper quantization is different because the constraints are on the final trade size, and thus one needs to do the profit adjustment prior to the quantization.

Most certainly! They are called adftest, pptest, kpsstest and vratiotest. They reside in the Matlab Econometrics Toolbox. You are very welcome to purchase a copy to include with your Matlab installation.

http://cdn.forexfactory.com/images/s..._the_floor.gifhttp://cdn.forexfactory.com/images/s..._the_floor.gifhttp://cdn.forexfactory.com/images/s..._the_floor.gif I'm not even going to dignify this one!!!


Yes.

Then make one!

Indeed.

Regards

Old Dog

EDIT: Guys - Please take heed of 7bit's warning in his thread! You can't just calculate these weights once then forget about them! You need to re-calculate regularly to see if the coefficients are changing. If so, you need to rebalance the basket in terms of trades. You can't just dive in without thinking! I don't want anyone losing real money until you are quite familiar with the principles here!

Alright, you're going to play with me.

Are you willing to discuss the name of your integer programming package / algorithm and any details?

For this is something quite complicated & technical to do well and far more subtle than convex optimization.

What sort of regularization terms have you found to be appropriate and how have you chosen them? What do you do for constructing cross-validation sets when underlying has autocorrelation?

Hello, MB

Seriously, I do appreciate your valuable input - you are clearly on top of your game here. All I ask is that, at this stage, you give me the benefit of the doubt that I know what I am doing too! I have been doing optimization like this for the best part of 40 years and started with a "slide rule"; thank the Lord the world has moved on a bit since then! http://cdn.forexfactory.com/images/s...m/thumbsup.gif

As you say, the optimization is not trivial and has to be done right.

All the stuff you need is in the Matlab....

-------------

Optimization Toolbox

Solve standard and large-scale optimization problems

http://www.mathworks.com/cmsimages/3...ain_w_3250.jpg Optimization Toolbox™ provides widely used algorithms for standard and large-scale optimization. These algorithms solve constrained and unconstrained continuous and discrete problems. The toolbox includes functions for linear programming, quadratic programming, binary integer programming, nonlinear optimization, nonlinear least squares, systems of nonlinear equations, and multiobjective optimization. You can use them to find optimal solutions, perform tradeoff analyses, balance multiple design alternatives, and incorporate optimization methods into algorithms and models.

 

1. Introduction and Key Features
2. Defining, Solving, and Assessing Optimization Problems
3. Linear Programming
4. Quadratic Programming
5. Nonlinear Programming
6. Nonlinear Least-Squares, Data Fitting, and Nonlinear Equations
7. Multiobjective Optimization
8. Solving Optimization Problems Using Parallel Computing

-------------

Since we are constrained by Lot sizes by the brokers (OandA not withstanding), I did my regressions subject to the constraints that (10 x weights) == INTEGER. So when I divide by 10 again, I get one decimal place of lots, which is what my broker uses. Very easy with the routines supplied.

I have tested many windows and looked both back and forward at out-of-sample data to see how long the regression can be expected to hold. If you plot the coefficients over moving windows, they do wander slightly, but the basket only needs to be adjusted when you get a quantised jump of 0.1 lot. Otherwise, leave it as it is.

Your questions on regularization etc are fully covered in the toolbox - you have many options for different hypotheses concerning the nature of the data.

I don't consider this for one moment a finished project - in many ways I have only just begun exploring the possibilities.

But one thing is critical. In my brief time on FF, I have learned that many people are here to expecting to be handed a profitable system "on a plate". As soon as an interesting new thread emerges, they jump all over it hoping this one is the "G-word".

They are fooling themselves!

As anyone with even limited experience will tell you, one trader's profitable system is another trader's losing system. IMHO, it is critically important to have "ownership" of the system you are trading.

You must deeply understand it, know why it works, know when it fails, and know how to handle the consequences.

That is why I INSIST that folk do they own homework and put the time in to master what is going on.

In this game, probably more than any other, a little knowledge is very dangerous!

Best regards,

Old Dog

hey canuckct...... was wonderin about your indicator........ you mentioned 'visual', how are you visualizing it......

are you actually viewing on the price bars or in standalone window such as below, of course i edited it slightly to allow that...... or are you just calling it via icustom or something...... if ya have the time thanks......h

here is my custom USDx indicator....I have an idea on how to use bands on this to get a standard deviation reading. Film at 11.
Cheers
CanuckCT

Hello all,

Well, today was the best trading day I can remember in ages http://cdn.forexfactory.com/images/s...m/thumbsup.gif.

All three live accounts up 11% or more. Even my young son's little £500 account was up almost 11%, (see attached pic).

The basket system is working flawlessly thus far, (see attached chart). Lovely break from the -1SD back to the centre. We all took profits half-way so could have had even more but chickened out on the jiggle in the middle!!

WARNING: Do not attempt to trade baskets that have been generated by simple linear regression! If you do, you may end up like this Post #77 here...

http://www.forexfactory.com/showthre...=262827&page=6

If you want to know why, look at this video! The coefficients fluctuate WILDLY every candle!! DO NOT ATTEMPT TO TRADE THIS.

http://vimeo.com/16342467

It is CRITICALLY important that you ensure that your basket is formally COINTEGRATED. If you can achieve this, your lot sizes will be stable for days if not weeks.

Otherwise you are just fooling yourself and will blow out your account.

I say again, a little knowledge is very, very dangerous!

Best regards,

Old Dog

LOL,

This did not happen because I used a linear regression. No optimization, no matter how complicated would have prevented this, it happened because I used only one week's data to fit the model because I was bored and wanted to quickly manifest a new instrument for immediate trading. And it happened after the 3rd profitable trade on this basket and it has reversed since this screenshot and is already on its way to bring be a 4th profit after slightly adjusting the coefficients.

Again, this is expected, if you look carefully you will see that it is using only a *small* sliding time window and of course the coefficients will drift, no matter what I use to fit the model when I permanently fit a new model to completely differently behaving data. Also this model in the video did not try to fit a cointegration, it tried to fit a straight diagonal line.

unfortunately the proper behavior of the basket can only be proven for the past. And this basket in the example was simply not carefully chosen and tested on anything other than these few bars. And it still gave me 3 out of 4 profitable trades. It was fitted to one week of data and stayed stable almost two more days.

Hello all,

Still alive - trading NOT posting!!

The basket is still extremely stable. The t-statistic of the cointegral kernel has only moved in the 4th decimal place.

I now have well over 6000 hours of data on the pairs (over a year). I have run several thousand tests on weekly blocks (120 hours), monthly blocks (~500 hours), etc. I have incremented the window by daily increments (24 hours) and checked the behaviour of the basket ex post for out-of-band performance based on the ex ante in-band analysis.

It is rock steady. The weights are unchanged and the mean and std dev are varying by only a few pips.

Attached is the chart and my Basket demo account, conservatively traded. Given the inherently hedged nature of the basket, and the fact that the weights have remained unchanged for a very long period, it is prudent in MM terms to trade the lot sizes in the statement. DD has been negligable and profits have been quite quick.

It is just a waiting game. Not many trades (only 3 in the week or so I have traded it) but worth it when they arrive!!

The account is up 20% in just over a week.

I go live on Monday with real money.

Best regards,

Old Dog

Hey Old Dog...is there any MT4 indicator that has come out of this process that you might share?
Cheers
CanuckCT