Optimized Trend Trading

Hi SeekingLight

Since I am just playing with the new robot, I am being kinda cautious so am doing £1/pip on the forex. My account started with £1K. The robot has an internal stop (so the brokers can't see it!!!). It is currently set to 50 pips but I have not yet seen it triggered. So I guess the "risk" is £50 (or 5%) and the return is around 42% which seems OK. If we assume a risk-free bank rate of 4%, folk who worry about such things would interpret this as a Sharpe ratio of 7.6 (and economist types in the Management School tell me anything over 3.5 is good).

BTW, the robot is a tad more sophisticated than just an MA crossover!!!!!!!!!!!!!!!!!! Read my posts.

CB.

I have received many, many PMs from F_F_L and other worthy folk on this forum who (a) have open minds, and (b) want to know about a bit more about what I am up to. So for the benefit of FFL and others who want to succeed (as opposed to those who just condemn what they clearly do not understand), some thoughts are as follows:

Mathematics is everything. Read that again - Mathematics is everything. There is nothing else (well, Beethoven, maybe, but I digress...). In the Medical School where I lecture, we recently had some visiting researchers who were discussing the problem of C. Diff in hospitals (a big problem in the UK). Among them were microbiologists, biochemists, chemists etc. C. Diff can be either "vegetative" when it acts like an ordinary bacterium and can easily be killed with Dettol or similar, or it can be a "spore" when it is kinda nuclear-hardened and almost impossible to kill. When C. Diff finds itself in a "hostile" environment it forms spores. When conditions are favourable again, it goes back to the vegetative state. This, in a nutshell, is why it is so hard to eliminate from hospitals. Without going into details, C. Diff likes moist and dislikes dry. So, how does it detect its environment? The microbiologists talked about "transfer processes in the phospho-lipid bi-layer". "Charge", says I, from the back of the room. The biochemists talked about surface potentials on membrane proteins. "Charge", says I, a little louder. To cut a long story short, one of them eventually asked what I was talking about. At the end of the day, the organism can ONLY sense the presence of water from the charge on the polar water molecule. When you cut through all the jargon of the different disciplines that serves only to obscure the issue, we find the truth. Which is always simple. The great mathematician David Hilbert said that he always knew if a proof was right. If it was simple, it was beautiful. And if it was beautiful, it was almost certainly true.

Remember --- Medicine is just applied biology, biology is applied chemistry, chemistry is applied physics. And physics is just applied maths. THERE IS ONLY MATHEMATICS.

So what has this to to with FOREX? I start from the assumption that the FX market MUST obey the laws of physics. If, indeed, this were not true, and we were to discover that the FX market is the only entity in the Universe that does NOT obey the laws of physics, this would be a discovery of such staggering proportions that the instant Nobel Prize and other speaking engagements would give one sufficient cash not to have to worry about FOREX trading.

So let us put to one side all the mumbo-jumbo and confine ourselves to the facts. The market is a stochastic time series, which is non-linear and non-Gaussian. 200 years ago the great Gauss showed that the best estimator (in the Maximum Likelihood [MLE] sense) of a RANDOM, STATIONARY process is its moving average. But we all know that moving averages don't work. Well, now we know why. The market is NOT a stationary process. (Look this up!! By stationary, I mean wide-sense-stationary in the statistical sense). So we can forget MAs and EVERYTHING based on them including RSI, stochastics, support and resistance, etc etc etc ad nauseam. SeekingLight is right, but I suspect for the wrong reasons. F-F-L is absolutely right that we must "keep it simple". Why - because if it is simple, it is probably right!

In the market, with hindsight we can see trends and periods of messing about. If we can capture the trends and avoid the messing about, we will win. So let us consider the market as a SIGNAL corrupted by NOISE. Unlike communications signal processing where the signal is known a priori (as is the statistical profile of the noise, eg AWGN [an additive white Gaussian noise] channel) in the market, we don't know what the signal is. That is what we want to find!

So let us arbitrarily DEFINE the signal to be trends, and the noise to be everything else. The problem now reduces to finding a matched filter to separate the signal and the noise. This is NOT a trivial task but perfectly feasible IF you know the mathematics underlying non-stationary, non-Gaussian filtering. Here we have to deal with not additive noise (which is easy) but MULTIPLICATIVE noise which is VERY different.

My robot is a first stab at this, and he is doing OK. Better, I suspect, than many schemes currently used by other forum members. But I could be wrong......

Enough for now -- I don't want to lecture the forum but this is a big subject that will need many posts to cover.

If, of course, anyone gives a damn....

CB.

I'm interested.

Wasn't saying you/robo was doing only MAs, was just pushing a pun back from the original stuff when you'd thrown on those two highly adaptive lines since that was what the fussing was over

I'm also curious as to what happened after you said charge a few times and got taken up on it. Did they figure they should find a way to electrocute around more and get it or rather the other way around? 2 AM here and I forget which way is to get it..whether it's soft in sleepy or wakey state
Basically I was wondering a) who was posting this stuff and b) whether you'd by shouting around a simple truth solved a health dilemma or not, since that kind of thing is always what I'm looking for / would like to happen more often.

Do keep in mind however that 99% of all readers are about as deeply mathematically inclined as the average chimp. Although I could be wrong; I am mostly drawing this conclusion from myself
Much rather would hear one banana, two banane rather than Gauss n Strauss. What I mean is that if you'd like to include simplified nutshell versions parallel to the complex stuff, there'd be more benefit to the masses for it.
Of course I'd also understand a need for exclusiveness simple due to the nature of the thing and an inherent communication level.
Either way - just trying to stay on board of what you're saying..always interested in science and scientific approachs to whichever..

Thank you for the edification and profound post, CB. I'm the type to read lightly on several diff. topics and one thing I've discovered coincides w/ exactly what you've said. Math is everything, including Beethoven (from what I've read). My brother used to tell me, when I was a younger youngster, that numbers are the only things that have remained constant throughout history. 1+1 will always = 2. He felt that such a fact meant it should be naturally easier to learn math, even in it's more complex subcategories. I, being more creatively mind than than logical, understood his rationale but was never able to assimilate the information quite as easily as he.

That being said, all the things you discussed (in laymen's terms (thank you)) I understand the completely and your philosophy makes perfect sense. Further proof, for those who disagree, is in the fact that most jobs for trading systems designers require proficiency in more than one coding language (math) and a degree in physics, engineering, or direct math and preferably knowledge of econometrics .


I, for one, (and sure many others) give a damn and am looking forward to you multi-post dissertation, if and when you're both willing and able, bro. You're knowledge in the field of mathematics is bested only by your benevolence. Building up some good karma

In my next post, I will explain my methods and experiences w/ de-noising as well as my analysis on CB's method (positive and simultaneously inquisitive ) and my attempts @ replication. If I'm not mistaken, he's explained most of what he does already on this as well as other threads. He done so directly (obviously) and, in a couple areas, indirectly (I believe). I maybe be and probably am wrong, though. Till then......

Codebreaker,

Congrats on a very worthwhile topic. FFL informed me about this post and told me i had to check it out. I agree with him and want to extend my sincerest thanks for your thoughts. We have been at it for a couple of months working with "filters" to lessen the "noise" that we see in trading. We have been using digital filters ( both high and low pass, along with band pass) to aid us in our methods.

I don't want to steal his thunder, so i will leave my comments brief. Looking forward to your input.

More later,

cl

Here we go:

From my experience, the best filters are indeed those that aren't built on algorithms or theories that assume that the financial markets are stationary and/or Gaussian in nature. The problem I've encountered is that I, as well as a few of my cohorts, have hit a roadblock as far as non-stationary time-series analysis goes. On the TSD forum, there are quite a few threads and individuals of interest.


For example, on this thread , Digital Filters are the topic. (Anyone who is interested can read, in-depth about the how's and why's via the information and links on the thread itself.) The method used to optimize the filters are a Maximum Entrophy Spectral Analysis included in the Digital Filters Generator software. Basically, you pick sample period of price that you wish to use for to determine the markets cycle amplitude, software calculates automatically in a lickety-split, displays a series of peaks which are said to be the dominant cycles unique for that particular pair and t.f. which then allows you tweak the parameters of and create several variations of de-noised, smoothed filters, when combined create a system w/ pre-determined rules. More thorough explanations w/ screenshots shown here and here WHEW!!!

The ROADBLOCK we just recently had placed before us occured when a member named robertinno (who is also a member here on FF and has commercial interest and as a result has since been banned AND who's site I referenced in the two most recent links ), who I might add seems to be very knowledgable in this area, explained here that using the SA (Spectral Analysis) in the aforementioned fashion was incorrect as we were using STATIONARY data (which we now know to be a synonym artificial) and had no idea of such. He briefly explained that we should use something similar to "Change In Price". He gave us a taste and then suddenly decided he didn't want to be as patient w/ those who weren't on his level and would only answer 1 out of 5 questions (if that) and then w/ a brief one sentence reply. Ultimately, progress has stopped and yet here we are. No one on the board seems to have the experience or background to understand where he was going w/ this. If you click the last embedded link, you'll see where several memebers including cl3307 (clahn04 on TSD) and myself have been racking our brains to figure it out. Though @ it's basic level, it seemed to make perfect sense. To take it a step further, I read that this method of spectral analysis was not only a poor selection for doing such work BUT also that the instance provided in the DFG software wasn't even true to correct MESA.

With all that being said, the reason I feel it's a topic that requires some serious attention is because these type of filters and their exit, order management, & entry logic (I listed those from most to least important )are mechanical in nature and have been proven, not only by robertinno, according to his site, (w/ several e.a.'s with supposed profitability), but also a very accomplished individual, detailed here as well as with my own two peepers on my charts, to give a very "sharp" edge. With more research and proper execution of data analysis and filter compilation, it'd be about as sure fire as applying econmetrics to financial markets can get IMHO.

*NOTE*

The filters will, depending on how you optimize them, require periodic re-optimization. The longer the sample from which you optimize, the less often re-optimization is required.

NOW, even w/ all of these current flaws, I believe that I've gotten into the ballpark of (using the above methods) CB's technique in a very basic form. PLEASE correct me if I'm wrong CB (and please note that in no way am I trying to discredit or belittle any of your work, but actually trying to get insight into your logic), but it seems as if w/ your "ULLMA", you are using the same "Length" (+/- 1)but different applications of price. One set to Close and the other set to High+Low/2 or something similar? I say this because no matter what indicator I used, be it the Low Pass Digital Filters or Jurik MA's, using that technique, it produced visually similar results w/ the Close line being the smoother (while lagging) and the High+Low/2 line being more reactive. I cannot achieve the low lag that you have.

Funny story. I had done this several months ago (the Close vs. H+L/2 thing w/ Digital Filters) and posted on a private 4X forum of which I'm a member, but everyone shunned it as a basic MA crossover doomed to fail miserably so I , being the newb I was, tossed it to the scrap pile because I thought they knew what they were talking about. Now on that same private forum, about 30%-50% of the members have either given up on trading, or disappeared from the forum. Go figure. Lesson learned......

Man, this is a long post.......

I'm looking forward to any and all feedback,

Peace,

F.F.L.

Reserved for next lengthy post once I get the time and energy.......

Another reserve......

Hi Guys

Just a quick post as I am at work :-(( Attached the trades file and equity
curve for the robot trading IG-Index 1-minute EURUSD live up to 3 minutes
ago. As you can see, he makes a little loss while he gets his act together,
then starts ramping up. Just gone through a bit of a rocky patch this last
hour.

CB

2 quick questions for you, CB (if you don't mind my asking):

1. On this thread, when you said you "smoothed OHLC streams" w/ "ULL-MA" and then applied the "HA calculations", were the original HA calculations altered to use ULL-MA instead of traditional MA?

2. On that same thread, when you said "price fidelity", what exactly did you mean?

Hi F_F_L

1. On this thread, when you said you "smoothed OHLC streams" w/ "ULL-MA" and then applied the "HA calculations", were the original HA calculations altered to use ULL-MA instead of traditional MA?

Nope - the bog-standard HA with my smoother on top.

2. On that same thread, when you said "price fidelity", what exactly did you mean?

Literally, how closely the smoothed signal follows the price. Of course, there is no point in making it follow the price too closely -- or we might as well not bother with the signal processing at all ;-)) If you read Jurik's site he makes a big thing of this and his indicator is very good at following price, but smoothly. Mine overshoots more but has less lag. Since I am not using the smoothed signal as a proxy for price, but just to time trades, the overshoot is irrelevant. But the low lag is essential.

CB

Thanks for answering my questions, CB.

In related news:

I was recently referred, by Igorad, to an article describing a couple of Jurik indicators created by Nikolay Kositsin, here, that allows one to preprocess OHLC data. Actually created from already existing technically correct functions that can be called up on, allowing easier creation of indicators in the future. Worth a read for those interested.

P.S.

Doesn't seem like there is alot of interest OR that it's just a few trying to contribue w/ a great number lurking.

Hi Guys

Here's the robot's trade file and equity curve for the whole week just gone from market open on Sunday evening to 2250 Thursday evening when I shut down for the Easter w/e. Made 375 pips on the £1K account at £1/pip so up 37.5%. As you can see, the equity curve is nothing to write home to Mother about, but at least it is upwards. With the current signal processing, the robot always makes a profit but has a bit of a rocky road getting there. Perhaps I should look into multiple-time-frames as the stochastic volatility (and hence Shannon mutual information) is quite different at different time frames and I might be able to do some kind of averaging to smooth the equity curve while preserving the profit.

(I promised some longer posts and plan to do these over the holiday w/e but have to sing the countertenor solos in Mozart's Coronation Mass in the Cathedral on Sunday and need to practice, so that is priority #1.....)

CB

Well, I promised forum members I would write something more "meaty" and so here goes. It is difficult to know where to begin with this topic as Signal Processing at the level where it is actually useful is a huge subject and almost everything is a pre-requisite for understanding almost everyhting else. So, I will just begin somewhere and let the forum feedback guide the direction.

F_F_L mentioned spectral analysis, particlarly MESA, in #46. (BTW, why mess up the thread with stubs #47 and #48??? Much better to maintain the chronological flow of the thread. I must ask the moderator to remove these silly posts).

Spectral analysis is an interesting idea and has been developed very extensively by John Ehlers in his MESA software package. MESA stands for Maximum Entropy Spectral Analysis and is based on quite different ideas to Fourier analysis. Essentially, MESA is an "all poles" method in that the filter transfer function is 1 over a polynomial. This means that at the roots (where the poly goes to zero) you can get really peaky functions which is great for spotting discrete frequencies among noise.

In Signals and Systems Theory 101 you would cover the Weiner-Kinchine Theorem (at my University anyway). This theorem states the (obvious) result that the Power Spectral Density of a signal is equal to the Fourier Transform of its Autocorrelation Function. This is great FOR STATIONARY SIGNALS. The problem is not with the theory but with the notion of correlation. The corr of two vectors is the cosine of the angle between them in the n-dimensional space they inhabit. To compute correlation, you first subtract the means of the vectors to centre them about zero. This is great for stationary signals as the mean is constant (this is one of the requirements in the definition of stationarity) but for non-stationary signals like financial time series, the mean is everywhere different. So correlation is not possible in any meaningful way. So the "power spectrum" does not really mean anything either.

That is not to say all is lost! Correlation may be meaningless but co-integration is NOT! This is a very powerful idea and if we combine this with cyclo-stationarity we have something infinitely more powerful but this is a huge idea for another 30-50 posts down the line somewhere.

Moving on..... In my system, I talk about trends and more than a few readers pour scorn on the idea of finding and trading trends. Let us be quite clear about what I mean by TREND. I mean a trend that is genuinely a trend and not an accident - ie, one we can identify and exploit.

Consider tossing a fair coin 1,000,000 times. If we assigned +1 to heads and -1 to tails and plotted the cumulative sum we would plot a Drunkard's Walk which looks alarmingly like a FOREX time series. In more than a few places in this long sequence, we would more than likely find long runs of heads and long runs of tails. These would appear in our plot as up-trends and down-trends. Aha!!! we would say! Here are some useful trends we can trade! NO WE CAN'T!! If the coin is fair, these trends are the result of pure statistical accident and we cannot EVER do ANYTHING to predict them in advance or say how long they will last. This why I don't use flags, pennants, trend channels or similar -- these fall into the "goat's entrails" category of system. It is just voodoo with no scientific basis.

These accidental trends are QUITE DIFFERENT from trends that arise due to external forces driving a financial market. These are the trends we want to capture and trade. So how do we do this?

Well, as I have said before, we DEFINE our wanted trends to be "signal" and define everything else to be "noise". We have a market time series that is (wanted) signal buried in tons of (unwanted) noise. We want to build a "matched filter" to pull these signals out of the noise. The sum of anything stationary with anything non-stationary is non-stationary. The market time series is non-stationary because it is the sum of non-stationary trends and (for simplicy we will define) stationary noise.

Building such a filter is non-trivial but quite straightforward for a competent mathematician. How do we know when we have succeeded? If we subtract the smoothed filter output from the input data we would expect to be left with the "noise". We can easily use a Dickey-Fuller co-integral unit-root test (also non-trivial but no problem) to determine if the resultant "noise" is stationary or not. If it is, we have, by definition, succeeded in our quest.

All the best

CB

CB,

This has become a very worthwhile thread indeed. Thanks for your comments. I'm looking forward to more.

As FFL has described, we have been using digital filters to decrease the "noise" that we see. Attached are Low Pass clones i created that mimic your lines. I'm looking forward to learning the "inner" reasonings behind your methods.

All the best,

cl

Hi cl,

Cloning?I don`t think so...one of the features of CB`s filter is overshooting,which is irrelevant if what you want to do is trade the slope of what you now define as signal(and I would agree with him 100%),but extremely interesting if you want to replicate his results.

CB:

1-You are probably using a phase advanced filter with very few parameters,and this causes the overshoot,since you have to accept overshoot in order to decrease lag..anyway,if and when you want to be more specific,we can explore this issue.

2-A big question..do you have any use for the Feigenbaum constant in your model?Looks like it has been proved(for the moment at least ) at being extremely useful when determining bifurcation points.

3-Please continue posting at this exceptional thread,your words are extremely appreciated by those among us that have invested years in exploring and using digital filters in trading.

Regards
Simba

By "cloning" i meant creating similar lines that filter "noise". The filtering out of noise is something that is just fascinating me. It seems like there are many different ways ( i'm sure not that many...but more than 2) to do it. I'm really looking forward to CB sharing more about his method ( if and when he so chooses).

Best,

cl

Yes,there are many different ways,and we share the fascination
I have found that what we worked at TSD with digital filters was interesting,but that there are much better ways to filter than using MESA,FFT,or even Goertzel.
When filtering noise,we exchange lag for noise,the ideal filter being one that gets zero lag and zero noise..the ones that approach this ideal,both in trending and ranging markets,usually have overshooting as their distinctive trait.
The 3 attached charts all have the same filter with the same length(20 periods) and phase(100),the behaviour of the filter looks different,because the overshoot(the reason behind the overshoot really ) is what makes it so adaptive..I use its slope to define trend,specially in D1/H4,have tried to do crossovers with same filter and different phases,and the usual apply to close,apply to high/low/midprice..etc and have found that slope is what works better for me.
In any case,it would be extremely interesting if CB can add to his explanations,both his filters and his "radar" are extremely interesting.

Regards,
Simba

Hi,
Is this a teaser or are you sharing your system? I thought everyone in this forum try to help others?

I am helping you. Particularly, by NOT giving you a "system". You will learn nothing from that. Far better you work to understand the principles and then design, build and "own" your own system. Trust me.

CB