Optimized Trend Trading

Hey ngawang-
Attached are two papers that should help you out. The first is on R-filters, a generalization of the HP filter. While HP uses only 2nd backward differences, (thus implying the series is I(2)), r-filters are a more general case for any (generally higher) order backward difference filters. The higher order R-filters have a much sharper frequency cutoff, but with the penalty that more future values are needed beyond the end of the series. There is, however, a tradeoff there which one can exploit.
The second paper is about TCS filters; essentially, the authors investigate the assumptions of the stochastic model underlying the HP filter. They present some ways to deal with endpoint problems, etc...
These articles are both innovative and well-written, hope they help.
-Gavner

Hi Ngawang

You are quite right. We just want to get rid of the high frequency "noise" that is polluting the clean price movement. Almost any filter will work provided, as you rightly say, that you don't distort the phase. After all, this is the very thing we want to recover!! Likewise, lag is a killer as lag introduces phase delay.

You can use a ULLMA, SSA, HP etc. My personal favourite at the moment is HP as it is easy to implement in MT (there is some great code posted a while back on this thread). SSA uses quite a lot of difficult matrix operations like finding eigenvectors which is painful. HP does a really good job.

Then (i) do the spectral analysis, (ii) pick your cycles, (iii) project each forward a few bars individually (you can't do this easily on the composite signal) then finally (iv) add them back together.

Good luck with your project - you seem to be doing great.

CB

First some context: I've been following this thread as a background task for several months now... I've only posted a few times, while doing my own research (which only partly overlaps the content in this thread) quasi-full-time. I have degrees in Electrical Engineering and Economics, and I'm a reasonably competent programmer, working toward becoming a professional trader. And my only contact with anyone on this thread (including CB) has been through reading and posting to the thread itself.

Now, I want to thank CodeBreaker for starting this thread and contributing so much to it. I respectfully disagree with the two contributors who have attacked him. If he wants to keep his identity and some of his techniques private, then I respect that wish. And for those of us who know a little about math and signal processing, his posts have been a treasure trove of ideas. He names algorithms by name, references specific papers, and clarifies points when people are confused.

Sometimes it goes fast, and it's hard to keep up, even for someone with the proper background. I've downloaded and pored over a dozen or more papers at the suggestion of this thread, and I've implemented a handful of the algorithms for use in my research. I don't expect to be spoonfed, because there's no way to learn this stuff without some intense thought and hard work. And if I didn't have an engineering background, this would probably be a waste of my time.

Anyone who cares to read this thread has free access to an accomplished professor with practical experience in applying advanced signal processing to trading systems. And more recently, the thread has attracted other smart and experienced people who are contributing as well.

If you dislike the thread or anyone on it, that's your right. But please don't ruin it for those of us who appreciate it. Thanks.

After having thought about it a bit, I will definitely second this. There is really nothing in this thread that compromises CB. If the opponents have legal issues with CB they should seek justice through the usual academic or juristic channels and not turn it into a public trial on a forum where facts gets distorted and people easily jumps to conclusions. And while I do find the referred websites a little...emm...eccentric, CB never made any references to any of them whatsoever or any commercial sites in general.

While I only have little of relevance to post (and therefor admittely little at stake), I value this thread a lot and would like to see it continue.

Hi Gavner,

Thank you for those papers. I read the Brazilian r-Filter paper. (On to the ECB paper next.) It's quite clear. I was able to get the following sharper frequency cutoff using the ideas from that paper essentially on the first try. See attached below.

Thank You!
Ngawang

No problem, always happy to share good research. Just as a message to others on the thread, if something is published in a refereed journal, there is no loss to you (and possibly benefit to others) in posting it. I realize that many have shared, and it's always appreciated. Fosters an academic environment and all

As an aside, I agree with what BarHopper and BlackMage have said. Forex is a competitive game, in case you all haven't noticed, and obviously some things people aren't willing to share. Get over it. We can still help each other learn, and mutually improve things we're working on. I again thank CB for starting a real thread with real information for intelligent folks on this forum.

Best,
Gavner

Well said BarHopper - I totally agree.

It's a shame because I thought FXTimer did have some useful contributions, but I believe it was right that he was banned.

Hi CB, can I ask you about selecting time frames?

In post 31 you said:

"I only EVER use 1h bars. The reasons for this are deep and based on my long analysis of the Shannon Entropy"

then later in post 230 you said:

"I currently like the 11min 43.2sec TF on the EURGBP"

This second post was a year later. Did you change your opinion during these twelve months, or am I missing something? (I'm not trying to trip you up here, just trying to make sure I understand what you're saying).

I guess the issue with time frames is the trade off between accuracy and timeliness. A DF on 30 minute data will give quicker signals than a DF on 1 hour data, but with less accuracy.

Incidentally I guess a 10 period SMA on 10 minute data will actually give quicker signals than a DF on 1 hour data (but the DF signal will be more accurate).

Most of us on these forums just pick one of the default TFs that are already available. These arbitrary TFs are almost certainly not optimal.

(Why do we count in base 10, or why is an hour divided into 12 five minute subdivisions? Possibly because we're pentadactyl, but certainly not because it's the optimum TF for trading fx) .

So, the question is: How do you determine the optimum TF? I think you answered this in post 230 - generate vectors for each size of TF over our data and pick the one that gives the smoothest trajectory. Am I right in thinking that this implies the optimum TF is independent of the type of DFs we're using.

I was subscribed to this thread before but decided to remove my subscription because I didn't seem to find any tangible stuff here, just discussion about abstract methods.

I have background computer science [edit: Read I have studied it, not been a professor or anything like that] and have been a CTO in couple of companies with systems that handle hundreds of thousands of users and because of that experience I'm quite strict and result oriented when evaluating trading algorithms.

I personally just don't seem to be able to understand how signal processing could be applied to trading [except by curve fitting past data of course ~ show me an indicator and I can show you a screen where it would make money even if the indicator would be based on random data]. I just don't seem to be able to grasp how you could predict movements of masses using these kinds of methods.

So my humble question is, is anybody able to produce longer term (10+ years) test data on any signal processing algo for all majors (with same parameters of course to eliminate possible curve fitting)? Is this method really profitable or is this thread more of an academic discussion of theoretically promising methods?

Note that this is not meant as an insult or anything like that, as I have said in many threads, I want to be proven wrong because new profitable trading algos are exactly what I'm looking for all the time.

I have completed the implementation of a Goertzel "browser" on MT4. There are still some quirks I need to work out...

This is EURUSD 30m. The indicator shows all cycles sorted by amplitude for a specified period. Only the top cycle minus all other cycles is shown. As has been discussed before, there is a descent lead to the waveform.

Beautiful Carl!

How did you select those 14 cycles? Was it purely by amplitude or did you require a certain minimum spacing?

It's great to see this thread back on track.

Hello everyone,

I am quite new to FX and to FF - I have only been a member 3 months. I started off in BabyPips but couldn't find what I wanted. Then a friend put me onto FF and this thread.

I don't post much on FF (and certainly nothing here) because I don't know anything yet and don't want to waste the time of the brilliant people here. I read Economics and Computer Science at Lancaster and so I know a bit about markets and I can program a computer.

I was loving this thread until the upset a few days ago. I was quite appalled by the flaming of CB. This is serious personality-disorder stuff! Bullies are never happy people and they expose nothing but their own deep inadequacies.

I don't know if CB is, or is not, a University Professor, but if he is GREAT!! Because:

(1) It is wonderful that someone so knowledgable but so busy would take so much time to contribute his expertise for nothing,

(2) A professor is used to teaching students and so can teach us in a way we can learn. That is why I am here, to READ READ READ and LEARN LEARN LEARN.

One of the flamers said CB never imparted any information. Nothing could be further from the truth. He posts the code of carefully designed and working algorithms. I programmed the Goertzel in Excel from his code in post #215 and it worked great! He gives links to papers we can download and read and answers questions in a detailed and clear way.

I am NOT interested in just downloading some MT4 indy, putting it on a chart and staring at it wondering how it works. I want to LEARN how stuff works and I have ever seen a better thread anywhere.

Please CB, don't let a few idiots kill this thread. All they did was to show themselves up as very very sad individuals. And others like BarHopper and BlackMage have rallied to your defence. We are more important than one or two ignoramuses.

OK - I have had my say. Now I will just go back to reading and saying nothing until I have something worthwhile to add.

Anne

Guys (and Gals)

Thank you for your kind comments. I will say no more on the matter.

CB

Hi all,

I would like to second the question by chamois (post 238). CB, in my attempt to understand your post 230, I've done a little reading, and have come up with a few questions:

My first question is to clarify terminology... an embedding is defined by two parameters: the dimension D and the delay d. It seems to me that a sliding window of 360 bars in a 10 minute timeframe represents an embedding with D=360 and d=10M. And these imply a 360 dimensional phase space. Is this how you are using the concepts?

My next question is, how do you measure the smoothness of a trajectory in phase space? A very high dimensional phase space (such as 360!) leads to all kinds of numerical problems (not least of which is the breakdown of the euclidean metric due to noise) so I'm guessing you use a smaller D for these smoothness tests.

One naive method would be to simply measure the distance between consecutive points in the time series, and take the mean or the standard deviation, etc. Another method would be to create recurrence plots for each embedding delay d, and eyeball the smoothness of the lines that appear. Or perhaps automate that idea with some image processing?

Or perhaps this is how you're measuring smoothness? If you look at the SNR of your end-to-end system (filter -> spectral estimation -> cycle selection -> reconstruction -> SNR estimation) the smoothest trajectory is the one with the lowest SNR? If you have a fast computer, then hey, why not?

Once again, thanks to you CB and the other collegial folks here for your contributions

Guys

Chamois raises and excellent question about timeframes. A year or so ago when I was very new to MT I was confined to just the 9 timeframes they allow. In those days, I always used the H1 timeframe simply because it "looked by eye" to be the "smoothest" (we will define in a moment what this really means!) and yet with not too much drawdown. The H4 was often "smoother" but the potential drawdowns were too much for my stomach!

More recently I have got a reliable interface working between MT and Matlab. A recent poster in this thread, brenta929, has mentioned that he also has a DLL-based system for getting data in and out of MT to Matlab.

Once in Matlab you can do SERIOUS programming!! I feed the tick-data from MT to Matlab in real-time and run a False Nearest Neighbour algorithm (loads of info on the web on this) to find an optimum "embedding" for the data.

It is critically important to realise that all timeframes are somewhat artificial. The market is NOT a regularly sampled system, so the concept of orthogonality in the time-domain does not apply. (Hence the problems with Fourier techniques!)

Trades happen as and when they happen and so the time intervals between ticks are random (although approximately normally distributed - see my earlier post with a link to a downloadable paper on High Frequency Trading). This is the "true" timeframe of the market.

When we gather up the ticks into bins of equal interval (M1, M5 etc) we do this just to make our lives easy and to make the X-axis of the charts have a nice linear scale.

BUT... There is no mathematical justification for this at all. It is purely convention! If you take the embedding approach, you will come closest to finding the most "natural" timeframe for the market. This changes from pair to pair and also in a given pair with time. This is the chaotic dynamics of the system coming out. But once established, the timeframe usually lasts a while. My 11m timeframe lasted around 3 days before needing tweaking.

IMHO, this timeframe business is ABSOLUTELY CRITICAL to success. It is a necessary, but not sufficient, condition. Of course, you need a strategy as well! (Without mentioning any names, this is why the more "successful" commercial EA's specify very carefully the currency and timeframe they need. And some even bring out monthly updates as this changes. Now we know why...)

This is a VERY important topic and I am more than happy to expand on it in future posts.

CB

(PS added in edit) - BarHopper just beat me with his post!! I will try to address his questions shortly!!

Hi

As I said, this is a deep and important topic. There are no textbook answers and I don't know all the answers. All I can do is give you my opinion and tell you what I do..

[1] I use the term "embedding" in the same sense, but when one is talking about lagged vectors it is conventional just to specify Dimension. Delay is assumed and is always 1 unless otherwise stated. So if D=10, you just take your price series in chunks of 10 and slide along by 1 bar each time. These vectors are, as you know from previous posts, nothing more than the columns of the circulant matrix, whose left and right singular vectors are our old friend SSA.

[2] I measure smoothness by the FNN algorithm.

[3] No. I don't measure smoothness from SNR or vice-versa. I just said they are closely connected. If you have a bad embedding dimension then the waveform appears ragged. This is why some TFs are better than others. They are just different embeddings of the tick data - ie the actual market!

CB

Would this material any help to subject?

CB, what you saying about about selecting time frames implies that the optimum time frame is independent of the method you use to process the data.

This is an interesting result because it runs counter to what most people on these forums think (that some methods work better on different time frames than other methods).

Thanks CB, I think I understand... as long as your D is big enough to "unfold" the dynamics, you can hold it fixed, and construct a plot of FNN% vs. timeframe, and pick the minimum. This is instead of plotting the delayed Mutual Information vs. timeframe as several papers on this topic are doing.

(And if timeframe is your goal rather than D, one approach is to use the MI and skip the embedding and FNN entirely, if I have understood correctly.)

I guess you have a reason for preferring the FNN to the MI? I've personally written code to estimate MI, entropy, etc. of financial timeseries. I've been mostly satisfied, but a bit troubled by the somewhat arbitrary process of binning the data to form an empirical discrete pdf to approximate the density of a continuous variable. I've seen mention of using the correlation integral instead, but have not delved into it. Do you have any opinions on this topic?

Cheers...

Hi All,

Here are two papers that seem to replicate the process and research conducted by Farmer and Sidorowich from the 1980s. The information and techniques appear relevant to the conversation. One of my long term goals is to understand and be able to use the methods they describe, but first... signal processing, filter desing and spectral analysis.

Hope this is helpful,
Ngawang