Optimized Trend Trading

Interesting thread,I have stumbled into it by chance,took a few hours to read the posts.
The level here is extraordinary,so,I will try to add something useful regarding my experiences with digital filters ,Fourier transforms and trading.

I won`t share my indicators,but due to the fact that I have been suffering the "digital filters illness" for years,and finally,around 3 years ago,I found the cure to it,I would like to just point out for you what I found to be the "better for the moment" tools to be used in our trading profession.

I started by trying most of the commercial packages,including ,among others the MESA software directly from Ehlers and a well known "cyclic software" that has bartels test to differentiate among cycles,used e-signal datafeed and this was presumed to be a money making machine.. nothing worked consistently,including the supposed money machine.

Then,I decided to do my own homework,so,I started by trying Goertzel,let just say that Meyers analytics papers convinced me that this was the way to go,end result,the instant phase problem and the continuous repainting,just make Goertzel a "special kind" of FFT,yes,one that can work slightly better with noisy data but that lacks the fast response needed to trade the markets with a modicum of consistency...so,I switched gears..again.

I tried FFT,MESA,Wavelets,as standalones or combined both with Neural Networks or Genetic Algorythms,final result:wavelets held some promise,specially for preprocessing data to feed the NNs,but the promises didn`t materialize in continuous profits.

Finally,I found something that has been working consistently...a special kind of FFT,its name is VFFT(Very Fast Fourier Transform).

VFFT handles the distribution of scales in a function(smaller scales must be resolved over the distance of the largest scale)in such a way that allows the optimal modelling of multi scale problems on a computer...and this works exceptionally well for Forex data.I was very lucky to find an ucranian programmer that translated these concepts into metatrader indicators,now he is busy doing so for other platforms,I will probably settle for a few APIs,but,anyway,we all know that metatrading coding qualities are not optimal.

Please ,do not ask for my indicators,they are propietary and not for sale,but I am fairly sure that you don`t need them,you can team and develop them if you become convinced that VFFT is the way to go...I will show you just a chart,so that you get the idea of what is possible with this method.

I want to thank both Alvin M Despain(inventor of the VFFT) and CB(for this unbelievable thread)..Thank you.

I won`t post much,and besides,Carnaval in Rio is approaching fast,and one has to prepare.

The result of EMD/DESA analysis on GBP/JPY M15

Anyone has the article by Despain?

fxtimer,

That is impressive. That is the problem with stuff that works consistently... you want to keep it to yourself.

Like you, I am convinced cycles are the best way also. The key of course is finding the right algorithm to handle the time series, as you have.

If you need representation in the USA, I can get you in contact with the right people. But personally, I would trade that thing into the ground!!!

An initial search of Mr. Despain or VFFT leads to mostly professional papers and sources.

It seems an extended search is worthwhile.

Carl

So, how can you project into the future with these complex waves?

Carl

Very interesting FxTimer- thanks for the post.

This seems to be the VFFT paper in question

http://ieeexplore.ieee.org/stamp/sta...isnumber=35181

At first glance it appears Mr Despain was aiming for a more efficient hardware based FFT calculation, but perhaps uncovered something more

There is more to VFFT than it meets the eye,as usual,you have to do your homework,if you can access these 4 papers you will realize its possibilities :


1-Novel design of multiplier-less FFT processors -Yuan Zhou, J.M. Noras and S.J. Shepherd(Special emphasis on the phase-amplitude splitting technique that allows conversion to cyclic convolutions and additions.)

2-S. J. Shepherd, P. W. J. van Eetvelt, S. M. R. Jones, J. M. Noras and H.

S. Rajamani, "A Lower Complexity Discrete Fourier Transform",
Mathematics Today

Not so relevant,but very interesting nonetheless :

3-S. J. Shepherd, P. W. J. van Eetvelt, S. M. R. Jones, J. M. Noras and H.
S. Rajamani, "The Very Fast Fourier Transform-A Tool for Science in the 21st Century"

4-Herron, James F,"Design and Development of a High-Speed Winograd Fast Fourier Transform"

Some of the most interesting articles,you will find in declassified material from military information databases,Bradford University in the UK has interesting materials too.

Multi-Scale Modelling ,with relevance to non linear models is the key conceptual tool one has to use to adopt the mindset that allows one to find the gold nuggetts,in that sense ,I can only suggest that you read the following introductory book :

Multiscale Analysis of Complex Time Series: Integration of Chaos and Random Fractal Theory, and Beyond
by: Jianbo Gao, Yinhe Cao, Wen-wen Tung, Jing Hu

It will give you an useful prism to view the possibilities where others see none.

Guys

Don't get too hung up on this one. Despain's paper was published in 1979 before the advent of the PC and general desktop computing power. As you know, the FFT can be implemented by decimation-in-time or decimation-in-frequency (or a combinatin of both - this is how most fast algorithms work).

All Despain really did was implement the rotations using CORDIC which is a very primitive (but efficient) way of doing trig on a calculator without much computing power.

The multi-resolution properties are quite general and can be implemented much more efficiently with modern algorithms.

The details of this method are irrelevant. It is the application and accuracy that are fascinating!

CB

This info was posted in another forex forum by Krzysztof(fajst k): noise filtering using multiscale analysis ,it has a wavelet and principal components implemented as indicators:
http://thames.cs.rhul.ac.uk/~multires/

http://www.multiresolution.com/cupbook.pdf

Yes,the details are irrelevant,unless you want consistency.

FxTimer- I notice from the two charts you posted that your indicator seems to be based around one dominant cycle, largely unvarying in frequency.

Would I be right in thinking that you are picking a single preferred cycle from your VFFT analysis and applying that?

If so, perhaps the edge your algorithm has is in how it determines which cycle to pick (LMS or something like that) rather than in the specifics of the VFFT analysis itself.

CB,

In one of your recent posts, you wrote...

"Once you have your chunk of say 360 closing prices, just work out the mean and subtract from every value. Crude, but extremely effective. (Remember Occam's Razor!). Then Goertzel will work fine, even in the presence of a lot of noise."

Alright. This is a very silly question, but do you mean rotating the data so that it has no slope from the first to the last observation, or do you mean simply translating the data (X - avg(X)) so that it's centered around a mean of 0?

Also, regarding the Signal to Noise Ratio using the Hilbert Transform. I programmed a version using this technique, and, just as you said, it doesn't seem to work very well. Any thoughts on why this doesn't work? Is it just a bad technique in the presence of noise, or are there other deficiencies?

Thanks,
Ngawang

FxTimer,

first of all congratulations for finding a system that works for you and thanks for posting here.

Now, regarding VFFT, my understanding is that it is just an approximation to FFT implemented sligtly differently for less computational requirements, and if anything it is less accurate than the real FFT. So the way I see it, whatever you do with VFFT you can also do with FFT (as long as your PC is not really slow). Have you tried using FFT instead of VFFT to compare your results? If you think VFFT is better than FFT, why do you think is that? Clearly you have devoted much more time than me on this one, so I would appreciate your opinion on the difference between the 2.

Thanks

adamant,You know that cycles are non stationary,so ,picking a single dominant cycle will only lead to continuous changes in amplitude and phase when the market switches from dominant cycle to new dominant cycle,no,the "trick" is to find a stable combination of cycles.


To do so,you have to design a filter that allows your indicator to decide on its own how many cycles to use,too many cycles and resonant noise will give you a lot of false signals,too few and you will be too early or too late.

I suggest that you read the papers,and, specially ,the book,I suggested they will help you in understanding the advantage VFFT provides in this realm.

I think it has more to do with his system, than his VFFT indicator. Which, of course he will not show you.

His first post has a polynomial curve fit onto bars, which may be an early indication of change in trend coincident with a change in cycles.

I have duplicated that indicator.

However, I need to finish my Goertzel cycles in MT4 as CB suggested I do.
Still working...

Carl

Firstly, many thanks to CB for the best thread I've ever read on these forums. At last, signs of intelligent life on FF

I've been down to my local university library and copied the Despain article on VFFT.

This isn't really my field, but my first thoughts were similar to CB's comment - that this paper is concerned with a quicker way of computing Fourier Transforms. FxTimer seems to hinting that there is more to it than this. I can't argue with him on this point because it's way over my head.

I've reduced the jpg's as much as possible but the scanned article is way too large to post in a zip. If ayone would like a copy, drop me an email and I'll send it to you.

FxTimer- thanks for the explanation. And thank you as well for the links to the book and the four papers. It does sound like your cycle selection heuristics are the really clever bit of your system, rather than the method of convolution as such, but I will undertand more when I get the chance to take a look at the literature.

Thank you so much for sharing your ideas with us.

There are so many compelling ideas in this thread it could be a lifetimes work to follow them all!

But for now...back to my Goertzels...

Thanks Challenger,Yes,I used FFT with no results,I mainly used it to find the main cycles and then applied the Bartels test to extract just the ones supposed to be persistent(commercial software,I presume I can`t post the link),in theory this is a good approach,in practice it didn`t work consistently enough.

Guys

FXtimer's post #287 is worth noting. For clarity, the algorithm described by Despain is NOT known as the "VFFT". He simply used the word "very" in front of the words "fast fourier transform" in the paper's title.

"The" VFFT was invented by Professor Simon Shepherd and his team at Bradford University in 2002 and patented by the University in two patents. This is a quite remarkable algorithm that offers the possibility of a speed-up in efficiency of substantial proportions. It has been described as the biggest contribution since Cooley and Tukey invented the FFT in 1960.

In addition, it allows multi-resolution analysis, allows for missing data and essentially extends the Winograd solution to larger sizes. It is unique in that it splits the amplitude and phase components into different number fields. It is based on deep ideas in Algebraic Number Theory and is quite a piece of work.

Crodzilla is quite right to start off duplicating the white indy at the end of the price bars with a polynomial fit - this will do just as well! The problem of cyclic extension, though, needs to be handled carefully. My method of choice for short extensions would be to try Pade Approximants.

Just a thought...

There is some fantastic work going on here - please keep it coming!!

CB

Crodzilla,my first post has an indicator that I found in a russian forum,made by Ang,its name is $RRR,I have found that when it is coincident with VFFT,in the high timeframes,it signals fairly good entries,I don`t use it ,even for confirmation,when below 60 minutes timeframe,noise kills it ,so,handle with care.

Please find it attached here,hope it helps you somehow.

Good luck with your Goertzels,I am sure that if CB suggested them it may be due to their inherent properties,so,you are in good hands.

I can`t attach the indicatorr,so,please copy and paste its code.

//+------------------------------------------------------------------+
//| $RRR.mq4 |
//| Copyright 2007, MetaQuotes Software Corp. |
//| http://www.metaquotes.net |
//+------------------------------------------------------------------+
#property copyright "[email protected]"
//---------ang_PR (Din)--------------------
#property indicator_chart_window
#property indicator_buffers 1
#property indicator_color1 White
//-----------------------------------
extern double hours = 12;
extern int m = 6;
extern int i = 0;
//-----------------------
double fx[];
double ai[10,10],b[10],x[10],sx[20];
double sum;
int p,n,f,sName,fs;
double qq,mm,tt;
int ii,jj,kk,ll,nn;
//*******************************************
int init() {
IndicatorShortName("at_PR (Din)");
SetIndexStyle(0,DRAW_LINE,EMPTY,2);
SetIndexBuffer(0,fx);
if (fs==0) {sName=CurTime(); fs=1;}
p=hours*60/Period();
nn=m+1;
return(0);}
//----------------------------------------------------------
int deinit() {ObjectDelete("pr"+sName);}
//**********************************************************************************************
int start(){
int mi;
//-------------------------------------------------------------------------------------------
if (f==1) { p=iBarShift(Symbol(),Period(),ObjectGet("pr"+sName,OBJPROP_TIME1));}
sx[1]=p+1;
SetIndexDrawBegin(0,Bars-p-1);
//----------------------sx-------------------------------------------------------------------
for(mi=1;mi<=nn*2-2;mi++) {sum=0; for(n=i;n<=i+p;n++) {sum+=MathPow(n,mi);} sx[mi+1]=sum;}
//----------------------syx-----------
for(mi=1;mi<=nn;mi++) {sum=0.00000; for(n=i;n<=i+p;n++) {if (mi==1) sum+=Close[n]; else sum+=Close[n]*MathPow(n,mi-1);} b[mi]=sum;}
//===============Matrix=======================================================================================================
for(jj=1;jj<=nn;jj++) {for(ii=1; ii<=nn; ii++) {kk=ii+jj-1; ai[ii,jj]=sx[kk];}}
//===============Gauss========================================================================================================
for(kk=1; kk<=nn-1; kk++) {
ll=0; mm=0; for(ii=kk; ii<=nn; ii++) {if (MathAbs(ai[ii,kk])>mm) {mm=MathAbs(ai[ii,kk]); ll=ii;}} if (ll==0) return(0);
if (ll!=kk) {for(jj=1; jj<=nn; jj++) {tt=ai[kk,jj]; ai[kk,jj]=ai[ll,jj]; ai[ll,jj]=tt;} tt=b[kk]; b[kk]=b[ll]; b[ll]=tt;}
for(ii=kk+1;ii<=nn;ii++) {qq=ai[ii,kk]/ai[kk,kk]; for(jj=1;jj<=nn;jj++) {if (jj==kk) ai[ii,jj]=0; else ai[ii,jj]=ai[ii,jj]-qq*ai[kk,jj];} b[ii]=b[ii]-qq*b[kk];}
}
x[nn]=b[nn]/ai[nn,nn]; for(ii=nn-1;ii>=1;ii--) {tt=0; for(jj=1;jj<=nn-ii;jj++) {tt=tt+ai[ii,ii+jj]*x[ii+jj]; x[ii]=(1/ai[ii,ii])*(b[ii]-tt);}}
//===========================================================================================================================
for (n=i;n<=i+p;n++) {sum=0; for(kk=1;kk<=m;kk++) {sum+=x[kk+1]*MathPow(n,kk);} fx[n]=x[1]+sum;}
//----------------------------------------------------------------------------------------------------------------------------
ObjectCreate("pr"+sName,22,0,Time,fx);
ObjectSet("pr"+sName,14,159);
ObjectSet("pr"+sName,OBJPROP_TIME1,Time);
ObjectSet("pr"+sName,OBJPROP_PRICE1,fx);
f=1;
//----------------------------------------------------------------------------------------------------------------------------
return(0);}
//==========================================================================================================================