Facing problem with Math of Harmonic Analysis | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Harmonic trading-harmonic patterns 527 replies

Setup frequency and probability - Math problem 8 replies

Math problem! How do you calculate hedge (hedging) in Excel? 0 replies

Margin level % math problem 3 replies

Little math problem for indicator 9 replies

Attachments: Facing problem with Math of Harmonic Analysis
Exit Attachments

Facing problem with Math of Harmonic Analysis

  • #1
  • First Post: Feb 5, 2018 12:33am Feb 5, 2018 12:33am
Hi,

Few days ago I read one old book: Economic Cycles: There Law and Cause By Henry Ludwell Moore (freely available on the internet). Moore in his book used Harmonic Analysis (Fourier Series) to ascertain the dominant cycle in Ohio Valley rainfall data. Moore uses 72 years (1839-1910) of Ohio valley rainfall data to predict the future trend of rainfall in that region. Moore shares rainfall data and formula for calculating coefficients. Then he shares final values after calculation and plots a graph of those values.

Moore writes that his results are based on invention of Joseph Fourier, but the perfection of Fourier's method is possible due to work of Professor Arthur Schuster. After searching about Arthur Schuster I found he is well known for Periodogram Analysis. Then, I started reading about Periodogram Analysis, but because of my lack of knowledge of higher mathematics I am unable to find the right way forward and thus decided to ask for help.

Moore calculated coefficient values for 3 to 36 cycle lengths. After applying the formula I am getting right values for cycles which completely divide 72, such as 3, 4, 6, 8, 9, 12, 18, 24 and 36. For all other cycles, with an exception of 5, I am getting wrong values.

I am attaching the excel sheet with all the relevant data. Formula is calculated in the sheet. I am also attaching the ebook 'Economic Cycles: There Law and Cause By Henry Ludwell Moore'. The rainfall data and periodogram values are given in page 32 and 33 of the book. Chapter 2 of the book (page 6 to 36) deals with calculation process. If someone who is good in math or have an understanding of this method can be kind enough to assist me in this, I would be truly grateful. I don't need to calculate exact values, but even if close values are calculated that's enough.

Thanks for reading
Attached File(s)
File Type: xlsx Harmonic.xlsx   149 KB | 651 downloads
File Type: pdf EconomicCycles.pdf   8.3 MB | 1,349 downloads
  • #2
  • Feb 5, 2018 3:16am Feb 5, 2018 3:16am
I didn't read the book and I didn't reverse engineer the excel sheet but I think your results are correct.
Fourier decomposition is about periodic time series implicitly infinite in length. The formula assumes that your data contains an integral number of cycles so it can be repeated infinitely. Your data is 72 samples so the formula works with the cycles of the periods which divide 72. To understand this just look at the sin(x) below. It was generated on the range 0-8 which is not a integral number of cycles (0-2π). The formula "rubber stamps" implicitly. The discontinuity creates artificial frequencies.
No greed. No fear. Just maths.
  • #3
  • Feb 5, 2018 3:59am Feb 5, 2018 3:59am
Quoting PipMeUp
I didn't read the book and I didn't reverse engineer the excel sheet but I think your results are correct. Fourier decomposition is about periodic time series implicitly infinite in length. The formula assumes that your data contains an integral number of cycles so it can be repeated infinitely.
Thanks PipMeUp for your reply. I know that according to Fourier Series my formula is right, but as I said Periodogram Analysis of Schuster further refines the Fourier method, which is what Moore says in his book. Moore does something with the coefficients, of the periods which don't divide 72, to get different readings than what I am getting. My brief reading of Periodogram Analysis tells me that it was used by Schuster to determine dominant sunspot cycle. I am unable to understand the mathematics involved in calculating Periodogram values.

Quoting PipMeUp
To understand this just look at the sin(x) below. It was generated on the range 0-8 which is not a integral number of cycles (0-2π). The formula "rubber stamps" implicitly. The discontinuity creates artificial frequencies.
Going by your reply, I assume Schuster was able to get rid of artificial frequencies. Am I right? I am not good at higher level mathematics, but you seem to have much more knowledge of this subject than I have. If you could please go through the relevant pages of the book and take a look at the excel sheet, maybe you could suggest some changes in the formula. Hoping you will help. Thanks.
  • #4
  • Feb 5, 2018 5:48am Feb 5, 2018 5:48am
This book was written almost a century ago. Signal processing theory has evolved a bit since then. I don't understand what you mean by "Periodigram Analysis of Schuster further refines the Fourier method". Can you point the page? The Fourier transform has nothing to be refined. It is just what it is: a dual representation of a signal in the frequency domain.

You can never get rid of the frequencies introduced by the discontinuities. You can try to limit their effect by using windowing. This technique multiplies the signal by a function which forces the start and the end of the signal to be zero. Multiplication in the time domain is a convolution (blurring) in the frequency domain. The peaks become bumps. => search google for "Fourier windowing". You'll mainly find DSP and electronics related stuff.

A few things you need to know if it is not already the case:
- The main cycle is not necessarily the one with the maximum value in the periodigram. Harmonics may have higher energies. This problem arises for instance in speech recognition.
- The sampling frequency can create phantom frequencies. This is called aliasing. See the picture below.
- You need a series spanning several cycles to detect them. Cycles come and go so quickly in financial time series that the time you acquired enough samples the cycle is gone and replaced by another.
- You may get interested in Goertzel algorithm. It is like Fourier but for a single frequency.

BTW I also tried to measure the cycles in the price and I completly failed. (https://www.forexfactory.com/showthread.php?t=474643)
Attached Image (click to enlarge)
Click to Enlarge Name: up-sampling-fig2-lg.jpg Size: 128 KB
No greed. No fear. Just maths.
  • #5
  • Feb 5, 2018 3:09pm Feb 5, 2018 3:09pm
Quoting PipMeUp
I don't understand what you mean by "Periodigram Analysis of Schuster further refines the Fourier method". Can you point the page?
That was my mistake to use the term 'refines', I should have used 'improves for testing probability'. In first line of Page 8, Moore writes, "The perfection of the method whereby the findings may be subjected to the test of probability is the work of Professor Arthur Schuster of Manchester".

Quoting PipMeUp
You can never get rid of the frequencies introduced by the discontinuities. You can try to limit their effect by using windowing. This technique multiplies the signal by a function which forces the start and the end of the signal to be zero.
I am completely new to this concept. Thanks very much for introducing me to 'Fourier Windowing'. From the time I read your post, I have been reading about it. From what I have gathered, there are several techniques for windowing. And what I am unable to ascertain is what values to use in which function of my formula to make sure that the data at the edges are zero, so there is no discontinuity. I think I have understood in theory what needs to be done, but I have no idea what changes needs to be made in the formula. That's because of my lack of understanding of Calculus.

I read your post in 'Cycle measurement: the easy way' you have deep knowledge of this subject. From what I got you were also trying to find the dominating cycle in a price movement. Moore also tries this and combines more than one cycle to predict the probable future. Now, I don't know about predicting future and I definitely don't want to get in the merits of a strategy I haven't tested. I just stumbled upon an old book, found its information interesting, put some hours to create an excel sheet and finally realized I am in a soup. I just want to solve this problem, but I also think there is some value to it.

I am truly grateful for all the help you are providing. Also, I will study about Goertzel algorithm, like you suggested, but will keep looking for the solution of this problem. I will need few more hours of study on the points you have mentioned, before I can actually come-up with a proper question. Again, I am really thankful for all your guidance.
  • #6
  • Feb 6, 2018 5:01am Feb 6, 2018 5:01am
I had to search about Arthur Schuster, probabilities and Fourier. I understand better now what you mean by "improvement".
Quote
what I am unable to ascertain is what values to use in which function of my formula to make sure that the data at the edges are zero, so there is no discontinuity.
You simply "stretch" the window so it spans exactly your samples. You multiply the signal by the window. The resulting transaformed data is what you plug in the formula. For instance with Hann window you get this below. I repeated the samples three times. Because the rainfall data is quite noisy you won't really see a difference.
Attached Image (click to enlarge)
Click to Enlarge Name: windowed.png Size: 93 KB
No greed. No fear. Just maths.
  • #7
  • Feb 7, 2018 8:06am Feb 7, 2018 8:06am
Quoting PipMeUp
You simply "stretch" the window so it spans exactly your samples. You multiply the signal by the window. The resulting transaformed data is what you plug in the formula. For instance with Hann window you get this below. I repeated the samples three times. Because the rainfall data is quite noisy you won't really see a difference.
I think I have solved the issue. The difference in values was due to boundary effect. So, I discarded extra values in the problematic series and values I am getting are very close to Moore's values.

Thanks very much PipMeUp for helping me.
  • #8
  • Edited 3:50pm Apr 23, 2018 2:54pm | Edited 3:50pm
Hello Dilmesh !!
I read the kind of problem that you were facing and its naice to know that you were able to solve the problem. I am facing the same problem as you were and for some reason I am not able to calculate correct values for the cycles which are not completely divisible by 72. The only exception being cycle #11 where it matches the figures derived by Mr Moore. I am not able to get correct figures for Cycle #3 which is completely divisible by 72.
Its been over 5 months I am stuck at this problem and I could nt seem to get an answer.
I dont have a strong mathematical background. I could nt fathom the meaning of your last post which states the difference in values was due to the boundary effect so you discarded extra values in the problematic series.
You have an understanding of the problem and it would serve me immensly if you can help me solve the problem mate. I would be grateful for your assistance.
Thank you.

Attached is my excel sheet on the problem and calculation.
Attached File(s)
File Type: xlsx Fourier-Mr Moore.xlsx   360 KB | 470 downloads
  • #9
  • Jun 1, 2018 1:19pm Jun 1, 2018 1:19pm
Quoting mercury63
I dont have a strong mathematical background. I could nt fathom the meaning of your last post which states the difference in values was due to the boundary effect so you discarded extra values in the problematic series.
Hi Mercury63,

I visit FF in gaps. Although I did came several times in past 2 months, but didn't see your post. I am very late in replying, I apologize, I hope this answer clears your doubt.

Suppose you are calculating for 10th cycle, then 72 divided by 10 leaves 2 as remainder. Here, 2 is boundary effect, so you take only 70 values, don't add last 2 values. For 15th cycle, remainder is 12, so you don't add last 12 values as they are extra values. You don't get the exact values of Moore, but the chart you will create will be very similar. After all we just want to find the dominant cycle and that purpose is served easily.

I haven't seen your excel file, but I have explained my solution, I hope you will be able to solve your problem.

Next time while answering please quote me. It will make it easier for me to see your reply.
  • #10
  • Sep 10, 2018 2:37pm Sep 10, 2018 2:37pm
Quoting Dilmeshsingh
{quote} Hi Mercury63, I visit FF in gaps. Although I did came several times in past 2 months, but didn't see your post. I am very late in replying, I apologize, I hope this answer clears your doubt. Suppose you are calculating for 10th cycle, then 72 divided by 10 leaves 2 as remainder. Here, 2 is boundary effect, so you take only 70 values, don't add last 2 values. For 15th cycle, remainder is 12, so you don't add last 12 values as they are extra values. You don't get the exact values of Moore, but the chart you will create will be very similar....
Hello Dilmesh !

Thank you for your reply. I guess its my turn to apologize I had taken a break from trading sometime around June and I resumed my trading activities around the first week of Sept. I didnt visit forex factory much after June although I had seen your reply. My apologies for the delayed reply. Your explanation makes sense and I didnt think about the fact that all we need to find are the dominant cycles through fourier series. I was focused in getting the same results as Mr Moores but anyways after reading your reply was able to derive a curve similar to Mr Moores. Thank you once again and I will be glad to share my trading resources with you, if you are looking for anything specific that I may have.

Cheers !!
  • #11
  • Jul 27, 2020 1:26pm Jul 27, 2020 1:26pm
[quote=mercury63;10991968]Hello Dilmesh !! I read the kind of problem that you were facing and its naice to know that you were able to solve the problem. I am facing the same problem as you were and for some reason I am not able to calculate correct values for the cycles which are not completely divisible by 72. The only exception being cycle #11 where it matches the figures derived by Mr Moore. I am not able to get correct figures for Cycle #3 which is completely divisible by 72. Its been over 5 months I am stuck at this problem and I could nt seem to get an answer. I dont have a strong mathematical background. I could nt fathom the meaning of your last post which states the difference in values was due to the boundary effect so you discarded extra values in the problematic series. You have an understanding of the problem and it would serve me immensly if you can help me solve the problem mate. I would be grateful for your assistance. Thank you. Attached is my excel sheet on the problem and calculation. {file}[/quo

Hi Mercury63, how are you? I was reading an article from a book and while I was struggle to find answers, it happened that I stumbled on your post here which is just same like what I am looking for. Looking at your excel sheet here for the rainfall data, I have some problems which I believe you already have the solution. Based on your Excel sheet 1, on the formula (2/N) * (rainfall amount) * COS(2*PI*(t)/(T)) what is the value of PI? For example based on your 3rd cycle, how did you obtain your first value of a1 (0.8311) and your second value of b1 (1.0306)? I am sure if I know how you got just the first value, then I can work my way all through. I honestly wait for your help.
  • #12
  • Last Post: Edited Dec 21, 2021 12:06am Dec 20, 2021 8:52am | Edited Dec 21, 2021 12:06am
Quoting Dilmeshsingh
{quote} I think I have solved the issue. The difference in values was due to boundary effect. So, I discarded extra values in the problematic series and values I am getting are very close to Moore's values. Thanks very much PipMeUp for helping me.
Dear Dilmesh,
My struggle on the same problem connected me to your answer.

Please help me understand:
In the remaining cycles (which don't divide 72 completely), do we need to enter '0' in the rainfall data column?
Anyway a^2 and b^2 comes to zero...but values obtained a bit away from Moore's values...hence in doubt.
Do we need to enter '0' in the same way in the analysis of the other things like stocks...is it working?

Are you able to find out a better way of handling the boundary effect?


Your reply will help a lot.
0 traders viewing now