The Great Pumpkin | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

What a great site! (Dailyfx.com rave) 4 replies

A Great Indicator needs a touch of a great coder 7 replies

Great Posts By Great Traders 79 replies

Average system + great discipline =great results? 44 replies

Attachments: The Great Pumpkin
Exit Attachments

The Great Pumpkin

  • #41
  • Nov 11, 2014 7:28am Nov 11, 2014 7:28am
Quoting kprsa
The Bands take the same number of samples (candles/bars) for estimating the mean (moving average) and the standard deviation (bands).
Yes, that's what I'm saying too. But your code allows this "same number" to be different.

Quoting kprsa
SMA calculates (a+b+c+d+e+f+g)/7, which is fine, and attributes it to price g, which is not correct.
I honestly don't see how attributing the average to any price matters during calculation of standard deviation. The formula doesn't involve time or index of the values (in our case, prices).

Attached Image


Attached Image


I fail to see any attribution.

As for the confidence interval, to me that says only that it is advised to use larger samples, i.e. 200 instead of 20 (using your examples). But it doesn't say that we should use 20 values for calculating the average and use 200 values to calculate the SD, starting from the "middle" value of the smaller sample. (I can only refer back to my example of using prices of 12-3 vs. the average of 5-0.)

One of us is really missing something here.
Even if you fall on your face, you're still moving forward.
  • #42
  • Nov 11, 2014 8:26am Nov 11, 2014 8:26am
Quoting Ralome
{quote} I honestly don't see how attributing the average to any price matters during calculation of standard deviation.
I agree that it should be no difference when the price mean is flat with respect to time. Then also the lag of the moving average will be equivalent to zero, so in this case the lag correction is unimportant. But things do change in a trend.

The blue line in the plot below is the standard SMA, the red is shifted back by a half of its period. Would you calculate the standard deviation based on subtracting the data off the first or the second line? Do you think it is the same in this case?

Attached Image


Quoting Ralome
{quote} As for the confidence interval, to me that says only that it is advised to use larger samples, i.e. 200 instead of 20 (using your examples). But it doesn't say that we should use 20 values for calculating the average and use 200 values to calculate the SD, starting from the "middle" value of the smaller sample.
In principle one part of the problem could be solved by using Bands with a large Period. However, that would enforce one to use the large period moving average, which one might not like to do. Additionally, there is that uncorrected lag business there.

k
  • #43
  • Nov 11, 2014 9:49am Nov 11, 2014 9:49am
Quoting kprsa
Do you think it is the same in this case?
Yes.

Attached Image (click to enlarge)
Click to Enlarge Name: same.jpg Size: 96 KB
Even if you fall on your face, you're still moving forward.
  • #44
  • Nov 11, 2014 10:11am Nov 11, 2014 10:11am
Quoting Ralome
{quote} Yes.
Well, the error of the standard deviation (600 samples) for the non-shifted case is 128% in my example above.
For the MA shifted by period/2, the error is 2.6%.
The ratio of the two calculated standard deviations for the two cases is 2.29.
I would say that is a significant difference.

Attached File(s)
File Type: pdf test_stdev.pdf   600 KB | 547 downloads


Cheers,
k
  • #45
  • Nov 11, 2014 10:33am Nov 11, 2014 10:33am
Quoting kprsa
{quote} Well, the error of the standard deviation (600 samples) for the non-shifted case is 128% in my example above. For the MA shifted by period/2, the error is 2.6%. The ratio of the two calculated standard deviations for the two cases is 2.29. I would say that is a significant difference. {image} Cheers, k
Inserted Code
ln[139]: i - period, i

Are both ends included? Because that would mean you average period + 1 points in time (x axis).

Inserted Code
ln[147]: data2 = (Take[y, {401, 1000}] - SMA)^2;

SMA = Average of 1-400 = past data
401-1000 = future data

???
Even if you fall on your face, you're still moving forward.
  • #46
  • Nov 11, 2014 10:49am Nov 11, 2014 10:49am
Quoting Ralome
{quote} ln[139]: i - period, i Are both ends included? Because that would mean you average period + 1 points in time (x axis).

The array indexing in Mathematica starts with 1. I calculated the 401st element of the MA as a sum of first 400 data points, etc. For example, 501st is calculated as a mean value of 101-500th data element etc.

Quoting Ralome
{quote} SMA = Average of 1-400 = past data 401-1000 = future data ???
No, these are the data points for which the SMA (400) was calculated so the y values are for x=401-1000. In Metatrader this would correspond to a standard moving average.

The trick here is, once again, to shift the MA by half of its period back in time (to x=201-800 in my example), and then use it as a mean value estimate. It is much better centered then (because the lag of period/2=400/2=200 is compensated) and approximates the data much better. Therefore the sum of squares of differences between the data and the mean estimate is much smaller. Otherwise, if you do not do this, the standard deviation is overestimated.

The only disadvantage of this shifting back in time is that the most recent data points are not taken into account in the calculation of the standard deviation. But since the standard deviation does not change much, that is not so important.

I am sorry, but I really cannot explain it any better. The only thing I can say is: add a linear drift to your data above, calculate the simple moving average and use it as an estimate of the mean value. Then shift it back in time half a period and calculate the values again. See if there is a difference.

Cheers,
k
  • #47
  • Nov 11, 2014 10:59am Nov 11, 2014 10:59am
Quoting kprsa
{quote} Try this version. k {file} {image}
Hey kprsa,
works perfectly!
Thanks a lot
  • #48
  • Edited 1:11pm Nov 11, 2014 12:19pm | Edited 1:11pm
Quoting kprsa
{quote} The array indexing in Mathematica starts with 1. I calculated the 401st element of the MA as a sum of first 400 data points, etc. For example, 501st is calculated as a mean value of 101-500th data element etc. {quote} No, these are the data points for which the SMA (400) was calculated so the y values are for x=401-1000. In Metatrader this would correspond to a standard moving average.
Okay, I think I got it. So, looking at this line again:

Inserted Code
 data2 = (Take[y, {401, 1000}] - SMA)^2;

Does this create an array of 600 elements, with each element being equal to the square of the difference of the Y value and the corresponding SMA value at the same point on the X axis? I.e.

data2[1] = (Y[401] - SMA[401])^2
data2[2] = (Y[402] - SMA[402])^2
...
data2[600] = (Y[1000] - SMA[1000])^2

And then this next line:

Inserted Code
sigmanonshifted = Sqrt[Sum[data2[[i]] / 600, {i, 600}]]

Takes the sum of data2[1], data[2] ... data2[600] and divides it by 600, then takes the square root of this quotient?
Even if you fall on your face, you're still moving forward.
  • #49
  • Nov 11, 2014 5:00pm Nov 11, 2014 5:00pm
Quoting Ralome
{quote} Okay, I think I got it. So, looking at this line again: data2 = (Take[y, {401, 1000}] - SMA)^2; Does this create an array of 600 elements, with each element being equal to the square of the difference of the Y value and the corresponding SMA value at the same point on the X axis? I.e. data2[1] = (Y[401] - SMA[401])^2 data2[2] = (Y[402] - SMA[402])^2 ... data2[600] = (Y[1000] - SMA[1000])^2 And then this next line: sigmanonshifted = Sqrt[Sum[data2[] / 600, {i, 600}]] Takes the sum of data2[1], data[2] ... data2[600] and divides it by 600,...
yes, all correct. the same stuff with (Take[y, {201,800}]-SMA)^2 corresponds to shift by 200=period/2.
today i was not at my hobby computer and did not have mt4. i will generate some histograms for mt4 so that one can directly compare the standard bb, bb with more samples for stdev, bb with shift of ma and more samples for stdev. likely tomorrow morning.

k
  • #50
  • Edited 2:47am Nov 12, 2014 2:36am | Edited 2:47am
Quoting kprsa
{quote} yes, all correct.
No, it's not. What I've described here:

Quoting Ralome
data2[1] = (Y[401] - SMA[401])^2
data2[2] = (Y[402] - SMA[402])^2
...
data2[600] = (Y[1000] - SMA[1000])^2
is NOT how you calculate standard deviation.

Attached Image (click to enlarge)
Click to Enlarge Name: sd.jpg Size: 46 KB

You can see that the mean ( μ ) has no index. So there's only ONE mean value you work with at each point.
Attached Image (click to enlarge)
Click to Enlarge Name: sd_diagram.jpg Size: 89 KB
Even if you fall on your face, you're still moving forward.
  • #51
  • Nov 12, 2014 3:12am Nov 12, 2014 3:12am
Quoting Ralome
{quote} No, it's not. What I've described here: {quote} is NOT how you calculate standard deviation. {image} You can see that the mean ( μ ) has no index. So there's only ONE mean value you work with at each point. {image}
Ralome,

For the tenth time, the moving average is an estimate of the mean. The standard deviation is then calculated as the square rooted (sum of squares of differences divided by number of samples for the mean). Simple moving average lags by half a period and that is why one needs to shift the MA back in time by half a period so that the lag is compensated.

On the Wiki link it is called the central moving average, and I think this is what you are essentially doing in the green labels.
Quoting Wikipedia
Hence a central moving average can be computed, using data equally spaced on either side of the point in the series where the mean is calculated.[3] This requires using an odd number of datum points in the sample window.
The point is that the standard Bands don't do it (they do the procedure you outlined in the red), so I needed to do this shift in the code. I think the period of the moving average you are using is 4, so shift the MA back in time by two and you will see that it coincides with what you call the mean value. And THAT is MY formula, the formula in the Bands is the red one.

k
  • #52
  • Nov 12, 2014 3:38am Nov 12, 2014 3:38am
Hmm as much as the topic of the above discussion eludes me...I have nothing but respect to see such an exchange at such a level!

R K Me
I want more brain power!
  • #53
  • Nov 12, 2014 3:49am Nov 12, 2014 3:49am
Attached Image (click to enlarge)
Click to Enlarge Name: Clipboard01.jpg Size: 41 KB
  • #54
  • Nov 12, 2014 4:11am Nov 12, 2014 4:11am
Quoting kprsa
{quote}they do the procedure you outlined in the red
Correction, the standard Bands does even worse than what I said. For the calculation of the stdev they calculate the differences between the current value of the SMA and all the past close prices within a period, which absolutely makes no sense (edit: it is possible to set the BandsShift to the MA, but that is 0 by default). It would be better if they would set oldval=MovingBuffer[k] within the while(k>=i) loop. My code just adds the shift to this modification.

for(i=0; i<limit; i++)
MovingBuffer[i]=iMA(NULL,0,BandsPeriod,BandsShift,MODE_SMA,PRICE_CLOSE,i);
//----
i=Bars-BandsPeriod;
if(counted_bars>BandsPeriod-1) i=Bars-counted_bars-1;
while(i>=0)
{
sum=0.0;
k=i+BandsPeriod-1;
oldval=MovingBuffer[i];
while(k>=i)
{
newres=Close[k]-oldval;
sum+=newres*newres;
k--;
}
deviation=BandsDeviations*MathSqrt(sum/BandsPeriod);;
UpperBuffer[i]=oldval+deviation;
LowerBuffer[i]=oldval-deviation;
i--;
}

k
  • #55
  • Nov 12, 2014 4:46am Nov 12, 2014 4:46am
Quoting Ralome
{quote} No, it's not. What I've described here: {quote} is NOT how you calculate standard deviation. {image} You can see that the mean ( μ ) has no index. So there's only ONE mean value you work with at each point. {image}
As for the calculation of the stdev, what you are saying is similar to the stuff done in the Bands indicator. You imply that one should calculate the standard deviation based on a simple mean of, say five points. This is in my view not precise, the moving average is a better estimate of the mean value than the simple mean, notably in a trend. In other words, the question I am asking is how much do the data deviate away from the (lag-corrected) moving average.
k
  • #56
  • Nov 12, 2014 4:50am Nov 12, 2014 4:50am
The Great Debate. My head is too small to digest all this.
Traders gain some knowledge, win, their emotions kick in, and they die.
  • #57
  • Nov 12, 2014 5:40am Nov 12, 2014 5:40am
Quoting kprsa
{quote} This is in my view not precise
Just to put the numbers on this, the error in estimating the standard deviation of the underlying process (normal distribution with σ=1) by not taking account the trend
is in my example in the Mathematica sheet equal to 104%.

k
  • #58
  • Nov 12, 2014 5:51am Nov 12, 2014 5:51am
Quoting kprsa
{quote} For the calculation of the stdev they calculate the differences between the current value of the SMA and all the past close prices within a period, which absolutely makes no sense.
After a second look, I am wrong in saying this, this would actually be a correct procedure in its world. The current SMA is of course the mean value over the past period so it does make sense, even though not taking trends into account is painfully imprecise.

Sorry,

k
  • #59
  • Edited 6:39am Nov 12, 2014 6:25am | Edited 6:39am
Okay, I think we've finally got to the end of the "debate"

Attached Image (click to enlarge)
Click to Enlarge Name: sd_diagram3.jpg Size: 143 KB


Taking a period of 5, original/standard deviation/bands/whatever formula can be calculated at point 9. Your formula can first be calculated at point 5 only (shifting not included in this example).

Quoting kprsa
As for the calculation of the stdev, what you are saying is similar to the stuff done in the Bands indicator.
Not similar, exactly the same!

Quoting kprsa
You imply that one should calculate the standard deviation based on a simple mean of, say five points.
It's not me implying it, it is per the definition of standard deviation!

Quoting kprsa
This is in my view not precise, the moving average is a better estimate of the mean value than the simple mean, notably in a trend. In other words, the question I am asking is how much do the data deviate away from the (lag-corrected) moving average. k
So, the point is, you're calculating "kprsa deviation" or "PipMeUp deviation", but NOT "standard deviation".

Quoting kprsa
subtract the moving average at position [i] from the price at candle [i] to obtain the standard deviation
Quoting PipMeUp
the variance is estimated by taking the average of the squared difference of the current value of the SMA and the current price. (...) dotted lines are their respective 2 standard deviations bands.
Having said that, I realize now that this whole debate originated from miscommunication (both from me and from you, in my opinion). But I acknowledge that it still didn't warrant the words I used in my starting post, deswegen bitte ich nochmals um Entschuldigung
Even if you fall on your face, you're still moving forward.
  • #60
  • Nov 12, 2014 6:46am Nov 12, 2014 6:46am
Quoting Ralome
Okay, I think we've finally got to the end of the "debate"
Apart from the fact that in my formula the red curve is shifted by half a period to the left. The red curve is what I was saying would be wrong. The standard deviation that I get is much smaller than the one you get using "the correct formula".

This is sometimes called a "moving" or "rolling" standard deviation, it is just that one acknowledges that the mean value changes across the time series.

k
0 traders viewing now