Disliked{quote} Are you using sqlite wrapper / mysql wrapper to connect to your database? there is new feature at mt4 build 670 called Webrequest, are you using it to connect to database via webserver (ex. Using Wamp)?Ignored
Fractals, ZZ & the Pissing Dog
I have been away working on other things but have turned my mind again to this thread. It seems that people are having issues with the next bar prediction bar of the concept, so I thought I would start fresh also and work my way through the concept. Maybe some things I find will help others or generate new ideas.
To start with, I have gone away from using time bars (eg 1min, 5min etc) and moved to using range bars. I create the range bars myself from raw tick data. From the range bars I create a big list of all the range bars, with up bars having a 1 and down bars having a 0. From this list I generate another list that contains the fractals that occur in the first list. These also are represented as 1 for high fractals and 0 for low fractals. Using the fractal list, I split the data into 2 sections, an in sample and an out of sample period.
Using the in sample period, I generate all the dna sequences for all linear and all sub samples for a range. Eg my range could be a dna string length of 4 to 6 (eg 0110 is length 4 and 000111 is length 6) and store a count for the occurrence of each dna string. So, for example I have a linear substring 0111 that occurs x times. I also have a non linear substring (that uses every second fractal) 0111 that occurs y times. I also have a non linear substring (that uses every third fractal) 0111 that occurs z times. You get the idea, I build a full collection of all the strings for a range of lengths and sub sample intervals. I call these genes.
The next problem is that it is difficult to know if 0111 from a linear string is more important then 0111 from a substring of step size 2 (for example). So, after I have created collection of all the strings and counts, I then walk forward again though the in sample data and at each fractal I use each gene to try to predict the next fractal. If the gene is correct I increment its weight (all genes start with a weight of 1000). If the gene is incorrect I decrease its weight. So at the end I have the genes that have the highest correct percentage with the highest weight.
Now comes the out of sample prediction. I simply walk forward through the out of sample data. At each new fractal I build the dna string using the previous fractals for each of the genes that we want to test. Then I simply count how many occurrences of the next fractal being a 0 or a 1. Each gene has a weight, so I add to a summed count a 1 (if the prediction is positive) or a -1 (if the prediction is negative) * the gene weight to a total summed value that each gene contributes to. At the end of processing all genes I have a summed total that is either < 0 or > 0 that gives me the weighted prediction for the next bar.
I'm sure this is different to what everybody else does but I can see no other way of determining the importance of each individual dna string. I am open to any other ideas or suggestions.
Onto the results. I have run a sweep of tests using dna bar lengths from 6 to 20 (all start from 4) and sub samples from 1 (no sub samples) to 10. Using 5 pip range bar data the best result I have had so far is 68% prediction accuracy with out of sample data. It is interesting that the best results come from using almost no subsampling. Increasing the range of historic data available did not increase the maximum accuracy but showed that the same accuracy could be achieved over a wider range or dna lengths and subsample sizes.
I am still playing with this so will report back any developments. I am curious, what sort of next bar (fractal) prediction rates are other getting?
To start with, I have gone away from using time bars (eg 1min, 5min etc) and moved to using range bars. I create the range bars myself from raw tick data. From the range bars I create a big list of all the range bars, with up bars having a 1 and down bars having a 0. From this list I generate another list that contains the fractals that occur in the first list. These also are represented as 1 for high fractals and 0 for low fractals. Using the fractal list, I split the data into 2 sections, an in sample and an out of sample period.
Using the in sample period, I generate all the dna sequences for all linear and all sub samples for a range. Eg my range could be a dna string length of 4 to 6 (eg 0110 is length 4 and 000111 is length 6) and store a count for the occurrence of each dna string. So, for example I have a linear substring 0111 that occurs x times. I also have a non linear substring (that uses every second fractal) 0111 that occurs y times. I also have a non linear substring (that uses every third fractal) 0111 that occurs z times. You get the idea, I build a full collection of all the strings for a range of lengths and sub sample intervals. I call these genes.
The next problem is that it is difficult to know if 0111 from a linear string is more important then 0111 from a substring of step size 2 (for example). So, after I have created collection of all the strings and counts, I then walk forward again though the in sample data and at each fractal I use each gene to try to predict the next fractal. If the gene is correct I increment its weight (all genes start with a weight of 1000). If the gene is incorrect I decrease its weight. So at the end I have the genes that have the highest correct percentage with the highest weight.
Now comes the out of sample prediction. I simply walk forward through the out of sample data. At each new fractal I build the dna string using the previous fractals for each of the genes that we want to test. Then I simply count how many occurrences of the next fractal being a 0 or a 1. Each gene has a weight, so I add to a summed count a 1 (if the prediction is positive) or a -1 (if the prediction is negative) * the gene weight to a total summed value that each gene contributes to. At the end of processing all genes I have a summed total that is either < 0 or > 0 that gives me the weighted prediction for the next bar.
I'm sure this is different to what everybody else does but I can see no other way of determining the importance of each individual dna string. I am open to any other ideas or suggestions.
Onto the results. I have run a sweep of tests using dna bar lengths from 6 to 20 (all start from 4) and sub samples from 1 (no sub samples) to 10. Using 5 pip range bar data the best result I have had so far is 68% prediction accuracy with out of sample data. It is interesting that the best results come from using almost no subsampling. Increasing the range of historic data available did not increase the maximum accuracy but showed that the same accuracy could be achieved over a wider range or dna lengths and subsample sizes.
I am still playing with this so will report back any developments. I am curious, what sort of next bar (fractal) prediction rates are other getting?
- #805
- Edited 7:35am Aug 8, 2014 7:18am | Edited 7:35am
{quote} elen,For the historical building of the database, you should ......Carl
Thank You Again Carl, sorry for my stupidity ....
need so long time to understand Your Clue ...
- #808
- Aug 8, 2014 8:26am Aug 8, 2014 8:26am
....
In your introduction you talked about the next bar prediction but you continue with talking about the direction of fractals.
Just a few questions: How do you determine if a longer fractal sequence is more likely than the once we currently have as this is really the information we are after.
I haven't looked at the actual probabilities but I assume that the probability of getting a 1 fractal after a 0 fractal is much higher than getting another 0 fractal and getting a 0 fractal after a 1 fractal is more likely than another 0 fractal. Can you beat that probability with your method?
I just picked out a random time period.
If you get a 70% next bar direction prediction that is pretty impressive. Also skipping fractals like taking every 2nd or 3rd fractal doesn't sound very logical to me to be honest. We did it at the very beginning but disregarded it quickly.
Has any one of us tried the Zigzag Height ??
I am just thinking like this :
This Zigzag Height is for replacing the Bars amount (vertically versus horizontally).
So, instead of using Bars Amount in Zigzag Leg & Fractals structure inside Zigzag Leg, may be we can try Zigzag Height & Fractals structure.
Converting Zigzag height by splitting it to some Height-Based value. The Height-Based value may be the smallest Zigzag height, or the Average of the Zigzag height in 2011-2012. I think i can not use a very big ATR value for this, or just calculate the simple average. Another approach is to split this average into smaller value (ex: may be it will be 1/2, 1/3, 1/4, 2/5 or other value ... of this ATR/simple average value).
Lets say the "calculated" Zigzag height Average is X pips. So, if i have a Zigzag with Y pips Height, it will be convert to : MathCeil (Y/X).
I do not know, this make sense or not ...
@Killian & @Beachloup,
Can we use "svm or ann or R or Matlab or any other tools" to get which average value is the most "Reasonable" ??
I think if there is a tool to calculate them, it will be faster than trying one by one using Mql ....
i never used them, do not know which one is easier for beginner. Any information is welcome ...
Thanks in advance
I am just thinking like this :
This Zigzag Height is for replacing the Bars amount (vertically versus horizontally).
So, instead of using Bars Amount in Zigzag Leg & Fractals structure inside Zigzag Leg, may be we can try Zigzag Height & Fractals structure.
Converting Zigzag height by splitting it to some Height-Based value. The Height-Based value may be the smallest Zigzag height, or the Average of the Zigzag height in 2011-2012. I think i can not use a very big ATR value for this, or just calculate the simple average. Another approach is to split this average into smaller value (ex: may be it will be 1/2, 1/3, 1/4, 2/5 or other value ... of this ATR/simple average value).
Lets say the "calculated" Zigzag height Average is X pips. So, if i have a Zigzag with Y pips Height, it will be convert to : MathCeil (Y/X).
I do not know, this make sense or not ...
@Killian & @Beachloup,
Can we use "svm or ann or R or Matlab or any other tools" to get which average value is the most "Reasonable" ??
I think if there is a tool to calculate them, it will be faster than trying one by one using Mql ....
i never used them, do not know which one is easier for beginner. Any information is welcome ...
Thanks in advance
Good friends are more valuable than gold ...
- #814
- Edited 10:50am Aug 8, 2014 10:36am | Edited 10:50am
I have just built a little chart app to visualise what my algo was doing (I don't use MT4). In the screen shot below, I have learnt the dna string for the range bar fractals from 2012 to 2014. The screen shot shows 1 week in June 2014. Green bars are up bars and red bars are down bars. The green and red arrows show the actual fractals (as used in the algo), they are draw in after the next bar has completed (because you always need an additional bar to determine if you have a fractal). The Purple bars show the predicted direction of the next fractal.
You can see that the purple bars occur 1 bar after the actual fractal. This is because you need the additional bar to confirm the real fractal. Once I have confirmed the real fractal I make a prediction for the next direction. The purple bars with the red circles show the predictions that were wrong (I added them in photoshop so I hope I got them all). From this I count 8 wrong predictions from the 38, so that's about 78% correct.
I'm showing this just to give people a heads up on my thinking. Maybe it is not helpful, but from my understanding of the complete concept getting the next direction prediction accurate is a major part of the whole process...
Edit: Looking at this again, the predictions that failed are where the next actual fractal is the same direction as the previous fractal. Crunching the numbers, in a dataset of the 2013 fractals, I have 9421 fractals in total of which 2993 are the same direction as the previous fractal. Ie 68% of the time the next fractal direction is just the inverse of the last fractal direction. So any algo needs to predict more then 68% of the directions correctly to be considered useful. At the moment my algo is as good as assuming that the next fractal is simply going to be the inverse of the last fractal....
You can see that the purple bars occur 1 bar after the actual fractal. This is because you need the additional bar to confirm the real fractal. Once I have confirmed the real fractal I make a prediction for the next direction. The purple bars with the red circles show the predictions that were wrong (I added them in photoshop so I hope I got them all). From this I count 8 wrong predictions from the 38, so that's about 78% correct.
I'm showing this just to give people a heads up on my thinking. Maybe it is not helpful, but from my understanding of the complete concept getting the next direction prediction accurate is a major part of the whole process...
Edit: Looking at this again, the predictions that failed are where the next actual fractal is the same direction as the previous fractal. Crunching the numbers, in a dataset of the 2013 fractals, I have 9421 fractals in total of which 2993 are the same direction as the previous fractal. Ie 68% of the time the next fractal direction is just the inverse of the last fractal direction. So any algo needs to predict more then 68% of the directions correctly to be considered useful. At the moment my algo is as good as assuming that the next fractal is simply going to be the inverse of the last fractal....
If I understand you correctly you want to replace the amount of bars with "a height based value"? ...... why don't we just take the "base" value pips.
Yes you understand it correctly ... and you define the "base" value very clear. How can we get the best/reasonable "base" value using other tools, like GG got the best setting 13/8/5 using matlab ??
But aren't we kind of doing that at the moment? We take the average of x bars and if a bar is 2 times as big as the average we (at least thats what I am doing) count it as 2 bars. This is a transformation from the horizontal bar count (time) to a vertical approach the pip count.
*) Carl has just given some other clue ... i think we are on the right track ...
i must think and think again .... no matter it takes long time, i will try it if i have free time ....
Thanks a lot Killian.
Good friends are more valuable than gold ...