Statistical mean of the market [quant corner] | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Explore Real (Statistical) Market Edges 21 replies

The Birth of a Quant Trader 107 replies

Quant trading & QuantLib 6 replies

Quant Corner 12 replies

questions from a future quant 39 replies

Attachments: Statistical mean of the market [quant corner]
Exit Attachments

Statistical mean of the market [quant corner]

  • #41
  • Sep 19, 2014 6:45am Sep 19, 2014 6:45am
Quoting Aws1962
{quote} No, I really am not confusing mean and median. Every 6 year old in the country knows the difference and what I showed was the mean. Also, the median is high+low/2, so I really don't know what the hell you are talking about. I can show you a chart of the mean for any currency over any time frame if you want, but there is no value to be gained from doing so.
Ok let me explain it again, if you look at the overall market , that is since 1950 or so since FX was born, the mid line is the median because thats the mid of the probability distribution aswell, as the market has confined between the HIGHEST HIGH and LOWEST LOW since inception.

However since the market is fractal nature, and we zoom in, the distribution of X bars is not equal to the distribution of the overall market,this is called Heteroscedasticity

I dont care if the mean of the overall market is a straight line in the mid, because guess what, chances are that the price wont go there for the next 10-15 years, so this information is useless to me.

Instead i want to estimate the weekly/intraweek median, and that is certainly not equal to the "mid line of the past week" (that would be called a weekly pivot point, but we all know that they dont work that well in live trading ), because there is a high chance that the intraweek probability distribution changes every day, or even more frequent, but i`m looking at H4 bars for now to filter out the majority of the noise.

So because of that i need a filter which can cut off the noise as good as possible and adapt to the curvature of the market, and that shall be a good mean for me.
"There's a sucker born every minute" - P.T. Barnum
  • #42
  • Edited 7:00am Sep 19, 2014 6:49am | Edited 7:00am
Quoting iamgotzaa
What is the win rate for above back test?
Winrate : 20.97%
Maximal drawdown: 17.89%
Average
consecutive wins: 1
consecutive losses: 5

Pretty rough, and lower WR but the expectancy is much much better.No SL&TP are used, because i dont have the SDs calculated yet, i open a trade at a signal and close it at the reverse.
"There's a sucker born every minute" - P.T. Barnum
  • #43
  • Sep 19, 2014 7:36am Sep 19, 2014 7:36am
Quoting Proximus
{quote} Ok let me explain it again, if you look at the overall market , that is since 1950 or so since FX was born, the mid line is the median because thats the mid of the probability distribution aswell, as the market has confined between the HIGHEST HIGH and LOWEST LOW since inception. However since the market is fractal nature, and we zoom in, the distribution of X bars is not equal to the distribution of the overall market,this is called Heteroscedasticity I dont care if the mean of...
lol, you've no idea have you?
  • #44
  • Sep 19, 2014 7:41am Sep 19, 2014 7:41am
Quoting Aws1962
{quote} lol, you've no idea have you?
I dont know what is not clear about my explanation, i explained it 2 times already.
"There's a sucker born every minute" - P.T. Barnum
  • #45
  • Sep 21, 2014 3:13am Sep 21, 2014 3:13am
Quoting Proximus
This one has an expected profit of 1.65, which is much better than 0.21 {image} {image} Although not a holy grail, yet, but we are getting there eventually {image}
Can you give us more details regarding this please?
What MA you used in these charts?
Where you enter and where you exit from a trade?
Regards,
Flaviu
  • #46
  • Sep 21, 2014 2:18pm Sep 21, 2014 2:18pm
Quoting Flaviu1981
{quote} Can you give us more details regarding this please? What MA you used in these charts? Where you enter and where you exit from a trade? Regards, Flaviu
It is not a conventional MA , it is a specially modified one, because the retail moving averages are very static and non-adapting to the market.
In a dynamic market static indicators cant really forecast anything.

The enter and exit was always at the opposite signals,so that is buy when the line is green and sell when it is red, and close the already open trade at the opposite signal.

No SL and TP was set, so there was always a minuscule risk of margin call, if there is a big spike,but until i can calculate the SD correctly i will backtest without SL and TP, because this model is anyway not ready for real not even demo trading.
"There's a sucker born every minute" - P.T. Barnum
  • #47
  • Sep 23, 2014 1:56pm Sep 23, 2014 1:56pm
Ok i`ve been experimenting in the last couple of days with other models for a mean, and just to set a standard i will use the 2ND generation MA (@ post 38) as standard benchmark.

So i will look for 3 criterias for my filter: > or = expectancy, > or = trade samples, < or = (%) drawdown , if any of the 3 criterias is not met, than that model will be discarded, and the benchmark as i said will be the generation 2 model.

So those criterias will actually measure the variance/DD, the trade sample will indicate better fitting to the price, and the expectancy will indicate more accuracy.

I will post screenshots about my new model soon, it will be the 3rd generation MA.

The current benchmark of the 2nd generation is:

Exp: 1.65
Sample: 391
% DD: 17.89%
"There's a sucker born every minute" - P.T. Barnum
  • #48
  • Sep 23, 2014 2:53pm Sep 23, 2014 2:53pm
I also thought about lowering the timeframe to H1, but unfortunately on H1 a simple optimization with basic magnification would take 6 hours.

This model requires already much CPU with only 2 parameters & 2 booleans, so i`ll have to stick to H4 unfortunately.

Also this will definitely make my strategy, when it will be ready, intraweek.Intra-day is just not an option with limited computing power
"There's a sucker born every minute" - P.T. Barnum
  • #49
  • Sep 23, 2014 3:37pm Sep 23, 2014 3:37pm
I'm really curious about the kind of optimizations you're doing that take so much time. As I showed you in my PM I could reproduce your MA with a simple 2nd order IIR filter. Are you sure your optimization problem is so much nonlinear that it need an GA to be solved?
No greed. No fear. Just maths.
  • #50
  • Sep 23, 2014 3:50pm Sep 23, 2014 3:50pm
I think it's more the unbelivable slow speed of MT4 back-testing.

I do my backtests in Python, and for example a simple strategy on one instrument takes 5 min to backtest on two years of ticks. Real market ticks (more than 50K ticks per day), not interpolated ticks from M1 candles like MT4 does. Python is one of the slowest languages to do this kind of things, and I was complaining that the test took 5 min. In C++ a similar test would take <10 sec.

Then one day I needed to test a simple EA in MT4. It took 1 hour to test over the last two years of M1 candles. Unbelivable. I can't image what the hell it is doing there for 1 hour. I thought that Python was bad, but wow, MT4 takes things to new extremes. No wonder that traders don't like to do backtests.
  • #51
  • Sep 23, 2014 4:01pm Sep 23, 2014 4:01pm
Quoting Adal
I think it's more the unbelivable slow speed of MT4 back-testing. I do my backtests in Python, and for example a simple strategy on one instrument takes 5 min to backtest on two years of ticks. Real market ticks (more than 50K ticks per day), not interpolated ticks from M1 candles like MT4 does. Python is one of the slowest languages to do this kind of things, and I was complaining that the test took 5 min. In C++ a similar test would take <10 sec. Then one day I needed to test a simple EA in MT4. It took 1 hour to test over the last two years of...
Yes i know, looks like i have to complete my platform.I already made a proto-platform before to test some patterns on M1 data ,and while in MT4 it took me 5-6 hours, my platform computed it in 3 minutes.

But the problem i think is not the platform itself, but the method how it reads the data.

You see there are 2 methods to read from the file, either you load it all into the RAM memory or you load only the part you need, which is called streaming .
I think because the MT4 is an older program, it was designed to not be compatible with newer file streaming functions.

I worked primarly in C# before using MQL4 and there were some very elegant ways to read files with maximum efficiency, than in MQL4.

I think the fastest was like 50-60 mb /milisecond streamed, no matter how big the file was.

And because the MQL4 language reiterates every instance the file reading function, the reading of a large file > 200 mb would take years to compute complex algorithms off it.Because at every single bar the code rereads the file completely...Its a monstruous ineficiency in the algorithm coding and resource wasting.

Quoting PipMeUp
I'm really curious about the kind of optimizations you're doing that take so much time. As I showed you in my PM I could reproduce your MA with a simple 2nd order IIR filter. Are you sure your optimization problem is so much nonlinear that it need an GA to be solved?
My new method puts more emphasis on autoregression, thus the computing requirements go up severely, but yes just as my answer to Adal, if i were to continue this project efficiently i need to switch to C# again
"There's a sucker born every minute" - P.T. Barnum
  • #52
  • Sep 23, 2014 4:42pm Sep 23, 2014 4:42pm
Quoting Proximus
{quote}You see there are 2 methods to read from the file, either you load it all into the RAM memory or you load only the part you need, which is called streaming .
That's not quite right. Loading all the info versus streaming doesn't affect speed/time, it only affects memory requirements. If you load 1 GB file in memory, your process will consume 1 GB of RAM. But if you stream a 1 GB file through memory, you can get by with much less, using as little as 1 MB for that.

And in reality, all kinds of file readings boil down to a single thing - memory mapped pages. All modern OSs work like this. But let's not get too technical.

Quoting Proximus
{quote}And because the MQL4 language reiterates every instance the file reading function, the reading of a large file > 200 mb would take years to compute complex algorithms off it.Because at every single bar the code rereads the file completely...Its a monstruous ineficiency in the algorithm coding and resource wasting.
I really really doubt that. But this is easily checkable - open Resource Manager (there is a button in the Windows Task Manager), switch to the Disk tab, and watch it during a backtest. If MT4 would reread all the file each time, you would see massive numbers there.

My guess is that MT4 updates the stats (winning trades, losing percentage, ... ) after each tick, otherwise I can't explain it.
  • #53
  • Sep 23, 2014 4:56pm Sep 23, 2014 4:56pm
Quoting Adal
{quote} That's not quite right. Loading all the info versus streaming doesn't affect speed/time, it only affects memory requirements. If you load 1 GB file in memory, your process will consume 1 GB of RAM. But if you stream a 1 GB file through memory, you can get by with much less, using as little as 1 MB for that. And in reality, all kinds of file readings boil down to a single thing - memory mapped pages. All modern OSs work like this. But let's not get too technical. {quote} I really really doubt that. But this is easily checkable - open Resource...
Yes it does, because if the array Close[] for example contains all data points, requires less CPU than if those unused are left blank.Especially if the array is populated dynamically as how i observed MQL4 does, it is not fixed like Close[99999999] for example.

For example if a basic moving average needs only the last 30 bars to compute, why you need load the entire history file on, especially if the last 30 bars are linearly indexed in the file, you dont need to open the entire file to search for those data points,and you also dont need to load all data into the array to find it.Hence file streaming is waaaaaaaaaay more efficient for time series reading

Attached Image (click to enlarge)
Click to Enlarge Name: kkk.png Size: 57 KB


Here are the specs on my laptop while backtesting a small sample.It seems like the MT4 has some limits as how much resouce can it use.It never uses more than 40% of my CPU.

But above that, that it doesnt use all my resources, its still uses a very inefficient algo to read the data.
"There's a sucker born every minute" - P.T. Barnum
  • #54
  • Sep 23, 2014 5:14pm Sep 23, 2014 5:14pm
Quoting Proximus
{quote} Yes it does, because if the array Close[] for example contains all data points, requires less CPU than if those unused are left blank.Especially if the array is populated dynamically as how i observed MQL4 does, it is not fixed like Close[99999999] for example. For example if a basic moving average needs only the last 30 bars to compute, why you need load the entire history file on, especially if the last 30 bars are linearly indexed in the file, you dont need to open the entire file to search for those data points,and you also dont need...
As someone who was paid big salaries in the last 8 years to write C++/Java or Python code for well known companies I can tell you that what you wrote above is non-sense. But this is not a place to go into a technical discussion about how computers or high-speed code works. If you don't believe me, fine, whatever.

Quoting Proximus
It seems like the MT4 has some limits as how much resouce can it use.It never uses more than 40% of my CPU. But above that, that it doesnt use all my resources, its still uses a very inefficient algo to read the data.
MT4 backtests are single threaded. MT4 can not use more than a single core in you machine at one time. Windows will give it a different one each time, this is why all four are showing activity on your computer.
  • #55
  • Edited 8:30pm Sep 23, 2014 5:36pm | Edited 8:30pm
Quoting Adal
{quote} As someone who was paid big salaries in the last 8 years to write C++/Java or Python code for well known companies I can tell you that what you wrote above is non-sense. But this is not a place to go into a technical discussion about how computers or high-speed code works. If you don't believe me, fine, whatever.
{quote} MT4 backtests are single threaded. MT4 can not use more than a single core in you machine at one time. Windows will give it a different one each time, this is why all four are showing activity on your computer.
Thats funny because i also have 7+ year experience in programming.Do you know how streamed data gets assigned to dynamic arrays? That is the problem with MT4 as the algorithm of any indicator populates the entire data into the array, instead of streaming only the data that it immediately needs.

Now this obviously, believe it or not, creates excessive iterations to solve, which requires more CPU.The RAM is not overused because the whole data package on H4 bars is only 0.4 megabytes.

But because the AR method i use has exponentially increasing iterations and i add more parameters to it, first it needs to iterate trough the entire historical data, then trough the array itself, then recursively has to calculate the next,next,next bars by reiterating itself trough the previous cycles, instead of saving some time and using only the historical data that its needed.

A full GARCH model has so many iterations in it, that with this inneficient method, i bet the supercomputers would also face some difficulty in calculating it tick-by-tick

I dont care how many cores the MT4 backtester uses, i can code a 1 core application that is 99999999x more efficient than the MT4 backtestes, just by using efficient file streaming tehniques.
"There's a sucker born every minute" - P.T. Barnum
  • #56
  • Sep 24, 2014 3:14am Sep 24, 2014 3:14am
FYI I'm 15+ yr senior software developer. I learned coding at 12 with Basic and the Z80 then 8080 assembly languages (yes I'm old piss off!). I know a bit on optimizing code. I can tell that before blaming the buffering of the data from the disk you should certainly look at the most basic stuff first. For what I know of MT4 it stores the candles and ticks data in text format (.hst files if I'm not mistaken). Not very efficient! All the indis I could read the code were written by monkeys turning a linear algo into a O(n^2) or worse. Each time recomputing the whole same formula over the same data!

Seriously a quad core to fit an ARMA model? That's basic linear algebra. Come on!
No greed. No fear. Just maths.
  • #57
  • Sep 24, 2014 9:35am Sep 24, 2014 9:35am
Quoting PipMeUp
FYI I'm 15+ yr senior software developer. I learned coding at 12 with Basic and the Z80 then 8080 assembly languages (yes I'm old piss off!). I know a bit on optimizing code. I can tell that before blaming the buffering of the data from the disk you should certainly look at the most basic stuff first. For what I know of MT4 it stores the candles and ticks data in text format (.hst files if I'm not mistaken). Not very efficient! All the indis I could read the code were written by monkeys turning a linear algo into a O(n^2) or worse. Each time recomputing...
Also because i used an EA to check the winrate,dd,etc while the model was in an indicator file, i observed that because the EA uses [0], [1] for the current bar and the prev bar, instead of using the real index of the indicator,calculated from 'Bars' to 0 (even if when the EA is run the bars are painted on the run), because this "index swap" it has to do 1 more wasteful iteration just to align the EA index with the indicator index.

I dont think the text files are so bad, there are probably better file formats to store optimally the data, but with the newest streaming technology even the TXT files are good.
Once i ran a check on M1 candlesticks to check if there is any patterns there like morning star,harami,engulfing, etc (how naive was i back then ) and i ran trough +6 million candles and it was done in 10-15 seconds, while also the data was in .txt .The test results were obviously dissapointing, i think the morning star had the highest edge of 50% + 0.5% which tells alot about these so called 'patterns'
"There's a sucker born every minute" - P.T. Barnum
  • #58
  • Edited 1:55pm Sep 24, 2014 1:31pm | Edited 1:55pm
Quoting Adal
I think it's more the unbelivable slow speed of MT4 back-testing. I do my backtests in Python, and for example a simple strategy on one instrument takes 5 min to backtest on two years of ticks. Real market ticks (more than 50K ticks per day), not interpolated ticks from M1 candles like MT4 does. Python is one of the slowest languages to do this kind of things, and I was complaining that the test took 5 min. In C++ a similar test would take <10 sec. Then one day I needed to test a simple EA in MT4. It took 1 hour to test over the last two years of...

Google "Birt's Patch" or "Eareview" and use the free version if you don't want to pay for the enhanced version. I back-test in MT4 all the time and I use converted data from my institutional platform using Birt's method. You can use tick data and you can set the spread as fixed or variable (real spreads). Once you get your backtesting station set-up correctly, its a breeze to use. Have lots of HDD space available as the raw data .csv loads can be heavy depending on your source.

Tick data conversion times are predicated on the size of the original data file and the capabilities of your CPU. You will run a CSV2FXT script (VBA .dll) locally which will convert your CSV files to MT4 FXT readable files for the backtester in whatever time frames you select. The Walk Forward Analyzer will not work with the free version, but it does work with the Tickdata Suite paid version.

If you are trading long range signals over longer time frames, using interpolated M1 data in MT4 should not adversely impact test results. However, using MT4's standard interpolation mode on M1 bars for Intra-Day Trading can adversely impact test results. Tick data is always best and variable or real spreads are optimal when intra-day signals are being generated by your EA.

After properly setting up the backtest environment my tests run very fast all the time. The problem for most people is that they never learned how to property set-up and use the MT4 backtester, so it lags when it could run a lot faster.

I do not endorse, sell or receive any compensation for anything on or off the internet, whatsoever.
F-X-C-M | A Classic Retail Bucket Shop Scam Artist Who Manipulates Prices
  • #59
  • Sep 24, 2014 1:50pm Sep 24, 2014 1:50pm
Dont even bother, MT4 strategy tester is really slow like a snail, i almost completed my own strategy tester in C#.

I wont code a chart in it, although i already made a chart viewer program earlier, i also showed prinscreens in this forum, because i will keep if very basic just for optimization and backtesting.

I already coded half of it, the data reading and loading section. And what can i say its lightspeed fast with filestreaming
I loaded a 7 gb tick data historic, and it loaded it into the memory in less than 15 seconds (because i only used 1/5th of it, as Bid/Ask so it only had 1.4 gb of my RAM)

I`m working now to code a function which can interpret indicators just as it is in MQL4 , so that i only need to COPY+PASTE from one code to the other and i can test it both in my strategy tester and see the results in MT4 charts.

This way i shall optimize it in my platform, and then view the optimized result in MT4. It will definitely save me some computing power to run trough all the strategies
"There's a sucker born every minute" - P.T. Barnum
  • #60
  • Sep 26, 2014 9:41pm Sep 26, 2014 9:41pm
My platform is almost finished so i will soon resume my research about this topic.

In the meantime i redownloaded my historical data, without swaps, as the normal hist files contain the swaps, but they always apply the latest swap to the entire dataset, so this gives a false result.Instead its better to not count the swaps at all, and then later on i will be careful when trading to not trade against the "heavy swapped pairs" like short NZD/JPY for example.

However because i mainly worked offline in MT4, because the online mode certainly decreased the computing speed even more, and i only used the strategy tester.
It seems like there was no difference.

I got the same identical results, line by line, data by data, from the offline swap containing hist data as from the offline non-swap containing hist data.
So i came to the conclusion, that just by using MT4 in offline mode the swaps are not calculated at all.Interesting.

Anyway i got what i wanted, so my research is not biased by the swap at all.

As for the spread, i dont know that if i mentioned it but i always used 20 pipettes = 2 pips swap for the EUR/USD test, because this is an above average swap rate that is very uncommon, so my filter is really tuned in into the worst case scenario events.

The spread is really > 20 pipettes about 30 minutes / day @ hot news, so that is less than 3% of time, but still as i said i`m preparing myself for worst case scenario.
"There's a sucker born every minute" - P.T. Barnum
0 traders viewing now