Disliked{quote} Just a question, the OpenMP/MPICH allow to spread the internal calculations of the ML algorithms (gradient descent, SVM...) through a cluster, or just parallel optimization of non-dependent input BT? I mean, the cluster will be useful to speed up the every bar re-training in a single setup BT?Ignored
Machine Learning with algoTraderJo
Daniel has allowed me to show a graph from a data-mining bias cloud experiment (using USDJPY hourly data from 1986-2015 in this case). In this experiment a machine learning algo is being evaluated to see if the generated systems (systems with R2>0.9 and placed within a CAGR/MD distribution where MD is Maximum Drawdown) have a low or high data mining bias (DMB for short). An optimization that has around 13,000 points is carried out (not genetic but normal so that all evaluated systems are always the same) on the real data first and then the same optimization process is repeated on random data sets created using boot-strapping with replacement from the real data. The probability for systems to come out from the data-mining process is then calculated from these two frequency distributions. The image you see next contains the frequency distributions for both runs (20+ random data runs have been made up until today for this experiment).
From this image we can see that the data-mining bias for the particular ML algo and optimization used here is low. The probability for systems to come out of the data-mining process is at most 1.8% (0.018). Still too high for my taste but reasonably low to contemplate trading the systems. The cloud mining idea is a very good one since many people can give computational time to find out the DMB of ML algos, it definitely eliminates the need for an in-site cluster (as I use). However I prefer the flexibility of having my own DMB analysis cluster, where I can just do anything as I please
From this image we can see that the data-mining bias for the particular ML algo and optimization used here is low. The probability for systems to come out of the data-mining process is at most 1.8% (0.018). Still too high for my taste but reasonably low to contemplate trading the systems. The cloud mining idea is a very good one since many people can give computational time to find out the DMB of ML algos, it definitely eliminates the need for an in-site cluster (as I use). However I prefer the flexibility of having my own DMB analysis cluster, where I can just do anything as I please
The USDJPY can also yield higher linearity systems using this multiple position machine learning. Here is an example:
I think we have seen enough examples of how high linearity and good sharpe ratios are possible using many different machine learning algos, using single or multiple positions, using only-SL or SL-with-TP configurations. We have also seen several examples of how portfolios improve results. We are however yet to go seriously into ensembles, probably what I'll go into next.
Inserted Code
SL=90% TP=390% A=16 B=275 C=19 D=120
I think we have seen enough examples of how high linearity and good sharpe ratios are possible using many different machine learning algos, using single or multiple positions, using only-SL or SL-with-TP configurations. We have also seen several examples of how portfolios improve results. We are however yet to go seriously into ensembles, probably what I'll go into next.
- #610
- Aug 14, 2015 1:18pm Aug 14, 2015 1:18pm
- Joined Oct 2005 | Status: Pip Slappa Extrordinaire | 1,013 Posts | Online Now
{quote} A member has actually modified the framework to compile on ARM (made it run on a Raspberry Pi!). I have never tried this myself but I know people have built F4 on different architectures.
It was really just a science experiment. Unfortunately it would take like 100 RPis to equal a single 8 processor i7. Might as well just buy the i7.
You are in a maze of twisty little passages, all alike.
{quote} That would be me.
Did you ever try the new raspberry pi 2? They are supposed to be 6x as fast as the original raspberries for the same cost! Not that this would make it cost effective, but at least it would lower the difference.
Look at this result on the USDJPY:
Often ensemble results will appear to be combinations of very similar strategies that create a result that is not much better than the original one. Here ensembles are not working, they are just providing us the same result as the original machine learning algo, distorted across two different algos. There are many parameters that will yield this type of result. Only one of the things to take into account when looking at ensembles.
Inserted Code
SL=80% TP=340% A=13 B1=75 C1=11 D1=190 B1=50 C1=20 D1=200
Often ensemble results will appear to be combinations of very similar strategies that create a result that is not much better than the original one. Here ensembles are not working, they are just providing us the same result as the original machine learning algo, distorted across two different algos. There are many parameters that will yield this type of result. Only one of the things to take into account when looking at ensembles.
I'm very interested in learning more about Ensembles so algoTraderJo, appreciate the topic coming back to this area! On this subject, I saw a working paper that addresses Backtesting strategies based on multiple signals (also attached). The author, Robert Novy-Marx refers to this as an overfitting problem in the paper and asserts that the problem is dramatically increased when large numbers of strategies are combined. I suppose this further underlines the need to perform data mining bias...
Sometimes we get true enhancing ensembles with very nice performance:
Performance for ensembles can be jaw-dropping! This is the type of performance that we would want if we expect ensembles to survive the evaluation of the mining bias. But how... How can we evaluate mining bias? Do we modify the data-mining bias evaluation process or the ensemble creation process? Ideas, Ideas, come on!
Inserted Code
SL=70% TP=40% A=23 B1=300 C1=6 D1=60 B2=125 C2=18 D2=80
Performance for ensembles can be jaw-dropping! This is the type of performance that we would want if we expect ensembles to survive the evaluation of the mining bias. But how... How can we evaluate mining bias? Do we modify the data-mining bias evaluation process or the ensemble creation process? Ideas, Ideas, come on!