Machine Learning with algoTraderJo | Page 36 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #701
  • Edited 10:17am Sep 30, 2015 10:05am | Edited 10:17am
Quoting ampleparking
Instead of that DMB analysis, can't we use the standard train/cross-validate/test dataset splitting? Train on the train set, tune hyper-parameters on the validation set, test on the test set.
No, that doesn't work to replace the DMB analysis. That type of test tries to remove another source of bias (not DMB) but curve fitting bias but it does not even work for that purpose. The train/cross-validate/test analysis does not work to do this in trading, this is what most academics (who never trade real money) fail to see. Suppose you build a system that fails the cross validation or the testing parts, you would simply change and repeat and change repeat and change and repeat until you find something that works across all the tests you want. You're simply doing a much more intensive mining exercise, a more complex curve fitting process. The fact that you can use data as much as you want makes these cross validation and testing procedures useless. Doing something as you propose leads to a gargantuan increase in the DMB as well.

But even if you did the above you would either way need to do the DMB analysis to see if your cross-validation/testing results are also likely the result of random chance. This means that you would need to perform a DMB analysis by performing the entire search process train/cross-validation/testing on random price series, etc.

There is no free lunch, you cannot skip DMB analysis.
  • #702
  • Sep 30, 2015 10:32am Sep 30, 2015 10:32am
Quoting algoTraderJo
Suppose you build a system that fails the cross validation or the testing parts, you would simply change and repeat and change repeat and change and repeat until you find something that works across all the tests you want. You're simply doing a much more intensive mining exercise, a more complex curve fitting process.
Well, that would be a wrong approach because you would fit the test set. You have to improve your validation set score, not your test score.
  • #703
  • Sep 30, 2015 10:36am Sep 30, 2015 10:36am
Quoting ampleparking
{quote} Well, that would be a wrong approach because you would fit the test set. You have to improve your validation set score, not your test score.
But would you go live with something with a bad test score and a good validation score? Of course not. You would tweak things until you had both a good validation score and test score. This tweaking increases DMB a gazillion times because of the continuous usage of the data.
  • #704
  • Sep 30, 2015 10:40am Sep 30, 2015 10:40am
Quoting ampleparking
{quote} Well, that would be a wrong approach because you would fit the test set. You have to improve your validation set score, not your test score.
Besides how good or bad the method you use is, the point is also that for any system generation process, no matter the method used, you still need to repeat the method on random data sets and perform the DMB analysis. You simply cannot skip this step. Whether your method is genetic optimization, normal optimization, train/cross validate/test you still need to do the DMB analysis for whatever method you use.
  • #705
  • Sep 30, 2015 10:45am Sep 30, 2015 10:45am
Quoting algoTraderJo
{quote} But would you go live with something with a bad test score and a good validation score? Of course not. You would tweak things until you had both a good validation score and test score. This tweaking increases DMB a gazillion times because of the continuous usage of the data.
If my validation set score is good and my test score is bad, I would try to improve my already good validation set score. When I reach my best validation score, if the test score is still bad, then my model is garbage. Never try to directly improve the test set score.
  • #706
  • Sep 30, 2015 10:55am Sep 30, 2015 10:55am
Quoting algoTraderJo
{quote} Besides how good or bad the method you use is, the point is also that for any system generation process, no matter the method used, you still need to repeat the method on random data sets and perform the DMB analysis. You simply cannot skip this step. Whether your method is genetic optimization, normal optimization, train/cross validate/test you still need to do the DMB analysis for whatever method you use.
I would probably perform a DBM analysis at the very end of my model analysis, but I think that a proper train/cv/test dataset splitting should already give some good hints.
  • #707
  • Sep 30, 2015 11:11am Sep 30, 2015 11:11am
Quoting ampleparking
{quote} If my validation set score is good and my test score is bad, I would try to improve my already good validation set score. When I reach my best validation score, if the test score is still bad, then my model is garbage. Never try to directly improve the test set score.
It's a pity I don't have more time to go into this in detail. Just a few more words. These methods always fail because they involve two things: one is reuse of data and the second is bias in the selection of the train/validate/test sets. I'm not trying to convince you but keep this in mind after you have some experience with this type of process in real trading.
  • #708
  • Sep 30, 2015 11:14am Sep 30, 2015 11:14am
Quoting ampleparking
{quote} I would probably perform a DBM analysis at the very end of my model analysis, but I think that a proper train/cv/test dataset splitting should already give some good hints.
You cannot perform a DMB analysis "just at the end". The DMB must include the entire creation process, otherwise you could just mine and modify as you please. The idea of the DMB is to show you how probable it is for what you have created to have come out of chance. Creation goes from start to finish. Therefore including the entire process in the DMB analysis is critical. If you include just the last step the process is useless because DMB on the last step is probably going to be minimal, the large DMB contributions usually happen in initial steps.
  • #709
  • Sep 30, 2015 11:15am Sep 30, 2015 11:15am
Quoting ampleparking
{quote} I would probably perform a DBM analysis at the very end of my model analysis, but I think that a proper train/cv/test dataset splitting should already give some good hints.
The train/cv/test process has nothing to do with DMB. If your process is successful it does not mean that an equal process wouldn't be successful on random data. The system creation process tells you nothing about DMB, this is why you need to measure it.
  • #710
  • Sep 30, 2015 11:45am Sep 30, 2015 11:45am
Quoting algoTraderJo
{quote} The train/cv/test process has nothing to do with DMB. If your process is successful it does not mean that an equal process wouldn't be successful on random data. The system creation process tells you nothing about DMB, this is why you need to measure it.
This is what I would do:
A) Start with a theoretically valid idea (don't try random non-sense things until something seems to work)
B) Train on the train set
C) Validate/Tune on the cv set
D) Test on the test set (never fit/train/tune on the test set!)

When my idea has a theoretical foundation, B score is good, C score is good and D score is good... then I could be on the right track.

At the very end, I can try to apply my model to some random data (e.g. random walk). If it does work on my original test data and it doesn't work on the random data, then it's likely that my model exploited some structure in the data which isn't present in the random data.
  • #711
  • Sep 30, 2015 12:04pm Sep 30, 2015 12:04pm
Quoting ampleparking
{quote} This is what I would do: A) Start with a theoretically valid idea (don't try random non-sense things until something seems to work) B) Train on the train set C) Validate/Tune on the cv set D) Test on the test set (never fit/train/tune on the test set!) When my idea has a theoretical foundation, B score is good, C score is good and D score is good... then I could be on the right track. At the very end, I can try to apply my model to some random data (e.g. random walk). If it does work on my original test data and it doesn't work on the random...
You're mistaken about what the DMB analysis tries to do. It does not try to evaluate whether your system sees something on the real data that is not present in random data. It seeks to find out whether your process can generate a similarly performing model on random data. The DMB is about your process, not about your ending model. This means that after you finish on real data you should:

  1. Create random data sets and divide them into train/validate/test
  2. Train on each random set
  3. Validate/tune on each set
  4. Test on each set

After doing this on many different random price series you'll be able to tell whether your generated model is or is not the consequence of mining bias. If you can always generate an equally well performing model on random data then your model is worthless. Maybe the inherent bias in your tuning step is so large that you can make it work on any random data set, meaning that your results are simply due to chance.

  • #712
  • Sep 30, 2015 12:06pm Sep 30, 2015 12:06pm
Quoting algoTraderJo
{quote} You're mistaken about what the DMB analysis tries to do. It does not try to evaluate whether your system sees something on the real data that is not present in random data. It seeks to find out whether your process can generate a similarly performing model on random data. The DMB is about your process, not about your ending model. This means that after you finish on real data you should: Create random data sets and divide them into train/validate/test Train on each random set Validate/tune on each set Test on each set After doing this on...
Of course I apply the whole process to the random data, not the model fitted to my original data. The starting point is the "theoretically valid idea".
  • #713
  • Sep 30, 2015 12:07pm Sep 30, 2015 12:07pm
Random walks also don't work well to generate random data for this purpose... The data must be exactly the same as your real data except that it should be shuffled to eliminate any potential for real predictive action to happen. This means use bootstrapping with replacement or Monte Carlo methods to generate the series.
  • #714
  • Sep 30, 2015 12:09pm Sep 30, 2015 12:09pm
Quoting ampleparking
{quote} Of course I apply the whole process to the random data, not the model fitted to my original data. The starting point is the "theoretically valid idea".
That's cool.
  • #715
  • Sep 30, 2015 12:13pm Sep 30, 2015 12:13pm
Quoting algoTraderJo
Random walks also don't work well to generate random data for this purpose... The data must be exactly the same as your real data except that it should be shuffled to eliminate any potential for real predictive action to happen. This means use bootstrapping with replacement or Monte Carlo methods to generate the series.
Yes, that's probably more indicative than a random walk, since it uses more realistic steps.
  • #716
  • Sep 30, 2015 1:31pm Sep 30, 2015 1:31pm
Quoting algoTraderJo
Random walks also don't work well to generate random data for this purpose... The data must be exactly the same as your real data except that it should be shuffled to eliminate any potential for real predictive action to happen. This means use bootstrapping with replacement or Monte Carlo methods to generate the series.
Quoting ampleparking
Yes, that's probably more indicative than a random walk, since it uses more realistic steps.
Technically speaking if you bootstrap the return of ticks you do get a RW. Not a drunkard (coin toss) or a Brownian (Gaussian) but a RW nonetheless. Any probability distribution can be used (Cauchy, Levy, skewed, multimodal, time variant...). With tick returns you get this: http://www.forexfactory.com/showthre...00#post7163700
No greed. No fear. Just maths.
  • #717
  • Oct 5, 2015 10:51am Oct 5, 2015 10:51am
Another way to attempt to improve your results is to include data from other symbols as inputs. You must be really careful when doing this as you must make sure that the two data sets match perfectly so that you don't get any wrong results due to data snooping or anything along these lines. You must make sure that a given shift on one symbol matches exactly the same timestamp on another symbol. Luckily the Asirikuy tester takes care of this automatically.

The results you see on this post use the same trailing stop and machine learning mechanism as before only that this time we use 17 bars from the GBPUSD and 17 bars from the EURUSD for each example used in training. Trading is done on the GBPUSD. As always retraining is performed before each trading decision (see previous posts for more info).

Inserted Code
SL = 150% daily ATR20
A=20 (GMT +1/+2)
B=85
C=17
D=65

Attached Image


Results are different than what you get if you use only GBPUSD data, they provide a good possibility for diversification because the correlation with results using only GBPUSD data for predictions is low. Just look at the trading hour (A) and the SL. Getting these results with these A/SL/D values using only GBPUSD data is not possible with this algo.

Next I'll move to examples on the GBPUSD 30 minute TF...
  • #718
  • Oct 5, 2015 3:15pm Oct 5, 2015 3:15pm
Hi algoTraderJo, Thanks for sharing the results. It would be very helpful for people who want to try machine learning techniques on forex trading.

One thing that keeps me wondering is that, since the forex market is such an efficient market, why don't we try these machine learning techniques on other markets as well? Other commodity or index futures markets could be much more inefficient, and trading in multiple markets with an edge can only smooth out the growing asset curve. I am interested to know your thought on this.

Another thing I want to know is, by subscribing to your group with Dennis, will the long term forex and swap data you used be available to members for testing?
  • #719
  • Oct 6, 2015 6:33am Oct 6, 2015 6:33am
Quoting dreamvalley
Hi algoTraderJo, Thanks for sharing the results. It would be very helpful for people who want to try machine learning techniques on forex trading. One thing that keeps me wondering is that, since the forex market is such an efficient market, why don't we try these machine learning techniques on other markets as well? Other commodity or index futures markets could be much more inefficient, and trading in multiple markets with an edge can only smooth out the growing asset curve. I am interested to know your thought on this. Another thing I want to...
By all means, try it on other markets! The techniques I have discussed on this thread can be applied to the equity, bond or futures markets without problems. There are however a few reasons why you might want to try FX first:

  1. Intraday data for futures/equities/bonds is more difficult to find and more expensive
  2. Trading commissions on these markets are higher
  3. Margin requirements are also higher
  4. Minimum trading sizes are also much higher
  5. DMB is commonly much higher, especially on the equity/bond markets due to the natural positive long term drift on these markets.

There are also many advantages to trading machine learning algos on equities/bonds/futures but if you have never done this you might want to try first on FX, then move to higher entry level markets.

About the Asirikuy community (with Daniel... There is no one named Dennis involved as far as I remember!), the long term forex swap and historical data is indeed available for anyone who joins. There is a weekly updated dataset available to all members that covers several pairs from 1987 to present with 1M resolution.

  • #720
  • Oct 6, 2015 7:05am Oct 6, 2015 7:05am
Going down to the 30M timeframe, compounding starts to get really serious. Using the previously described trailing stop based system on the 30M TF we can get the following result (remember that the ML algo is retrained before making each trading decision):

Quote
SL = 50% daily ATR20
A=8 (GMT +1/+2)
B=85
C=33
D=80

Attached Image


Linear correlation coefficient is now 0.97 (very high!). This means that trading results for a 28 year period are considerably stable. See that the C parameter (number of bar returns used to construct each example used for training) has increased importantly from other EURUSD results using this same type of strategy on the 1H. This makes sense if you think about it, if previously we needed 10 1H bars to build each training example we would now need 20 30M bars to read the same market information. Since all this information is important to make a prediction using higher resolution info does in fact lead to a better predictive power. We have a better picture of the relevant info.

Some hints when going to low TFs:

  1. Differences between results using data from different brokers increase because the OHLC of lower TF data is more sensitive. A difference of 1 pip on the daily is nothing, on the 15M it might be 10% of the bar. You run the risk of curve fitting to very particular aspects of the historical feed structure that may not be present on your broker. Even if you use your own broker's data you might fall into curve fitting to something like a filtering algo used by your broker to smooth the historical data that might not be present on real market data.
  2. The DMB is almost always lower on a lower TF because there is a lot of additional data which makes it difficult to get systems with high Sharpe ratios on random data, while the Sharpe ratio of the systems on the real data tends to remain constant.
  3. Simulations are much more expensive. A simulation on 30M timeframe generally requires double the data than a simulation on the 1H. Not only does this mean going through twice as many bars but often it means that on each execution you need to load double the historical data. This means a decrease in performance in the order of 3-5X when compared with a 1H simulation. You can of course make this a bit better by employing some tricks, we may go into this later on.

0 traders viewing now