Machine Learning with algoTraderJo | Page 9 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #161
  • Dec 19, 2014 9:05am Dec 19, 2014 9:05am
Quoting algoTraderJo
{quote} Your rules and definitions are ok. Note that I do all calculations (including the ATR calculation) on daily data that has been corrected to GMT DST +2, non-DST +1. If you can please post some balance graphs so that I can take a quick look and see what might be wrong (sadly do not have time to go through and process txt files). Have you compared your predictions with the ones I posted before? Are they in alignment? Differences could be because of a few reasons, the ATR calculation (mine is done with the ATR as implemented in TA-lib) or the...
I have very different prices than yours. My source is FXCM for this BT. When I'm back at home I try again with Dukascopy data. I think this is the reason. When I feed my linear regression algo with your
data (post #99) I get exactly the same result as you get. The algorithm I use for the ATR is the one described on Wikipedia except that I skip the week-end candles. I tried with and without the week-end with no noticeable change.

Note that I exclusively use tick data for the BT. My simulator simply can't use candles: I emulate a broker which feeds the ticks. My SL are executed at the bid or ask of the breaking tick. The spread is the spread at the time the order is executed. This may also make a difference.

EDIT: I forgot to say that I don't take any commission or swap into account.
Attached Image (click to enlarge)
Click to Enlarge Name: BT-EURUSD.png Size: 170 KB
No greed. No fear. Just maths.
  • #162
  • Dec 19, 2014 9:14am Dec 19, 2014 9:14am
Quoting PipMeUp
{quote} I have very different prices than yours. My source is FXCM for this BT. When I'm back at home I try again with Dukascopy data. I think this is the reason. When I feed my linear regression algo with your data (post #99) I get exactly the same result as you get. The algorithm I use for the ATR is the one described on Wikipedia except that I skip the week-end candles. I tried with and without the week-end with no noticeable change. Note that I exclusively use tick data for the BT. My simulator simply can't use candles: I emulate a broker which...
I get the same results for 2007-2014 when executing tick-for-tick simulations using Dukascopy tick data and using candles. Since we're on the daily TF and the stops are so wide this does not make a very large difference. The difference we get must therefore be largely due to data differences. My simulations do include swap charged at 17 PM, 3x on Wednesdays. Swaps are calculated from historical interest rate data (from Oanda).
  • #163
  • Dec 19, 2014 9:31am Dec 19, 2014 9:31am
To end the week I want to leave with a new regression based strategy. In analogy to what we did with classifiers we can also attempt to make regression models that try to predict targets that are more strongly related with our ability to profit from a trade. To do this I created a linear regression model that uses the past 2 bar returns as inputs and uses the highest and lowest excursions after 5 days as targets. This is our first model that attempts to predict 2 targets instead of a single target, giving us a prediction of what we should expect to happen within the next 5 days. The model is retrained on every bar using the past 100 examples. We take a long trade only if the highest excursion is 20% bigger than the prediction of the lowest excursion and vice versa for shorts. Here are the results:

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_LR_targets_first.png Size: 30 KB


We now have a smoother balance curve than we did with the linear regression model trying to predict the next bar's return as output and we have overall better results across several market conditions. Since the variable we are now attempting to predict is now strongly related with our performance, this makes some sense. However the relationships we are trying to find here are quite complex and therefore the linear regression model is not the ideal candidate. Here we have a ton of flexibility in the entry rules we can device (the one above is just an example) so we can play a lot and see how this changes our model's results. Next week we will go deeper into what we can do with these models.

See you next week guys. Leave your thoughts
  • #164
  • Dec 22, 2014 4:35am Dec 22, 2014 4:35am
We can also play with the prediction frontier to see what horizon prediction brings the most favorable results. Changing the target to predict the lowest and highest returns within a 2, 5, 10 and 20 day period gets us the balance curves shown below. I have used a linear regression model that retrains on each bar using the past 80 examples, with the past 2 bar returns as inputs and a stoploss equal to 60% of the ATR20.

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_lr_different_targets.png Size: 41 KB


As you can see the results are worst for the prediction of values two bars into the future while it is best for the prediction of 5 bars. This happens because movements within the next two bars are more difficult to accurately predict than movements within the next 5 bars. At longer time predictions our results also start to deteriorate although they never deteriorate as strongly as for the 2 day target prediction. The results we achieve here for the prediction of highest/lowest return targets in 5 days is the best result I have shown for a linear regression model yet.
  • #165
  • Dec 22, 2014 6:08am Dec 22, 2014 6:08am
We can also try to make predictions using a random forest model instead of a linear regression model. I use a target that attempts to predict the highest and lowest return points within the next 10 bars, using the return of the past 2 bars as input, remember that retraining happens on every bar. You can see here a comparison of the linear regression and random forest models:

Attached Image (click to enlarge)
Click to Enlarge Name: machineLearning_lr_vs_randomForest_2.png Size: 34 KB


Surely this image shows that the results are much better for the linear regression model, the random forest model is not able to predict properly with this limited amount of information. You will see that random forest models tend to overfit when you have few inputs because of the degrees of freedom within the modeling process. This does not mean that the linear model is better in all cases, as a matter of fact the random forest model can give us much better risk adjusted returns given a more complex set of inputs that cannot be adequately processed using the linear regression model.
  • #166
  • Dec 22, 2014 11:16am Dec 22, 2014 11:16am
I'd like to see a comparison using different bar durations. For example, selecting a single test methodology, can you plot side by side the results using daily, 4 hour, one hour, 30 mins and 5 mins data as an example?
  • #167
  • Dec 22, 2014 12:59pm Dec 22, 2014 12:59pm
Quoting beachloup
I'd like to see a comparison using different bar durations. For example, selecting a single test methodology, can you plot side by side the results using daily, 4 hour, one hour, 30 mins and 5 mins data as an example?
We are only using daily bars at the moment. Lower timeframes have to be treated differently because of daily volatilty cycles caused by market opening|closing times. When we move to lower timeframes on the thread you will see how this is done. However we will only proceed in this direction after we have explored daily timeframes in detail (we still have a long way to go!). But keep on posting any thoughts or questions you might have.
  • #168
  • Dec 23, 2014 5:39am Dec 23, 2014 5:39am
Hi Jo,

first of all Merry Xmas for all of you here in the forum!

some remarks: did you try Theil-Sen estimator instead of linear regression (there is good R package "Zyp")

I try to reproduce your number in full R, could you define the algorithm short please!

Anyway I am dealing with a similar system for a long time and think the most important is the taking the money in time,
and finding a good stop distance.
  • #169
  • Dec 23, 2014 7:20am Dec 23, 2014 7:20am
Quoting emeafx
Hi Jo, first of all Merry Xmas for all of you here in the forum! some remarks: did you try Theil-Sen estimator instead of linear regression (there is good R package "Zyp") I try to reproduce your number in full R, could you define the algorithm short please! Anyway I am dealing with a similar system for a long time and think the most important is the taking the money in time, and finding a good stop distance.
Theil-Sen is a more robust linear regression estimator so it obtains better results for some sets of data but I am unsure if it will give you better predictions in the case of this exact linear regression algorithm. I have not used Theil-Sen before, mainly because more complex regression algorithms (like SVM and randomforest) do a much better job when you move into more complex cases.

For the system rules you can look at this post by PipMeUp, where he gave a quite good description of what I explained on previous posts. Remember that on every bar I retrain the linear regression model using the past 100 examples (for what he was trying to reproduce). It is very important that you have daily data constructed with the same GMT shift and DST (also see other posts on that page) if you want to reproduce results. If you want another algorithm's description do let me know (perhaps I did not understand your request well enough).

About stop and profit targets, this is an interesting subject, we'll move to it when we start discussing the MFE and MAE of trading systems developed using ML. This thread will take me years to develop with all the things on this field that are worth discussing!
  • #170
  • Dec 23, 2014 7:32am Dec 23, 2014 7:32am
sorry for being unclear.

How can you train a linreg model on 2 inputs?
For my understanding the linreg takes the inputs and gives you back a slope and intercept what
does not depends on your target.
If it is an NN I understand the process.
  • #171
  • Dec 23, 2014 7:42am Dec 23, 2014 7:42am
There are many ways in which we can attempt to get better trading results. So far we have discussed portfolios, ensembles, changing predictions targets, changing models and changing our training parameters (how many examples to use for training on each bar). Now that we have a more complex model for regression (random forest) we can start looking at the effect of using more complex arrays of inputs for the regression mechanism. A random forest has the advantage that it is quite robust to the inclusion of inputs that do not provide a lot of information so it is a good model from this point of view as any mistakes in input selections are not too costly.

If you think about it most additional data we include for the EURUSD daily (like indicators, price derivatives, etc) will give us information naturally contained in the bar returns, therefore the information provided by other EURUSD inputs is limited (although there are some that are useful, we'll see this in a small while). In order to include more information I have decided to add GBPUSD daily bars to our model. The results below use the last 8 bar returns for the EURUSD and the last 8 bar returns for the GBPUSD in order to predict the highest and lowest return excursions within the next 5 trading days. The random forest model is retrained on each bar. These are the results:

Attached Image (click to enlarge)
Click to Enlarge Name: machineLearning_rf_gbpusdeurusdinputs.png Size: 27 KB


Notice how this is a much better graph than the ones we had previously obtained for linear models (from a Sortino ratio perspective). We now have a more powerful regression algorithm that by using more information is able to make much better predictions. The linear model we used before could not accommodate the use of these 16 inputs, simply because the input proposition becomes too complex for what the model is able to do. The random forest model, which can easily accommodate large arrays of inputs and find non-linear relationships, has no problem with all this information. We can also apply the things we have already learned to improve these results. What would you do?
  • #172
  • Dec 23, 2014 7:45am Dec 23, 2014 7:45am
Quoting emeafx
sorry for being unclear. How can you train a linreg model on 2 inputs? For my understanding the linreg takes the inputs and gives you back a slope and intercept what does not depends on your target. If it is an NN I understand the process.
I use the shark library for training using linear regression. You can check the library's linear regression page for more information on how the algorithm works. Shark is open source so you can also take a look at the library's source code if you want access to more specific information detailing how the algo works.

Please note that the model is not trained using 2 inputs. The model is trained using 100 examples, each with two inputs and a target. You have 100 input-to-target relationships for each input and prediction target. They are all used at the same time for the training of the model on each bar.
  • #173
  • Dec 23, 2014 7:56am Dec 23, 2014 7:56am
Thanks!

I will look it!
  • #174
  • Edited 11:22am Dec 23, 2014 11:05am | Edited 11:22am
So now I got around to testing EU with lm and then princomp in R for comparison purposes as a follow up of the knn model I previously created. I haven't used exactly the same settings because my daily data is different, so it won't match anyway. The picture below shows equity curves created using principal components implementing total least squares (top), linear model aka linear regression (middle) and a 50/50 combination (bottom). The strategy based on princomp outperforms linear regression and the combined model in terms of total return.

One reason princomp outperforms lm is that linear regression tends to be unstable. Total least squares doesn't suffer from the same type of instability as linear regression. For more information on this instability and how to implement princomp in R, search for Paul Teetor's paper "Better Hedge Ratios for Spread Trading".

The combined model has a lower maximum drawdown (red vertical line) than either of the other models, even though its total return is lower than Princomp. One reason I like to combine equity curves is described by the optimal f values for each of the 3 systems below (higher means you have a smoother equity curve and can increase leverage more for that strategy):
Inserted Code
> optimalf(Princomp)    > optimalf(LM)          > optimalf(Combined)
1] 5.983692              [1] 4.338143              [1] 10.44123
As you can see, the optimal f value for the combined strategy is nearly double that of Princomp even though its final return is less. This basically means you could trade the same or larger size (up to a certain point) with the combined model with lower overall risk than either of the other two models. To me this indicates that reducing drawdown (or increasing smoothness) may be more important than increasing absolute returns, in terms of how much leverage you can apply to a given strategy.

algoTraderJo: What method are you using to come up with your weights between equity curves? Also, have you noticed that out of sample the performance of a given set of weights degrades much the same as a system that is heavily optimized tends to degrade rather quickly? Do you do in/out of sample testing when you calculate your equity curve weights or how do you guard against this type of overfitting? For this reason I have typically taken the naive view and weighted my equity curves equally. Thanks for any info you can provide!
Attached Image (click to enlarge)
Click to Enlarge Name: LM_Princomp1.png Size: 11 KB
  • #175
  • Dec 23, 2014 3:37pm Dec 23, 2014 3:37pm
Quoting algoTraderJo
Surely this image shows that the results are much better for the linear regression model, the random forest model is not able to predict properly with this limited amount of information. You will see that random forest models tend to overfit when you have few inputs because of the degrees of freedom within the modeling process. This does not mean that the linear model is better in all cases, as a matter of fact the random forest model can give us much better risk adjusted returns given a more complex set of inputs that cannot be adequately processed...
I have some suggestions for better prediction for random forest model : Influence of Hyperparameters on Random Forest Accuracy http://link.springer.com/chapter/10....26-2_18#page-1 for optimizing hyper parameters http://www.johnmyleswhite.com/notebo...uning-methods/
http://rmcantin.bitbucket.org/html/

Also I have suggestion for different random forest models: Mondrian Forests: Efficient Online Random Forests http://arxiv.org/abs/1406.2673 from Prof Yee Whye Teh
Extremely randomized trees : http://docs.opencv.org/modules/ml/doc/ertrees.html

It is worth to mention that there are some very talented data scientists participating in kaggle competitions.Check the kaggle info for random forest models in general and the random forest models participated in kaggle competitions https://www.kaggle.com/wiki/RandomForests
  • #176
  • Dec 23, 2014 5:36pm Dec 23, 2014 5:36pm
Quoting FXEZ
algoTraderJo: What method are you using to come up with your weights between equity curves? Also, have you noticed that out of sample the performance of a given set of weights degrades much the same as a system that is heavily optimized tends to degrade rather quickly? Do you do in/out of sample testing when you calculate your equity curve weights or how do you guard against this type of overfitting? For this reason I have typically taken the naive view and weighted my equity curves equally. Thanks for any info you can provide! {image}
Those are actually very good questions. In sample/out of sample testing is useless IMHO because you would simply find some way of weighting the systems that would work on your "out of sample" period with equal effectiveness. There is no real out of sample testing when using historical data, Daniel wrote an article explaining this point. I strongly agree with his way of looking at things on this topic, any "out of sample" period that you use is ultimately ineffective because repeating use of the "out of sample" data makes it a part of an iterative development process (ends up being an inefficient in sample data source).

When weighting systems I usually combine them in order to minimize variance (what is known as a Markowitz optimization) since my tests generally use 25 years of data I do not see a substantial deterioration of the performance when I move to live trading (it could happen but I have not seen it yet). Nonetheless I keep rebalancing the portfolio to the weights that reduce variance the most every month using the updated backtests. Since the tests are so long the changes are very gradual if they happen at all.

One thing the Markowitz optimization process does that makes things much better than an equal weights distribution is that it compensate naturally for trade frequency (something that is likely to remain similar). If your weights are equal you are probably under-weighting and over-weighting systems depending on how frequently they trade. If you have a system that trades 10 times per week and another that trades 2 times and you weight them equally the 10 times per week system is actually getting a much bigger cut, simply because it trades more (from a P/L versus time look). A Markowitz balancing approach is therefore very valuable, even if it does not end up reducing variance as it does in your historical tests, simply because it compensates factors such as frequency between systems.

There are more advanced things you can do but the above is what I consider the most basic sound approach to portfolio management. I will be happy to further discuss this if you have other questions.
  • #177
  • Dec 23, 2014 5:38pm Dec 23, 2014 5:38pm
Quoting olsen-yersen
{quote} I have some suggestions for better prediction for random forest model : Influence of Hyperparameters on Random Forest Accuracy http://link.springer.com/chapter/10....26-2_18#page-1 for optimizing hyper parameters http://www.johnmyleswhite.com/notebo...uning-methods/ http://rmcantin.bitbucket.org/html/ Also I have suggestion for different random forest models: Mondrian Forests: Efficient Online Random Forests http://arxiv.org/abs/1406.2673 from...
All those are valid suggestions and links, however before we go into exploring better random forests algorithms there are many things you can do to improve the results of the random forest model. When working in ML (even more in FX) I have always found that the simpler approaches work better, before attempting to do something more complicated see if something simpler can lead to a much better result. What do you think, do you have any simple modifications that you think could improve the random forest results?
  • #178
  • Dec 23, 2014 6:46pm Dec 23, 2014 6:46pm
We can also test adding more inputs to the random forest model to see if we can get even better predictions by looking at more data sources. For example what happens if we attempt to add 8 bar returns for the USDJPY to our previous simulations? This new model uses 24 inputs to make predicitons, the past 8 returns for the EURUSD, the past 8 returns for the GBPUSD and the past 8 returns for the USDJPY. Our prediction targets continue to be the highest and lowest returns within the next 5 days (the largest excursion to either side). Remember that the model is retrained using the past 80 examples on each daily bar. Here are the results (comparing with our previous result):

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_rf_eu_gu_uj.png Size: 34 KB


Our results are now better in terms of risk adjusted return for 80% of the test. I would like to remark that the linearity of the log(balance) versus time graph is higher, pointing to a larger level of stability of returns on different market conditions. But not everything is better, our results with this model continue to be poor from 2011 to 2014 and in fact the maximum drawdown is worse for this new model with more information. What can we do to get better results for recent market conditions without compromising our results for previous periods?
  • #179
  • Edited 7:47pm Dec 23, 2014 7:32pm | Edited 7:47pm
Quoting algoTraderJo
{quote} I have always found that the simpler approaches work better, before attempting to do something more complicated see if something simpler can lead to a much better result. What do you think, do you have any simple modifications that you think could improve the random forest results?
I think hyper parameter optimizing for kernel(svm, svr etc..) and decision tree models is a good suggestion to avoid overfitting and for better prediction.

My other suggestions are you can apply Student's t test to see how likely the trading system’s results occurred by chance, have a look at the link: http://www.adaptrade.com/Articles/article-sig.htm and entry -exit efficiency ratio https://www.mql5.com/en/articles/137 it may look similar to mae/mfe analysis.
  • #180
  • Dec 23, 2014 8:45pm Dec 23, 2014 8:45pm
Quoting algoTraderJo
{quote} Those are actually very good questions. In sample/out of sample testing is useless IMHO because you would simply find some way of weighting the systems that would work on your "out of sample" period with equal effectiveness. There is no real out of sample testing when using historical data, Daniel wrote an article explaining this point. I strongly agree with his way of looking at things on...
Regarding the overfitting issues even when doing insample/out of sample, yes when you start testiing multiple combinations, you bring out of sample data into insample.

Check this paperhttp://papers.ssrn.com/sol3/papers.c...act_id=2308659 to see how overfitting is so hard to avoid and how many iterations you should do to have any confidence in your test results. this guys is at Guggenheim doing systematic work.
Think : Trade : Live Life
0 traders viewing now