Machine Learning with algoTraderJo | Page 7 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #121
  • Dec 12, 2014 7:21am Dec 12, 2014 7:21am
Since the linear regression offers a prediction of return values we can see what happens if we try to filter the results by only taking into account predictions that are above 5 pips. In theory this should eliminate all border-line predictions (+/- 4 pips) and offer us a more certain set of predictions (those where we are predicting bars of more than 5 pips in size). You can see the results below (normal LR compared with LR with filter):

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_lr_filter.png Size: 34 KB


From these results it is crystal that we don't have any significant edge in the prediction of the bar's size but we get all of our edge simply due to the accurate prediction of directionality and the distribution of these predictions. How could we have a better prediction of the returns? Should we change our model? Is the linear regression simply not good enough here? Are we suffering from the problems exposed above by fellow traders? What do you think?
  • #122
  • Edited 10:13am Dec 12, 2014 9:53am | Edited 10:13am
Hi algoTraderJo,

Do you mind positing some samples of your R code?
  • #123
  • Dec 12, 2014 11:10am Dec 12, 2014 11:10am
What is the timezone used in the file detailed_trade_log.txt? I can't match with the prices I have.

The SL can get hit at any time. Why are all the trades in the file stopped at an o'clock hour?

This is what I get. I assumed the file contains GMT times. The trades are open at 00:00 GMT except for the Monday where they are open at 4:00 GMT and except for the week-end where no trade is open (like in the file). My tick data starts in 2007 so I can't test more. Honestly I don't see a big difference with a lucky realization of coin flipping trades.
Attached Image (click to enlarge)
Click to Enlarge Name: BT-EURUSD.png Size: 89 KB
No greed. No fear. Just maths.
  • #124
  • Dec 12, 2014 12:14pm Dec 12, 2014 12:14pm
Quoting Ultimate6
Hi algoTraderJo, Do you mind positing some samples of your R code?
I only use R code to make the graphs and show you the analysis. As I have posted before all simulations are carried out using the Asirikuy programming framework and tools.
  • #125
  • Dec 12, 2014 12:20pm Dec 12, 2014 12:20pm
Quoting PipMeUp
What is the timezone used in the file detailed_trade_log.txt? I can't match with the prices I have. The SL can get hit at any time. Why are all the trades in the file stopped at an o'clock hour? This is what I get. I assumed the file contains GMT times. The trades are open at 00:00 GMT except for the Monday where they are open at 4:00 GMT and except for the week-end where no trade is open (like in the file). My tick data starts in 2007 so I can't test more. Honestly I don't see a big difference with a lucky realization of coin flipping trades. {image}...
The Timezone used on this file is GMT +1 non-DST and GMT +2 DST. All the files I post will be using this TZ.

The SL can be hit at anytime within the daily bar but the timestamp is rounded to the daily bar.

Also you're right. For the 2007-2014 period I also don't see a big difference with a lucky realization from random trades for this linear regression example. You need to use a much longer sample to see the difference with a random outcome (as in my 1990 to 2014 test). It is obvious that the model loses a good portion of its edge under more recent market conditions. We will need to evolve to more advanced methods to have something that performs well under recent market conditions (you will see some examples of this as the thread advances).
  • #126
  • Edited 1:28pm Dec 12, 2014 12:57pm | Edited 1:28pm
I now want to post some experiments related with a more complicated input/target structure using classifiers. Let's say that we now want to predict something that is more useful to our trading, we want to predict whether a + or - given target will happen first, which should lead to a direct prediction related with trade success. We use the bearish/bullish character of the last 3 bars (0 bearish, 1 bullish) as inputs to predict whether a target of 50% of the ATR(20) is reached within the next 20 daily bars (I check what was hit first using a 1H TF resolution). If a +ATR(20) value is reached first from the last open the classifier puts the example in category 1 and if it reaches -ATR(20) first it puts it in category 0. We enter trades with a stoploss equal to 50% of the ATR(20) and hope the machine learning model gives us an edge in trade success (we enter a trade trying to avoid hitting the stop). I don't use a takeprofit because I want to take advantage of luck (favorable moves beyond this expected movement) when they happen (I only want to exit when I have an expectation for the following trade to be losing). The model is retrained on every bar using the past 85 examples (remember that each example has the 3 last bar directions (1 or 0) as inputs and whether the + ATR20 (1) or -ATR20 (0) price are reached first as target).

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_exittargetKNN.png Size: 26 KB


I also attach the trade-by-trade detailed log in case you want to take a deeper look (remember that all simulations and files are given in GMT +1 non-DST, GMT +2 DST TZ). In this case we definitely reach a much better result than with the previous algorithms we have studied, since we are now using a prediction target that is more closely related with trade success this makes sense. We can also push this to better performance under recent market conditions. Any ideas on how this could be done?
Attached File(s)
File Type: txt machineLearning_resultlog.txt   391 KB | 518 downloads
  • #127
  • Dec 12, 2014 1:17pm Dec 12, 2014 1:17pm
This is just a comparison of some of the things we have studied up until now:

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearing_comparison.png Size: 38 KB


We are yet to get a regression approach that is as successful as our classifiers and to get better performance under recent market conditions.
  • #128
  • Dec 12, 2014 1:32pm Dec 12, 2014 1:32pm
I'm traveling now for the weekend! Leave your thoughts
  • #129
  • Dec 12, 2014 3:28pm Dec 12, 2014 3:28pm
Quoting olsen-yersen
{quote} I will test your prediction results tonight with chaos hunter to see if I can improve it further.
I get slightly better results than the PipMeUp but as he said it is coin flipping.
  • #130
  • Dec 12, 2014 3:51pm Dec 12, 2014 3:51pm
Quoting algoTraderJo
We use the bearish/bullish character of the last 3 bars (0 bearish, 1 bullish) as inputs to predict whether a target of 50% of the ATR(20) is reached within the next 20 daily bars (I check what was hit first using a 1H TF resolution). If a +ATR(20) value is reached first from the last open the classifier puts the example in category 1 and if it reaches -ATR(20) first it puts it in category 0.
Quote
(remember that each example has the 3 last bar directions (1 or 0) as inputs and whether the + ATR20 (1) or -ATR20 (0) price are reached first as target).

Hi algoTraderJo, I have a question about how you prepared the data for training, specifically the class label or actual outcome that is used in training. As you prepare your data, are you assuming that future price (20 bars) hits the +/- ATR20 * 0.5 on every starting bar? Because without looking at the data I would guess that for some starting bars it probably doesn't hit either of those +/- ATR targets 20 bars into the future. But maybe I'm mistaken. But in either case, it is conceivable that I might wish to select future targets that may not be hit on every starting bar x bars into the future. So how would you approach this?

If it occasionally doesn't hit the future target, do you just use the last direction or have you got a third class or something else? Or are you collapsing your data so that only those points that correspond to an actual +/- ATR20 * 0.5 hit are in the series and the missing items are taken out? Collapsing the data to only coincide with ATR hits would seem problematic to me as you probably introduce data snooping bias, so I doubt you are doing this.

Quote
We can also push this to better performance under recent market conditions. Any ideas on how this could be done?
You've had good luck combining equity curves so this seems like an obvious thing to try again now that you have more variety in the equity curve pool. You could also change your parameters or increase the # of features you are using, all of which you've tried before so I'm not really giving you any new ideas. By the way, would you classify the recent performance problem for this equity curve as a bias or variance issue? Because it seems that bit of information should drive the suggestions.
  • #131
  • Dec 12, 2014 4:34pm Dec 12, 2014 4:34pm
Quoting olsen-yersen
{quote} I get slightly better results than the PipMeUp but as he said it is coin flipping.
"GMT +1 non-DST, GMT +2 DST"
I'm always confused with this GMT offsets... To me this is Paris/Berlin timezone. I corrected my strategy to use local time taking DST into account. MM = fixed lot size 1 microlot. Starting with 100€.

When I use London locale time the result is great... in the beginning! The account grows from 100€ to 370€ then big DD then back to 350€ then back to square one 100€....

When I use Paris local time the algo backrupts in 2011.
No greed. No fear. Just maths.
  • #132
  • Dec 15, 2014 6:29am Dec 15, 2014 6:29am
Quoting FXEZ
{quote} {quote} Hi algoTraderJo, I have a question about how you prepared the data for training, specifically the class label or actual outcome that is used in training. As you prepare your data, are you assuming that future price (20 bars) hits the +/- ATR20 * 0.5 on every starting bar? Because without looking at the data I would guess that for some starting bars it probably doesn't hit either of those +/- ATR targets 20 bars into the future. But maybe I'm mistaken. But in either case, it is conceivable that I might wish to select future targets...
You talk about a valid issue. When I train the model on each bar I start looking at past data to build trading examples. The first valid examples exist 20 bars in the past (because I need to be able to look 20 bars forward to build the target) if the output is not defined (a target of +/- ATR20 *0.5 is not reached) I simply do not use that example and move to the next example in the past. This ensures that I have as many historical examples as I need where the target actually has a valid value. However using a +/- ATR20 * 0.5 you almost never have to skip invalid examples when rebuilding the model on each bar.

Your suggestions are all valid and they are all things I do to improve model results. You will see some more of that as the thread progresses. About the recent performance problem, I think you'll be able to draw a conclusion with some aspects we will be studying this week. Tell me what you think is the cause as the thread advances.
  • #133
  • Dec 15, 2014 6:33am Dec 15, 2014 6:33am
Quoting PipMeUp
{quote} "GMT +1 non-DST, GMT +2 DST" I'm always confused with this GMT offsets... To me this is Paris/Berlin timezone. I corrected my strategy to use local time taking DST into account. MM = fixed lot size 1 microlot. Starting with 100€. When I use London locale time the result is great... in the beginning! The account grows from 100€ to 370€ then big DD then back to 350€ then back to square one 100€.... When I use Paris local time the algo backrupts in 2011.
I have to emphasize here that the money management aspects of the strategy are critical. The strategy (in fact all strategies I post) adapt their lot sizes against market volatility. Trading a constant lot size will always bring you worse results when simulating these systems. Trading 1 microlot with 100€ is also an extremely large risk (that would translate to more than 10-20% per trade, which guarantees a margin call in the long term for this strategy). It is critical to accurately represent my results that you risk the same amount of money or account percentage (depending if you want or don't want compounding) on each stoploss hit (meaning that you must adapt your traded lot size against market volatility). This adapting of lot sizes against volatility is madly important!
  • #134
  • Dec 15, 2014 6:52am Dec 15, 2014 6:52am
It is evident that we wouldn't want to trade something that has unstable performance under recent market conditions (gone flat or started losing money) so I will now go a bit into some measures to attempt to increase our performance statistics for the past 7 years. You could think that this bad performance shows up because our machine learning strategy lost the ability to predict the target with the inputs we gave it (it lost its edge) mainly because either the information is not relevant, not enough to make a decent prediction or the relationships between inputs/targets have become too complex for the K-NN model. In line with this I have simulated an SVM system using the directional character of the last 12 bars as input, as well as the 4 bar K-NN strategy I posted before for comparison. The target is whether a +/- ATR20*0.5 target is reached within the next 20 bars.

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_knn_svm_12bar.png Size: 44 KB


Using more data as input definitely improves results under recent market conditions while results for the 1991-1999 period deteriorate substantially for the SVM. This shows you that an increase in both model and input complexity has improved results. Where we previously had the worst historical drawdown and a 4 year drawdown period we now have a steady balance uptrend and a new balance high every 6 months. Can we lead the less complex K-NN model to similar results? Is it enough to add more inputs? Can we have something that is stable in 1990-1999 and 2000-2014?
  • #135
  • Dec 15, 2014 7:59am Dec 15, 2014 7:59am
The previous post begs to ask whether the results are simply due to more information or due to a better model for the description of the input to target relationship. Running tests for strategies using SVM models using the past 2, 4, 6, 8, 10 and 12 bar directions (0,1) as inputs we get the following result:

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_svm_inputcomparison.png Size: 39 KB


Results do improve very significantly from 2 to 4 bars (recent market conditions) and then they continue to improve up to a value of around 10, time after which we see some small variations in performance (but it remains fairly constant). This means that although we have some gain from the additional input complexity most of the gain relative to the 4 bar K-NN is actually due to a model that is able to better describe input/target relationships for more recent market conditions.
  • #136
  • Dec 15, 2014 8:15am Dec 15, 2014 8:15am
Quoting algoTraderJo
{quote} I have to emphasize here that the money management aspects of the strategy are critical. The strategy (in fact all strategies I post) adapt their lot sizes against market volatility. Trading a constant lot size will always bring you worse results when simulating these systems. Trading 1 microlot with 100€ is also an extremely large risk (that would translate to more than 10-20% per trade, which guarantees a margin call in the long term for this strategy). It is critical to accurately represent my results that you risk the same amount of...
The 1 microlot with 100€ was only to have the graph scaled properly ;-) (this is on my "to be fixed" list)

It is true that a poor MM can deteriorate and even kill a strategy. But it is more importantly true that no MM in the world can turn an edgeless stategy into a winner. This is the reason why fixed lot size must be used for a BT. If the fixed lot size can't make it work, no MM will. If your BT only works because of a given MM your strategy is simply edgeless.

BTW: I modified my code to use 1% MM. The result: both BT (Paris and London local times) turn losers!

------

I can still not figure out how a SVM may be used for this kind of vectors. All of the vectors fall exactly on one of the cells of the (hyper-)grid. Any query vector as well. What difference can it make where the frontier actually lays?
No greed. No fear. Just maths.
  • #137
  • Dec 15, 2014 10:04am Dec 15, 2014 10:04am
Quoting PipMeUp
{quote} The 1 microlot with 100€ was only to have the graph scaled properly ;-) (this is on my "to be fixed" list) It is true that a poor MM can deteriorate and even kill a strategy. But it is more importantly true that no MM in the world can turn an edgeless stategy into a winner. This is the reason why fixed lot size must be used for a BT. If the fixed lot size can't make it work, no MM will. If your BT only works because of a given MM your strategy is simply edgeless. BTW: I modified my code to use 1% MM. The result: both BT (Paris and London...
I agree with you in that MM does not generate any edge. However it does affect the results significantly so I would advice you to use the same MM to reproduce my results. The GMT offset is far more critical however, since it directly affects the trading edge generated. It's a pity that you don't have access to the Asirikuy framework, that way you could directly reproduce all the results I post with the exact same tools/code. Do ask me anything if you need information to reproduce this using your own tools.
  • #138
  • Dec 15, 2014 11:12am Dec 15, 2014 11:12am
The number of examples used to train on each bar is also important. Below you'll see a graph with the results using 40, 60, 100 and 120 examples to train the SVM model on each bar (using 10 bar directions as inputs). This means that we actually need to train the model with a given window of market conditions or we start to lose our edge in predicting the +/-ATR20 outcome direction.

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_svm_examples.png Size: 39 KB


Results are indeed only good on all tests through the 2003-2007 period while results across other periods depend on the number of examples used for training to a larger extent than the number of inputs used. How you train also has importance on whether you make or don't make money under some market conditions.
  • #139
  • Dec 15, 2014 11:28am Dec 15, 2014 11:28am
We can attempt to eliminate our dependency on the number of inputs and the number of learning examples used on each bar by making decisions using a group of models instead of a single model. If the overall premise of input and target relationship is valid then we can suppose that the accuracy can increase by taking trades only when a group of models agree on the decision to make. For example I can build models using inputs from 8 to 12 using training examples from 60 to 120:

SVM - 8 bars as inputs, 60 examples used for training on each bar
SVM - 9 bars as inputs, 60 examples used for training on each bar
SVM - 10 bars as inputs, 60 examples used for training on each bar
SVM - 11 bars as inputs, 60 examples used for training on each bar
SVM - 12 bars as inputs, 60 examples used for training on each bar
SVM - 8 bars as inputs, 70 examples used for training on each bar
SVM - 9 bars as inputs, 70 examples used for training on each bar
.... etc ....

All the 20 models are retrained on each bar (using their defined number of example) and I only take signals then if the 20 models tell me to trade in the same direction. This should generate a result that is better than the result of the single SVM models if the input/target relationship holds some predictive power. Here are the results for this ensemble (an ensemble is a group of models):

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_svm_ensemble.png Size: 27 KB


Our trade frequency is lower due to the filtering but our information ratio increases (the loss in frequency is worth the increase in risk adjusted return) and the results under recent market conditions suddenly become much better. As a matter of fact the best trading period for this system is in the 2007-2014 period. We still have the issue of dealing with results in the 1991-1997, which has now become the issue. However since we have K-NN models that perform very well within that period it seems a no-brainer to try out portfolios to reduce variance. What do you think?
  • #140
  • Dec 15, 2014 4:09pm Dec 15, 2014 4:09pm
Loving this thread. Will continue to read...
0 traders viewing now