Machine Learning with algoTraderJo | Page 10 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #181
  • Dec 24, 2014 10:53am Dec 24, 2014 10:53am
algoTraderJo: Many thanks for all your comments. This has helped me clarify several issues I've been working with over the past few years.
Quoting algoTraderJo
{quote} When weighting systems I usually combine them in order to minimize variance (what is known as a Markowitz optimization)
I've used Markowitz portfolio optimization in the past. My results may have suffered because 1) I wasn't using enough 'data', and 2) I didn't take into account the need to periodically reoptimize the weights. You bring up another interesting point about data and in particular, having enough data:
Quote
since my tests generally use 25 years of data I do not see a substantial deterioration of the performance when I move to live trading (it could happen but I have not seen it yet). Nonetheless I keep rebalancing the portfolio to the weights that reduce variance the most every month using the updated backtests. Since the tests are so long the changes are very gradual if they happen at all.

When I started out using Markowitz I had the impression that 'data' meant bars. However I'm now of the opinion that it means examples or trades because it is the consistency of a strategy that is important, and when you combine equity curves you are making a bet that the relative relationship between the curves will stay fairly static. However it could also mean time. When you think of 'lots of data', what measure do you use?

Quote
One thing the Markowitz optimization process does that makes things much better than an equal weights distribution is that it compensate naturally for trade frequency (something that is likely to remain similar). If your weights are equal you are probably under-weighting and over-weighting systems depending on how frequently they trade. If you have a system that trades 10 times per week and another that trades 2 times and you weight them equally the 10 times per week system is actually getting a much bigger cut, simply because it trades more (from...
I was aware that more frequently trading systems tended to dominate less frequent systems in the combined portfolio but was unaware that equal weighting may have implicitly exacerbated this tendency, or that Markowitz helped to compensate for this. I don't see trade count in the formula, is this compensation done through the covariance matrix or some other means?
  • #182
  • Dec 24, 2014 11:25am Dec 24, 2014 11:25am
Quoting stt
{quote} Regarding the overfitting issues even when doing insample/out of sample, yes when you start testiing multiple combinations, you bring out of sample data into insample. Check this paperhttp://papers.ssrn.com/sol3/papers.c...act_id=2308659 to see how overfitting is so hard to avoid and how many iterations you should do to have any confidence in your test results. this guys is at Guggenheim doing systematic work.
stt, I'm a huge fan of Marcos Lopez de Prado. I've read many of his papers but don't think I've seen this one, so thanks for the citation!

The paper seems to consider a 'large amount of data' as time, as shown in Figure 2 rather than either bars or trades. I skimmed over some at the end so maybe the paper covers other measures. So how do HFT firms or those developing intraday strategies justify the use of shorter training periods in years, given this consideration? Are they defining "lots of data" in some other way, and who is right?

Also, I've been conflicted on this idea for some time. On the one hand let's say there is 1 good system that is waiting to be found in a pool of a 100 trillion systems. There may also be a number of 'time bomb' systems that blow up after they are implemented. The paper basically takes the view that the more backtests that are run, the greater the likelihood that a time bomb will show up. One major problem is that it is difficult (or perhaps even impossible) to differentiate a time bomb from a good system. If it were possible to differentiate between the two with a high degree of reliability, then all of these points would be moot and brute force optimization would be the way to go.

On the other hand, doing 100 trillion backtests won't turn a good system into a time bomb, it just has to do with how likely it is that a time bomb will be found and mistaken for a good system. Am I off base?
  • #183
  • Dec 26, 2014 6:54am Dec 26, 2014 6:54am
Quoting FXEZ
algoTraderJo: Many thanks for all your comments. This has helped me clarify several issues I've been working with over the past few years. {quote} I've used Markowitz portfolio optimization in the past. My results may have suffered because 1) I wasn't using enough 'data', and 2) I didn't take into account the need to periodically reoptimize the weights. You bring up another interesting point about data and in particular, having enough data: {quote} When I started out using Markowitz I had the impression that 'data' meant bars. However I'm now of...
Data means information. You can think about information as the amount of ticks that were used to build the data you are using. This is why 300 daily candles contain more information than 300, 5 minute candles. To build a daily candle you need far more ticks than to build a 5 minute candle, although both bars are represented by 4 OHLC values, the daily candle values contain a lot of additional market information (more market conditions). This is different from the number of trades, having 5000 trades across a period of 1 month does not mean that your strategy is less subject to curve-fitting than a strategy that has 500 trades across 10 years. The strategy that was created for a 10 year period has survived much more varied market conditions, therefore it is more robust despite having less trades. The number of trades matters (especially when determining data-mining bias), but it does not determine how robust a system is.

The Markowitz optimization balances differences in frequency implicitly, due to the effect that higher/lower frequencies have on portfolio variance. A markowitz optimization procedure will give you a portfolio that balances both trading frequencies and system correlations. Systems that trade less will tend to have larger weights than less frequently trading systems. In general however I try to make my portfolios such that their weights are similar as I don't like to put a lot of weight on a few trades.
  • #184
  • Dec 26, 2014 7:07am Dec 26, 2014 7:07am
Quoting FXEZ
{quote} stt, I'm a huge fan of Marcos Lopez de Prado. I've read many of his papers but don't think I've seen this one, so thanks for the citation! The paper seems to consider a 'large amount of data' as time, as shown in Figure 2 rather than either bars or trades. I skimmed over some at the end so maybe the paper covers other measures. So how do HFT firms or those developing intraday strategies justify the use of shorter training periods in years, given this consideration? Are they defining "lots of data" in some other way, and who is right? Also,...
Consider that HFT firms do not trade the same type of systems we're concerning ourselves with here. The HFT firms trade statistical arbitrage strategies for which there is no need to demonstrate an inefficiency exists, because the inefficiency they exploit is purely statistical. This is why HFT do not have to concern themselves with things like mining bias, the very existence of the phenomenon they try to exploit guarantees that is is exploitable. The HFT firms play a market making game, provided they are able to reduce execution risk enough (execute fast enough) the edge is guaranteed (the market pays for this liquidity).

Discussing your other concern, you should concern yourself with determining mining bias, the probability that the results of your system search process have been the result of random chance (you found a "time bomb" instead of the result of a real inefficiency). Suppose you have 100 systems and from these you have 8 that are real while 2 are simply "time bombs". You cannot distinguish one from the other but you can tell that the probability that you have selected a time bomb is 20% (you can use several different techniques to measure data mining bias, I use a technique similar to the one described here). You can never tell if you do have time bomb or not (you cannot see the future) but you can make your system creation process such that the mining bias becomes extremely low. As a matter of fact I never trade anything unless the probability of having a "time bomb" is less than 0.001%.

Think that there are also several reasons why a system might fail. It might fail because it is a "time bomb" (the result of mining bias) but it can also fail because the market conditions it was created on are not a general reflection of the instrument you are trading (curve fitting bias). Imagine you create a system using only 2 months of data, you can get a result telling you that the probability that you have found a time bomb is very low (your mining bias can be close to zero) but then it fails when you trade it simply because market conditions for those two months are not a general representation of the pair's behavior (you curve fitted to a very limited set of market conditions). You should develop things that have both a very small probability to come from randomness and reflect general aspects of the financial instrument's behavior.

Success for me has been trading systems with very low mining bias created across a very wide range of market conditions (at least 20 years). I am by all means not saying this is the only way, but this is what has worked for me and it makes a lot of sense IMHO. This does not create systems that never fail (because future market conditions can be totally new, even when compared with the past 20 or more years) but it makes failing much less common, at least in my experience.
  • #185
  • Dec 26, 2014 11:53am Dec 26, 2014 11:53am
Quoting algoTraderJo
I am starting this thread hoping to share with you some of my developments in the field of machine learning. I will share with you ideas, results of my experiment and possibly other aspects of my work. I am starting this thread in the hopes that we will be able to share ideas and help each other improve our implementations.
I enjoyed reading all your posts in this thread. I am following your work with a great interest. I worked on machine learning based prediction of the market price few years ago. At the beginning I tried to predict the next day's close price (i.e., regression), but all my attempts failed: the predictive models return the previous day's close as the best guess for today's close!
Then I changed my target to predict the direction of H1 candles. I tried Bayes Classifer, K-Nearest Neighbors, Random Forest of Decision Trees, Support Vector Machines and Artificial Neural Networks. I chose H1 candles because they are independent of the broker's platform GMT time difference (as long as the time difference is a multiple of one hour). The result of classification was much better, a correct prediction rate of 64%. Unlike you, I do not use the previous N candles's direction as the input feature vector. I use the normalized difference values between two consecutive candles's high, low and close as the input features. What is the best prediction rate of all your experiments so far?
Trading is the hardest way to make easy money...
  • #186
  • Dec 26, 2014 12:13pm Dec 26, 2014 12:13pm
Quoting MathTrader7
{quote} I enjoyed reading all your posts in this thread. I am following your work with a great interest. I worked on machine learning based prediction of the market price few years ago. At the beginning I tried to predict the next day's close price (i.e., regression), but all my attempts failed: the predictive models return the previous day's close as the best guess for today's close! Then I changed my target to predict the direction of H1 candles. I tried Bayes Classifer, K-Nearest Neighbors, Random Forest of Decision Trees, Support Vector Machines...
I'm glad you have found the thread interesting!

Yes, this is a well known and somewhat expected problem. If you try to predict the next day's close using closing prices machine learning algorithms will always predict the best prediction to be the previous close because the close price series is heavily auto-correlated. Since variations between consecutive closes are most often very small, the machine learners train to predict that price won't move (which is not very useful!). In order to make predictions you must eliminate the auto-correlation in the series by using returns (or something alike returns) instead of prices. If you want to predict using regressions you should predict a return target, rather than a future price (this is what I have done with the linear regression and random forest models on this thread). There are other tricks, which we will talk about later on.

You can use a wide array of inputs to make predictions. I decided to stick with returns for regression models and binary directions for classifiers to make things simpler, but inputs can get more and more complicated. Nonetheless you should always consider whether an input adds or doesn't add any information (see for example what happens with the RSI example I posted in the beginning of the thread). Random forests are particularly good at judging input quality but we can do so in many other ways (another topic for the future of this thread).

About prediction accuracy, this is mostly irrelevant. You can have models with more than 80% accuracy that end up losing money because in the end you don't care about whether you can predict or not but whether you make more money than you lose from your predictions. The market is not efficient to successful prediction, it is most efficient to making more money than you lose from predictions. Unless your predictions are related very tightly with the amount of pips you make or lose per trade (your prediction target is exactly your trade success) then your accuracy from a machine learning stand-point is largely irrelevant. A few of my best models have sub-40% accuracy but they are very profitable, largely because the predictions that are correct lead to very profitable trades (almost 4 times as much as losers).

When building ML systems do not consider your model and your trading independently, you should always build systems so that you can take a look at the whole picture. Do ask other questions if you desire to have my opinion on something else.
  • #187
  • Dec 26, 2014 12:27pm Dec 26, 2014 12:27pm
On the last posts using random forests for regressions we implemented a model using returns from the EUR/USD, GBP/USD and USD/JPY to make predictions. In the same manner we can create a more complex classifier that uses past bar directions (1 bullish, 0 bearish) from these three symbols to make predictions. We can use an SVM with a C=10 and a gamma=0.001 to do this (Gaussian Kernel). The model is trained on each bar with the directions of the past 5 bars on the three symbols (a total of 15 inputs per example) using the past 75 input/target examples. The target for this model is simply the next day's direction (as we had initially used for our SVM). We enter trades based on the predicted direction at the start of each daily candle.

I also want to take this opportunity to look at the effect of different SL values. The SL has an important effect on system statistics mainly because it strongly affects trading frequency. A tighter SL generates more trades while a more relaxed SL makes trade exits rely more heavily on reverse signals. You can see a comparison of 40, 50, 60 and 80% of the ATR20 stops for this system.

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_svm_15inputs_stops.png Size: 40 KB


If you look at the daily return Sharpe ratios (calculated using the standard deviation) you have the following values (multiply by sqrt(252) to get yearly values):

40% ATR20 = 0.035
50% ATR20 = 0.058
60% ATR20 = 0.068
80% ATR20 = 0.074

The best system from a risk adjusted point of view is definitely the system with the highest SL because despite having a much lower total profit it also has a much less pronounced standard deviation. This also highlights why it is so important to consider risk adjusted returns instead of only total profit, the absolute final profit is strongly misleading. The Sharpe and Sortino ratios are my favorite statistics when evaluating trading strategy performance.
  • #188
  • Dec 26, 2014 1:05pm Dec 26, 2014 1:05pm
Quoting algoTraderJo
{quote} About prediction accuracy, this is mostly irrelevant. You can have models with more than 80% accuracy that end up losing money because in the end you don't care about whether you can predict or not but whether you make more money than you lose from your predictions.
I agree with you that the prediction accuracy is (to some extent) irrelevant to the profitablity of the model in the Forex market trading. However, the accuracy has a great impact on, for example, Binary Options trading.

Quoting algoTraderJo
{quote}Do ask other questions if you desire to have my opinion on something else.
I noticed that you train a machine learning paradigm at each bar. In comparison, I divide the data (<feature vectors, labels>) into two parts, 70% of the data is randomly chosen for training while the rest of 30% are used to examine the performance of the model on unseen samples. In other words, I'm looking for a general model of the market that has been trained once and can be used for some duration in the feature before re-training will be done (similar to the idea of Walk Forward Optimization).
Trading is the hardest way to make easy money...
  • #189
  • Dec 26, 2014 1:16pm Dec 26, 2014 1:16pm
Quoting MathTrader7
{quote} I agree with you that the prediction accuracy is (to some extent) irrelevant to the profitablity of the model in the Forex market trading. However, the accuracy has a great impact on, for example, Binary Options trading. {quote} I noticed that you train a machine learning paradigm at each bar. In comparison, I divide the data (<feature vectors, labels>) into two parts, 70% of the data is randomly chosen for training while the rest of 30% are used to examine the performance of the model on unseen samples. In other words, I'm looking for a...
This is the manner in which most academic papers approach machine learning. However it creates many problems related with the selection (what data is used for training, how often to train, when to train, etc), in my experience this leads to mining bias problems and poor forward trading results (live results with statistics very different from the "test period" because the test period covers only a very limited set of market conditions). You also get the classic problem of data reuse with in sample/ out of sample testing (that I have mentioned before on this thread), because you will most likely try many models before you arrive at something that happens to give you the statistics you want across both training and testing sets. For me all these issues were insurmountable.

I have found that developing systems using constant retraining (retrain on each bar) gives the best overall results. The simulations also tend to have realistic performance (I tend to get similar results in live trading as I do in backtesting, considering I use very long simulations and test the systems to have low data-mining bias). I would also like to make it very clear that I am not saying that what I do is the best thing that can be done, but it is what has worked for me. With enough due diligence you might be able to get a method like the one you suggest to work, I just have found it much harder when compared with my constant retraining method.
  • #190
  • Dec 26, 2014 1:40pm Dec 26, 2014 1:40pm
Quoting algoTraderJo
machine learning algorithms will always predict the best prediction to be the previous close because the close price series is heavily auto-correlated. Since variations between consecutive closes are most often very small, the machine learners train to predict that price won't move
Where have you seen this heavy auto-correlation? I never found any. ARMA models' coefficients change almost completely at every bar. Redlion opened a thread because up to 160 bars he couldn't find any significant auto-correlation. The few articles I found about this consider the auto-correlation so small that it is totally neglictable, even for SP500 which clearly shows a positive drift.

The reason why the best prediction is the current price is exactly due to the fact that there is no auto-correlation. Even during trends. The empirical probability distribution of the price return is almost perfectly symmetric about 0 which is also the mode of the distribution: the maximum likelihood of the price return is therefore 0.
Attached Image (click to enlarge)
Click to Enlarge Name: histogram_EUR_H4.png Size: 28 KB
No greed. No fear. Just maths.
  • #191
  • Dec 26, 2014 1:41pm Dec 26, 2014 1:41pm
Quoting PipMeUp
{quote} Where have you seen this heavy auto-correlation? I never found any. ARMA models' coefficients change almost completely at every bar. Redlion opened a thread because up to 160 bars he couldn't find any significant auto-correlation. The few articles I found about this consider the auto-correlation so small that it is totally neglictable, even for SP500 which clearly shows a positive drift. The reason why the best prediction is the current price is exactly due to...
Yes, returns have very little auto-correlation. Do the same exercise with close prices (not returns) you will see a very heavy auto-correlation.
  • #192
  • Dec 26, 2014 1:48pm Dec 26, 2014 1:48pm
This is the auto-correlation of Oanda daily close prices for lags from 0 to 26 on the EURUSD. It is huge. Returns are very poorly correlated but prices are very heavily correlated because variations from one price to another are extremely low. This gives absolutely no predictive power, it just means that you have a series where the next value tends to be close to the last. This is why it is so important to always use returns instead of absolute prices.

Attached Image (click to enlarge)
Click to Enlarge Name: acf_oanda_eurusd_daily.png Size: 10 KB


You can very easily reproduce this in R using quantmod,

Inserted Code
library(quantmod)
getSymbols("EUR/USD", src="oanda")
acf(EURUSD)
  • #193
  • Dec 26, 2014 1:49pm Dec 26, 2014 1:49pm
I don't expect to find anything different from
Attached Image

EDIT: BTW I've never been able to make some use of this fact. It only makes really harmful problems with Kalman filters. I call that "flip a coin paradox" as I don't know if the phenomenon has a name or not.
No greed. No fear. Just maths.
  • #194
  • Dec 26, 2014 1:52pm Dec 26, 2014 1:52pm
getSymbols("EUR/USD", src="oanda")
pacf(EURUSD) # <--- (partial ACF,I don't know the name in R)
No greed. No fear. Just maths.
  • #195
  • Dec 26, 2014 2:01pm Dec 26, 2014 2:01pm
Quoting PipMeUp
I don't expect to find anything different from {image} EDIT: BTW I've never been able to make some use of this fact. It only makes really harmful problems with Kalman filters. I call that "flip a coin paradox" as I don't know if the phenomenon has a name or not.
We're talking about the same thing. You're saying that the next prediction is always close to the last price (a zero return) because a zero return is the most likely scenario. This is precisely the problem that is exacerbated when you attempt to predict a price instead of a return. You can use any regression model to see the big difference that there is when using the past N close prices to predict the next close price or the past N returns to predict the next return value. Returns will always give you better results. Even though the information contained is fundamentally the same, the machine learning models training/targeting prices have a much harder time in getting away from this zero return issue than the models using returns.

I think this also has a lot to do with the additional noise contained due to the price's absolute value. If you train using 100 examples some examples will be far away from the current price and therefore the machine learning model will see them as quite different. Two examples might have fundamentally the same input/target in terms of returns while in terms of absolute price the values are different, this creates a much more complicated case for ML.

With all that said, my conclusion from experience has been that you should always use returns (or something along these lines), never use prices as inputs or targets.
  • #196
  • Dec 26, 2014 2:01pm Dec 26, 2014 2:01pm
Quoting algoTraderJo
{quote} I have found that developing systems using constant retraining (retrain on each bar) gives the best overall results. The simulations also tend to have realistic performance (I tend to get similar results in live trading as I do in backtesting, considering I use very long simulations and test the systems to have low data-mining bias).
I have to admit that I have never examined constant retraining in my experiments, but your idea seems very promising for me. One reason is that, your idea can somewhat deal with the problem of non-stationary of the market time series. I will modify my codes accordingly such that at each bar they will retrain the model. I will post my results later on this thread. Hopefully the trick will increase the successful prediction rate. Thanks for your comments and hint! :-)
Trading is the hardest way to make easy money...
  • #197
  • Dec 26, 2014 2:06pm Dec 26, 2014 2:06pm
Quoting MathTrader7
{quote} I have to admit that I have never examined constant retraining in my experiments, but your idea seems very promising for me. One reason is that, your idea can somewhat deal with the problem of non-stationary of the market time series. I will modify my codes accordingly such that at each bar they will retrain the model. I will post my results later on this thread. Hopefully the trick will increase the successful prediction rate. Thanks for your comments and hint! :-)
You are most welcome. Keep the questions and discussion coming
  • #198
  • Dec 26, 2014 2:25pm Dec 26, 2014 2:25pm
Quoting algoTraderJo
Let us make this thread a little bit more interactive Post some ideas you would want me to try to improve the machine learning results I have posted so far (predictions on the daily timeframe on the EURUSD), I will try them out and will post the results here!
Have you examined the performance of a Fuzzy classifier in your experiments?
Trading is the hardest way to make easy money...
  • #199
  • Dec 26, 2014 2:45pm Dec 26, 2014 2:45pm
Quoting MathTrader7
{quote} Have you examined the performance of a Fuzzy classifier in your experiments?
Yes, but I want to reserve Fuzzy classifiers for when we move to lower time frames when they become more important.
  • #200
  • Dec 26, 2014 4:09pm Dec 26, 2014 4:09pm
Quoting algoTraderJo
This is precisely the problem that is exacerbated when you attempt to predict a price instead of a return.
Can you explain this sentence much better please? Preferably with mathematical proof. Predicting the next price is exactly the same thing as predicting the next return since the return is the difference between two consecutive prices. Based on the price a lin.reg. model will simply predict the continuity of a straight line; while based on price return it will predict the continuation of the mean of this series, which is nothing else than the slope of the straight line.
Also taking the difference of the signal increases the variance of the noise.

I wish I could check by myself but I can still not reproduce any of your results. I extended the dataset to July 2002 by using 1min data interpolation. Irregardless of the timezone (I tried Paris, London, NY and GMT) the result is edgeless.
No greed. No fear. Just maths.
0 traders viewing now