Machine Learning with algoTraderJo | Page 15 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #281
  • Jan 21, 2015 2:14am Jan 21, 2015 2:14am
Quoting algoTraderJo
To continue moving in the exploration of NN I am now going to show you the effect of making variations in the hidden and output functions as well as the number of optimization cycles per training for the NN using 100 examples to train (last three bar returns as inputs, next bar's return as target), remember that training of the models happens on every bar. The balance chart below shows you a comparison between FastSigmoid (FS), Linear (L) and Hyperbolic Tangent (HT) functions at different optimization cycle numbers. {image} There is a very large...
Regarding ensemble: there is a strong correlation between the two equity curve, so it (the ensemble) can not reduce the drawdown significantly.
My idea: if we could find two NN algorithm with not so strict correlation than the ensemble could function better.

Reminds me if you had short position in two correlated pairs than it is almost the same as you had double in one pair (ok, slightly less loss).
  • #282
  • Jan 21, 2015 5:56am Jan 21, 2015 5:56am
Quoting anfego22
Hi, great thread I have been reading all the day! I have something I like to comment: Foreach day of the sample, you are taking the 100 past days and training three different kinds of NN with one hidden layer, using feed forward estimation and stopping in different cyclyes (500, 1500), is that right? Can you explain me how ensembles is done? it is related to combine the forecast of the two models? One good thing todo do is to compare this model against a random walk. In ecomometrics this is usually done when you are trying to proof the predictibility...
I am not taking the past 100 days, I am taking the past 100 examples. Each example uses the past three day returns as inputs and the next day's return as target. I am using an IRpropPlus optimizer to optimize the feed forward neural network to a given number of cycles (see Shark library page for more information on the IRpropPlus optimizer).

The ensemble simply combines model forecasts to make decisions only if the forecasts of both models align in the same direction.

Is it more important to predict volatility? Is the model giving a worse performance than what you call a "naive" model? Run some experiments and post your results, I will be happy to discuss them.
  • #283
  • Jan 21, 2015 6:00am Jan 21, 2015 6:00am
Quoting emeafx
{quote} Regarding ensemble: there is a strong correlation between the two equity curve, so it (the ensemble) can not reduce the drawdown significantly. My idea: if we could find two NN algorithm with not so strict correlation than the ensemble could function better. Reminds me if you had short position in two correlated pairs than it is almost the same as you had double in one pair (ok, slightly less loss).
This is actually a very common misconception. Correlation does not determine whether an ensemble will work or not work. The correlation coefficient between the two models in the ensemble that gives worse results is actually not very high (0.53) which tells you that correlation is not telling you the entire story. Sometimes you have models with very high correlations (>0.9) that actually give better performance when combined in an ensemble. Whether you get better or worse results in an ensemble is not so easy to predict because an ensemble is not a portfolio (where correlations do play a big role).

An ensemble modifies trading results because we are changing signals, therefore the results of the ensemble are not the addition of the results of the models used, it is more complex than that. An ensemble works better if signals within drawdown periods mismatch (trading is avoided) while signals in profitable periods align (we take trades). Many times a high correlation is even a better predictor of an ensemble that works than one that does not. Can you think of any other metric that can help us predict the success of an ensemble? (hint: unrelated to correlations).
  • #284
  • Jan 21, 2015 7:11am Jan 21, 2015 7:11am
Quoting algoTraderJo
{quote} This is actually a very common misconception. Correlation does not determine whether an ensemble will work or not work. The correlation coefficient between the two models in the ensemble that gives worse results is actually not very high (0.53) which tells you that correlation is not telling you the entire story. Sometimes you have models with very high correlations (>0.9) that actually give better performance when combined in an ensemble. Whether you get better or worse results in an ensemble is not so easy to predict because an ensemble...
OK, got it. :-)

What about combining faster reacting a slower reacting systems (maybe not 2 but more) to an ensemble.
My biggest concern to avoid long deep DD periods even if it makes lower (but steadier ) returns.
Ideally the ensemble should produce a straight line as an equity curve.
  • #285
  • Jan 21, 2015 8:12am Jan 21, 2015 8:12am
Quote
We can actually see a deterioration in results. Why do you think this is the case?
My guess is that the two NN kill each other. Say we have an up trend in place.
- Both Algo1 and Algo2 say LONG. You open a LONG position.
- Let's say the next two days they keep agreeing on LONG. The position is kept open.
- The day after Algo2 says Short but Algo1 keeps saying LONG. There is a divergence so you close the position. Market pulls back a little.
- Algo1 and Algo2 both say LONG. You re-open a LONG position. This position is stopped out for a loss.
- After the trend resumes.

In this scenario, Algo1 had his position closed too eary and Algo2 missed a winning trade which would have covered the loser...
No greed. No fear. Just maths.
  • #286
  • Jan 21, 2015 9:24am Jan 21, 2015 9:24am
Quote
We can actually see a deterioration in results. Why do you think this is the case? Why are sometimes ensembles so beneficial and other times so counter-productive? Can we tell in advance if an ensemble will be better or worse? Penny for your thoughts

Yes I've seen that phenomenon as well as the inverse. When system results are worse as a result of a combination, it is sort of like an inverse Parrondo's Paradox. I guess the answer as to why is that it depends on the exact rules you are using for your ensemble combination. PipMeUp gave an example where opposing orders are interpreted as a flat signal. You could also interpret opposing orders as a 'status quo' signal to stay with the last signal. Just this small change in rules will have a big impact on the end results all else being equal.

Additionally, the case presented is of 2 systems forming an ensemble. Because there are offsetting ties (long vs short) it has a very different dynamic than combining 3 strategies, for instance. In the case of 2 strategies the system that gives slower signals will dominate the system that gives faster signals because both must agree to have a position. In the case where opposing signals are interpreted as 'status quo' or stay with the last signal, and where you have one system constantly switching positions faster and one system constantly slower in switching positions, the ensemble will essentially just replicate the slower system, so the slower system totally dominates the faster system.

If you have 3 systems, the slowest switching system is dominated by the other two systems so the dynamic is quite different. With an odd number of combined systems, you will have a higher trade count with the ensemble. With an even number of combined systems, the trade count is lower and as a result, the average profit per trade tends to be higher. This even/odd effect decreases as more systems are added to the ensemble. However, I know of no way to anticipate which set of combined systems will improve results other than to test the combinations and the specific rules of the ensemble. If you know of a way, I'd really love to hear it.

And you're correct - correlation of returns or equity curves isn't really that important here - its more correlation of trade signals that matters. But even there you can't tell the profitability of the outcome just by the correlation calculation - you have to perform the combination to know if it improves or worsens the result.
  • #287
  • Jan 22, 2015 12:16pm Jan 22, 2015 12:16pm
Quoting emeafx
{quote} OK, got it. :-) What about combining faster reacting a slower reacting systems (maybe not 2 but more) to an ensemble. My biggest concern to avoid long deep DD periods even if it makes lower (but steadier ) returns. Ideally the ensemble should produce a straight line as an equity curve.
All systems here are equally reacting. They all make one prediction on every daily bar and they always make either a short or long prediction. In this manner all systems generate an equal amount of signals, therefore success or failure of the ensemble has in this case nothing to do with frequency.
  • #288
  • Jan 22, 2015 12:17pm Jan 22, 2015 12:17pm
Quoting PipMeUp
{quote} My guess is that the two NN kill each other. Say we have an up trend in place. - Both Algo1 and Algo2 say LONG. You open a LONG position. - Let's say the next two days they keep agreeing on LONG. The position is kept open. - The day after Algo2 says Short but Algo1 keeps saying LONG. There is a divergence so you close the position. Market pulls back a little. - Algo1 and Algo2 both say LONG. You re-open a LONG position. This position is stopped out for a loss. - After the trend resumes. In this scenario, Algo1 had his position closed too...
Depends strongly on how you interpret ensemble results. In this case I did not take a lack of agreement as a close signal but merely as a "don't do anything" signal. Therefore open positions remain open if the ensemble components disagree.
  • #289
  • Jan 22, 2015 12:20pm Jan 22, 2015 12:20pm
Quoting FXEZ
{quote} Yes I've seen that phenomenon as well as the inverse. When system results are worse as a result of a combination, it is sort of like an inverse Parrondo's Paradox. I guess the answer as to why is that it depends on the exact rules you are using for your ensemble combination. PipMeUp gave an example where opposing orders are interpreted as a flat signal. You could also interpret opposing orders as a 'status quo' signal to stay with the last signal. Just this small change in rules...
You have made some excellent points. You are right in that ensemble results depend heavily on how you interpret different scenarios, it also depends heavily on how many systems you use. There are many different ways to interpret ensemble results and translate them into signals. You can take majority votes, you can take an "only act on unanimous predictions" approach, etc. You can however anticipate if an ensemble will give you favorable results... Tell me if anything comes to mind!
  • #290
  • Jan 22, 2015 4:31pm Jan 22, 2015 4:31pm
Quoting algoTraderJo
{quote} You have made some excellent points. You are right in that ensemble results depend heavily on how you interpret different scenarios, it also depends heavily on how many systems you use. There are many different ways to interpret ensemble results and translate them into signals. You can take majority votes, you can take an "only act on unanimous predictions" approach, etc. You can however anticipate if an ensemble will give you favorable results... Tell me if anything comes to mind!
Thanks. I didn't come up with anything so I cheated by looking on Wikipedia:

Quote
Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7]...

So it appears to be "diversity" but there is some question regarding how to measure it. After following one of the footnotes to a paper it appears that finding the "best" measure of diversity that leads to high predictions is unclear. So I presume you have learned what works from experimentation in this area, rather than from research. I'm on the edge of my seat!
  • #291
  • Jan 23, 2015 6:59am Jan 23, 2015 6:59am
I would like to propose an exercise. Here are the results of a neural network that uses the next bar's return as target and a neural network which is equal in every other way except for the fact that it predicts the bullish/bearish excursions within the next 10 bars (uses two output neurons instead of one). On the first we enter a long trade if the next bar is predicted to be bullish while we enter a short if the next bar is predicted to be bearish. On the second we enter a long trade if the bullish excursion is larger than the predicted bearish excursion and we enter a short trade in the opposite scenario. Both neural networks make one prediction per daily bar and have the networks retrained on each bar using the past 100 examples (3 bar returns as inputs). These are the results:

Attached Image (click to enlarge)
Click to Enlarge Name: machine_learning_nn_6.png Size: 29 KB


I also attach a zip file with the daily returns for the entire test for both neural networks. The challenge is to predict (giving a mathematical argument) if an ensemble will give better or worse results. Both tests use the exact same exits on trades (60% of the ATR20) and manage signals in the same way (signals in opposite directions cause trades to be closed, signals in the same direction cause the SL to be updated). All analysis welcome
Attached File(s)
File Type: csv NN_results.csv   157 KB | 344 downloads
  • #292
  • Jan 23, 2015 1:48pm Jan 23, 2015 1:48pm
Quoting algoTraderJo
I would like to propose an exercise. Here are the results of a neural network that uses the next bar's return as target and a neural network which is equal in every other way except for the fact that it predicts the bullish/bearish excursions within the next 10 bars (uses two output neurons instead of one). On the first we enter a long trade if the next bar is predicted to be bullish while we enter a short if the next bar is predicted to be bearish. On the second we enter a long trade if the bullish excursion is larger than the predicted bearish...
in this case, looks like ensemble will give worse performance. At a simple level, if sum of your returns on days when the sign on two model returns is opposite is > 0 then your ensemble will underperform.

so essentially you say if sum[(sign(m1_return)*sign(m2_return)<0?1:0)*(m1_return+m2_return)] < 0, then ensemble will be better.

this is simplified (because your ensemble carry the position from last day when there is a disagreement instead of flat position when disagreement) but i think this still applies.
Think : Trade : Live Life
  • #293
  • Jan 23, 2015 3:04pm Jan 23, 2015 3:04pm
algoTraderJo,

I see you have given us a homework assignment. Thank you teacher! I used Excel for this analysis.

If I sum the two cumulative return series I get the following in terms of return and max dd:
Inserted Code
Column        MaxDD         LastPnL       PnL/DD
NN1:          0.3152        3.0843        9.8
NN2:          0.3385        2.3624        7
(NN1+NN2)/2   0.2007        2.7234        13.6
Markowitz     0.1982        2.7073        13.7
Where the last column is ending pnl / maxdd or a measure of risk adjusted return where higher is better. Adding the two curves clearly shows some promise, at least in terms of risk adjusted return. So while this doesn't answer the question that was posed, the linear equity curve combination at least shows promise in terms of calculating equal weights and Markowitz / Kelly weights to fit to the efficient frontier and gives an idea of what numbers to beat in order to show improvement.

stt is right about under-performance when the signs of the returns differ between NN1 and NN2: =IF(AND(B3<>0,C3<>0),(B3+C3)/2,0)
Inserted Code
Column        MaxDD         LastPnL       PnL/DD
Column: 6      0.3971        0.2643        0.7
So this probably rules out an ensemble that stays with status quo when the signals disagree as stt states.

If I assume that trades are only taken when both returns are opposite then I get: =IF(SIGN(B3)=-SIGN(C3),0,(B3+C3)/2)
Inserted Code
Column        MaxDD         LastPnL       PnL/DD
Column: 7      0.2126        2.0509        9.6
So these results are really no better than just trading NN1 from a total return and a risk adjusted return basis, so no improvement here.

And finally, if I only take trades when the signs agree (100% unanimous ensemble), then I get: =IF(SIGN(B3)=SIGN(C3),0,(B3+C3)/2)
Inserted Code
Column        MaxDD         LastPnL       PnL/DD
Column: 8      0.0682        3.1315        45.9

This last example would be similar to an ensemble that requires both to agree in order to place a trade. Results here are off the charts better. So assuming that this analysis is correct, and assuming this example is representative of how the rules would be implemented with the actual trade series, it pays to check out different ensemble rules.
  • #294
  • Jan 23, 2015 5:45pm Jan 23, 2015 5:45pm
Quoting stt
{quote} in this case, looks like ensemble will give worse performance. At a simple level, if sum of your returns on days when the sign on two model returns is opposite is > 0 then your ensemble will underperform. so essentially you say if sum[(sign(m1_return)*sign(m2_return)<0?1:0)*(m1_return+m2_return)] < 0, then ensemble will be better. this is simplified (because your ensemble carry the position from last day when there is a disagreement instead of flat position when disagreement) but i think this still applies.
Quoting FXEZ
algoTraderJo, I see you have given us a homework assignment. Thank you teacher! I used Excel for this analysis. If I sum the two cumulative return series I get the following in terms of return and max dd: Column MaxDD LastPnL PnL/DD NN1: 0.3152 3.0843 9.8 NN2: 0.3385 2.3624 7 (NN1+NN2)/2 0.2007 2.7234 13.6 Markowitz 0.1982 2.7073 13.7 Where the last column is ending pnl / maxdd or a measure of risk adjusted return where higher is better. Adding the two curves clearly shows some promise, at least in terms of risk adjusted return. So while this doesn't...
Good posts! Nonetheless there are some important conceptual problems with the hypothesis you have made. I think it would be most interesting if you look at the actual results of the ensembles and tell me how these compare to your predictions and what you might have misinterpreted or miscalculated. Here are the results for doing an N1+N2 equally weighted portfolio, an ensemble when trades are closed on disagreement and an ensemble when we do nothing on disagreement. Since we have only two neural networks we take long/short signals when they both agree (since the possible outcomes are +2 (both say long), -2 (both say short) or 0 (they disagree)).

Attached Image (click to enlarge)
Click to Enlarge Name: machine_learning_nn_7.png Size: 40 KB


Attached Image


To understand ensembles we must understand why the results are so different.

  1. Why does the decision to keep or close trades on disagreement have such a small effect?
  2. Why are portfolio and ensemble results so different?
  3. Why is the total return of the ensembles so much lower than the return of the individual NN and the portfolio?

Tell me what you think are the answers to these questions. Having clear answers to these questions will allow you to see in a much more transparent way how you can predict whether an ensemble will be better or worse by looking at the individual results of the different models.

  • #295
  • Jan 24, 2015 10:54am Jan 24, 2015 10:54am
This is my guess

The portfolio combines linearly the two forecast and creates a new one (which in teory is unbiased and has less variance), the ensembles is a rule that needs both forecast agree in order to trade, in order to have a correct trade, then you'll need not only that both forecast agree but that the two forecast came with a right prediction, wich I believe this event has less probability than a individual model.

Also I would expect NN1 to have greater profitability than NN2 because NN2 uses a bigger forecast period which results in a bigger forecast error.

PS: Have anyone try to backtest a strategy in ? I'm trying to use package quantstrat but I haven't manage to do it. Does someone know a way to backtest a strategy in R or in other free software? Thanks.
  • #296
  • Jan 25, 2015 7:35am Jan 25, 2015 7:35am
Quoting anfego22
This is my guess The portfolio combines linearly the two forecast and creates a new one (which in teory is unbiased and has less variance), the ensembles is a rule that needs both forecast agree in order to trade, in order to have a correct trade, then you'll need not only that both forecast agree but that the two forecast came with a right prediction, wich I believe this event has less probability than a individual model. Also I would expect NN1 to have greater profitability than NN2 because NN2 uses a bigger forecast period which results in a...
When you put two models in an ensemble you will surely get a lower trading frequency (because the two models are bound to disagree under some conditions), however this does not in principle guarantee worse results (read about statistics like the "information ratio" that allow you to measure if a reduction in frequency is worth it or not). I have showed in several places within this thread that an ensemble can indeed outperform the outcomes of the two separate systems and the two system portfolio. If an ensemble performs better or worse than the separate systems or the portfolio has nothing to do with the things you mention. I suggest you look deeper into the data and give a mathematical argument.

Forecasting ability also does not have a clear direct relationship with the forecasting horizon. It is true that forecasting error increases as a function of forecasting time if you are attempting to forecast a random walk, but if price series don't follow a perfect random walk then forecasting accuracy is not necessarily tied to the forecasting horizon in this manner. You can easily see this if you run experiments using neural network or SVM models on real price series. Try to forecast returns for 1, 2, 3, etc bars in the future, you will see that your forecasting ability reaches a peak beyond bar 1 (follows a somewhat parabolic function). Look at the SVM experiments on this thread at different target prediction values, peak forecasting ability is reached when attempting to predict 4 bars into the future (not 1). You need to be very careful to take hypothesis as facts.I advice you always run experiments before making statements.

If you're looking into reproducing these results I would strongly advice against using R alone. I used R for a long time but then it was a pain when I needed to move to live trading. If you're going to go into machine learning use a programming framework that you can use for both back-testing and live trading (so that you have the exact same code for both things). This means that you will need to probably code something from scratch (as several traders here have mentioned) or, if you don't have the time/skill to do this, simply buy a solution that already provides the needed functionality (like the Asirikuy framework).
  • #297
  • Jan 26, 2015 3:27am Jan 26, 2015 3:27am
Hi algoTraderJo

I have just found your thread after getting interested in the topic of forex and using some sort of statistical/machine learning to find predictors. But wanted to ask are you doing statistical learning on the past market performance or looking at the financial calender and using the expected vs actual numbers as predictors?

Thanks
Giles
  • #298
  • Jan 26, 2015 5:25am Jan 26, 2015 5:25am
Quoting gileslewis
Hi algoTraderJo I have just found your thread after getting interested in the topic of forex and using some sort of statistical/machine learning to find predictors. But wanted to ask are you doing statistical learning on the past market performance or looking at the financial calender and using the expected vs actual numbers as predictors? Thanks Giles
Please read the thread, it explains in detail the input/targets I am using for the training of machine learning models.
  • #299
  • Jan 26, 2015 9:52am Jan 26, 2015 9:52am
This info will only be of use to members who write their own software. If you want to use the functions in R, such as plotting or NN such as AMORE, you can do a data dump from your software to a file format which can be read into R with the read.table or other function. Then your software can shell out an R script, run it on R using a batch file (R scripts can be run from the command line) and have the R script dump the result to a file which is read by your software. My description sounds more complicated than it really is. The only caveat is that the file exchanges between R and your software will slow the overall process. But in many cases that will not be a serious drawback.

PM.
  • #300
  • Jan 26, 2015 12:10pm Jan 26, 2015 12:10pm
Quote
When you put two models in an ensemble you will surely get a lower trading frequency (because the two models are bound to disagree under some conditions), however this does not in principle guarantee worse results

Yes, thats right if what I was saying was true, then any ensembles wouldn't work because of that. What I think now, is that an ensembles in this case wouldn't work because both NN are constructed the same way, so if there is a behaviour of the series that can't be capture by the NN structure, both NN will miss that in their forecast and could have bad trading results in the same time. I think that an "ideal" ensembles will be one that when NN1 get losses then NN2 take profits, what I wouldn't prefer is that both neural networks get losses.

This is a table of the proportion of the signs of returns:

Attached Image


If you see 41% of the time both NN give positive results, 23% of the time both give bad result, 21% NN2 give positive returns and NN1 gives negative results. We can't see how bad/good the returns are in this table, so I divide the returns in three quantiles and here are the results:

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot2.png Size: 20 KB


What I can see from this table is that 16% of the time when NN1 have the worst returns, NN2 have the worst results too and only 3% of the time when NN2 give the worst performance NN1 gives the best returns.
I think this is the reason why ensembles doesn't work well in this case, I'm going in the right path here?

Quote
You need to be very careful to take hypothesis as facts.I advice you always run experiments before making statements.
I will follow this advice. Thanks
0 traders viewing now