Machine Learning with algoTraderJo | Page 12 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #221
  • Jan 6, 2015 7:41am Jan 6, 2015 7:41am
So far when using classifiers to predict returns (1 bullish, 0 bearish) we have only looked at predicting the direction of the next bar's return. However we can also look at attempting to predict the return of the next N bars, making a prediction every day and applying the same rules for trading as we did before. The following are examples using only the past 8 bar directions (1, 0) as inputs and retraining using 30 examples on each bar. The target is the direction of the total return over the next N bars (1 bullish, 0 bearish). These results are on the EURUSD daily chart. I used C=1 and gamma=1 for these runs.

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_differentTargets_svm.png Size: 40 KB


The target has a big impact on the results of the simulation. We can see that when predicting direction for the first 1, 2, 3 and 4 bars we tend to get the bulk of our profit in the 1992-2011 period while for the 5 bar return we tend to get the bulk of positive performance on the 2008-2014 region. We have now found something that makes a quite dramatic difference in returns under recent market conditions. Can we make the 4 day target work for 1992-2000? Could we merge the models somehow to obtain better results? How would you approach this problem?
  • #222
  • Jan 6, 2015 8:54am Jan 6, 2015 8:54am
By increasing the number of examples used for training, the number of inputs used per example (using more past bar directions) and placing the C at 10 and the gamma at 0.001 (train much harder at a finer level of detail) we can get much better results for the 4 bar direction target model. Remember that we always retrain the model on every bar. The image below shows you a comparison between the two results:

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_better4bartarget_svm.png Size: 30 KB


We now have a result that is homogeneous across the last 22 years, with high linearity of time Vs log(balance) (remember that the Y axis is logarithmic on the graph). This is a good base we can use to introduce further improvements to refine our classification efforts. We can try some of the things we have already seen within the thread, such as adding inputs from other symbols. What do you think could further improve these results?
  • #223
  • Edited 4:28pm Jan 6, 2015 3:58pm | Edited 4:28pm
Let's experiment with how the results change when adding more inputs. The following results show you the same model as on my last post although the inputs have been changed to include a series of input variations. We try making predictions with EU data, with GU data, with UJ data and with different mixtures of EU/GU/UJ sets. All the backtests are executed on the EURUSD so the tests with only GU and UJ inputs are in fact attempting to predict EU outcomes by looking only at data from other pairs. Remember that our target is the net direction for the next 5 bars and we retrain on everyday using 145 past examples.

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_svm_4bar_manyinputs.png Size: 36 KB


This graph tells you a few things. For starters we can see that adding GU or UJ information to EU inputs makes little difference for this model, we cannot justify the increase in model complexity with better trading results. Moreover we can see that we can still get good results on EU predicting only with GU inputs and/or UJ inputs, as they both seem to provide a similar amount of information (information from the USD, which is most critical here).

I like this exercise to show the high degree in which the FX market is related. There is little information available on other FX pairs that is useful to increase ML profitability on the EURUSD when using an SVM classifier like the one we tested here (as the model is able to get all information efficiently) . Perhaps there is some information on the EURUSD we still haven't looked at that can be used to increase prediction accuracy? What other inputs unrelated with the Open-to-Open relationships can you think about?
  • #224
  • Jan 6, 2015 4:20pm Jan 6, 2015 4:20pm
The above results also make us wonder about the need for such complexity on the SVM model. Would a K-NN perform in a similar manner? You can see a comparison of the above same input/target structure using a K-NN. Predicting a 5 day direction Vs a 1 day direction might require different levels of complexity.

Attached Image (click to enlarge)
Click to Enlarge Name: machine_learning_4bar_svm_vs_knn.png Size: 29 KB


What we have here is that the complexity of the SVM classifier (the ability to find highly non-linear relationships) is now fundamental to our trading results. We can also use random forest classifiers to see if we can improve performance. Do you think this will work?
  • #225
  • Jan 6, 2015 7:54pm Jan 6, 2015 7:54pm
u need a speed
  • #226
  • Jan 7, 2015 5:42am Jan 7, 2015 5:42am
Quoting algoTraderJo
{quote} It's very nice to see your results They actually closely match what I get for that same period using an SVM and the settings you have suggested. I was in fact very suspicious when I saw your 1H results as in my experience the lower TFs are much trickier (we will eventually get to them).
I'm still very suspicious to all the backtesting in the fact. I tried to SVM with prediction 3rd day based on two previous with training on 10 previsou days. With stop loss 0.5 ATR and Take profit 5 ATR. And I'm very suprised because of the difference in results. I'm attaching the result from MT4 on Daily 1989-2015 with Open Prices and Control points. The difference is weird.
I do not have enough memory for backtest of control points for all the data. So I'm a bit unsure what to trust.
Attached Image(s) (click to enlarge)
Click to Enlarge Name: StrategyTester_SVM_DAILY_EU_89-14.gif Size: 6 KB
Click to Enlarge Name: StrategyTester_SVM_DAILY_EU_89-14_OP.gif Size: 7 KB
  • #227
  • Jan 7, 2015 6:02am Jan 7, 2015 6:02am
Quoting th0th
{quote} I'm still very suspicious to all the backtesting in the fact. I tried to SVM with prediction 3rd day based on two previous with training on 10 previsou days. With stop loss 0.5 ATR and Take profit 5 ATR. And I'm very suprised because of the difference in results. I'm attaching the result from MT4 on Daily 1989-2015 with Open Prices and Control points. The difference is weird. I do not have enough memory for backtest of control points for all the data. So I'm a bit unsure what to trust. {image} {image}
Control points and open price should give you the exact same results if your systems is coded correctly in MT4. This is probably an issue with your code executing multiple times for the same bar or something along these lines. Make sure you're only executing trades once on open. Also make sure you're not doing any data snooping (which is possible to do in MT4 if you're not careful). As I have mentioned in the past, GMT shift and whether or not you use DST is also very important. To avoid all these nightmares I would advice anyone to use the Asirikuy framework, where all these issues are already taken care of.
  • #228
  • Edited 6:56am Jan 7, 2015 6:45am | Edited 6:56am
We can also make classifiers more complex to attempt to filter predictions that are too small to be of practical interest. Suppose we want to enter trades only when the total net movement within the next 5 bars is predicted to be larger than 30% of the ATR20. Since we are using a classifier we are not predicting the exact return so we need to divide our classifier into two models (one for longs and another for shorts) each with its own training on each bar. Our two models would have the following structure:

inputs are direction (0, 1) of the last 20 bars, target is whether the return of the next 4 bars is > 30% of the ATR20
inputs are direction (0, 1) of the last 20 bars, target is whether the return of the next 4 bars is < -30% of the ATR20

We end up with two predictions that tell us if we should expect a movement larger than +30% of the ATR20 and/or a movement larger than -30% of the ATR20. We enter trades whenever we have a positive prediction of one model and a negative prediction of the other model (1,0) or (0,1) we don't enter trades when predictions are (1,1) or (0,0) as this gives either a contradiction or an absence of the desired behavior (the market is predicted to remain flat). These are the results:

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_4bar_filtering1.png Size: 28 KB


daily Sharpe ratio normal model = 0.088
daily Sharpe ratio two model classifier approach = 0.077

The result is a lower trade frequency (expected from filtering) and therefore the total profits are inferior to our previous strategy, as a matter of fact the information ratio decreases, telling us that the filtering we're doing does not compensate with an increase in risk-adjusted-return (evident also by the Sharpe showed above). The above exercise as a matter of fact equates to filtering random trades from the simpler model, this means that our ability to properly predict market movement magnitude is not significant. The information ratio is critical in helping you determine if a filtering strategy is worth implementing (sometimes filtering is the way to go, sometimes it is not worth doing). Never compare total profit (this is distorted by the change in trading frequency) always compare risk adjusted returns (Sharpe, Sortino, etc).
  • #229
  • Jan 7, 2015 6:58am Jan 7, 2015 6:58am
These are very nice curves, but for me a little bit disturbing the long (6months?) heavy drawdown
period.
Did anyone try to investigate (some papers maybe) what property of the time series changing causing this
drawdowns?
I tried several change point detection but these are slow, detecting the CHP very late, not really reducing DD.
  • #230
  • Jan 7, 2015 7:06am Jan 7, 2015 7:06am
Quoting emeafx
These are very nice curves, but for me a little bit disturbing the long (6months?) heavy drawdown period. Did anyone try to investigate (some papers maybe) what property of the time series changing causing this drawdowns? I tried several change point detection but these are slow, detecting the CHP very late, not really reducing DD.
Drawdowns are caused mainly by changes in information which take time to digest by the models. If the local distribution of returns of the time series starts to change significantly it takes time for the model to be able to predict something in line with the new distribution, this lag while the model gets enough information to make predictions is what creates the drawdown periods. There are ways to attempt to get around this, like being smarter about how your model learns (search for past examples that match the most recent market conditions instead of looking at X past examples in chronological order) but this only works to a certain extent. Finding faster reacting models (more complex models) can also help. Having ensembles that start to disagree when market conditions change is also a key strategy. We can get much lower DD period lengths but for this we need much much more than what we have discussed on the thread up until now. We'll get there with time.
  • #231
  • Jan 7, 2015 7:16am Jan 7, 2015 7:16am
I would like to point out that the SVM model using 5 bar targets already has a 22 year annualized sharpe ratio of 1.39 while the sharpe ratio of the SPY is 1.20, this is already a strategy that has better risk adjusted returns vs the broad market.
  • #232
  • Jan 7, 2015 7:27am Jan 7, 2015 7:27am
If you remember last time we looked at target results changes we were using a C=1 and a gamma=1 using 30 examples and 8 bar inputs. It is interesting to revisit the target comparison now using the configuration of the last SVM results. Using 145 learning examples with 20 inputs and a C=10 and a gamma=0.001. This will let us know how the different results due to target changes also change as a consequence of introducing these differences in how the models train and the quantity of data they use. These are the results:

Attached Image (click to enlarge)
Click to Enlarge Name: machinelearning_newtargets_highlearning.png Size: 41 KB


You can see that results are still in agreement with what we had found before, although the larger direction predictions do have a much better outcome thanks to the more demanding learning we are now using on the SVM (C and gamma). In terms of risk adjusted returns we see little improvement for the low bar target predictions, with higher absolute returns compensated by higher standard deviation values.
  • #233
  • Jan 8, 2015 3:16am Jan 8, 2015 3:16am
Jo,

where do you get your data (maybe Dukas)? I would make some stat. analysis on it.

Or would you mind to share it?
  • #234
  • Jan 8, 2015 3:47am Jan 8, 2015 3:47am
Quoting emeafx
Jo, where do you get your data (maybe Dukas)? I would make some stat. analysis on it. Or would you mind to share it?
I use at least 4 different data sources. I use 1M data 2000-2014 from Asirikuy, 15M 1986-2014 data from forex-historical, Dukascopy tick data from 2007-2014 and Oanda tick data from 2007-2014. I always confirm that all 4 data sources give similar trading results. I have found that in FX you always need to check your results against at least several different sources of data to ensure everything works properly (no different results for different feeds). The Asirikuy framework makes GMT/DST conversions and candle refactoring to have the exact same candle structure (weekly starting and ending times) completely automatic so I don't need to worry about doing this by myself when changing data sources.
  • #235
  • Jan 8, 2015 4:25am Jan 8, 2015 4:25am
[quote=algoTraderJo;7978243]
inputs are direction (0, 1) of the last 20 bars, target is whether the return of the next 4 bars is > 30% of the ATR20 inputs are direction (0, 1)

Could You please explain how do You prepare the algorithm? What is the next bars in in the future? How do You handle uncertainity. I mean e.g. In case we are calculating from 4th bar in history that means direction of 24th-4th bar and prediction on bars 4th - 0. However how do You use the 2nd bar in calcualtion? History would be from 22nd-2nd bar. And what about the prediction it would be from 1st bar to three bars in the future. Any hint on how to solve that?

Thanks in advance
  • #236
  • Jan 8, 2015 4:27am Jan 8, 2015 4:27am
Quoting th0th
Quoting algoTraderJo
inputs are direction (0, 1) of the last 20 bars, target is whether the return of the next 4 bars is > 30% of the ATR20 inputs are direction (0, 1) Could You please explain how do You prepare the algorithm? What is the next bars in in the future? How do You handle uncertainity. I mean e.g. In case we are calculating from 4th bar in history that means direction of 24th-4th bar and prediction on bars 4th - 0. However how do You use the 2nd bar in calcualtion? History would be from 22nd-2nd bar. And what about the prediction
...
You only include complete examples when building your examples for training. If you are training an algorithm that makes a prediction that needs 4 bars you simply cannot use the last 3 examples for training because, as you say, they are incomplete.
  • #237
  • Jan 9, 2015 10:22am Jan 9, 2015 10:22am
Quoting algoTraderJo
{quote} Control points and open price should give you the exact same results if your systems is coded correctly in MT4. This is probably an issue with your code executing multiple times for the same bar or something along these lines. Make sure you're only executing trades once on open. Also make sure you're not doing any data snooping (which is possible to do in MT4 if you're not careful). As I have mentioned in the past, GMT shift and whether or not you use DST is also very important. To avoid all these nightmares I would advice anyone to use...
I went with Your advice and obtained asirikyu framework as advices. It was a good decision. Now I'm trying to catch up with Your progress. And at first the basic & simple SVM gave some results.
It is 0.5 ATR for Stop, 5.0 ATR for Takeprofit, Period for learning 100 and bars used 10. Althrough I did not Yet implemented the trailing on same signal and reversing on contrary it gave some results. Just the end of testing period did not end very well.
Attached Image (click to enlarge)
Click to Enlarge Name: 201501_svm_results.png Size: 70 KB
  • #238
  • Jan 9, 2015 10:33am Jan 9, 2015 10:33am
Hi algotraderjo,

Great thread, thanks for sharing.
Since good inputs are the key, i am courious what really adds value.

1. Normalize target values, e.g. Express the target not in absolute but in terms of atr(). Probably Little difference?
2. how about different indicators? Most of them are derived from price, but maybe some cycle or stochastic indicators add value?
3. Volume? Even for forex - Tick count. Should add value.

How would you go about adding e.g. a stochastic, one value/input or like returns an entire sequence?

Cheers,
KaBo
  • #239
  • Jan 9, 2015 2:07pm Jan 9, 2015 2:07pm
Quoting th0th
{quote} I went with Your advice and obtained asirikyu framework as advices. It was a good decision. Now I'm trying to catch up with Your progress. And at first the basic & simple SVM gave some results. It is 0.5 ATR for Stop, 5.0 ATR for Takeprofit, Period for learning 100 and bars used 10. Althrough I did not Yet implemented the trailing on same signal and reversing on contrary it gave some results. Just the end of testing period did not end very well. {image}
The reversing on trailing on the same signal and reversion on contrary is implemented automatically on the F4 framework if you use the openOrUpdateLongEasy/openOrUpdateShortEasy functions for entering trades. These functions automatically implement updating of the SL/TP when signals in the same direction happen and exits when signals in the opposite direction happen. There are a lot of things you can do with the ML functions that the native Asirikuy framework comes with, you can already do a lot of things with the SVMs that are implemented. The input/output library also has several examples on the building of datasets.

You can also create functions to generate new datasets as you wish. Here is the one I am using for the last SVM examples (I modified one of the native IO examples):

Inserted Code
LabeledData<RealVector,unsigned int> classification_i_returnBinary_o_returnBinary(int period, int barsUsed){
    vector<RealVector> inputs(period,RealVector(barsUsed));
    vector<unsigned int> labels(period);
    
    int i = 0;
    int j = 0;
    int lag = 4;
    for(i=0;i<period;i++){
                for (j=lag;j<barsUsed+lag;j++){
                inputs[i](j-lag) = (iOpen(0,i+1+j) >= iOpen(0,i+2+j));
                }
                labels[i] = (cOpen(i) >= cOpen(i+1+lag));
    }
    
    ClassificationDataset dataset = createLabeledDataFromRange(inputs,labels);
    return dataset;
}

You can just add this function to the machine learning project InputOutputGeneratorLibrary.cpp and hpp files. See how I used "lag" to introduce the target effect discussed where I explored prediction of different targets returns (more than one day). You can then use the p_classification_i_returnBinary function to obtain a prediction (this one is included I believe). I am happy to help you reproduce anything you want, since we're using the same framework it is easy for me to test anything or to share code with you if you want to.
  • #240
  • Jan 9, 2015 2:16pm Jan 9, 2015 2:16pm
Quoting KaBo
Hi algotraderjo, Great thread, thanks for sharing. Since good inputs are the key, i am courious what really adds value. 1. Normalize target values, e.g. Express the target not in absolute but in terms of atr(). Probably Little difference? 2. how about different indicators? Most of them are derived from price, but maybe some cycle or stochastic indicators add value? 3. Volume? Even for forex - Tick count. Should add value. How would you go about adding e.g. a stochastic, one value/input or like returns an entire sequence? Cheers, KaBo
Good inputs are an important part of the problem but not the whole problem. You can see for example that sometimes using more inputs has apparently helped (look at the examples where I used data from other symbols to predict EU values and obtained an improvement) but this was only because the model I was using was not complex enough to get all the information from some of the inputs. It is not only a matter of having relevant inputs but a matter of being able to extract information from the inputs. For your questions:

 

  1. Very little difference if you normalize inputs against volatility, unless your targets are normalized against volatility as well and you're doing a regression. This is only relevant for regression models because for classifiers that receive X > Y as inputs it does not matter if things are normalized or not (the input is either 1 or 0 -- or a more complex categorization). You can see already in some examples on this thread that normalization makes almost no difference (for the models studied so far!!)
  2. They also add noise because they are numerically very different from price and returns. Do they add more information or simply add more noise that makes predictions even harder ? Take a look at the RSI example I gave during the beginning of the thread. In most cases indicators do much more harm than good.
  3. Real volume is interesting but tick volume is a nightmare. The problem is that volume is very dependent on your data. For classifiers adding something like volume1 > volume2 as inputs may help somewhat but for regressions the values of volume across feeds are simply too different, causing problems in the modeling. There are other sources of information that are much more interesting.

When considering inputs it is also important to consider everything that might affect the value of a currency pair. How about using indexes, bonds, oil, gold? Are these instruments relevant? are they not? can we use them? What about seasonality? Does the month/day/day of the week matter? What do you think?

2 traders viewing now