Machine Learning with algoTraderJo | Page 32 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #621
  • Aug 15, 2015 9:05am Aug 15, 2015 9:05am
Quoting KaBo
{quote} Yes, I see that and get the point, but already to try and explore the search space of 2 systems is huge - I don't see how I could possible calculate the mining bias. However, the mining bias is based on the process. So if there is a way to combine (e.g. in pairs of 2) just an extract (lets say top 100) from all the resulting single systems (all same hour) by some method, the mining bias calculation would still be in reach. {quote} I did not mean to combine more than 2 or 3 systems at a time - but I do see and my computer feels the curse...
As I said, a valid idea but such an idea does not lead to the desired ensemble results as I posted here, due to the bias introduced in the selection of these "top systems". Often the best ensembles come from mixing signals that perform rather badly by themselves. Ohhhhh, there are more clever solutions! Keep digging...
  • #622
  • Aug 15, 2015 9:38am Aug 15, 2015 9:38am
Quoting algoTraderJo
{quote} As I said, a valid idea but such an idea does not lead to the desired ensemble results as I posted here, due to the bias introduced in the selection of these "top systems". Often the best ensembles come from mixing signals that perform rather badly by themselves. Ohhhhh, there are more clever solutions! Keep digging...
How about reducing the search space for each system? E.g. System-1 only searches Sample-Size 60 -100 and System-2 searches Sample-Size 110 - 150.
It would still end up in a huge number of system combinations, but a little better.
  • #623
  • Aug 15, 2015 9:43am Aug 15, 2015 9:43am
Using a correlation filter to pre-screen (eliminate) systems before combination makes a lot of sense in terms of building an ensemble. One of the strengths of ensembles is the ability to combine diverse signals in a synergistic way. If the system signals are too similar there may not be enough unique information to allow the ensemble to synergize properly.
  • #624
  • Aug 15, 2015 11:26am Aug 15, 2015 11:26am
Quoting KaBo
{quote} How about reducing the search space for each system? E.g. System-1 only searches Sample-Size 60 -100 and System-2 searches Sample-Size 110 - 150. It would still end up in a huge number of system combinations, but a little better.
Yes this is what I thought.

For arguments sake, suppose our model has 5 parameters each ranging in the search space from 1 to 100 in steps of 10. Also suppose that we wish to construct an ensemble of dimension 10.

Then, as you say, we can restrict the search space for each parameter in each 'constituent' model as follows;

Model 1: 1-10, Model 2: 11-20, ... , Model 10: 91-100

Though thinking about it now this search space is entirely different to that of a brute-force optimisation, and won't yield the same results;

Consider the existence of an 'optimal' ensemble where each constituent model has all of its parameters between 1 and 10. This will certainly not be discovered using the proposed method.
  • #625
  • Edited 3:25pm Aug 15, 2015 11:43am | Edited 3:25pm
Now suppose we have found 10 prospective systems (parameter sets) from the original search space which satisfy our performance objective(s) and each have a low DMB. We wish to create an ensemble model.

Could we conduct a brute-force optimisation where each iteration constructs an ensemble from a subset of these 10 systems (so that all possible combinations are tested), and then, of course, perform the same operations on random data sets as we would do normally in order to measure DMB? Note that each subset must be of dimension 2 at the very least for the concept of an ensemble to still hold.

Using this method we would test 10C2 + 10C3 + ... + 10C10 = 1265 ensembles for the 10 system example above - this is a considerably smaller number than if we were to combine the parameters (understatement!).
  • #626
  • Aug 16, 2015 1:57am Aug 16, 2015 1:57am
Quoting algoTraderJo
We can use the following parameters on the EURUSD. As stated above, the ML is retrained on every bar: A=10 B=50 C=19 SL=40% of the ATR20 TP=210% of the ATR20 To generate these results for different horizons (D): {image}
Hi Jo,

1. I'm trying to reproduce your results in post #564 using A=10, B=50, C=19, D=100 without success. I've attached my F4 code, can you please have a look and tell me what am I doing wrong?

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot1.png Size: 64 KB


2. Can you please attach the R script for producing your multiple curves charts?

Thanks,
Ruby

Attached File(s)
File Type: zip ML.zip   2 KB | 296 downloads
  • #627
  • Aug 16, 2015 3:01pm Aug 16, 2015 3:01pm
Ok, so I've found a bug in my implementation - I havn't follwed this rule: "If it predicts that a buy would have a positive result and a sell a negative result it then enters a long and vice versa"
Here is the fix for those who are interested:

Inserted Code
#include "Precompiled.h"
#include <ta_libc.h>
#include <stdlib.h>
#include "AsirikuyMachineLearningCWrapper.hpp"
#include "EasyTradeCWrapper.hpp"
#include "OrderSignals.h"
#include "OrderManagement.h"
#include "AsirikuyTime.h"
typedef enum additionalSettings_t
{
    TRADING_HOUR         = ADDITIONAL_PARAM_1,
    LEARNING_PERIOD      = ADDITIONAL_PARAM_2,
    BARS_USED            = ADDITIONAL_PARAM_3,
    FRONTIER             = ADDITIONAL_PARAM_4
} AdditionalSettings;
 
AsirikuyReturnCode runSH004(StrategyParams* pParams)
{
    double stopLoss, takeProfit, atr;
    int prediction[2];
    atr = iAtrWholeDaysSimpleShift(PRIMARY_RATES, (int)parameter(ATR_AVERAGING_PERIOD), 1);
    stopLoss   =  atr * parameter(SL_ATR_MULTIPLIER);
    takeProfit =  atr * parameter(TP_ATR_MULTIPLIER);
 
    if (hour() != (int)parameter(TRADING_HOUR))
        return SUCCESS;
      
    LR_Prediction_i_simpleReturn_o_tradeOutcomeSLTP((int)parameter(LEARNING_PERIOD),
                                                    (int)parameter(BARS_USED),
                                                    (int)parameter(TRADING_HOUR),
                                                    parameter(SL_ATR_MULTIPLIER),
                                                    parameter(TP_ATR_MULTIPLIER),
                                                    (int)parameter(FRONTIER),
                                                    &prediction);
    //hit TP Long && hit SL Short
    if (prediction[0] == 1 &&  prediction[1] == 0)
        openSingleLongEasy(takeProfit, stopLoss, 0);
  
    //hit TP Short && hit SL Long
    if (prediction[1] == 1 && prediction[0] == 0)
        openSingleShortEasy(takeProfit, stopLoss, 0);
 
    return SUCCESS;
}


And here are the results:
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot1.png Size: 61 KB


It looks better now, but still not completely similar to algoTraderJo's curve.
Any input regarding the differences would be appreciated.
Thanks
Ruby
  • #628
  • Aug 16, 2015 3:09pm Aug 16, 2015 3:09pm
University of Oxford Machine Learning course.
  • #629
  • Aug 16, 2015 3:33pm Aug 16, 2015 3:33pm
Quoting beachboy44
{quote} Then, as you say, we can restrict the search space for each parameter in each 'constituent' model as follows; Model 1: 1-10, Model 2: 11-20, ... , Model 10: 91-100
beachboy44 and KoBo, If you create the categories manually like this, you implicitly assume that you will get more diversity and less related systems, an assumption which may be true or false. In reality, how do we know in advance which systems in what ranges will combine well, unless we have data snooped?

Why not simply make no assumptions of this sort and just create 10 non-overlapping random categories from the whole pool instead. This will reduce the variance without imposing artificial assumptions on the process (because random selection is unbiased). This is similar to how it is done in the bagging algorithm. Due to randomness, the results may be somewhat different each time you repeat this process, but the average result could be used as an estimate of how well this process works.
  • #630
  • Aug 17, 2015 1:53am Aug 17, 2015 1:53am
Quoting FXEZ
{quote} beachboy44 and KoBo, If you create the categories manually like this, you implicitly assume that you will get more diversity and less related systems, an assumption which may be true or false. In reality, how do we know in advance which systems in what ranges will combine well, unless we have data snooped? Why not simply make no assumptions of this sort and just create 10 non-overlapping random categories from the whole pool instead. This will reduce the variance without imposing artificial assumptions on the process (because random selection...
I would be hesitant to use only a single random categorisation which then, as you say, necessitates repeating the process and averaging the result. I suppose we would need to investigate the added computational cost which this method entails (on average).

However I'm sure algoTraderJo is hinting at a much more elegant solution!
  • #631
  • Edited 7:22am Aug 17, 2015 2:22am | Edited 7:22am
With regards to my post above (#625), Daniel from Asirikuy pointed out that this method does not achieve the desired result since, as algoTraderJo mentioned, often the best ensembles are comprised of systems with inferior individual performance.

Daniel also noted that we can reduce the ensemble parameter space by thinking in terms of combinations as opposed to permutations. Let me clarify with an example;

Suppose our model consists of 3 parameters, each ranging from 1 to 10 in increments of 1.
For a 2-system ensemble, let's call them X1, Y1, Z1, X2, Y2, Z2. Then the parameter space has a dimension of 10^6 = 1,000,000 (this is thinking in terms of permutations).

Now thinking in terms of combinations, we want to calculate how many ways there are of choosing 2 different systems from the possible 1,000 (10^3) single systems;
1000C2 = 499,500.

Hence we have reduced the parameter space by more than a factor of 2. Note that by thinking in terms of combinations, we have eliminated two types of ensembles from our search space;

1.) Those where X1 = X2, Y1 = Y2, Z1 = Z2, which of course makes no sense by the very nature of an ensemble.
2.) Those where the pair of systems contained in the ensemble has already been included in a previously tested ensemble (but just the other way round).

So ultimately what we end up with are results identical to those that we would have obtained if we had just done a brute-force optimisation, but at (less than half) the computational expense.

Furthermore, note that the reduction factor increases (non-linearly) as the number of systems within the ensemble increases. Using the same 3-parameter ranges described above;

3-system ensemble: Original space = 1,000,000,000. Reduced space = 166,167,000. Reduction factor = 6.02 (to 2dp).
4-system ensemble: Original space = 1,000,000,000,000. Reduced space ≈ 4.14x10^10. Reduction factor = 24.14 (to 2dp).

And so on...
  • #632
  • Aug 17, 2015 4:06am Aug 17, 2015 4:06am
Hi Beachboy, Kabo and others who are using the F4 framework.
Have you successfully implemented the algorithms described in post#564 and post#588 ?
Personally speaking, I'm not achieving any of the results using these algos.
Thanks
Ruby
  • #633
  • Aug 17, 2015 4:46am Aug 17, 2015 4:46am
Hi Ruby,
no, I have not done it yet. Right now I'm using some variation of the algorithm idea and run experiments with it.
Since I don't have hundreds or thousands of cores, it takes time.
Would be great if Jo posts the code. Always got it going then.
  • #634
  • Aug 17, 2015 5:09am Aug 17, 2015 5:09am
No I haven't done it yet either. I will try when I get the chance... might be a couple of days though
  • #635
  • Aug 17, 2015 5:59am Aug 17, 2015 5:59am
Thanks guys, looking forward for some sort of confirmation.
Ruby
  • #636
  • Aug 17, 2015 6:49am Aug 17, 2015 6:49am
Hi, I've been following this thread for a while. Thank you algoTraderJo for this fantastic approach. I got some ideas for ensemble testing. When building ensembles on agreement of algos there are 3 possibilities for every trading day: LONG, SHORT and FLAT. So supposing we got N trading days there are 3^N possible trading sequences. Now every ensemble (consisting of arbitrary many algos) can be mapped to one of those sequences. The point seems to be that there could be many ensembles mapping to a joint sequence. So at the end we have to calculate the trading criterion (e.g. sharpe ratio) only for a reduced number of sequences.
The mapping and looking up can be efficiently done with a boolean array of size 3^N.
The complexity is bigger with ensembles built on addition of trades (instead of agreement) but could be reduced with the same procedure.
  • #637
  • Edited 9:59am Aug 17, 2015 7:44am | Edited 9:59am
Quoting cyberjag
Hi, I've been following this thread for a while. Thank you algoTraderJo for this fantastic approach. I got some ideas for ensemble testing. When building ensembles on agreement of algos there are 3 possibilities for every trading day: LONG, SHORT and FLAT. So supposing we got N trading days there are 3^N possible trading sequences. Now every ensemble (consisting of arbitrary many algos) can be mapped to one of those sequences. The point seems to be that there could be many ensembles mapping to a joint sequence. So at the end we have to calculate...
I'm not sure that I understand exactly what you mean. Particularly, what do you mean by "So at the end we have to calculate the trading criterion (e.g. sharpe ratio) only for a reduced number of sequences."?

Besides, when testing on 28.5 years of data as we do in Asirikuy, 3^7410 is a ridiculously large number.

Also, by analysing the trading sequences in this way we limit ourselves to only building ensembles with machine learning algorithms which take a trading decision on each bar (no stoplosses/trailing stops etc).
  • #638
  • Aug 17, 2015 8:47am Aug 17, 2015 8:47am
Quoting beachboy44
{quote} I'm not sure that I understand exactly what you mean. Particularly, what do you mean by "So at the end we have to calculate the trading criterion (e.g. sharpe ratio) only for a reduced number of sequences."? Besides, when testing over 28.5 years of data as we do in Asirikuy, 3^7410 is a ridiculously large number. Also, by analysing the trading sequences in this way we limit ourselves to only building ensembles with machine learning algorithms which take a trading decision on each bar (no stoplosses/trailing stops etc).
3^7410 would only be the theoretical number of possibilities. Considering for example ensembles with 2 algos we would only have to calculate the sharpe ratio of k over 2 sequences out of these 3^7410 even by brute force, where k would be the size of the parameter space for the single test. But these k over 2 sequences might not all be different! So we have to test less! This is the idea behind mapping to these sequences.
Stoploss and so on are no problem. The difference lies only in the trading decision (long,short,flat). The trade can then take place according to some special rule. Since the ensembles are built on agreement the lots are constant or depend only on the ATR.
  • #639
  • Edited 11:50am Aug 17, 2015 11:14am | Edited 11:50am
Quoting cyberjag
{quote} 3^7410 would only be the theoretical number of possibilities. Considering for example ensembles with 2 algos we would only have to calculate the sharpe ratio of k over 2 sequences out of these 3^7410 even by brute force, where k would be the size of the parameter space for the single test. But these k over 2 sequences might not all be different! So we have to test less! This is the idea behind mapping to these sequences. Stoploss and so on are no problem. The difference lies only in the trading decision (long,short,flat). The trade can then...
But surely to construct the map you would need to backtest every possible ensemble within the parameter constraints, since we do not know in advance which ensembles lead to which sequence?

On an abstract level, it's not the calculation of the trading criterion in itself which is computationally demanding, it's the determining which parameter combinations lead to which sequence (since the parameter space is so large). It's just incidental that these determinations are made by calculating the trading criterion.

My post above (#631) shows that for a brute-force optimisation of a 2-system ensemble, there are (kČ/2) - (k/2) different possibilities - derived from the Binomial Coefficient. I'm not sure where you get k/2 from?

I apologise for not understanding what you mean, I must be having a bad day
  • #640
  • Aug 17, 2015 4:10pm Aug 17, 2015 4:10pm
Quoting beachboy44
{quote} But surely to construct the map you would need to backtest every possible ensemble within the parameter constraints, since we do not know in advance which ensembles lead to which sequence? On an abstract level, it's not the calculation of the trading criterion in itself which is computationally demanding, it's the determining which parameter combinations lead to which sequence (since the parameter space is so large). It's just incidental that these determinations are made by calculating the trading criterion. My post above (#631) shows that...
Ok, I'll try to explain it in detail. One thing I disregarded above was that there might be parameters not only in the opening trades decision but also in the closing trades decision. Therefore I now split the parameter space: Let S1 resp. S2 be the space of those parameters responsible for opening resp. closing trades. Let k1(k2) be the size of S1(S2). I will consider ensembles of only 2 systems.

  1. For each element in S1 we determine the long/short decision for each trading day and save this as an element of {long, short}^N, thus getting k1 of these sequences. (k1 * N computationally demanding jobs)
  2. From every two of these sequences we build an "ensemble sequence" as an element of {long, short, flat}^N by the rule of agreement, thus getting k1C2 ensemble sequences.
  3. Checking for equal ensemble sequences yields m <= k1C2 different ensemble sequences.
  4. Since there are parameters in the closing decision we have to combine each of the m ensemble sequences with an element of S2 which leads to m * k2 different ensembles.
  5. For each of the m * k2 ensembles we calculate the sharpe ratio (or whatever).

Of course the computationally demanding part is regression/SVM/... but this is the same as with testing the k1 single systems. I take this as a basis. In my understanding of "ensemble" it's just combining existing precomputed single systems in some way. The only lengthy thing is evaluating the combinations, so I strip those combinations that lead to equal trades. That's about all.

3 traders viewing now