Machine Learning with algoTraderJo | Page 33 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #641
  • Aug 17, 2015 6:46pm Aug 17, 2015 6:46pm
Quoting cyberjag
{quote} Ok, I'll try to explain it in detail. One thing I disregarded above was that there might be parameters not only in the opening trades decision but also in the closing trades decision. Therefore I now split the parameter space: Let S1 resp. S2 be the space of those parameters responsible for opening resp. closing trades. Let k1(k2) be the size of S1(S2). I will consider ensembles of only 2 systems. For each element in S1 we determine the long/short decision for each trading day and save this as an element of {long, short}^N, thus getting...
Thanks for the explanation. I was thinking in terms of 'mining ensembles' from scratch, whereas you were thinking in terms of combining existing systems.

It makes more sense now
  • #642
  • Aug 19, 2015 6:46am Aug 19, 2015 6:46am
Hi algoTraderJo,

Is it practical to use ensembles with ML strategies ( increase degree of freedom) instead of using portfolio of simple ml strategies.
Testing ensembles is much more expensive.
What do you "WIN" from your practice ?
  • #643
  • Aug 19, 2015 7:12am Aug 19, 2015 7:12am
Quoting beachboy44
{quote} Thanks for the explanation. I was thinking in terms of 'mining ensembles' from scratch, whereas you were thinking in terms of combining existing systems. It makes more sense now
As I understand it it's the difference between "parameter ensembles" (combined input vector) and "system ensembles" (combined systems).

The concept of parameter ensembles is clear to me from an abstract point of view. However in a specific implementation there are some questions arising.
Suppose we have these two parameters LEARNING_PERIOD and BARS_USED as in the examples above.
Say LEARNING_PERIOD_1 = 100, LEARNING_PERIOD_2 = 200, BARS_USED_1 = 12, BARS_USED_2 = 24.
How is the input matrix constructed? In a "combined matrix" in every row there would be the past 12 returns followed by the past 24 returns. So there is a redundance, the first 12 columns could be stripped. And how many rows will the matrix have? 100 or 200? In any case there would be no other result as would have been achieved with single input vectors.
If two matrices were constructed this would in fact be nothing other than a system ensemble. There we get really new systems.

Or do I get something wrong?
  • #644
  • Aug 19, 2015 9:36am Aug 19, 2015 9:36am
At post Post614
TP and SL are the same for the systems in the ensemble as they are important for the same exit... I guess.
  • #645
  • Aug 19, 2015 9:38am Aug 19, 2015 9:38am
Quoting cyberjag
{quote} As I understand it it's the difference between "parameter ensembles" (combined input vector) and "system ensembles" (combined systems). The concept of parameter ensembles is clear to me from an abstract point of view. However in a specific implementation there are some questions arising. Suppose we have these two parameters LEARNING_PERIOD and BARS_USED as in the examples above. Say LEARNING_PERIOD_1 = 100, LEARNING_PERIOD_2 = 200, BARS_USED_1 = 12, BARS_USED_2 = 24. How is the input matrix constructed? In a "combined matrix" in every row...
When constructing a 'parameter ensemble' with 2 systems, we still train two seperate models on each bar/trading hour - we do not combine the inputs. Constructing a 'parameter ensemble' is done in the same way as the search of the original space, only there are twice as many parameters (and of course the code is adjusted to only trade on agreement between the two distinct models).

You are correct in that if you were to construct every possible ensemble (all combinations) using the 'system ensemble' method, then you will end up with the same results as when using the 'parameter ensemble' method.

The only difference between the two is that the 'parameter ensemble' method actually conducts a backtest for each possible ensemble as soon as they are found, whereas, from what I understand, the 'system ensemble' method does not (instead the trade decision sequences are analysed beforehand). So I do see that there is perhaps some benefit to the 'system ensemble' method from a computer-resource perspective.
  • #646
  • Aug 19, 2015 1:05pm Aug 19, 2015 1:05pm
Quoting beachboy44
{quote} When constructing a 'parameter ensemble' with 2 systems, we still train two seperate models on each bar/trading hour - we do not combine the inputs. Constructing a 'parameter ensemble' is done in the same way as the search of the original space, only there are twice as many parameters (and of course the code is adjusted to only trade on agreement between the two distinct models). You are correct in that if you were to construct every possible ensemble (all combinations) using the 'system ensemble' method, then you will end up with the same...
I see, so if the algo works independently on every item of the parameter ensemble the algo outputs for the (simple) state space could be calculated and saved beforehand and then be used in the testing. That seems to me quite inevitable for performance reasons. Concerning the algo job this would be the same calculation time as with non-ensemble testing.

In my proposal above I considered it sufficient to save only the long/short information since the ensemble logic worked on agreement. Evaluation needn't be done as long as there is no restriction to special systems. Although some restriction could become necessary if the parameter space was too large for the RAM or drive to take all the precomputed outputs...
  • #647
  • Aug 19, 2015 1:20pm Aug 19, 2015 1:20pm
Quoting cyberjag
{quote} I see, so if the algo works independently on every item of the parameter ensemble the algo outputs for the (simple) state space could be calculated and saved beforehand and then be used in the testing. That seems to me quite inevitable for performance reasons. Concerning the algo job this would be the same calculation time as with non-ensemble testing. In my proposal above I considered it sufficient to save only the long/short information since the ensemble logic worked on agreement. Evaluation needn't be done as long as there is no restriction...
Yes, I think that storing the long/short decisions is somewhat necessary for the majority of people to be able to measure data-mining bias for ensembles, due to the computational complexity involved otherwise.

Thanks for posting your idea!

Also, since the individual calculations of the 'system ensemble' method are quite simple and parallelizable (after the initial search has been done), it may be worth looking into implementing them in CUDA/OpenCL and leveraging the power of GPUs
  • #648
  • Sep 16, 2015 3:27am Sep 16, 2015 3:27am
Hi everyone,
Thanks to those that have contributed and algotraderjo for starting the thread. It's been very interesting.
I have a question about bootstrapping and data mining bias testing. After some poking around online I think I get the general outline of what needs to be done. However I couldn't find anything particularly about walk forward machine algos, only in/out sample bootstrapping (like what had been raised here as a poor idea by algotraderjo).

So...when we bootstrap returns we do it with replacement as this seems to be the consensus (even though it changes the statistical properties of the return series).
The problem I have is WHEN to do the N times bootstrap.
Say we decide N = 100.

Do we bootstrap the entire return series N times, running the entire walk forward backtest each time OR
do we bootstrap each training windows returns N times to gain a distribution of that window before we step forward to the next window.
Option 1 results in 100 runs of the backtest regardless of the available data whereas option 2 results in N * number of steps (and is obviously very computationally expensive for even just 1 year of data, let alone 20.

This is confusing the hell out of me so if anyone coukd help out that would be great. If I've confused you too I'll try and make myself a little clearer.

Cheers,
SteepCurve
  • #649
  • Sep 16, 2015 10:09am Sep 16, 2015 10:09am
Quoting SteepCurve
Hi everyone, Thanks to those that have contributed and algotraderjo for starting the thread. It's been very interesting. I have a question about bootstrapping and data mining bias testing. After some poking around online I think I get the general outline of what needs to be done. However I couldn't find anything particularly about walk forward machine algos, only in/out sample bootstrapping (like what had been raised here as a poor idea by algotraderjo). So...when we bootstrap returns we do it with replacement as this seems to be the consensus (even...

Hi SteepCurve,

I don't know if you are aware of http://mechanicalforex.com, but it includes some great articles that describe exactly what data-mining bias (DMB) is in the context of this thread, and also how you can measure it. I probably won't be able to explain it as well as Daniel - the author of the blog - does, so I will include various links within this post.

The first thing to note is that different people use the terms 'data-mining bias' and 'curve-fitting bias' to mean different things. DMB within this thread is the type I bias described here: http://mechanicalforex.com/2014/12/t...trategies.html

Therefore DMB is a property of a system creation process, not of a particular trading system itself: http://mechanicalforex.com/2015/05/b...-property.html

The DMB test used by Daniel and algoTraderJo has nothing to do with walk-forward testing. Instead, the entire process used to generate a system (eg searching a parameter space for systems which satisfy predetermined performance objectives) is repeated on N random time series - not just a backtest/walk-forward test. As you said, these time series are created by bootstrapping the returns (with replacement): http://mechanicalforex.com/2014/03/a...-creation.html

Having completed the test, you end up with a graph like the one in post #602.

The following articles may also be of interest to you:
http://mechanicalforex.com/2014/01/d...-part-one.html
http://mechanicalforex.com/2014/01/d...-part-two.html
http://mechanicalforex.com/2015/06/d...ok-part-1.html
http://mechanicalforex.com/2015/06/d...ok-part-2.html

Let me know if anything is still unclear; I'll do my best to help
  • #650
  • Sep 16, 2015 12:35pm Sep 16, 2015 12:35pm
A quite simple introduction to the idea of DMB tests is summarized at
http://www.timothymasters.info/Permutation_tests.html
The tests alluded to in previous posts are closest to the test called "method 3" in the link.
  • #651
  • Sep 17, 2015 6:51pm Sep 17, 2015 6:51pm
50 first testing trades with Raccoon SVC... 62.7% win rate.

My backtesting is at 58%, l'm over my simulation.. I need to lost more..

:-)

http://www.forexfactory.com/forexsamsam#40
  • #652
  • Sep 23, 2015 6:39pm Sep 23, 2015 6:39pm
So is anybody able to show results from live demo account of any systems discussed in this thread ?? AlgoTraderJo ??

after this I stopped to believe in backtests...

http://link.brightco...d=3878525144001

http://eranraviv.com/multiple-testing/

http://www.financial-math.org/

Inserted Video


Krzysztof
  • #653
  • Sep 23, 2015 6:42pm Sep 23, 2015 6:42pm
Quoting fajst_k
So is anybody able to show results from live demo account of any systems discussed in this thread ?? AlgoTraderJo ?? after this I stopped to believe in backtests... http://link.brightco...d=3878525144001 http://eranraviv.com/multiple-testing/ http://www.financial-math.org/ https://www.youtube.com/watch?t=23&v=KKduDAbZ4Uk Krzysztof
zzzzzzzzzzzzzZZZZZZZZzzzzzz

i dont trust dude in tie..

:-)
  • #654
  • Sep 24, 2015 6:08am Sep 24, 2015 6:08am
Quoting fajst_k
So is anybody able to show results from live demo account of any systems discussed in this thread ?? AlgoTraderJo ?? after this I stopped to believe in backtests... http://link.brightco...d=3878525144001 http://eranraviv.com/multiple-testing/ http://www.financial-math.org/ https://www.youtube.com/watch?t=23&v=KKduDAbZ4Uk Krzysztof
This is precisely why we measure data-mining bias.

I have attached a screenshot of the performance of a EURUSD D1 system which has been developed using some of the techniques within this thread.

Attached Image (click to enlarge)
Click to Enlarge Name: Screen Shot 2015-09-24 at 11.53.51.jpg Size: 161 KB
  • #655
  • Sep 24, 2015 6:27am Sep 24, 2015 6:27am
The problem is that if you run many of such systems and accounts and show me results of selected one is like trowing coin 10000 times, getting 5 heads in the row in one of trials and saying i have a method for predicting coin result. So how many system you try in pararell on demo account and they fail ??

Krzysztof
  • #656
  • Sep 24, 2015 6:34am Sep 24, 2015 6:34am
Quoting fajst_k
The problem is that if you run many of such systems and accounts and show me results of selected one is like trowing coin 10000 times, getting 5 heads in the row in one of trials and saying i have a method for predicting coin result. So how many system you try in pararell on demo account and they fail ?? Krzysztof
This is the only machine learning system that I trade/test live. You are right to be sceptical, but I can assure you that I am not trying to fool you - I'm not selling you anything

I am merely showing you that it is possible to be profitable using the methods discussed in this thread.

One thing I should tell you is that DMB has not been measured for this system, and so the results you see above are 'lucky' in that respect.
  • #657
  • Sep 27, 2015 4:18am Sep 27, 2015 4:18am
Quoting beachboy44
{quote} Yes, I think that storing the long/short decisions is somewhat necessary for the majority of people to be able to measure data-mining bias for ensembles, due to the computational complexity involved otherwise. Thanks for posting your idea! Also, since the individual calculations of the 'system ensemble' method are quite simple and parallelizable (after the initial search has been done), it may be worth looking into implementing them in CUDA/OpenCL and leveraging the power of GPUs
That's what I'm talking about! Those are solid ideas that would allow you to realistically test the mining bias of ensembles. It will still take a lot of time but it is now in the realm of possibility.
  • #658
  • Sep 27, 2015 4:23am Sep 27, 2015 4:23am
Quoting fajst_k
So is anybody able to show results from live demo account of any systems discussed in this thread ?? AlgoTraderJo ?? after this I stopped to believe in backtests... http://link.brightco...d=3878525144001 http://eranraviv.com/multiple-testing/ http://www.financial-math.org/ https://www.youtube.com/watch?t=23&v=KKduDAbZ4Uk Krzysztof
You must really be kidding. Think about what you're asking.

You're asking me to show you how I risked my own money and achieved a profitable result so that you can make a clearer decision to trade the same family of methods I use. I get absolutely nothing in return. As beachboy44 said we are not selling you anything here so if you don't want to trade these methods, then don't, my intention is not to convince you of anything. Read the thread, if you like what you see then I suggest you risk your own money to see if anything here can really be profitable. Sorry but I won't be showing you the results of my own tests. If I risk my money, then it's to get information for myself, not for you or anyone else.
  • #659
  • Sep 27, 2015 4:27am Sep 27, 2015 4:27am
Quoting ForexSamsam
50 first testing trades with Raccoon SVC... 62.7% win rate. My backtesting is at 58%, l'm over my simulation.. I need to lost more.. :-) http://www.forexfactory.com/forexsamsam#40
The probability to get those results out of random chance is still too large as your testing period is still way too small. Calculate the confidence with which you can say that your method is performing according to your backtesting results Vs. luck (null hypo = my system is equivalent to a random trading system), if it's below 95% then you cannot say anything. The win rate is also irrelevant on its own, you need to look at win rate times risk to reward ratio to get a sense of your profit expectation per trade.
  • #660
  • Edited 4:49am Sep 27, 2015 4:28am | Edited 4:49am
Well I asked for demo account statement not real account. It would confirm efficiency of such methods and calculation of DMB as according to statements of guys in ties its kind of impossible.

Krzysztof
1 trader viewing now