Forex Factory (https://www.forexfactory.com/forum.php)
-   Trading Discussion (https://www.forexfactory.com/forumdisplay.php?f=11)
-   -   Machine Learning with algoTraderJo (https://www.forexfactory.com/showthread.php?t=516785)

algoTraderJo Mar 16, 2017 7:10pm | Post# 881

If any here are using pKantuML for ML strategy mining we can post the csv files for the systems we find here and share... Anyone up for it?

algoTraderJo Mar 29, 2017 11:02am | Post# 882

2 Attachment(s)
This is an algo I like, it has been trading for a bit more than a year now for me:

Attachment 2250251

It uses a two algo linear classifier ensemble. I post the csv for this system, if anyone wants to explore it more.
kantu_ml_1.csv

000 Mar 29, 2017 4:44pm | Post# 883

How do you make the renko charts immune to this problem? Any takers?
There is something in the same idea as renko charts, but much better (and immune to this problem). It's called directional change.

https://arxiv.org/abs/0809.1040

I'm using this as input into my convolutional networks.

algoTraderJo Mar 29, 2017 5:19pm | Post# 884

1 Attachment(s)
There is something in the same idea as renko charts, but much better (and immune to this problem). It's called directional change.

Cool, check this recent paper out.
kampouridis2017.pdf

PipMeUp Mar 30, 2017 3:36am | Post# 885

{quote} There is something in the same idea as renko charts, but much better (and immune to this problem). It's called directional change. https://arxiv.org/abs/0809.1040 I'm using this as input into my convolutional networks.
I remember reading this paper. The author says that he takes a trading decision at every lambda. If you break the bars at every lambda the building method he proposes generates a constant range bar chart. The difference is that lambda is in percentage rather than pips/ticks/points. With FX data the difference is very small.

CRB are immune to the problem algoTraderJo describes because they "re-synchronize" at every pivot. I also prefer CRB to renko because the renko are somehow biased: they need twice the size to change direction. CRB have the interesting property that all the information is contained in the close price (except when a gap happens).

000 Mar 30, 2017 6:31am | Post# 886

{quote}CRB are immune to the problem algoTraderJo describes because they "re-synchronize" at every pivot.
Hmm, are you sure about that? The CRB construction rules that I found do not mention any kind of pivot/extreme point. They only care about range. So for example, if you start computing them in the middle of a constant up trend, when the trend finishes, the CRB would switch direction at different points, depending on exactly when it was started, so on average they will be "desynched" by half the range.

Halley Mar 30, 2017 7:59am | Post# 887

{quote} Hmm, are you sure about that? The CRB construction rules that I found do not mention any kind of pivot/extreme point. They only care about range. So for example, if you start computing them in the middle of a constant up trend, when the trend finishes, the CRB would switch direction at different points, depending on exactly when it was started, so on average they will be "desynched" by half the range.
He was right, the range bars get chart synchronised with every single reversal candle.

FXEZ Mar 31, 2017 2:16pm | Post# 888

{quote} Hmm, are you sure about that? The CRB construction rules that I found do not mention any kind of pivot/extreme point. They only care about range. So for example, if you start computing them in the middle of a constant up trend, when the trend finishes, the CRB would switch direction at different points, depending on exactly when it was started, so on average they will be "desynched" by half the range.
PipMeUp is correct that constant range charts converge from different starting points. From my tests you can start multiple constant range charts at different points and in approximately 3-5 bars they will synchronize. The number of bars to synchronization may be a factor of the range chosen. Try the bar creation logic with multiple random starting points and you'll see that constant range charts converge.

000 Mar 31, 2017 4:22pm | Post# 889

{quote} PipMeUp is correct that constant range charts converge from different starting points. From my tests you can start multiple constant range charts at different points and in approximately 3-5 bars they will synchronize. The number of bars to synchronization may be a factor of the range chosen. Try the bar creation logic with multiple random starting points and you'll see that constant range charts converge.
I couldn't find the CRB creation logic clearly described anywhere, the general description that I found seems susceptible to sync problems. Doesn't matter anyway, they don't interest me. Directional change "bars" converge in exactly 1 bar.

FXEZ Mar 31, 2017 4:58pm | Post# 890

{quote} I couldn't find the CRB creation logic clearly described anywhere, the general description that I found seems susceptible to sync problems. Doesn't matter anyway, they don't interest me. Directional change "bars" converge in exactly 1 bar.
CRB creation logic:
After you select a CRB range amount (like 20 pips), the logic for constant range bars (CRB) is quite simple and goes something like:
1) start a new bar at the next/current price
2) keep track of bar's range (high and low prices)
3) When high-low >= CRB range amount, go to 1) above

Note with #3 above you will not get all equally sized bar ranges (high - low), but you will get convergence.

olsen-yersen Apr 3, 2017 1:44pm | Post# 891

This looks interesting; https://blog.zhaw.ch/sef/files/eco2_script.pdf
https://www.amazon.com/Real-Detectio.../dp/3656396388
https://artax.karlin.mff.cuni.cz/r-h...l/00Index.html

bernacek Apr 10, 2017 6:37pm | Post# 892

Hi algoTraderJo,
it's really good to see you back on this thread with new machine learning ideas.

I have specific question for you that bothers me a lot recently. When measuring DMB of a strategy that uses returns from multiple symbols as inputs, how do you bootstrap the symbol datasets? Is it safe to bootstrap the bar returns of all symbols independently (effectivelly destroying any correlation between symbols)? Doesn't this lead to underestimation of the datamining bias? Alternative approach is to group bar returns that occurred at particular time from all symbols together and bootstrap the groups of bar returns.

algoTraderJo Apr 16, 2017 9:33pm | Post# 893

Hi algoTraderJo, it's really good to see you back on this thread with new machine learning ideas. I have specific question for you that bothers me a lot recently. When measuring DMB of a strategy that uses returns from multiple symbols as inputs, how do you bootstrap the symbol datasets? Is it safe to bootstrap the bar returns of all symbols independently (effectivelly destroying any correlation between symbols)? Doesn't this lead to underestimation of the datamining bias? Alternative approach is to group bar returns that occurred at particular...
This is a toughy. You want to maintain the symbol correlations present but you want to randomize all the data. The solution is not complicated, you sample and always pick the same index for the different symbols at the same time. If your symbols have their datetime stamps aligned then this means drawing say the bar with index 100 and then picking index 100 for all symbols for that bar. This keeps the correlation between symbols inside each timeframe but effectively destroys any inefficiencies in the series.

It surprised me the first time, but this preserves the general correlations between the original time series while completely eliminating any predictable behavior within each one of them. I have used this process since I started doing multi-symbol DMB and can't say I've had issues with it.

algoTraderJo Apr 16, 2017 9:34pm | Post# 894

Kids are keeping me busy but I think I'll have time for some more ML fun this week!

andrea182 Apr 17, 2017 5:52am | Post# 895

This is an algo I like, it has been trading for a bit more than a year now for me: {image} It uses a two algo linear classifier ensemble. I post the csv for this system, if anyone wants to explore it more. {file}
Hi algoTraderJo,

can you explain just a little bit the csv that you've posted here?

Thanks,
Andrea182

slowbrain Apr 19, 2017 12:52am | Post# 896

[qu
ote=PipMeUp;9721578]{quote} I remember reading this paper. The author says that he takes a trading decision at every lambda. If you break the bars at every lambda the building method he proposes generates a constant range bar chart. The difference is that lambda is in percentage rather than pips/ticks/points. With FX data the difference is very small. CRB are immune to the problem algoTraderJo describes because they "re-synchronize" at every pivot. I also prefer CRB to renko because the renko are somehow biased: they need twice the size to change direction. CRB have the interesting property that all the information is contained in the close price (except when a gap happens).[/quote]

PipmeUp

If you are interested about DC , this is a link with all the major publications about Directional Change. its really interesting to have a look on the scaling laws introduced by richard Olsen .

I believe to trade DC , the best method is so called multi-agents .... trend following at Dc and counter trade at overshoot
It is important that you use two different thresholds for 'increasing a position (bigger threshold) and reducing positions (smaller threshold). The difficulty is to correctly calibrate the 'size' of trades for the two sides....as a rough approximation: if you have a threshold for reducing is half as big as the threshold for increasing positions, then the trade size needs to be 1/4.

for your information my 2 favorite threshold are 0.46% and 0.96% on Eurusd

http://www.bracil.net/finance/DirectionalChanges/

slowbrain May 5, 2017 8:35pm | Post# 897

2 Attachment(s)
{quote} Cool, check this recent paper out. {image}
For your reference the latest document released by Richard Olsen on Directional Change
SSRN-id2951348.pdf
Click to Enlarge

Name: Olsen Agents.JPG
Size: 73 KB

VTMLtrading Jun 9, 2017 10:31am | Post# 898

3 Attachment(s)
I briefly followed all this. Initially my intention was to work on a machine learning tool and connect metatrader to it. I then had problems finding the right dataset as input for my model. With your ideas I was able to at least reproduce the initial screenshots I saw here multiple times:

Click to Enlarge

Name: 3days.gif
Size: 9 KB

So that is a backtest from 1990-2017 with my machine learning api. You can download the metatrader EA source code here:

3days.zip

You can play around with the model and also add your own. A description of the api is here: http://ml.vanillatech.de/
What I found out is that backtesting that exact model with data from e.g. 1980-1990 the whole thing does not work anymore.
I also played around with more complex models, e.g. Momentum10, ATR20 as additional inputs (as someone suggested here) - the result is getting worse:

Click to Enlarge

Name: atr-momentum.gif
Size: 16 KB

and what I can see is that in most screenshots here (e.g. algotraderjo) you are also testing from 1990. Is that for a reason?
There are 45 pages here already. But if I understand right, your favorite model, AlgoTraderJo, is still solely based on directional change (25 bars?) and using ATR20 only to set a SL. Is that correct?

windgod123 Jul 5, 2017 7:48am | Post# 899

algoTraderJo what a wonderful thread you started!!

it's 900 posts already but I just happened to find this on the internet. Hope I am not too late to the Machine Learning World in FX!!

I am going to test & reproduce some of your results and will share the findings.

EADevChris Jul 11, 2017 6:53pm | Post# 900

Hi algoTraderJo,

I'm new here - but I've been reading your posts with interest and I'm trying out some of your techniques using the Accord .NET ML library in C# with nQuotes on MT4.

First off - thanks for all of your efforts - very interesting reading!

I'm trying to replicate some of your results (or at least something close) and I have a question which makes me think I've fundamentally misunderstood part of your algorithm (note that I'm referring to some of your earlier examples).

As I understand it, you make a prediction, and based on that prediction you open a trade with a size and stoploss such that you are risking 1% of your balance. You then make another prediction and, if it's in the same direction as the open trade you move the stoploss (only to more favourable positions). If it's in a different direction you close the trade and open in the new direction - again risking 1% of your balance.

So if I've got that right, you only ever have 1 trade open with a maximum risk of 1%... so how do you end up in situations where you have drawdown of 20% - 40%?!

As I say... I'm obviously missing something!?

Thanks,
- Chris


© Forex Factory