Machine Learning with algoTraderJo | Page 26 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #501
  • May 28, 2015 8:56am May 28, 2015 8:56am
Quoting ampleparking
Hello everybody, algoTraderJo, is there any reason for using 2 output units (MLE and MSE) instead of using 1 unit (the MLE/MSE ratio)?
You tell me
  • #502
  • May 28, 2015 9:02am May 28, 2015 9:02am
Quoting jcl366
I've just found this thread and scanned over it - very, very interesting! I would like to reproduce your results, but with different software than F4. As it uses simple linear regression and hourly bars, I suppose the results should be reproducible with any other platform. My question is - what is your experience with the robustness of the systems? Doe they completely fall apart when the time zone is a little different, such as by 1 hour, or when the weekend starts or end a little sooner or earlier? Also, have you checked if the results still hold...

  1. Yes, the timezone can make the systems fall apart. Cyclical volatility in the 1H timeframe is EXTREEEMELY important, I cannot say this enough. It is different to predict a market opening time than the market one hour before or an hour later. This is why you should correct things to use DST and why you should change the TZ. The weekly opening/ending times are less important but also have some influence (cyclical volatility...).
  2. Depends, the parameters here are definitely not like the period of an MA crossing system, changing the parameters changes the actual logic (the way in which trading decisions are made) significantly. Changing parameters is not like trading a different flavor of the same system it's more like trading an entirely different system most of the time. But I encourage you to do the exercise and see how parameters are affected.

Also I want to point out that things like "are the parameters like a broad hill or a peak" in my experience matter very little for real live trading performance when using machine learning algos. In my experience the most important thing is to use lots of data (20 or more years) and for the data mining bias analysis to give a low probability for systems to come from spurious relationships.

I have always failed when I have used little data or when I have attempted to trade without doing a very meticulous data mining bias analysis (as explained before on this thread).

I know that my experience is not everything but I am just telling you what has worked for me (perhaps others use different techniques that work as well).

  • #503
  • May 28, 2015 9:26am May 28, 2015 9:26am
Quoting algoTraderJo
{quote} You tell me
I don't know in my opinion a single output model would be easier to train, especially if you use algorithms like backpropagation, rprop etc....
  • #504
  • May 28, 2015 9:30am May 28, 2015 9:30am
Quoting ampleparking
{quote} I don't know in my opinion a single output model would be easier to train, especially if you use algorithms like backpropagation, rprop etc....
Easier to train, definitely. But better? Do some experiments and share your findings with us! Perhaps I have always been wrong about using 2 targets!
  • #505
  • May 28, 2015 9:38am May 28, 2015 9:38am
Quoting algoTraderJo
{quote} Easier to train, definitely. But better? Do some experiments and share your findings with us! Perhaps I have always been wrong about using 2 targets!
Yes, I mainly use Python (for backtesting too) so I'm trying to reimplement your models in Python... then I'll be able to share some experiments
  • #506
  • May 28, 2015 10:20am May 28, 2015 10:20am
You can even get good results for the GBPJPY using this linear regression model. (A=2 GMT+1/+2, B=80, C=3, D=12, SL=50% ATR(20)). Remember that as always the model is retrained before taking each trading decision.

Attached Image


This basic technique is already a generic mechanism to create profitable machine learnings algos across almost the entire forex space. Of course, this is of little use without the mining bias assessment (which is veeery critical) but the map of how to get to the final strategies for live trading should now be more than clear to those who listen... Or is it not?
  • #507
  • Edited May 29, 2015 4:12am May 28, 2015 11:54am | Edited May 29, 2015 4:12am
Quoting PipMeUp
On E/U with SL=30% ATR from 01/01/2001 to 27/05/2015, changing one parameter at a time:
A B C D
9 110 4 8 147.24%
8 110 4 8 -81.52%
10 110 4 8 -24.50%
9 109 4 8 198.52%
9 111 4 8 131.50%
9 110 3 8 113.25%
9 110 5 8 2.81%
9 110 4 7 192.90%
9 110 4 9 37.78%
Hmm, this looks as if the system, if it's not a random result, has indeed been trained on some seasonal effect. A=9 is the opening time in Europe. When the system is trained on the daily MFE/MAE, 7 or 8 would be the best D value. If we assume that this is the edge of the system, can these two parameters then be eliminated from the parameter space? This would make data mining bias estimates a lot easier.
  • #508
  • Edited 5:55am May 29, 2015 5:43am | Edited 5:55am
Quoting jcl366
{quote} Hmm, this looks as if the system, if it's not a random result, has indeed been trained on some seasonal effect. A=9 is the opening time in Europe. When the system is trained on the daily MFE/MAE, 7 or 8 would be the best D value. If we assume that this is the edge of the system, can these two parameters then be eliminated from the parameter space? This would make data mining bias estimates a lot easier.
A few things:

  1. It does not work on the daily timeframe (the system trains each example using hourly returns), this system requires use of the hourly timeframe and all the specified degrees of freedom. You can test this yourself.
  2. You cannot simply "fix and eliminate" parameters after a mining exercise. Finding a system through a complicated mining technique and then reducing parameters to find the same system on a smaller space is fooling yourself. When you reduce the mining space around an already known profitable system you are creating additional mining bias. When you do data-mining bias you need to use the same technique that originally produced the system you are evaluating, the process on random data needs to have the exact same freedom. There is no way around doing the mining bias assessment properly, sorry.

  • #509
  • May 29, 2015 7:41am May 29, 2015 7:41am
But the linear regression model is not a holy grail. There are complicated cases where the linear regression algo does not produce good results. The USD/CAD is an interesting example. Here are the best results that the linear regression algo could get:

Attached Image


No one would ever trade this. Basically no profit from 1998 to present. What could we possibly do to improve this result? Why is the USD/CAD much harder to predict than the other pairs? Why doesn't linear regression work here as well as it does on others? If you get the answers to these questions you will be able to produce something that works on the USD/CAD and improve results on other pairs as well... Give it some thought folks...

I'm listening!
  • #510
  • May 29, 2015 9:05am May 29, 2015 9:05am
Hello everybody,

I'm using Machine Learning algorithms for trading since 2010, my current system trading on 1min charts have 8 intsruments EURUSD GBPUSD AUDUSD GBPJPY AUDJPY USDJPY XAUUSD US500, the training size is 20000 bars, retraining every 24h. So my question is: Because of the size of training set and multiple intstruments is data mining bias removed from my results ??


Krzysztof

http://www.trade2win.com/boards/trad...-machines.html
  • #511
  • May 29, 2015 9:12am May 29, 2015 9:12am
Quoting fajst_k
Hello everybody, I'm using Machine Learning algorithms for trading since 2010, my current system trading on 1min charts have 8 intsruments EURUSD GBPUSD AUDUSD GBPJPY AUDJPY USDJPY XAUUSD US500, the training size is 20000 bars, retraining every 24h. So my question is: Because of the size of training set and multiple intstruments is data mining bias removed from my results ?? Krzysztof http://www.trade2win.com/boards/trad...-machines.html
Mining bias is never "removed" it is always present as a consequence of the mining process. If anything using more data and having more degrees of freedom introduces even more data-mining bias. This bias problem is always present in any strategy building process and you need to properly account for it. The amount of data has nothing to do with mining bias (it has to do with curve-fitting bias which is a different problem). You can always find something profitable in historical tests using some sort of machine learning algo (the more complex the easier) but the question is whether your results are spurious. This becomes more likely as you increase complexity (inputs, targets, topological variables, etc).
  • #512
  • May 29, 2015 9:34am May 29, 2015 9:34am
OK. From my experience I see that efficiency of my system depends of hard to say what ? There are days and weeks that it work well for all instruments (correlation ??) and days and weeks that it doesn't. Only explanation which comes to my mind is that when it doesn't signal to noise ratio is very low
or randomness very high of the market. usually it occurs for all instruments together. Do you think it has something with data mining bias ??

Krzysztof
  • #513
  • May 29, 2015 10:04am May 29, 2015 10:04am
good read on data mining bias http://www.priceactionlab.com/Blog/2...a-mining-bias/.

Jo, what do you think of "Myth No. 1: Data-mining bias can be measured" here specially the part about generating random data and applying mining to that.
Think : Trade : Live Life
  • #514
  • May 29, 2015 10:31am May 29, 2015 10:31am
Quoting stt
good read on data mining bias http://www.priceactionlab.com/Blog/2...a-mining-bias/. Jo, what do you think of "Myth No. 1: Data-mining bias can be measured" here specially the part about generating random data and applying mining to that.
Harris is just confused. Of course you can measure data-mining bias, this is just basic statistical hypothesis testing, anyone who knows statistics will tell you that. What is true is that low data-mining bias is a necessary but not sufficient condition for profitable trading under new market condition. You can create a system within a low mining bias process that uses data for only the past two months, of course it will fail going forward, it is curve fitted to just two months of data. The system may address some real (non-random) phenomena that happened during the past two months, but those are not general market characteristics. This is why I use more than 20 years of data. Curve fitting bias and data mining bias are two completely different things. I say it again, low data mining bias is necessary but not sufficient, it is one of the many things you need to be profitable in the real market.
  • #515
  • May 29, 2015 12:12pm May 29, 2015 12:12pm
Quoting fajst_k
OK. From my experience I see that efficiency of my system depends of hard to say what ? There are days and weeks that it work well for all instruments (correlation ??) and days and weeks that it doesn't. Only explanation which comes to my mind is that when it doesn't signal to noise ratio is very low or randomness very high of the market. usually it occurs for all instruments together. Do you think it has something with data mining bias ?? Krzysztof
I can only tell you that I would never trade without measuring data mining bias. The only way to discard it being an issue is to measure it and get an idea of whether your process suffers from bias issues. Before measuring you are just guessing and guessing never ends well in quantitative finance.
  • #516
  • May 29, 2015 1:21pm May 29, 2015 1:21pm
Quoting algoTraderJo
Of course you can measure data-mining bias, this is just basic statistical hypothesis testing, anyone who knows statistics will tell you that.
Can you explain us how you make this miracle? Machine learning is about infering a model from the data. Per definition you cannot measure how wrong your model is (=data mining bias) since you don't know the real underlying model; not even if there is one to be found. You can always perfectly fit any model to your data by increasing the allowed variance. An extreme example being EUR/USD is always 1.3000, plus or minus 7500 pips. As well as you can always make this variance as low as you want by increasing your tolerance to the bias (aka overfit). For example EUR/USD = a levy random walk, the model is the current price (variance=0). Not even you can estimate the variance of the noise since its depends on the definition of what the signal is, which is the model, which is what you search. This is no more than the bias-variance tradeoff.

Which basic null hypothesis can you offer us? What makes you believe the H0 itself is relevant?
I fully agree with the text
Quote
... such methods essentially rank the performance of an algo with respect to the performance of a large number of algos mined on random data and nothing more than that.
Essentially rejecting the validity of White's test
No greed. No fear. Just maths.
  • #517
  • May 29, 2015 1:40pm May 29, 2015 1:40pm
Quoting PipMeUp
{quote} Can you explain us how you make this miracle? Machine learning is about infering a model from the data. Per definition you cannot measure how wrong your model is (=data mining bias) since you don't know the real underlying model; not even if there is one to be found. You can always perfectly fit any model to your data by increasing the allowed variance. An extreme example being EUR/USD is always 1.3000, plus or minus 7500 pips. As well as you can always make this variance as low as you want by increasing your tolerance to the bias (aka overfit)....
I am not trying to measure how wrong my model is against the "real market model", as you have said, this cannot be done. I am referring to this as curve fitting bias, not data mining bias. We should be on the same page to discuss anything. This is what I mean:

1. curve fitting bias = how my model deviates from the general market model

2. data mining bias = whether the relationships found by my model within the historical data are spurious or not

You can have a model that does not approach the general market model (high curve fitting bias) but that did not find spurious but real relationships (low data mining bias). The example is the model built on 2 months of data. You can build a model over a period of 2 months that you can say did not find spurious relationships but the relationships found by the model are not general, your model describes the real, non-random behavior over those two months but fails to grasp the general market model. These are two very different things. There is a lot of confusion because many people use both terms interchangeably but they represent two entirely different types of bias (at least I separate both terms in this manner, to make things clearer).

Harris has the same problem you're having, when people talk about using random data and things like White's Reality check they refer to addressing the problem of spurious relationships not the problem of whether or not you are capturing the general market model. Obviously if you look at data mining bias (what I am calling data mining bias) as if it was curve fitting bias, the tests don't seem to make sense.
  • #518
  • May 29, 2015 2:24pm May 29, 2015 2:24pm
Quoting algoTraderJo
This is what I mean:
1. curve fitting bias = how my model deviates from the general market model
2. data mining bias = whether the relationships found by my model within the historical data are spurious or not
Ah ok. English language tends to be too weak sometimes. Like the word "random" for which I found more than 10 different translations in French (!).

Do I understand correctly?
1. Curve fitting bias is about the "degree of freedom" (complexity) of your model (and is what I understood indeed)
=> like the order of a polynomial to fit a set of points.

2. Data mining bias is about the relevance of the variables
=> like using the level of the sea in Venice to estimate the price of the bread in London.
No greed. No fear. Just maths.
  • #519
  • May 29, 2015 3:41pm May 29, 2015 3:41pm
Just to make it even more simple what i understand

curve fitting bias - if you have just 3 historical data points you can always fit some model to it but it will not generalize well as it catched pattern just form those points nothing more

data mining bias - you can fit a lot of models to those 3 points and one by luck will generalize well for a while due to luck. Like a flipping the coin, if you try enough times you can always get serie of ten heads but still this result was just luck.

So AlgoTradeJo. Do you measure data mining bias or not finally ?? If yes maybe you will give us a procedure ??

Krzysztof
  • #520
  • May 29, 2015 4:50pm May 29, 2015 4:50pm
Hi everyone. I have been following this thread for a while and would like to share my findings. Hopefully this will make things more useful and consistent.

I have attached some files. The first one is the E/U time series pre-processed from what PipMeUp uploaded to correct the time to GMT+1/+2 using DST. I have checked that the price agrees more or less (+/- 1 pip) with the trading records uploaded by algoTraderJo. I also uploaded a Matlab script test_params.m that computes MLE and MSE with parameters A = 9, B = 110, C = 4, D = 8 to compare with the values given by algoTraderJo, with the results uploaded in results_9_110_4_8.txt in case some people don't have Matlab.

From what I see, there is quite some discrepancies between my values of MLE and MSE compared to the values uploaded by algoTraderJo, which is worse than I expected. Of course it could be that my implementation was wrong. It would be great if PipMeUp could also upload these values computed from the same parameters, or algoTraderJo could provide even a short excerpt of the E/U time series for us to compute some values. Then we can hopefully be on the same page.
Attached File(s)
File Type: zip findings20150529.zip   728 KB | 288 downloads
1 trader viewing now