Machine Learning with algoTraderJo | Page 41 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #801
  • Dec 17, 2015 12:01pm Dec 17, 2015 12:01pm
Quoting ForexSamsam
{quote} Every day on A hour, how your machine know to not entrer in a trade ? In my understanding, to enter long, you use long>0 and short<0 but did you use minimun return value to enter ? If the idea is to predict the outcome of a trailing stop, why you dont set the minimun value as the trailing stop itself (20day ATR) ? You could get [1pip,-1pip], this case you will enter for long and you could never hit the trailing stop or close in reverse If the data is not continues and you consdider every A hour for trading, then you introduce difference...
Sometimes the ML algo will output long>0 and short>0 or long<0 and short<0 in that case you do not enter a trade. You also do not predict the outcome of the trailing stop, you predict the outcome of the trade that uses a given trailing stop. Since you are predicting the outcome of a trade as long as the output is positive, meaning that you will make money, you should enter the trade. The trailing stop can move such that price then reverses and hits the stop at a level that is equivalent to a 1 pip profit on the trade, if you predict this scenario then you should enter, just to make 1 pip. There is no minimum to make because the trailing stop can lead to trade exits that make any level of profit. There is no contradiction between training and trading if you're doing things correctly.
  • #802
  • Dec 17, 2015 12:04pm Dec 17, 2015 12:04pm
Quoting ForexSamsam
when you say A=8 .. This means A = 8PM in us system (20h in HH24) or 8AM ? thanks sam
8 means 8AM GMT+1/+2. This is actually before the European open which is at 10 AM GMT +1/+2 so a very relevant time for the EUR/USD. Also 20 GMT +1/+2 is not close to the London or NY closing times. London closes at 18 GMT +1/+2 and NY closes at 23 GMT +1/+2. I suggest you review your datetime offsets.
  • #803
  • Dec 17, 2015 12:13pm Dec 17, 2015 12:13pm
Quoting jorfermo
Hi algoTrader First of all thanks for such an interesting thread. I've been able to reproduce some of your results with F4 but I've noticed this algos are very feed dependant as you can see in the chart. Settings: SL = 160% ATR20 A = 1 GMT +1/+2 B = 90 C = 5 D = 90 {image} Is it always like that with this kind of logic? Is there any way to reduce the feed dependancy? Thanks!
Those differences seem acceptable to me. Don't they? What feeds did you use? There is a feed dependency you cannot escape but it never makes a long term difference in my live trading, even at the 30M timeframe. Just make sure you use some metric like the Sharpe/Sortino ratios to compare your systems and you should be ok. Reducing it can be very very very hard. To do this go to 1D or 4H, use simpler ML algos, use simpler trade handling... Bunch of things you can try.
  • #804
  • Dec 17, 2015 12:23pm Dec 17, 2015 12:23pm
Quoting dreamvalley
{quote} Hi algoTraderJo, If I understand this correctly, the outputs of the NN's would be the MLE and MSE in the following D hours, where C is the number of most recent returns to use? In all the examples that follow, I see that D is much bigger (115 in this case) than those used in the linear regression examples (8 or 10 or 12 etc.). Why would one want to use such big D when the strategy trades on H1 data? Also, beside looking at the final return achieved using the profit target / stop loss in trading, I think one fundamental benchmark is the accuracy...
I have talked about many ML algos, some using MLE/MSE but many using other outputs. The latest NN algos do not use the MLE/MSE they use the hypothetical result of trades taken with a trailing stop. Read those posts again if you want to get a better grasp about how the inputs/outputs are generated. You can use the MLE/MSE but you will get worse results IMHO.

Accuracy is also meaningless because accuracy does not mean profit, if you measure your algos in terms of prediction accuracy you will fail IMHO. Academics love to talk about accuracy in ML but in our case this does not matter because we don't care if we are right or wrong about whether the trade would have made 20 or 50 pips but what we care about is to be right when the trade would have made 500 pips and to be wrong when the trade would have made only 5. I said it before, focusing on accuracy is wrong because it does not matter if you are 90% accurate if whenever you are wrong you lose 10x what you win when you're right. What matters is the profit you make, that's how you should measure success in ML in trading.

As far as weights go I always start from random weights that are initialized between -0.5 and 0.5 on each training, no weights are saved or reused (this would introduce a ton of bias). Although this random initialization does mean that results could be different between backtests in principle in reality they are not. This is because the training always converges to roughly the same solutions due to the input/output structure that I am using. I can backtest the NN ML system I have shown here 1000 times and I always get the exact same thing. If I reduce training to something like only 10 iterations then I do see different results on each run, as long as your input/outputs have the right structure and you train for enough iterations you should always get the same result.
  • #805
  • Dec 21, 2015 2:53am Dec 21, 2015 2:53am
Hi algoTraderJo

How much time you trade live ML portfolios?
What the difference between (R^2,Sharp) backtest and live?

From my tests real and backtests don't have similar R^2 and sharp retio .
And the performance is feed dependent.
  • #806
  • Dec 22, 2015 6:37am Dec 22, 2015 6:37am
Quoting levkly
Hi algoTraderJo How much time you trade live ML portfolios? What the difference between (R^2,Sharp) backtest and live? From my tests real and backtests don't have similar R^2 and sharp retio . And the performance is feed dependent.
I have years of experience with the live trading of machine learning algos but just round 1-2 year trading algos created in the F4 framework. I cannot say that I have experienced any drop in R2 or Sharpe when going from backtesting to live trading. You need to learn how to compare this if you are going to say that your R2 or Sharpe dropped. Ar you using proper statistical testing or are you just comparing your backtesting values with values measures on a much shorter period in live trading?

You need to recalculate the distribution of expected Sharpe or R2 values for whatever period you have been live trading using your backtesting data. You will see that on your data for a 2 month period you will have sharpe values from close to 0 to probably above 50. Judging statistic using anything less than a year of trading and proper comparable statistics from the backtests adjusted to the live trading period is none-sense. For a 30 year simulation in 6 months you can experience a gazillion different Sharpe/R2 combinations. You need to calculate their distribution and then see if you're actually there. Use hypothesis testing, that's what it's there for.

If your performance is feed dependent then use your own broker's data to mine the strategies, that solves it. If that is not an option then test for robustness before you live trade, use something like oversampling to see how sensitive your system really is. Also make sure you properly measure the data mining bias of the strategies, if you don't do that then you have a high probability of long term failure.
  • #807
  • Dec 22, 2015 7:01am Dec 22, 2015 7:01am
Thanks

What hypothesis testing to you use (Only DMB test from asirikuy)?
How you select which systems to trade from the (sharpe distribution )? All system SR > 0.8 and number of live system > 100* random?
  • #808
  • Dec 22, 2015 7:12am Dec 22, 2015 7:12am
Quoting levkly
Thanks What hypothesis testing to you use (Only DMB test from asirikuy)? How you select which systems to trade from the (sharpe distribution )? All system SR > 0.8 and number of live system > 100* random?
For any question you have about a strategy or method you should establish a null hypothesis and create a test to evaluate it. These are the steps:

  1. Create the question
  2. Create the null hypothesis related to the question
  3. Create a test to attempt to negate the null hypothesis

Take this example:

  1. Question : Is my Sharpe in live trading worse than my Sharpe in back-testing?
  2. Null Hypothesis: The sharpe I have in live trading belongs to the distribution of sharpe results for that period length within back-testing results
  3. One possible test: Create a distribution of expected Sharpe values for the trading period length to be evaluated using the back-testing data, then see if your live trading sharpe ratio for that period length actually falls into that distribution. If it falls outside of the 4 sigma level then you can negate the null hypothesis with a 99% confidence.

I use many different hypothesis tests, for any question I have I follow the procedure above. I use the same data mining bias testing procedure they use at asirikuy but I use a whole other variety of tests. For example I use hypothesis tests related with oversampling, with broker dependency, etc. If you have the question "are my results feed dependent?" then you can create a hypothesis test to find out.

If you want to really be successful doing this then getting yourself some basic training on building and evaluating hypothesis tests is veeeery important. I would suggest taking a few of the basic statistics courses available at coursera where you can learn all the basics about how to do this if you don't know already.

  • #809
  • Dec 24, 2015 3:09am Dec 24, 2015 3:09am
Quoting algoTraderJo
But the linear regression model is not a holy grail. There are complicated cases where the linear regression algo does not produce good results. The USD/CAD is an interesting example. Here are the best results that the linear regression algo could get: {image} No one would ever trade this. Basically no profit from 1998 to present. What could we possibly do to improve this result? Why is the USD/CAD much harder to predict than the other pairs? Why doesn't linear regression work here as well as it does on others? If you get the answers to these questions...
Hi algoTraderJo and others,

First of all, thank you for running such an insightful thread! To me this place is truly a gold mine - I have re-read this thread half a dozen times and can still find meaningful knowledge in it. Perhaps this is because I am running my own little ML experiments (on futures data though) that reflects my learning curve.

The quoted post is from page 26 by the way, so if Jo and others think my current post is obstructing the thread's flow please kindly disregard it.

I know the more recent discussion on lower timeframes / switching to a trailing stop output structure somewhat tackles the problem already, but since a major theme of the thread is about output structure, I would very much like to have your comments on using the following output (coerced into a single 0/1 variable for starters):

  1. Calculate MLE/MSE the usual way, but skipping all bars between [12, 0] GMT +1/+2 in the calculation
  2. Output is 1 if MLE > MSE, 0 otherwise

The reason being that the pair seems to have a lot of volatility during the US hours but makes little net movement throughout the whole US session. This potentially distorts the MLE/MSE calculations, causing any model to fit to noise when trained on the samples when the outputs are constructed in the innocent way.

Of course we can see that the values 12 and 0 are arbitrary and will need to be included in the search space. This may lead to harder DMB analysis - but perhaps there are ways to trim down the search space a little bit, so that might not be a big concern either. My main concern is the soundness of this idea, so your comments are most appreciated!

  • #810
  • Dec 24, 2015 4:31pm Dec 24, 2015 4:31pm
Perhaps it was mentioned somewhere in this thread, but what software are you using to do your analysis?

Thank you.
  • #811
  • Dec 25, 2015 7:34am Dec 25, 2015 7:34am
Quoting forextrades
Perhaps it was mentioned somewhere in this thread, but what software are you using to do your analysis? Thank you.
F4 framework - http://asirikuy.com
  • #812
  • Dec 25, 2015 1:39pm Dec 25, 2015 1:39pm
Hi Nghei,

this is a good idea, at least I thought so, too. In my experiments I couldn't get any edge though.
But then again, I haven't done it exactly like you described. I tried omitting or packing data e.g. in one bar.

Cheers
KaBo
  • #813
  • Jan 6, 2016 5:32am Jan 6, 2016 5:32am
Quoting beachboy44
{quote} F4 framework - http://asirikuy.com
the problem with Asirikuy platform is poor live market access (Oanda, Dukascopy)
  • #814
  • Jan 6, 2016 12:24pm Jan 6, 2016 12:24pm
Quoting nghei
{quote} Hi algoTraderJo and others, First of all, thank you for running such an insightful thread! To me this place is truly a gold mine - I have re-read this thread half a dozen times and can still find meaningful knowledge in it. Perhaps this is because I am running my own little ML experiments (on futures data though) that reflects my learning curve. The quoted post is from page 26 by the way, so if Jo and others think my current post is obstructing the thread's flow please kindly disregard it. I know the more recent discussion on lower timeframes...
If you have an idea then I think you should code it and give it a try, this will give you more information than anything I can tell you. But think carefully about the things you're saying, you're already saying things that might not be true. For example you are implying that the "little net movement" under non-volatile hours is noise, when perhaps it is not. Do you have any real statistical evidence that says that this is noise and that no valuable information comes from these low volatility trading periods? I would advice you start building your ML algos without any assumptions, before you start building filters and modifications. If something is noise, then make sure you prove it's noise before you remove the information it may contain. Perhaps these "low volatility hours" contain vital information and you're discarding it only because of a preconception you have.
  • #815
  • Jan 6, 2016 12:27pm Jan 6, 2016 12:27pm
Quoting m.kreslik
{quote} the problem with Asirikuy platform is poor live market access (Oanda, Dukascopy)
This is actually not true. Using the F4 framework you can also trade in MT4 so you can trade the systems on any Metatrader 4 broker. The Asirikuy Trader that uses the direct broker API implementations only works with Oanda or Dukascopy but then it's not mandatory to use that program if you want to trade F4 systems. You can trade F4 systems through MT4, there are many members there who do.
  • #816
  • Jan 6, 2016 12:31pm Jan 6, 2016 12:31pm
Quoting forextrades
Perhaps it was mentioned somewhere in this thread, but what software are you using to do your analysis? Thank you.
There's the software from Asirikuy to do backtesting and live trading but I also use a lot of R for statistical tests and plotting. I also have some custom built libraries in python that I use to manage my mining operation.
  • #817
  • Jan 6, 2016 12:56pm Jan 6, 2016 12:56pm
Quoting beachboy44
{quote} F4 framework - http://asirikuy.com
Coming back from the holidays I see you have made some very clever advances in ML at asirikuy. I will give your ideas a try, see if anything interesting comes up
  • #818
  • Jan 6, 2016 1:06pm Jan 6, 2016 1:06pm
I want to float an idea that I now want to develop going forward on the thread. Up until this point we have built ML algos that have some characteristics in common:

  1. A system always uses the same number of examples through a test
  2. Examples are built sequentially in the past (for 1H systems we look for all examples to start on the same hour in the past while for daily systems we simply build examples sequentially by shifting one bar into the past)

What if we were more demanding in our example building? If we want to trade on a given bar then it makes sense that we attempt to trade our ML algo with examples that are somewhat similar to what has been seen in the recent past. What if we attempt to build an ML algo where examples are built using only past cases that "look similar" to present market conditions? Wouldn't this lead to more specific training and therefore better results? What do you think?

When we do the hourly systems we might not look for examples that start on the same hour but we can look for examples where recent market structure is similar, say we only train using examples where the OHLC arrangements happen to be the same for the past 3 bars... What will happen then? Can we get something totally different? How far can we scale in OHLC arrangement complexity? Can we even scale this to multiple timeframes?

Keep watching to see an example of this

Doesn't this scream pattern recognition immediately?

  • #819
  • Edited Jan 7, 2016 9:36am Jan 6, 2016 8:12pm | Edited Jan 7, 2016 9:36am
Quoting algoTraderJo
{quote} Coming back from the holidays I see you have made some very clever advances in ML at asirikuy. I will give your ideas a try, see if anything interesting comes up
Thank you - the more testers the merrier! In particular I am interested to know how the multiple trade outcomes affect DMB. On the one hand the search space is reduced as you no longer have to do a brute force optimization over the exit mechanism variables, but on the other hand it introduces a lot of complexity in the outputs.

If you do end up measuring DMB for any multiple trade outcome spaces, then I would be very grateful to know the result. Obviously I'm not asking for any specific values for any specific systems, it just would be nice to get a rough idea in case I'm fighting an uphill battle. My current computing resources are far too limited for me to be able to do many DMB measurements

Anyway, happy new year!
  • #820
  • Jan 7, 2016 11:54am Jan 7, 2016 11:54am
Quoting algoTraderJo
I want to float an idea that I now want to develop going forward on the thread. Up until this point we have built ML algos that have some characteristics in common: A system always uses the same number of examples through a test Examples are built sequentially in the past (for 1H systems we look for all examples to start on the same hour in the past while for daily systems we simply build examples sequentially by shifting one bar into the past) What if we were more demanding in our example building? If we want to trade on a given bar then it makes...
I think this is a good idea! When I was experimenting with using low DMB, historically profitable price patterns as inputs, I thought it would be nice to not be restricted to only those pairs for which we have such price patterns. This gives us a way to do just that (in a sense).

Furthermore, I suppose that this is more akin to how a human trader would approach the market. I'm definitely going to have a go at implementing this once I've finished the things I'm currently working on.
0 traders viewing now