Machine Learning with algoTraderJo | Page 20 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #381
  • May 7, 2015 9:46am May 7, 2015 9:46am
Quoting cipx90
I wonder what the result will be ,if USDJPY use EURJPY's settings?
These are the results for the A=0, B=130, C=9, D=12 configuration on the EURJPY and USDJPY:

Attached Image


Machine learning is not a "one size fits all" problem.
  • #382
  • May 7, 2015 10:23am May 7, 2015 10:23am
Quoting algoTraderJo
{quote} These are the results for the A=0, B=130, C=9, D=12 configuration on the EURJPY and USDJPY: {image} Machine learning is not a "one size fits all" problem.
Yes,It is a optimization problems about 4 parameters.
and What do you think about trading in Asian sessions based on M15 or M5 TF,due to the long-term trend will be influenced by the fundamentals and our model did not consider it.
  • #383
  • May 7, 2015 10:26am May 7, 2015 10:26am
Quoting PipMeUp
{quote} Not so much of a difference actually. FX doesn't have that many gaps. The bars usually open very near of the close of the previous one. I use the close price simply because I cannot use the open of the bar the strategy receives. Per definition the bar it receives is closed (otherwise it is not a bar). Its open price is the price as of one hour ago in the simulated time. Therefore I need the open price of the NEXT bar. I made the exercise of shifting all of the indexes by one in order to use the open price instead. I take the current (simulated)...
I don't see why you cannot use the bar open. If you are doing an event-driven tick simulation you should have access to the current bar open and all ticks since open without any problems. You should also have access to all previous bar OHLC values for the training. Using opening prices is fundamental, if I use closes my results are completely different. While you don't use opening prices we simply cannot compare apples to apples.

However I am completely sure that the F4 simulator does not allow data-snooping. That because of both my own tests in backtesting and due to my live trading comparisons with backtesting which give me identical results when using the same data. I use several ML mechanisms similar to the ones I have posted in live trading and after many months of testing I am confident in the accuracy of the results.

About the data, the log is quite complicated so I'm not sure I'll have the time to build it up. If I happen to get sometime this week I'll get it done and post it here (no promises however). I will use 2015 data to make it easier for you. Man, I feel tempted to buy you an F4 license just so that we can truly compare things up
  • #384
  • May 7, 2015 10:33am May 7, 2015 10:33am
Quoting cipx90
{quote} Yes,It is a optimization problems about 4 parameters. and What do you think about trading in Asian sessions based on M15 or M5 TF,due to the long-term trend will be influenced by the fundamentals and our model did not consider it.
I did not understand the question, please rephrase.
  • #385
  • May 7, 2015 10:42am May 7, 2015 10:42am
Quoting algoTraderJo
{quote} No cross validation. The machine learning method is already cross-validated since I retrain before each signal, the whole backtest is a cross-validation exercise. I see how the optimization of the machine learning variables can introduce curve fitting bias but I don't do anything to attenuate this fact besides using more than 20 years of data. This works for me, if I use less data (2007-2015 for example) I do get into curve fitting issues when going live. I use many different things for building of machine learning algorithms but this has...
I understand your view about using walk forward to avoid over optimization. However what do you think of these two papers on this subject:

http://papers.ssrn.com/sol3/papers.c...act_id=2528780
http://papers.ssrn.com/sol3/papers.c...act_id=2308659

Data mining bias can come in even when you are not optimizing parameters directly. The fact that you tested many methods before finalizing one is itself a sort of optimization. I guess if you are doing a lot of experiments, then hurdle rate should be higher than usual (t-stat > 2) to call it significant.

I will be very interested in how you do data mining bias assessment. The first paper here has one way (i should warn you that when that method was applied to most known factors for equities returns, none of them survived including fama french's 5 factor model :-)).
Think : Trade : Live Life
  • #386
  • May 7, 2015 10:50am May 7, 2015 10:50am
Quoting stt
{quote} I understand your view about using walk forward to avoid over optimization. However what do you think of these two papers on this subject: http://papers.ssrn.com/sol3/papers.c...act_id=2528780 http://papers.ssrn.com/sol3/papers.c...act_id=2308659 Data mining bias can come in even when you are not optimizing parameters directly. The fact that you tested many methods before finalizing one is itself a sort of optimization. I guess if you are doing a lot of experiments, then hurdle rate should be higher than usual...
Yes, I do an extensive data mining bias evaluation. I use a variation of White's reality check plus a method similar to the one suggested on the paper you posted. Still too early on the thread to deviate to this topic. I'll start talking about mining bias evaluation much later on.
  • #387
  • May 7, 2015 10:58am May 7, 2015 10:58am
Quoting algoTraderJo
{quote} I don't see why you cannot use the bar open. If you are doing an event-driven tick simulation you should have access to the current bar open and all ticks since open without any problems. You should also have access to all previous bar OHLC values for the training. Using opening prices is fundamental, if I use closes my results are completely different. While you don't use opening prices we simply cannot compare apples to apples. However I am completely sure that the F4 simulator does not allow data-snooping. That because of both my own...
Certainly just a wording misunderstanding. When you say "the current bar" I think you mean the candle which is currently forming and not yet closed. It is now 17:54 my time so you mean the bar that started at 17:00 and will close in 6 minutes. To me this is just not a candle! It only exists in the internal state of the chart building the H1 candles. To me "the current bar" is the bar that opened at 16:00 and closed at 16:59:59.999.

When the strategy gets this bar it can of course ask to the chart for its internal candle, but it may be a null value (depending on the implementation of the chart: time, renko, ...). Especially in simulation it is always null for performance reason. The strategy can ask the instrument what is its current price, which is the last tick received/simulated and is the tick of the open of the next bar to come.

Quote
Man, I feel tempted to buy you an F4 license just so that we can truly compare things up
If it was written in java I might have considered. But it is quite expensive to just give a quick eye and be unable to use it after...
No greed. No fear. Just maths.
  • #388
  • May 7, 2015 11:01am May 7, 2015 11:01am
Quoting algoTraderJo
{quote} Yes, I do an extensive data mining bias evaluation. I use a variation of White's reality check plus a method similar to the one suggested on the paper you posted. Still too early on the thread to deviate to this topic. I'll start talking about mining bias evaluation much later on.
I don't want to be mean but the GA you used to get A, B, C and D before backtesting over the whole set is already data snooping.
No greed. No fear. Just maths.
  • #389
  • May 7, 2015 11:11am May 7, 2015 11:11am
Quoting PipMeUp
{quote} I don't want to be mean but the GA you used to get A, B, C and D before backtesting over the whole set is already data snooping.
Any observation that you carry out on historical data to build a trading system is some form of data snooping and introduces mining bias. I account for all of this when I measure data mining bias.
  • #390
  • May 7, 2015 8:46pm May 7, 2015 8:46pm
Quoting algoTraderJo
{quote} I did not understand the question, please rephrase.
I mean that the model may have a better performance when doing short-term predict based on M15 or M5 TF.
  • #391
  • May 8, 2015 4:04am May 8, 2015 4:04am
Quoting KaBo
Hello algoTraderJo, glad to see you back I am also following with great interest! As I subscribed to the Asirikuy framework I wondered sometimes if algoTraderJo was Daniel himself since, if I am not mistaken, you left us with similar questions and ML achievements/results as realised in the community There is a lot of valuable info in your thread as is in Daniels and the community forum - great! But the community seems to put more effort on the price action systems. I enjoy more the approach with ML and try to learn from you...
Can you reproduce the results posted by Jo with the Asirikuy framework?
No greed. No fear. Just maths.
  • #392
  • May 8, 2015 6:34am May 8, 2015 6:34am
Quoting PipMeUp
{quote} Can you reproduce the results posted by Jo with the Asirikuy framework?
I sent him all the code to reproduce the USDJPY ensemble results yesterday (which he requested) there should be no problem reproducing. If there is it should be easy to spot.
  • #393
  • May 8, 2015 9:25am May 8, 2015 9:25am
Yes, I have everything to reproduce the results (thanks for the files). I just have been so tied up in job and family the past few days that I had no chance to really follow up. With more practice it might be plug and play - for me the framework is still a huge world of dependencies to understand.
This weekend I will have time and post my results.
  • #394
  • May 8, 2015 11:54am May 8, 2015 11:54am
Quoting KaBo
Yes, I have everything to reproduce the results (thanks for the files). I just have been so tied up in job and family the past few days that I had no chance to really follow up. With more practice it might be plug and play - for me the framework is still a huge world of dependencies to understand. This weekend I will have time and post my results.
If you have any trouble send me a PM and I'll help you out.
  • #395
  • Edited May 11, 2015 5:44am May 10, 2015 2:04pm | Edited May 11, 2015 5:44am


After a few little starting issues, I got it going perfectly (thanks TraderJo!),
and I am now able to replicate the results of TraderJo in the framework. My display is not quite there yet (cosmetics), as TraderJo does it in R.

The framework is big, powerful and a whole world of dependencies, certainly not like you take regular Software for a ride - got to be up for a little more homegrown scripting with all the advantages and disadvantages it brings.

I'm glad I can now replicate the systems, and experiment myself. I haven't given up on my own stuff (simple framework with other software), but this is great right now. Looking forward to continue the learning
  • #396
  • May 11, 2015 4:49am May 11, 2015 4:49am
Just for clarification, is the D value in hours or days?
  • #397
  • May 11, 2015 5:42am May 11, 2015 5:42am
Quoting beachloup
Just for clarification, is the D value in hours or days?
In hours
  • #398
  • May 12, 2015 10:43am May 12, 2015 10:43am
I am very interested in how you go about choosing systems for ensembles.
From what I understand, you

1. Have an idea about target (mse/mle regression in the last case), inputs (bars returns) and setup (hourly / daily with 4 Parameters to optimize)
2. Generate lots of systems with different parameters optimizing for Sharpe / R2
3. Pick individual systems that perform good
4. ... but how do you pick systems that perform good together as ensembles.

As Asirikuy generates the system and does the Backtest at the same time, do you save generated systems for possibly later including them in ensembles?
Or is the generation of ensembles always from scratch?

In my previous process, I figured to generate lots of systems and then try to combine them for an ensemble. But it ends in the problem of dimensionality.
e.g. taking 500 systems - picking the best 10 to work together (if my math is right) = 500! / (500! - 10!) = too many combinations to test in a lifetime.

How do you go about it? Are you combining systems randomly or do you preselect systems? Is there some mathematical approach to selecting the systems that work good together besides the actual backtest with the combination.
This is actually a question you asked us before, but we always answered with the actual backtest to make a decision whether a combination works good together.
  • #399
  • May 12, 2015 4:22pm May 12, 2015 4:22pm
Quoting KaBo
I am very interested in how you go about choosing systems for ensembles. From what I understand, you 1. Have an idea about target (mse/mle regression in the last case), inputs (bars returns) and setup (hourly / daily with 4 Parameters to optimize) 2. Generate lots of systems with different parameters optimizing for Sharpe / R2 3. Pick individual systems that perform good 4. ... but how do you pick systems that perform good together as ensembles. As Asirikuy generates the system and does the Backtest at the same time, do you save generated systems...
This is really where things get the most complicated. You cannot possibly do this based on brute forcing combinations, as you point out there are too many possible ways to combine things to figure out how to build an ensemble. Your questions:

  1. I do not combine machine learning algorithms randomly for the building of ensembles, I have a clear methodology for this. Take into account that combining systems is not the same as combining machine learning algorithms. A system is an entire trade decision mechanic and combining systems generates a portfolio while combining machine learning algorithms generates an ensemble (a single system that makes trading decisions based on the prediction of different ML techniques).
  2. You need to make a clear distinction between system combinations (portfolios) and machine learning algorithm combinations (ensembles). There are different techniques to do both. I have a clear method to know how to build fruitful ensembles and a method to build fruitful portfolios as well. Yes, there are clear mathematical methods to do both. Tip: it is much harder to build ensembles.

As you say, these are questions I have asked before because they are important. I will not spoon feed the answers on the thread (as it took me years to get them!) but I would love to guide you folks through the process and help you with any guidance I can. For those who want to do the work I am more than willing to be helpful. Ask questions, do experiments.

  • #400
  • May 13, 2015 12:05am May 13, 2015 12:05am
Hi algoTraderJo,


Thanks for your posts ,It's very helpful.

1. You told that you combine other fin markets to ML input. How do you choose that markets ,Any known tests?
2. Do you use any other inputs except last price returns ?
3. Do you have recommended book to read about ensembles that can be useful for fin markets.
4. In order to reduce long drawdowns , What do you recommend to improve the ML algo, use lower TF, Use ensembles ?

Thanks
Lev
1 trader viewing now