Machine Learning with algoTraderJo | Page 35 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #681
  • Sep 29, 2015 7:33am Sep 29, 2015 7:33am
Quoting nonlinear
{quote} If the trade count of each system is the same and of a statistically significant amount, then I am not so certain that a daily system that traded for 10 years is much more curve fitted than the system that traded for 2 months on M15. The thought is that that the system only needs to see a statistically relevant number of trades over a given test window, whatever that window may be (I don't know this to be true yet). All this keeps leading me to question how many trades are a statistically significant sample size.
TraderJo has a point. 2 months of M15 data would never come close to representing the various and changing market conditions over 10 years of daily data.

I would put my confidence in a system that generates a net profit from 10 trades over 10 years than 1000 trades in 1 year of M15 data.
  • #682
  • Sep 29, 2015 8:49am Sep 29, 2015 8:49am
Quoting nonlinear
{quote} If the trade count of each system is the same and of a statistically significant amount, then I am not so certain that a daily system that traded for 10 years is much more curve fitted than the system that traded for 2 months on M15. The thought is that that the system only needs to see a statistically relevant number of trades over a given test window, whatever that window may be (I don't know this to be true yet). All this keeps leading me to question how many trades are a statistically significant sample size.
You're partly wrong. I will explain. Imagine there is a trading instrument that went up from 2005 to 2010 and then down from 2010 to 2015. You create a system for 2 months in 2015 when the instrument was going down. Even if you have 10,000 trades, your system is curve fitted to a set of market conditions where the instrument was going down. If the drift changes and it goes up as it did in 2005-2010, you go broke. If you have a system built with data from 2005 to 2015 then you have the up and the down condition included -- your less fitted. Of course this is a very black&white example but it validates my point.

Think about market conditions as the number of ticks faced by your strategy. A strategy you built using two months faced 2 months worth of ticks, a strategy you tested on 10 years had to pass through 10 years worth of tick data. If both have the same number of trades then the strategy over 10 years is much less curve fitted. it had to face many more ticks.

What matters in curve fitting is not the number of trades but the actual number of market conditions (ticks) you face. You're correct in that a system needs to have a statistically significant sample size but it also needs to face enough market conditions. You need both a trading frequency as large as possible and a backtesting period as long as possible. It is super super hard to get this.
  • #683
  • Sep 29, 2015 9:00am Sep 29, 2015 9:00am
Quoting Atokys
{quote} TraderJo has a point. 2 months of M15 data would never come close to representing the various and changing market conditions over 10 years of daily data. I would put my confidence in a system that generates a net profit from 10 trades over 10 years than 1000 trades in 1 year of M15 data.
It depends on ones point of view, I guess. One approach is to represent the various and changing market conditions over 10 years of daily data, which has much merit. Another might be to represent what the market is doing recently -- this also has some merit. Nearly everything I read says the longer the historical test window the better, but without test results I am questioning that approach, especially with systems that trade more often and that relearn/adjust. It might sounds counter-intuitive, but I am curious to learn more about a robust sample size, regardless of the size of the test window and trading TF. Perhaps the sample size should dictate the historical test period.

System one: Learned over 10 years of data, trades on average 30 times per year, for 300 trades.

System two: Learned over 6 months of data, trades on average 50 times per month, for 300 trades.

Testing system two over 10 years and relearning daily, might be "robust", but it also seems like statistical overkill. I hope am not detracting from this thread too much, just topics I have been thinking about lately.
  • #684
  • Sep 29, 2015 9:00am Sep 29, 2015 9:00am
Quoting levkly
Hi AlgoTraderJo, 1. Do you paper trade the strategies before going live? if yes for how long? 2. How you build the portfolios? Equal weight ?
1. Never, I always go live with strategies as soon as I validate them. I have come to believe that strategies have "expiration dates" after you find them, if you take your time to paper trade them you're wasting precious time you could be reaping profits.

2. I do different types. Now I'm testing merit based approaches. I start trading them all the same. I then rebalance risk every week against live trading performance. I put more weight on systems that make money and take away from those that have bad times. But there are tons and tons of things you can do.
  • #685
  • Sep 29, 2015 9:06am Sep 29, 2015 9:06am
Quoting nonlinear
{quote} It depends on ones point of view, I guess. One approach is to represent the various and changing market conditions over 10 years of daily data, which has much merit. Another might be to represent what the market is doing recently -- this also has some merit. Nearly everything I read says the longer the historical test window the better, but without test results I am questioning that approach, especially with systems that trade more often and that relearn/adjust. It might sounds counter-intuitive, but I am curious to learn more about a robust...
IMHO retraining very often (on every decision) and testing this for long periods of time is not statistical overkill, it's what's necessary. Keep on testing, you'll start seeing this yourself, especially when you face live trading.
  • #686
  • Sep 29, 2015 9:08am Sep 29, 2015 9:08am
Quoting algoTraderJo
{quote} You need both a trading frequency as large as possible and a backtesting period as long as possible. It is super super hard to get this.
Hhm, I know what you are saying, algoJoe, but I want to test these points (difficult to do, of course) -- I am not sure this is the case and have not seen it proven otherwise.

Excellent contributions.
  • #687
  • Sep 29, 2015 9:18am Sep 29, 2015 9:18am
Quoting nonlinear
{quote} Hhm, I know what you are saying, algoJoe, but I want to test these points (difficult to do, of course) -- I am not sure this is the case and have not seen it proven otherwise. Excellent contributions.
Definitely, test and test and test!
  • #688
  • Sep 29, 2015 9:21am Sep 29, 2015 9:21am
Quoting nonlinear
{quote} It depends on ones point of view, I guess. One approach is to represent the various and changing market conditions over 10 years of daily data, which has much merit. Another might be to represent what the market is doing recently -- this also has some merit. Nearly everything I read says the longer the historical test window the better, but without test results I am questioning that approach, especially with systems that trade more often and that relearn/adjust. It might sounds counter-intuitive, but I am curious to learn more about a robust...
In my opinion the only significant data point is the daily OHLC as it represents a full trading day. Anything lower is just noise.

Anyway, regardless of the size of the historical test window, your backtest results will never be an accurate representation of future results. Backtesting only serves the purpose of validating a system's logic and estimating risk. If you fully understand your system you would not have to even backtest it.
  • #689
  • Sep 29, 2015 9:29am Sep 29, 2015 9:29am
Quoting Atokys
{quote} In my opinion the only significant data point is the daily OHLC as it represents a full trading day. Anything lower is just noise. Anyway, regardless of the size of the historical test window, your backtest results will never be an accurate representation of future results. Backtesting only serves the purpose of validating a system's logic and estimating risk. If you fully understand your system you would not have to even backtest it.
lol, all my tests are live forward tests
  • #690
  • Sep 29, 2015 9:31am Sep 29, 2015 9:31am
Quoting nonlinear
{quote} lol, all my tests are live forward tests
Whats the difference? May as well do a backtest and not have to wait.
  • #691
  • Sep 29, 2015 9:38am Sep 29, 2015 9:38am
Quoting Atokys
{quote} Whats the difference? May as well do a backtest and not have to wait.
Night and day, my friend.
  • #692
  • Sep 29, 2015 9:40am Sep 29, 2015 9:40am
Quoting nonlinear
{quote} Night and day, my friend.
Wait, are you referring to a methodological difference or just a difference in data recency?
  • #693
  • Sep 29, 2015 10:00am Sep 29, 2015 10:00am
This is off topic here, but numerous variables affect live forward trading that are not present in testing, especially in crap mt4 and if you are using indicators, etc (I don't). I run some stats analyses with good historical data on dynamic and multiple price-action based systems (mostly for exits and not with mt4's back tester) and then forward test live on low leverage. If it works, I ramp up. It's a learning process like everything else. Because I'm working with stats, I'm keenly focused on my sample size, particularly if I am getting similar exit levels over variable sample sizes, which led me to interrupt algojoe's thread
  • #694
  • Edited 10:33pm Sep 29, 2015 2:05pm | Edited 10:33pm
Quoting nonlinear
{quote} Nearly everything I read says the longer the historical test window the better, but without test results I am questioning that approach, especially with systems that trade more often and that relearn/adjust.
I used to hold the exact same view. Then I tested and found that my strategies with high trade counts on shorter term data didn't hold up very well on new data. So I went back and took a lot of time and did testing on longer periods. I found that the strategies I had found didn't perform that well over a longer period, meaning that their performance was likely overfit to the shorter time period and this was the main reason that they didn't generalize to new data. In my view overfitting is enemy #1, and one of the only ways to really have confidence you aren't overfitting is to use a longer data period, as in more days, not just more data points (ticks).

In addition to high trade count, a second component is equally important: whether the data sample is representative of the unseen future data population. So when you test on 6 months of data, you have much less assurance that the conditions present in that short period will be reflective of all possible future conditions because it may be dominated by a finite and incomplete set of conditions. Systems and system performance are products of the limited data set that they are trained and evaluated on, meaning they can only "learn" something that is in the data.

So to validate that your 6 month period is reflective of future periods, a rolling window could be used to backtest this process over a longer time period when it is reoptimized walking forward. I learned this from algoTraderJo who reoptimizes his models each bar so that the rules/analysis being applied change on each bar, and retraining in this manner seems to be working well for my models. This gives the models an adaptive component and validates the process because the reoptimization process is tested thousands of times, making it much more difficult to overfit the process.
  • #695
  • Sep 29, 2015 4:35pm Sep 29, 2015 4:35pm
Thanks FXEZ, always helpful to hear others' insights on these topics. I am currently rerunning a full year every week, but that is only because it takes my i7 a full day to run some serial computations on one pair. It's a real pain.
  • #696
  • Edited Sep 30, 2015 4:10am Sep 29, 2015 10:31pm | Edited Sep 30, 2015 4:10am
I hear you nonlinear, I've got 8 machines cranking away right now and it just isn't enough processing power. Dealing with tick data really limits the speed at which this "factory" can produce trading strategies. They're still amping up - maybe a couple more weeks then they'll hit critical mass. I've been browsing for info on GPUs but technically it looks like a bit of a steep hill to get something up and running based on where I am currently (windows based, C#) as a one man development shop.
  • #697
  • Sep 30, 2015 6:39am Sep 30, 2015 6:39am
Yep, I am in the same boat, more or less. Been looking a CUDA based GPU's off and on, salivating. lol
  • #698
  • Sep 30, 2015 7:03am Sep 30, 2015 7:03am
Quoting FXEZ
I hear you nonlinear, I've got 8 machines cranking away right now and it just isn't enough processing power. Dealing with tick data really limits the speed at which this "factory" can produce trading strategies. They're still amping up - maybe a couple more weeks then they'll hit critical mass. I've been browsing for info on GPUs but technically it looks like a bit of a steep hill to get something up and running based on where I am currently (windows based, C#) as a one man development shop.
Quoting nonlinear
Yep, I am in the same boat, more or less. Been looking a CUDA based GPU's off and on, salivating. lol
At Asirikuy they do have GPU system mining. With that software you can do 1 billion of system tests a day... Mind boggling. The software is limited to price action based strategies but still, have never seen that sort of mining power elsewhere.

I use GPU mining for the testing of ensembles and can achieve 500-700 million ensemble tests per day... I had to hire a couple of specialized programmers to get this done, it's really hard to code those openCL/CUDA things for backtesting so I preferred to get that done professionally.
  • #699
  • Sep 30, 2015 8:41am Sep 30, 2015 8:41am
Quoting KaBo
Hi Joe, great to see you back with more ml-ideas. I have been developing and testing the past months. As you say, there is a lot of effort required. Even when you have a working framework there are still a million ways to adjust the input-output-system entry-exit structure and computing time frustrates on top of it. {quote} On one of my systems, which is very loosely a variation of the systems you presented in this thread, I did measure the DMB. However, at this time not to the extend I wish I could. I calculated the original on 28 years of data...
I want to add that the current "Munay" system in the F4 framework contains some nice trailing stop functions that give you a lot of room to explore. IMHO low DMB systems for the GBP/USD can be found using this system if you use the right set of variables to explore. You can probably use this as a guideline of where you can find highly profitable systems that might lead to low DMB. Using the set file I attach you can get this result:

Attached Image
Attached File(s)
File Type: zip set-gu1-munay.zip   < 1 KB | 349 downloads
  • #700
  • Sep 30, 2015 9:34am Sep 30, 2015 9:34am
Instead of that DMB analysis, can't we use the standard train/cross-validate/test dataset splitting? Train on the train set, tune hyper-parameters on the validation set, test on the test set.
1 trader viewing now