Finding The Most Robust Parameters Using Optimization | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Backtesting Optimization parameters 0 replies

Optimized Parameters in Metatrader Testing? 3 replies

OrderSend parameters change? 3 replies

pivot points and MACD parameters 0 replies

Can parameters run on different periods in the same EA? 4 replies

Attachments: Finding The Most Robust Parameters Using Optimization
Exit Attachments

Finding The Most Robust Parameters Using Optimization

  • #1
  • First Post: Edited 12:58pm Dec 19, 2021 6:32am | Edited 12:58pm
Introduction

Most Expert Advisors have one or more parameters that affect its behavior. Most trading platforms offer functionality to "optimize" these parameters over a given date range. The goal is to search for the best parameters of which we believe it remains profitable for while even after the backtest date range. The question however is: "How can we select the most robust setting from our backtest results?". Robust meaning here: most likely to also be profitable in the near future.

Simply selecting the most profitable parameters from a backtest is not a good idea. They are often not the most robust parameters, as these were most likely the result of a few "lucky" trades. So which parameters should we choose? Choose the 90th-percentile most profitable parameters? Look at Sharpe? Some combination of parameters?

Example

As an example to try out some ideas, lets consider a very simple breakout strategy for MetaTrader 5. This Expert Advisor only has two parameters. If we try all combinations of the parameter values and backtest over a 6 month date range, we get the following profits:
Attached Image (click to enlarge)
Click to Enlarge Name: Backward 3D.png Size: 195 KB

Using these same parameters for the following 2 months, gives the following profits:
Attached Image (click to enlarge)
Click to Enlarge Name: Forward 3D.png Size: 173 KB

(the 6 month backtest interval obviously had higher profits than the 2 month forward test interval, but somehow MetaTrader chose weird red/green colors)

Looking at the backtest, the plateau around parameter value 30 looks nice and stable, and values around 15 are to be avoided. In the forward test the plateau turned out to be a good choice, but surprisingly value 15 would have been a very good choice!

In both graphs the highest profits can be found at the upper righthand side, but the chaotic nature clearly shows these are mere luck and certainly not stable, robust settings. We can look at it another way by sorting the profits and creating a profit distribution:
Attached Image

In my opinion, the excessive rise of the profits at the righthand side is a sign that the corresponding parameters were just "lucky". Similarly, the lefthand side was just unlucky. Personally I avoid the parameter settings outside the 10th- to 90th-percentile interval as these are probably not robust in future markets. What do you think?

It is reassuring to see that more than 63% of the possible parameter values resulted in profit. The strategy is clearly profitable, but how do we tune the parameters?

Can we perhaps use the various metrics provided by the MetaTrader Strategy Tester software?
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 40 KB

In the following graphs, I plotted the profits during the backtest interval versus the various metrics:
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 127 KB

These graphs look reasonable (e.g. higher Sharpe Ratio is higher profits).

Now see what happens if we plot the profits during the forward test interval versus the metrics of the backtest interval (note that the backtest interval is 6 months, and the forward test interval 2 months, hence the profits are lower):
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 229 KB

There is almost no connection between the metrics and the profit in these point clouds. For clarity, here we overlay both graphs:
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 188 KB


How can we possibly select the best parameters based on the backtest metrics?
  • #2
  • Dec 19, 2021 9:31am Dec 19, 2021 9:31am
Here is the data used to generate all the graphs:
Attached File(s)
File Type: xlsx Test Results.xlsx   104 KB | 479 downloads
  • #3
  • Dec 19, 2021 1:38pm Dec 19, 2021 1:38pm
@yoriz -- I like your data analysis. However, I suspect you are using 'Open' prices when performing calibrations, since using tick data would require years. Calibrations using 'Open' prices rarely (if ever) play out in real-time as they appear in the Strategy Tester simulations. This is a well-traveled road -- you are probably human #570,000 who has gone down this path.

A more realistic approach would be to create a portfolio EA -- that is, a single EA that executes multiple strategies, each with their own MagicNumber so you can keep track of the bookkeeping. Here -- the different strategies in the portfolio EA are really the same strategy / algebra but using different parameter settings. Run the portfolio EA in a demo account for a month to compare how the different parameter settings perform in real-time with tick input data.

And then, if you are lucky, you choose the parameter settings that performed best over the past month in your demo account. And, if you are unlucky, the market pattern may have changed over the past month and you have to start over again. Calibrated settings don't last forever -- they are only valid over the data horizon used when they were first computed. And back-testing over many years of data is one of the dumbest things people do -- as if an algebraic formula (i.e. your strategy) learns and gets smarter the more historical data you feed it. This is not AI -- it's just simple algebra. The more data you feed it only skews the results to favor older data.
  • #4
  • Dec 19, 2021 2:00pm Dec 19, 2021 2:00pm
If you want to find robust parameters, you should also include time factor. When to take a trade? On Monday? Wednesday?
I have the secret sauce, but different (from broker to broker) market datas make it difficult. So reliable market data also should be considered. I'm struggling to get reliable market data, so I compare bunch of brokers for similarity. Forex is nothing if reliable price data can be found. Especially candle open and close price.
  • #5
  • Dec 19, 2021 2:50pm Dec 19, 2021 2:50pm
Quoting robots4me
I suspect you are using 'Open' prices when performing calibrations, since using tick data would require years.
No, I am using real ticks including spread and slippage simulation (1000 ms) in the MT5 strategy tester. This is just a simple breakout strategy using only price action. Without any machine learning involved backtesting is very fast. There are 1250 different parameter combinations. MT5 needs 15 minutes to try them all for a 6 month interval consisting of 32 million EURUSD ticks (2020-01-01 to 2020-07-01).

Quoting robots4me
A more realistic approach would be to create a portfolio EA [...] Run the portfolio EA in a demo account for a month
Thanks for the suggestion, but why would I wait a month if there are years of historical data available? Unless you are doing HFT, the normal MT5 strategy tester is accurate enough to test your average retail strategy.

Quoting robots4me
back-testing over many years of data is one of the dumbest things people do
That depends how you use the data; walk forward simulation makes perfect sense. What is a forward test today, is a backtest tomorrow. Why should I not use historical data to see what parameters I would have used if I would have gone live a few years ago? And then use the historical data to check what profit that would have given? And then move a few months later and do the exercise again. Working my way through time until today will give me the same equity curve as if I had done that actual process in the live market over the last years.

The topic of this thread is how to determine what parameters to use in each step. Also, how many history to use in the backtest (now 6 months, but should we take more)? How long remain the parameters valid (should we re-optimize every weekend)? This can all be tried out using historical data.
  • #6
  • Dec 19, 2021 3:02pm Dec 19, 2021 3:02pm
Quoting Ihlas
If you want to find robust parameters, you should also include time factor. When to take a trade? On Monday? Wednesday?
How do you suggest to include the time factor? Should I search for different parameters per day of week?

This is just a simple EA, but for more advanced EAs using machine learning I always avoided to use time-of-day, time-of-week or day-of-week as an input because I fear that will either lead to curve fitting, or specialization of the NN/RF learning to see larger moves during London and NY session. I loose part of my NN/RF to encode all these different regimes. I rather normalize the data (scale by average price range per time of day) and don't tell the NN/RF what time it is.

Quoting Ihlas
I'm struggling to get reliable market data. Forex is nothing if reliable price data can be found.
I hear this more often, but don't quite understand. Unless you are doing HFT and need sub-pip accuracy, how is market data so important? My strategies have SLs/TPs between 10 and 100 pips. At that scale price data does not need to be so accurate anymore, right?
  • #7
  • Dec 19, 2021 3:56pm Dec 19, 2021 3:56pm
Quoting yoriz
{quote} How do you suggest to include the time factor? Should I search for different parameters per day of week? This is just a simple EA, but for more advanced EAs using machine learning I always avoided to use time-of-day, time-of-week or day-of-week as an input because I fear that will either lead to curve fitting, or specialization of the NN/RF learning to see larger moves during London and NY session. I loose part of my NN/RF to encode all these different regimes. I rather normalize the data (scale by average price range per time of day) and...
Searching for different parameters per day of week can be incredibly eye-opening and profitable. Also if we know the price is in the corner. Even it's just EA. Price doesn't reverse in a trend, but time. Time (I have seen this) is the leading indicator for reversal or continuation. IMHO any EA that doesn't include day of the week can't be profitable in the long run.
Look at different brokers, they all have different opening price. How can we know if it retested the daily open?
At your situation it doesn't need to be that right.
  • #8
  • Edited Dec 20, 2021 2:46am Dec 19, 2021 10:24pm | Edited Dec 20, 2021 2:46am
Quoting yoriz
{quote} No, I am using real ticks including spread and slippage simulation (1000 ms) in the MT5 strategy tester. This is just a simple breakout strategy using only price action. Without any machine learning involved backtesting is very fast. There are 1250 different parameter combinations. MT5 needs 15 minutes to try them all for a 6 month interval consisting of 32 million EURUSD ticks (2020-01-01 to 2020-07-01). {quote} Thanks for the suggestion, but why would I wait a month if there are years of historical data available? Unless you are doing...
@yoriz -- I apologize for assuming you were not using tick data. On MT4 you can run a simulation using tick data but it takes several hours -- and forget about using tick data for calibrations. I'm less familiar with MT5 than MT4 and it appears its simulation is orders of magnitude faster -- which is great.

The reason why you shouldn't use lots of historical data is because it skews the calibration results to favor the old data. For example, suppose I were to use 20 years of historical data -- in that case the most recent 2 months represents less than 1% of the data. In other words, 99% of the data being used to calculate the best settings (for today's market conditions) is hopelessly old and has nothing to do with current market conditions. Unless you have a time machine and can return to market conditions of 20 years ago, why would you use irrelevant data? Unless your software can learn and store past patterns in a database -- i.e. employs AI -- then all you are doing is polluting your data source with data that is no longer valid. You always, always, always want to use the most recent data to get the settings that work best with current market conditions.

Our strategies employ simple algebra and the input data is used to compute constants that will yield the best profit -- for the input data. If your data is 20 years old then those computed constants are most suitable for the data from 20 years ago. But since you trade today, then the data from 20 years ago doesn't help you because algebra isn't a database that can store and learn from past patterns. That 99% of old data simply serves to mask the 1% of good data. The common misconception is that algebra will "learn" from past data -- but it doesn't.

Since you trade today then the goal is to simulate conditions closest to today -- hence, forward testing. You are correct -- if you forward test for 1 month then a month from now your forward testing will be old -- by one month. Since we don't have a crystal ball then we can't forward test into the future. If the market pattern changes every couple of months then your calibrated settings may still be good for a few weeks. But forward testing for 1 month is a heck of a lot better than polluting your input data with 20 years of old data that has absolutely zero relevance to **today's** market conditions. Again -- a strategy is simple algebra: y = mx + c. There's no pattern storage, no database. If your strategy has a period of '14' then your calibration simply computes the best constants 'm' and 'c' for every sliding window of 14 numbers. The only benefit of using lots of data is to check for bugs -- e.g. divide-by-zero or index-out-of-range.

@yoriz -- I wish you luck. And I'm done...
  • #9
  • Edited 6:08am Dec 20, 2021 2:56am | Edited 6:08am
Quoting Ihlas
Searching for different parameters per day of week can be incredibly eye-opening and profitable. Also if we know the price is in the corner. Even it's just EA. Price doesn't reverse in a trend, but time. Time (I have seen this) is the leading indicator for reversal or continuation. IMHO any EA that doesn't include day of the week can't be profitable in the long run.
Interesting suggestion. I could try that by adding an input parameter to the EA to only allow it to trade on a certain day-of-week and do the full optimization process 5 times. Five times more work, but interesting to try once based on your experience.

Quoting Ihlas
Look at different brokers, they all have different opening price. How can we know if it retested the daily open?
Ah, you mean broker time zones? I agree, it is crazy that my D1 bars can be different from your D1 bars. Even worse, platforms like MetaTrader do not provide a good means to compensate for broker offset. It would be so convenient if the server always gives UTC time, and let the client software select the preferred time zone.

And then we also have the Daylight Savings Time chaos. Even worse, often London switches to DST in another week than NY, while Tokyo does not use DST at all during the year. Check this (one color is DST, the other is standard time):
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 101 KB

Here you can see the difference between summer and winter. Since Tokyo does not shift, we see different curves during the Asia session. Madness!

You also see a big difference between Monday and Friday. Probably that was what you were referring to.
  • #10
  • Edited 5:51am Dec 20, 2021 5:36am | Edited 5:51am
Quoting robots4me
The reason why you shouldn't use lots of historical data is because it skews the calibration results to favor the old data. For example, suppose I were to use 20 years of historical data -- in that case the most recent 2 months represents less than 1% of the data. In other words, 99% of the data being used to calculate the best settings (for today's market conditions) is hopelessly old and has nothing to do with current market conditions.
Yes. absolutely! I fully agree with you that you should not use all 20 years of historical data in one go to optimize. Indeed, what happened in 1999 is hardly relevant for today's market. However, what happened in 1999 was relevant for the market of 2000! So by using the data from 1999 we can figure out what optimization metrics we should look at to select the best parameters and verify that for 2000. Given the noisy character of the Forex market and the very thin edge of most strategies, the results in 2000 might be sheer luck. Therefore, we can repeat the process on 2000 data and verify on 2001, and so on, to make sure the strategy remains profitable in most years.

This method is commonly known as "Walk Forward Analysis". By using more historical data we can reduce the chance we were just lucky, and can better see the actual performance of both the strategy itself and the optimization method. In each step, we only use a small part of history, say 6 months, to optimize for next simulated interval, say 1 month:
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 75 KB

What do you think? Does it make sense to backtest using 20 years of historical data if we do it that way?
  • #11
  • Dec 20, 2021 7:01am Dec 20, 2021 7:01am
Quoting yoriz
{quote} Yes. absolutely! I fully agree with you that you should not use all 20 years of historical data in one go to optimize. Indeed, what happened in 1999 is hardly relevant for today's market. However, what happened in 1999 was relevant for the market of 2000! So by using the data from 1999 we can figure out what optimization metrics we should look at to select the best parameters and verify that for 2000. Given the noisy character of the Forex market and the very thin edge of most strategies, the results in 2000 might be sheer luck. Therefore,...
@yoriz: " Given the noisy character of the Forex market and the very thin edge of most strategies, the results in 2000 might be sheer luck. Therefore, we can repeat the process on 2000 data and verify on 2001, and so on, to make sure the strategy remains profitable in most years."

True -- you could do that -- but why? So what if 2000 settings "work" in 2001. And what does "work" mean? When you include 2000 data then maybe the settings continue to kind of "work" for 2001, but those settings won't be nearly as profitable in 2001 compared to the profitablility you would have seen if you had used only 2001 data.

Again -- the point is to use the most recent data so your settings have the best chance to succeed with the **current** market patterns. Including old data only skews the results and "dummies down" the settings to kind of work with older data.

The goal is the best settings for the **current** conditions and NOT settings that kind of work with older data. True -- the market may return to 2001 patterns, but we have no proof of that. There is zero evidence that the market returns to previous patterns -- and even if it did, how frequently does it do that? Every year? Every 5 years? Every 20 years? You don't know. And are you going to run that strategy for 10 years waiting for a market pattern to return? So why include OLD market patterns when the goal is the best settings for TODAY?
  • #12
  • Edited 9:35am Dec 20, 2021 8:51am | Edited 9:35am
Quoting robots4me
True -- you could do that -- but why? So what if 2000 settings "work" in 2001. And what does "work" mean? When you include 2000 data then maybe the settings continue to kind of "work" for 2001, but those settings won't be nearly as profitable in 2001 compared to the profitablility you would have seen if you had used only 2001 data.
Yes, of course! You can not use 2001 data to find the best settings for 2001. That would be cheating.

However, of all the possible parameter values we tried out on the 2000 data, which are the ones that most likely give good profits in the following year (i.e. 2001)? Choosing those parameters that gave the highest profits in 2000 is not smart, because these parameters were likely just the result of luck (see profit profile in post #1). Instead, we should probably look at Sharpe, Recovery Factor, etc. to select the most promising parameter values.

Quoting robots4me
Including old data only skews the results and "dummies down" the settings to kind of work with older data. [...] So why include OLD market patterns when the goal is the best settings for TODAY?
Indeed. We don't. Please have a look at the diagram in post #10. For each test we only use the data from the previous 6 months. All other data is unused. So we never use data that is more than 6 months old.

Perhaps I am misunderstanding your concerns. Perhaps you mean that the inefficiency we are trying to capture in our strategy will no longer exist in a few years from now. That can indeed be the case, so we need to keep monitoring the performance our live EAs and balance our capital over the EAs according to that. Perhaps Goldman Sachs (who hire hundreds of quants) can generate a new strategy every day and benefit from short-lived inefficiencies in the market, but as a retail trader I can only come up with a few per year at best. Therefore, I like my strategies to remain valid for a few years to come. The best way to check that, is to see if the inefficiency was already there a few years ago. If so, we found something that likely will continue to exist for a while.

Or perhaps you mean we should only focus on the future, and therefore only use the last 6 months and nothing else. But then how do we compensate for data-mining bias? How can we know whether our new strategy is really profitable or was just a lucky coincidence that matched the last 6 months? By doing a WFA over several years, we can reduce the data-mining bias. A failing strategy now needs to be extremely lucky to be profitable in all test steps. In other words, if a strategy is profitable when backtesting over several years, chances are high that the strategy is really profitable, and not just a lucky curve fit.

There are tools out there (e.g. EA Studio, StrategyQuant, etc.) that can generate millions of random strategies per hour. Many of these strategies have fantastic linear equity curves. That is the beauty of data mining: even in random noise we can fit fantastic strategies, if we like. And perhaps some of these strategies might even prove to be profitable in the coming months (although I have yet to see someone consistently being succesful with that). How do we know which onces to select? Typically, you do that by using more historical data to see how the strategy would have performed in the past (and by try to mine a similar strategy from randomized data to test the null hypothesis). Unfortunately, the above mentioned tools can generate so many strategies that even when pruning the strategy list using WFA and other methods, a lot of failing "lucky" strategies remain.

Can you please elaborate a bit more on your approach? I really like to understand your concerns and how you develop/optimize strategies.
  • #13
  • Edited 12:03pm Dec 20, 2021 10:41am | Edited 12:03pm
Quoting yoriz
{quote} Yes, of course! You can not use 2001 data to find the best settings for 2001. That would be cheating. However, of all the possible parameter values we tried out on the 2000 data, which are the ones that most likely give good profits in the following year (i.e. 2001)? Choosing those parameters that gave the highest profits in 2000 is not smart, because these parameters were likely just the result of luck (see profit profile in post #1). Instead, we should probably look at Sharpe, Recovery Factor, etc. to select the most promising parameter...
Quoting yoriz
There are tools out there (e.g. EA Studio, StrategyQuant, etc.) that can generate millions of random strategies per hour. Many of these strategies have fantastic linear equity curves.
I'm glad you mentioned EA Studio -- I'm familiar with it, which is one reason I mentioned the use of a portfolio EA in an earlier post. Using simulations and data analysis you may think you have a great strategy, but when you actually start trading it in a live account you'll find that things don't go the way you expected. The algebra and signal generation may appear to be correct, but the window between success and failure is very small and the difference between simulation and real-time is sufficiently large to cause even the best-looking, simulated strategies to fail. The more interesting question -- and one that deserves a thread of its own -- is what factors play a role in causing simulated trading to differ from real-time trading.

To make a point I exaggerated about the use of 20 years of data -- your use of 6 months of data is more reasonable, but still too large for my taste. My preference is to use a data horizon just large enough to yield 50 trades -- typically around 2 months using H1 data. 50 trades is large enough for the results to be statistically significant. Less than 50 is gambling. More than 50 is a waste of time and skews the results to favor older data.


Quoting yoriz
Therefore, I like my strategies to remain valid for a few years to come.
That's a dream. You haven't been trading long enough to realize that will never happen. The closest you'll come is to choose a few strategies and a favorite pair (e.g. GBPJPY) and recalibrate the settings every month.


Quoting yoriz
Or perhaps you mean we should only focus on the future, and therefore only use the last 6 months and nothing else. But then how do we compensate for data-mining bias? How can we know whether our new strategy is really profitable or was just a lucky coincidence that matched the last 6 months?
I did not mention anything about the future. The way you avoid data-mining bias is to focus on the smallest chunk of recent data that will yield statistically valid results -- e.g. 50 trades. Even though it is recent data it is still historical. So, there is no way to guarantee it will work under current market conditions. Market conditions are always changing -- choppy, ranging, swing and trend. No simple algebraic formula can handle all conditions. Choose a chunk of data the represents each of the 4 types of market conditions and calibrate different settings for each. When the market is choppy, apply the "choppy" settings. When the market is ranging, apply the "ranging" settings, and so on.

As for my approach -- it continues to evolve. I trade a combination of algorithmic and discretionary with a friend -- the friend is the "magic sauce"...


PS: To prove my point about the difference between simulated vs real-time -- and to save yourself a huge chunk of time and disappointment -- take your favorite simulated strategy and begin trading it in a live, micro account. Right now, this very moment, confirm for yourself that trading that simulated strategy with the fantastic, linear equity curve also gives a fantastic, linear equity curve in a live micro account. So what if you lose a few bucks -- that's nothing. The insight you'll gain (and time you'll later save) is worth way more than the few bucks you'll lose.
  • #14
  • Edited 3:10pm Dec 20, 2021 1:25pm | Edited 3:10pm
Thanks for further elaborating your ideas!

Quoting robots4me
I'm glad you mentioned EA Studio [...] Using simulations and data analysis you may think you have a great strategy, but when you actually start trading it in a live account you'll find that things don't go the way you expected.
Yes, we humans tend to under-estimate the raw computing power we have today. These tools can find really nice linear equity curves in completely random data. Literally! I demonstrated that once to a skeptical friend. Take historical EURUSD H1 data, shuffle the returns and reconstruct the new H1 bars. Now, there can not be any causality between subsequent bars. Yet, if we feed that random data to data-mining tools they will find hundreds of "good looking" strategies!

Snake oil? Perhaps. Or perhaps some people really know how to pick out the successful strategies, but don't post on public forums...

Quoting robots4me
what factors play a role in causing simulated trading to differ from real-time trading.
Actually the JForex and MT5 strategy testers are not bad. I have been live trading strategies and later compared with a backtest over that same date range. They are very close! They differ a few seconds in entry/exit time, and differ a few pipettes in open/close price. For my strategies that difference is neglectable because I use large TP/SLs of 10 to 100 pips.

Quoting robots4me
My preference is to use a data horizon just large enough to yield 50 trades [...] 50 trades is large enough for the results to be statistically significant
Only 50? I am surprised by that. I guess your strategies must have a very high win rate to make it statistically significant.

How do you optimize your parameters? With only 50 trades the differences between the various settings must be small. What criteria do you use to select the parameters for the upcoming live period?

Quoting yoriz
Therefore, I like my strategies to remain valid for a few years to come.
Quoting robots4me
The closest you'll come is to choose a few strategies and a favorite pair (e.g. GBPJPY) and recalibrate the settings every month.
Sorry, sloppy wording. I didn't mean remain valid for years with the same parameters. I did mean that the strategy mechanism/algorithm remains valid for years, but obviously with regular reoptimization of its parameters.

Quoting robots4me
Choose a chunk of data the represents each of the 4 types of market conditions
Can you capture all that in just 50 trades? Or do you mean 50 trades of each type (200 in total)?

How do you arrange that? Do you manually label each date range with the market condition class? Or do you have an indicator for that?

Quoting robots4me
the friend is the "magic sauce"...
Hahaha. You are blessed with such a magic ingredient!
  • #15
  • Dec 20, 2021 3:28pm Dec 20, 2021 3:28pm
Could we perhaps train a Neural Network using the backtest metrics (e.g. Sharpe, Recovery Factor, Profit Factor, etc.) as inputs to learn how to predict the performance of a strategy? NN's look like a good tool to combine several inputs into one output.

What you do think? Worth a try?
  • #16
  • Dec 20, 2021 6:17pm Dec 20, 2021 6:17pm
Hey @yoriz I can offer some thoughts since I am currently building the same thing as what you are describing here.

In essence you are building a neural net of sorts ... but just be aware the number of different combinations measure in the trillions so the limiting factor will be the practical aspects of modelling rather than data or even processing power. For example as already described above even simple time daylight savings time zone differences can confound the prediction engine if not modelled correctly. Additionally, for all the items you can model there are zillions that are unknown that you can't model ... i.e. invisible for all practical purposes - it's simply not practical to model every newspaper headline that the algos read and map their response to it.
Because of this chaos in price movement I've found simple neural nets cannot provide predictive power on price movements although they are great for pattern recognition.

The biggest challenge with those is lack of precision when it comes to modelling i.e. so many parameters to choose to model? To catalogue everything you must be systematic and rigorous - I've found the existing softwares will not meet my requirements so I've built my own, which allows me to focus on risk exposure and probability. This custom chart shows a simple heatmap of stop loss vs. using a simple time-based entry (in XAUUSD across 365 daily samples). It shows what you would expect ... a tight stoploss creates a stack of losses and a tight take-profit will let you keep quite a few wins but you lose a lot in spread and then there are regions where the correlation vanishes. This is obviously too simplistic because it ignores so much other market information but it does indicate that the domain can be mapped enough to provide a probabilistic edge.

My strategy at the minute is to maintain a variety of different optimized strategies, and choose each one to match current market conditions.
Attached Image
  • #17
  • Dec 21, 2021 2:25am Dec 21, 2021 2:25am
Quoting robots4me
It's not about win-rate -- most of my super-trend following strategies have a win-rate of about 35%. The 50 trades comes from statistics. If you are going to compute a statistical metric then you need at least 50 data points for the result to be statistically valid.
For a coin flip we can use the binomial distribution to figure out what sample size we need to know with p=0.995 certainty that the coin is fair (i.e. 50% win rate). If the coin was unfair (e.g. 35% win rate), wouldn't we need more samples to know that with p=0.995 certainty? I could be wrong here, as I am a bit rusty in statistics :-(
  • #18
  • Dec 21, 2021 3:02am Dec 21, 2021 3:02am
Quoting PercyJames
In essence you are building a neural net of sorts ...
The example strategy used here is just simple price action (breakout strategy using a Donchian channel) without any ML algo, but in post #15 I wondered whether we can use ML to predict the forward test performance based on the backtest performance and metrics. So I consider using ML here not for the strategy itself, but for selecting the optimal parameter values from a backtest.

Quoting PercyJames
The biggest challenge with those is lack of precision when it comes to modelling i.e. so many parameters to choose to model?
Indeed, neural nets are picky about the features you feed it. Give it the wrong feature and performance of the NN plummets. I am having moderate success using Random Forests. These are not picky at all. In fact, I feed it tons of features and it will figure out itself which ones have any predictive value and simply ignores the irrelevant ones. You can ask a trained RF the significance of each feature and that way learn which features are useful and which are not.

Come to think of it, perhaps I might use a RF here for the backtest results too ;-)

Quoting PercyJames
This custom chart shows a simple heatmap of stop loss vs. using a simple time-based entry
Interesting study, although a bit off-topic, perhaps. Assuming price does a random walk, isn't the theoretical win rate simply SL/(SL+TP)? You can clearly see that in your plot for low SL and low TP ranges. Are the disturbances/noisy parts just coincidence, or are they systematic and reproduce month after month? If so, you might be able to use that edge to make profits.

Back to the topic: how would you select the most robust parameters from backtest results?
  • #19
  • Edited 5:35am Dec 21, 2021 4:34am | Edited 5:35am
The noisy parts indicate the variability of conditions and they change depending on other macro conditions like trends and news events; but it was good to see some parameters are consistent regardless of conditions and it's made a difference for me in the sense if I don't like the price action then I can still reliably bank 100 pips even if I'm giving the broker 25 on spread.

After those heatmaps I did weekly variations and found they were different depending on the daily trends. So take that research direction and then skip ahead a few years to today ... my secret sauce is evolving my agents organically, each for different market conditions - I call that 'habitat' and it's modelled on nature. Example is a penguin being a perfectly performing animal ... but take him into the desert he won't last long ... so you'll only deploy him in wet snowy conditions. If I build a large enough ecosystem of agents then natural selection (of parameters) will reveal the most robust parameters for each market habitat. My parameters are a proxy for genetics and each successful agent can adopt a new gene variant. Initially curve fitting (screening) is used to get your initial population but from there it's just mutation (in labs they use similar mechanisms to find favourable bacterial strains). The goal is to find the fittest population and then see what they all have in common.

Basically it's a hand-crafted Neural Net .. effectively the outcome is the same thing. The answer to your original question - 'you have to be selective'. Here's an example: Train your random forests for a narrow set of conditions only; such as the first two hours after NYSE Close and when the Day closes with a red candle magnitude greater than x. If no consistent patterns from that then start introduce Bollinger Bands (x,x) ... or SMA(x). But be sure to index everything so you can keep track of what you've tested.

You can see it's a lot of work and there are literally trillions and trillions of combinations. In the example above you could substitute 'NYSE' for any other stock exchange and see how the same combinations perform but it's 5x the effort (most of the work is packaging and handling the data, not processing). I'm hoping my organics approach will provide me with direct shortcuts but the work is ongoing. I hope this provides some thought as you search for answers to your question .. I'm writing this publicly because I see some value in ideas exchange.


p.s. In this thread I do acknowledge Ihlas and robots4me are describing. Those are useful distinctions because Monday market definitely different to (say) Wednesday. In my system those are modelled as habitat boundaries although my granulation is finer.

p.p.s. Don't forget to include broker spread into your models. Although the market is a giant balancing machine, 50:50% result should not exist in any models since the broker always takes a risk free slice of the pie, however small.
  • #20
  • Edited 7:23am Dec 21, 2021 4:56am | Edited 7:23am
Quoting yoriz
For a coin flip we can use the binomial distribution to figure out what sample size we need to know with p=0.995 certainty that the coin is fair (i.e. 50% win rate). If the coin was unfair (e.g. 35% win rate), wouldn't we need more samples to know that with p=0.995 certainty? I could be wrong here, as I am a bit rusty in statistics :-(
I tried to simulate two "neutral" strategies (i.e. strategies without an edge, breaking even on average). One with 50% win rate, risking 2% equity per trade, RR 1:1, and one with 33.3% win rate, risking 2% equity per trade and RR 2:1. When simulating many test runs of 50 trades each (hence the staircase shape), you see that half of the backtests result in profit (as expected). But you see that the one with win rate 33.3% has higher profits in the high percentiles (in fact, the graph is cropped because the best strategies even reached 25,878.
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 44 KB


For comparison, these are strategies that have a slight edge and are profitable (52% and 36% win rate):
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 45 KB

The differences are very small. So how do you determine with backtesting only 50 trades whether your strategy and parameters are really profitable and not just a lucky backtest on a neutral or losing strategy? Suppose your backtest gives an equity of 12,000. Then there is still a 19% chance this is pure luck.
0 traders viewing now