Automated trading algo generation | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Automated or semi automated trading? 3 replies

Crowd-sharing model for open source algo trading platform 2 replies

Growing Neural Gas Algo for FX trading 2 replies

Veloce algo trading 13 replies

Best of four ECNs for algo trading? 3 replies

Attachments: Automated trading algo generation
Exit Attachments

Automated trading algo generation

  • #1
  • First Post: Edited 6:15pm Dec 20, 2015 6:03pm | Edited 6:15pm
Greetings Folks,

I created a thread on automated system creation last week but it was promptly moved to the commercial section although I had absolutely no intention to sell anything -- I guess because we mentioned the programs people sell that can do this (you can look at my threads if you want to follow that one as well). No problemo, I fully understand that but I want to have a proper "trading discussion" thread about the theory of this as well.

Soooooo, to get things on the right track this is what I want on this thread:

  1. Talk about the theory of automated trading system generation (things like data mining bias, look ahead bias, etc)
  2. Talk about features that might be useful or not useful in an implementation (without mentioning anything commercial)
  3. Talk about freely available (non-commercial) software for this purpose

As you nice folk may already know I am heavily involved with machine learning (http://www.forexfactory.com/showthread.php?t=516785) and this is how I have created all the systems that I currently use live. However there are many other venues for algo trading system generation. I hope we can go through them on this thread and give some worthwhile discussion to the merits and problems that each may have.

I am a busy guy so this thread will advance in "bursts" (as the one on ML has up til now) in whatever time my kids and trading business allow for,

Cheers,

Jo

  • #2
  • Dec 20, 2015 6:14pm Dec 20, 2015 6:14pm
What am I talking about when I say automated system generation?

In very simple terms I'm talking about software where you can simply say the simulation results you want and the software delivers an algo that makes that happen.

Say I want to have a system that makes money from 2005 to 2015 with a Sharpe ratio of 0.8, so I tell the program "give me a trading system that has made money in this time and has a Sharpe greater than 0.8" the program then works its magic and spits a piece of code that when simulated shows whatever performance metrics I have asked for...

There are a gazillion problems that come out when you program software to do this magic trick... This is what I want the thread to be about!
  • #3
  • Dec 20, 2015 6:22pm Dec 20, 2015 6:22pm
When a program does algo generation ask yourselves these questions. How does the program...

  1. decide what logic to initially use for the systems?
  2. implement money management?
  3. implement trade management?
  4. carry out the simulations?
  5. include trading costs?
  6. behave through the searching process?
  7. evaluate the data mining bias of the search process?

Besides these you might also want to consider that an implementation...

  1. is fast enough for what you want
  2. is efficient enough finding systems
  3. makes efficient usage of computational resources
  4. generates code for your trading platform
  5. gives the same simulation results as your trading platform

  • #4
  • Edited 7:54pm Dec 20, 2015 7:41pm | Edited 7:54pm
Interesting.... a field i don't have any real knowledge in so forgive the following question based on lack of understanding.

With your example "a system that makes money from 2005 to 2015 with a Sharpe ratio of 0.8" - How is that any different than just curve fitting and tweaking indicator settings on any particular strategy to meet that criteria? I guess since its simulated the algo will do that automatically but am i correct in this thinking?
Luck is where preparation meets opportunity.......
  • #5
  • Dec 20, 2015 9:12pm Dec 20, 2015 9:12pm
AlgoTraderJo quoted from the other thread:
Quote
IMHO there is a very very very very big problem when you use genetic programming for the building of algo trading systems.

Quote
In [genetic programming] software you have genetic searching that introduces a bus load of bias into the process that is terribly hard to account for.

Are we talking about bias in the statistical sense where high bias means high performance/accuracy and low bias means low performance/accuracy? I ask because it makes sense that to test for high bias you would compare against randomly created strategies and see if your results are very much superior to the random system results. But doesn't this also imply that a system that is more complex will also perform better when comparing against this random system standard?

However, when complexity increases, the chance that the system will have high variance also increases. From my experience high variance seems to be a larger problem than high bias for traders. The reasoning is that the act of maximizing system performance based on some performance metric is in and of itself a means of both increasing performance and decreasing bias. Done to the extreme, this may lead to increased variance which will show up as poor results (bad generalization) on unseen data. In other words with high variance, the system performance is so good on the training data that it is overfit and performance doesn't generalize to unseen data.

As this site shows (section 4.4 picture) as model complexity increases bias always decreases. However at some point variance starts to increase. High variance can be as big of a problem for systematic traders as high bias. Questions:

1) Does the data mining bias test address variance either indirectly or directly?
2) Is there any test that is specifically designed to target the variance of system results? Why is only the data mining bias test mentioned?

Thanks for your contributions.
  • #6
  • Edited 10:04pm Dec 20, 2015 9:17pm | Edited 10:04pm
Hi Guys

Interesting thread.

The way I see it and remember I am a total novice in this area and the guidance from those skilled in machine learning with these assumptions would be greatly appreciated is as follows.

I see that there are three broad roads that could be taken to EA development that attempts to achieve the objectives of machine learning, namely to be able to navigate uncharted waters (out of sample data):

1. An unsupervised machine learning pattern recognition algorithm that evaluates a given current pattern against a history of it's occurrence (stored in memory), and if materially significant against a given confidence level, predicts the most efficient trade method to harvest opportunity from that pattern. Such an algorithm essentially therefore is forward looking and able to navigate out of sample data and hone in on the optimal way to exploit that new terrain. An approach like this will necessarily be 'lagging' to new emerging conditions as it requires a sufficient historical sample to allow for pattern recognition..... but sufficiently responsive to 'learn' new market conditions within a relatively short period of time. Possible curve fitting however would be prevalent as the faster the response required to adapt to new conditions, the less history in memory which therefore reduces the sample size for testing and makes the approach very subject to curve fitting (data mining bias).

2. An instructed algorithm in which key performance metrics are given to the algorithm and then by interrogation of a sample dataset and extensive processing and iteration commencing from a key set of variables, creates an coded outcome that achieves those performance metrics on the data sample. Obviously the ability for data mining bias arises from the size of data sample chosen (if sufficiently small to not be representative of the population), however there may be workarounds with this approach by ensuring the sample data is sufficiently large to be representative of the majority of possible out of sample market conditions that may arise and not just a random or ephemeral feature. Note: This is describing the option that algoTradeJo want to discuss in detail in this thread.

3. A comprehensive history of data is made available to an algorithm in addition to a range of strategies that can be successful employed for those historical market conditions whereby the algorithm simply searches using pattern recognition for formative patterns that are statistically estimated as having to potential to mimic past history...and thereby an existing trading solution is used to trade that nascent pattern. There is not much difference here to that achieved by an experienced discretionary trader. The problem with this approach is the inherent assumption that historical market conditions are used as a guide for the future. Obviously with a limited sample dataset, like the other options, this too could suffer from data mining bias.

Surely the potential for all three possible approaches could be overcome from the utilisation of a sufficiently large historical sample that holds within it the vast majority of potential market conditions that could be thrown at it in the future. If so, then it is a problem of memory and processing grunt as opposed to the theory of the approach itself.

I would be interested to hear other thoughts.
  • #7
  • Edited 9:59pm Dec 20, 2015 9:32pm | Edited 9:59pm
Quoting FXEZ
AlgoTraderJo quoted from the other thread: {quote} {quote} Are we talking about bias in the statistical sense where high bias means high performance/accuracy and low bias means low performance/accuracy? I ask because it makes sense that to test for high bias you would compare against randomly created strategies and see if your results are very much superior to the random system results. But doesn't this also imply that a system that is more complex will also perform better when comparing against this random system standard? However, when complexity...
Nice FXEZ. I need to chew on this a bit and get my head straight between what is termed 'bias' and what is termed 'variance'.

PS Thanks for the link. I will throw in the article's definitions as they are helpful for this discussion.

Conceptual Definition

  1. Error due to Bias: The error due to bias is taken as the difference between the expected (or average) prediction of our model and the correct value which we are trying to predict. Of course you only have one model so talking about expected or average prediction values might seem a little strange. However, imagine you could repeat the whole model building process more than once: each time you gather new data and run a new analysis creating a new model. Due to randomness in the underlying data sets, the resulting models will have a range of predictions. Bias measures how far off in general these models' predictions are from the correct value.
  2. Error due to Variance: The error due to variance is taken as the variability of a model prediction for a given data point. Again, imagine you can repeat the entire model building process multiple times. The variance is how much the predictions for a given point vary between different realizations of the model.

PS FXEZ based on the simple illustration below from your article, aren't we seeking both low bias and low variance? You might need to help me out here.

Attached Image (click to enlarge)
Click to Enlarge Name: Capture.JPG Size: 56 KB
  • #8
  • Edited 11:33pm Dec 20, 2015 10:23pm | Edited 11:33pm
Quoting algoTraderJo
When a program does algo generation ask yourselves these questions.

How does the program

... decide what logic to initially use for the systems?
Their must be at least one initial condition assumption upon which it then iterates using key variables to do so using correlation as a method to cross compare iterations against performance metrics.

For example, this is just a punt guess.......I am assuming you start with a time interval range established for your historical data sequence (eg. 1-2 years of data using a 10 bar time interval period to start) with say 3-4 key variables being say entry, timeframe, profit target and stop (or just trailing stop). Set one to a non-random condition and the others to random and iterate for each time interval segment across the dataset in a walk forward manner on each new bar (which then applies for the subsequent time interval range). Determine correlation between input performance metrics and result of each iteration. Choose closest correlation result for non-random condition and then interrogate another variable in the same way. Once correlations established between variables and performance metrics then rank them and run tests on grouped non-random variables to determine result against performance metrics. Progressively add to non-random variable inputs until each variable is non-random.

Then select another time interval range (eg. 20 bars) and go again....and so on and so forth until you hone in on the desired performance metrics.

Scale up or down accordingly with position size to achieve closest result ????????
  • #9
  • Dec 20, 2015 10:50pm Dec 20, 2015 10:50pm
Quoting Copernicus
{quote}PS FXEZ based on the simple illustration below from your article, aren't we seeking both low bias and low variance? You might need to help me out here. {image}
Hi Copernicus, yes that's how I see it. Ideally you want models that are highly accurate (low bias) and with small dispersion (low variance) so they hit the target right in the center.

P.S. I like that site because it has really good graphics that make a fairly complex subject, intuitively clear.
  • #10
  • Dec 20, 2015 10:51pm Dec 20, 2015 10:51pm
Quoting FXEZ
{quote} Hi Copernicus, yes that's how I see it. Ideally you want models that are highly accurate (low bias) and with small dispersion (low variance) so they hit the target right in the center.
Thanks mate. That clears it up for me. :-)
  • #11
  • Dec 22, 2015 6:43am Dec 22, 2015 6:43am
Quoting Ozzie Taz
Interesting.... a field i don't have any real knowledge in so forgive the following question based on lack of understanding. With your example "a system that makes money from 2005 to 2015 with a Sharpe ratio of 0.8" - How is that any different than just curve fitting and tweaking indicator settings on any particular strategy to meet that criteria? I guess since its simulated the algo will do that automatically but am i correct in this thinking?
You're right in your thinking, it's basically the same thing and falls into the same caveats. How to overcome these problems? That's the main point that has to be addressed.
  • #12
  • Dec 22, 2015 6:52am Dec 22, 2015 6:52am
Quoting FXEZ
AlgoTraderJo quoted from the other thread: {quote} {quote} Are we talking about bias in the statistical sense where high bias means high performance/accuracy and low bias means low performance/accuracy? I ask because it makes sense that to test for high bias you would compare against randomly created strategies and see if your results are very much superior to the random system results. But doesn't this also imply that a system that is more complex will also perform better when comparing against this random system standard? However, when complexity...
1. It does not address variance either directly or indirectly. Data mining bias simply refers to the probability that the system comes from spurious relationships in the data instead of real causal relationships. It therefore relates to how strongly the process derives signals from noise. This bias is evaluated by comparing the number of systems with similar statistical characteristics in real data from those generated on data where no causal relationships are possible (a.k.a, random data). This is all explained further on my ML thread.

2. It is difficult to target the variance of results. You can check the variance of results on the backtesting data but this is always unrelated with variance in real live trading which is what really interests us. A certain number of systems will tend to follow negatively biased random walks when you start live trading and these systems will always increase your variance. This comes from the natural limitation of not knowing the future, you cannot discard strategies that follow spurious relationships a priori, you can simply attempt to reduce the probability that they exist but they will always have a chance to be there. This is to me a frontier problem, you cannot successfully solve it because its solution is related to future data which is always uncertain. The best you can do is do things to attempt to alleviate it, like ensuring little variance in backtesting results and using as much data as you can.
  • #13
  • Dec 22, 2015 7:01am Dec 22, 2015 7:01am
Quoting Copernicus
Hi Guys Interesting thread. The way I see it and remember I am a total novice in this area and the guidance from those skilled in machine learning with these assumptions would be greatly appreciated is as follows. I see that there are three broad roads that could be taken to EA development that attempts to achieve the objectives of machine learning, namely to be able to navigate uncharted waters (out of sample data): 1. An unsupervised machine learning pattern recognition algorithm that evaluates a given current pattern against a history of it's...
It is interesting to read your classification. They are all methodologies to attempt to automatically derive some relationships that can be used for profitable trading so in some sense they are all doing the same thing. Due to this common goal they are all exposed to the same sources of bias and therefore have solutions that are often transferable. For example tests for the evaluation of a process' data mining bias can be transferred to all of them.

As you also cleverly say they all face the frontier limitation of not knowing about the future. This is also why you always need to use as much data as possible. Often traders are reluctant to use old data, saying that it is not relevant, so I was thrilled to read this blog post about how data from 1986 is actually relevant to today's market. You need to use all the data you can, because it's all market data, regardless of any technological developments that might have changed things to some degree, it's all data generated by humans trading some form of value.

But it's super super important to point out that any amount of market data will always be minuscule compared to the realm of potential new market conditions. Everything fails eventually and this is also why having some formal criteria for removing systems is just as important as having criteria for adding them.
  • #14
  • Dec 22, 2015 8:15am Dec 22, 2015 8:15am
Thanks for the response Jo, your link to that blog post (very interesting) and the guidance on your last point. "Everything fails eventually and this is also why having some formal criteria for removing systems is just as important as having criteria for adding them".

It gives me stuff to ruminate on. The hardest question to answer is in defining when your system is broke or not. At the moment I use a historically derived drawdown threshold as my guidance to tell me when to remove a system. Not that it necessarily is broke (as the future is always uncertain), but that my risk tolerance threshold for that system has been exceeded. There are probably far more effective methods but you have to draw a line in the sand somewhere and risk tolerance is perhaps a great place to start.
  • #15
  • Dec 22, 2015 3:54pm Dec 22, 2015 3:54pm
Hi algoTraderJo, thanks for taking the time to reply. Some of your thoughts jumped out at me:
Quoting algoTraderJo
{quote}You can check the variance of results on the backtesting data but this is always unrelated with variance in real live trading which is what really interests us.
Quote
The best you can do is do things to attempt to alleviate it, like ensuring little variance in backtesting results and using as much data as you can.
When you say "ensuring little variance in backtesting results" are you referring to variance of a single system's results or variance between multiple systems' results created via the same process? In addition to using more data, can you expand further on what steps could be taken to ensure little variance in backtesting results?

Quote
A certain number of systems will tend to follow negatively biased random walks when you start live trading and these systems will always increase your variance.
Yes I have noticed this rather unpleasant phenomenon as well. As system results often are negatively autocorrelated, your statement seems to imply that these particular negatively biased systems are instead showing positive autocorrelation. Does that sound reasonable?
  • #16
  • Jan 6, 2016 12:38pm Jan 6, 2016 12:38pm
Quoting Copernicus
Thanks for the response Jo, your link to that blog post (very interesting) and the guidance on your last point. "Everything fails eventually and this is also why having some formal criteria for removing systems is just as important as having criteria for adding them". It gives me stuff to ruminate on. The hardest question to answer is in defining when your system is broke or not. At the moment I use a historically derived drawdown threshold as my guidance to tell me when to remove a system. Not that it necessarily is broke (as the future is always...
Failure needs to come from some formal criterion, lines in the sand that are psychological in nature have many problems. A drawdown threshold that you simply come up with based on what you can tolerate might be a premature exit point because it might be part of what the system can normally give you. Drawdowns derived from backtests are terrible thresholds, I can tell you that. Monte Carlo sims will show you that the real expected drawdowns are often far worse than those you see on a test.

There are many ways to define system failure in a formal manner but they are all basically ways of statistically saying that your distribution of returns in live trading and your distribution of returns in backtesting are not the same thing.

Simply think of ways to negate the null hypothesis "My live trading returns and my backtesting returns come from the same distribution". I hope you catch my drift
  • #17
  • Jan 6, 2016 12:50pm Jan 6, 2016 12:50pm
Quoting FXEZ
Hi algoTraderJo, thanks for taking the time to reply. Some of your thoughts jumped out at me: {quote} {quote} When you say "ensuring little variance in backtesting results" are you referring to variance of a single system's results or variance between multiple systems' results created via the same process? In addition to using more data, can you expand further on what steps could be taken to ensure little variance in backtesting results? {quote} Yes I have noticed this rather unpleasant phenomenon as well. As system results often are negatively...
1.1. I would be referring to the first, reducing the variance within single systems. I don't see a lot of advantage in reducing the variance from systems that come from the same process because the fact that a process generates a varied array of systems is not bad per se. It can be a process with a very wide outlook that generates several perfectly valid solutions. Do you see some purpose in reducing the variance between systems? If so, why?

1.2. Using more data in backtesting helps reduce variance in live trading, meaning trading after you finish the backtesting process. In backtesting reducing variance is easy because you can use some statistic to measure variance and simply only take results that are above whatever variance threshold you desire. You can use something like a variance calculated Sharpe ratio and only use systems above a certain point. Reducing variance after the backtestings ends is the hard thing because you enter the realm of the unknown.

2.1. To be sincere I have never seen if the systems that fall into a negatively biased random walk have a positive autocorrelation (do you mean a positive autocorrelation of the return series?). I have some strict criteria for discarding systems and once they are discarded I no longer monitor them. I generally discard systems at a 99.5% confidence for the tests I use for this purpose. Why would you want to learn more about the autocorrelation? Do you see this as being relevant in some manner? As far as I can remember there is no reason for a negatively biased random walk to be autocorrelated (all results are random and therefore independent?). Do correct me if I am wrong.
  • #18
  • Jan 6, 2016 5:08pm Jan 6, 2016 5:08pm
Quoting algoTraderJo
{quote} Failure needs to come from some formal criterion, lines in the sand that are psychological in nature have many problems. A drawdown threshold that you simply come up with based on what you can tolerate might be a premature exit point because it might be part of what the system can normally give you. Drawdowns derived from backtests are terrible thresholds, I can tell you that. Monte Carlo sims will show you that the real expected drawdowns are often far worse than those you see on a test. There are many ways to define system failure in a...
Thanks for the guidance Jo. Your rationale makes a lot of sense and gives me a much more definitive measure to assess whether a system is 'broke' or not. Greatly appreciated. :-)
  • #19
  • Jan 7, 2016 12:36am Jan 7, 2016 12:36am
AlgoTraderJo, are you actually using Metatrader ?
I believe the tool is actually a BIG part of the overall problem.
  • #20
  • Jan 7, 2016 1:07am Jan 7, 2016 1:07am
Quoting algoTraderJo
{quote} 1.1. Do you see some purpose in reducing the variance between systems? If so, why?
No, I agree with your points. Thanks for the clarification. If we have no variety between systems then the diversification effect is also be nullified. An example and an extension:

If process A selects systems based on top performance while discarding most systems from a set, and process B takes the same set of systems and just excludes those systems with really poor performance, an interesting thing happens. While the average performance of the systems selected via process A may be much higher than the average performance of systems selected via process B, when these systems are traded together it is often the case that the systems from process B will provide a superior diversification effect and thus better risk adjusted returns than those selected by process A.

Why? It may be that the top systems are too correlated or non-diverse and thus don't do as much to diversify the group as some lower performing systems. And a further question and possible extension, will the systems from process B provide better generalization on future data due to the fact that they are less homogeneous than the systems from process A? If you include a larger pool of systems in your combination, and if your systems are low or uncorrelated, then the addition of new systems should provide new features that could describe future market states more accurately. This may make the performance more robust going forward.
0 traders viewing now