Systematic trading | Page 17 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Trade another day through Systematic Trading 203 replies

Nonparametric Association Measures in Systematic Trading 57 replies

Anyone use Systematic from Alpari? 1 reply

A Systematic Approach to Markets 68 replies

Systematic Trading vs. Discretionary Trading 14 replies

Attachments: Systematic trading
Exit Attachments

Systematic trading

  • #321
  • Aug 30, 2009 2:55pm Aug 30, 2009 2:55pm
Quoting mikkom
I think it's the power of simplicity and the fact that I dump all algos that don't work on all the pairs I test with.

I'm still unsure about the results, this kind of testing is always problematic because I have done some really serious caching so there might be some bug somewhere that allows algos to cheat [this has happened for me a lot in the past], I think I'm wiser few weeks from now.
It's true that the simpler the idea the more chance it has of overcoming the variance/bias problem, I could just never make it work consistently. I remember back in the day I had an MT4 EA which only used one indicator, I did some back/forward testing which convinced me that it held up in forward testing. So I went live, first month it made some $$$ then started to deteriorate. So I re-optimized in the same fashion, but this time it totally bombed live. In the end I just concluded that any $$ I made OOS was simply down to luck. I banged my head against the back testing wall from every angle for years, in the end I just gave up, I'm simply not smart enough to make it work. This however does not invalidate the approach, I know people who do very well with back testing and optimization but their approaches tend to be quiet sophisticated.
The breaking of a wave cannot explain the whole sea.
  • #322
  • Aug 30, 2009 3:26pm Aug 30, 2009 3:26pm
Okay here is how it looks after I corrected the bug - this is still pretty crude, there is no covariance optimization or anything like that but still, in-sample (pre 2005) and out sample look quite the same. Lots of more volatility now - this run is about 5 minutes long so it will get better but doesn't look nearly as beautiful as the ones before the bugfix..
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot-AlphaFactory-25.png Size: 97 KB
  • #323
  • Aug 30, 2009 4:44pm Aug 30, 2009 4:44pm
Here is an example of using a combined curve instead of requiring algo to work on all pairs separately. Anyone notice anything different?

(pre 2005 is the training period)
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot-AlphaFactory-26.png Size: 75 KB
  • #324
  • Aug 31, 2009 9:02am Aug 31, 2009 9:02am
hi mikkom, the bug you remove seem to have made a huge difference.
you were right, it looked too good to be true the current results seem much more realistic.

Quote
Here is an example of using a combined curve instead of requiring algo to work on all pairs separately.

i am not sure what you mean by "using combined curve"... care to elaborate?
anyway the result from the last picture is clear... out of sample the result seems to be more or less random, so the algos you have picked have probably no predictive power. Forcing the algos to work on all pairs (by "work" you mean "be profitable"?) you seem to be better filtering out the algos that have worked only on some pairs by chance.

I have also written my own framework (data processing, genetic optimizer, backtester etc.) but the whole thing from ground up is targeted at battling the dreaded overoptimization/overfitting. if you think about what actually causes the overfitting you can find methods to battle it... you have to do it everywhere, on all fronts.
I'm not insane; my mother had me tested.
  • #325
  • Edited 11:20am Aug 31, 2009 11:05am | Edited 11:20am
Quoting RoBiK
hi mikkom, the bug you remove seem to have made a huge difference.
you were right, it looked too good to be true the current results seem much more realistic.
Of course, the first results seemed absurdly good so it was very hard to believe them.

By combined curve I meant filtering by combined curve of all pairs and yes, it will overfit very easily. It's also important to note that this is only few days old so the strategy "generator" is very crude and more importantly I don't have any kind of covariance filter/adjustment which I think will make a huge difference.

edit: just fyi, I have been playing with genetic algos etc for nearly 10 years so I do understand the reasons for overoptimization quite well, I think I first played with genetic algos over 15 years ago (on alife simulator).
  • #326
  • Aug 31, 2009 11:25am Aug 31, 2009 11:25am
Quoting mikkom
I don't have any kind of covariance filter/adjustment which I think will make a huge difference.
don't expect wonders from it. the covariance you calculate from the backtest is only an estimate, as is variance and the profit (usually the profit estimate has the biggest errors, errors in variance are lover and errors in covariance estimates are the lowest)... there can be a lot of error in those estimates. Besides, fiddling with the weights can also be a source of overoptimization... it is just an another set of parameters that you optimize although on a higher level than the strategy parameters.
I'm not insane; my mother had me tested.
  • #327
  • Edited 11:52am Aug 31, 2009 11:38am | Edited 11:52am
Quoting RoBiK
don't expect wonders from it. the covariance you calculate from the backtest is only an estimate, as is variance and the profit (usually the profit estimate has the biggest errors, errors in variance are lover and errors in covariance estimates are the lowest)... there can be a lot of error in those estimates. Besides, fiddling with the weights can also be a source of overoptimization... it is just an another set of parameters that you optimize although on a higher level than the strategy parameters.
I understand what you mean very well but without covariance filter there will be too many very similar algos and that's not what I want.

edit: just to clarify, I meant covariance between algos, not covariance with pair.
  • #328
  • Aug 31, 2009 11:53am Aug 31, 2009 11:53am
Quoting mikkom
I understand what you mean very well but without covariance filter there will be too many very similar algos and that's not what I want.
i did not mean that you shouldn't use it at all... i am just saying that it is only an estimate with some error in it, so one has to be cautious and not expect wonders. you say that all of the algos are only a variation on some basic strategy so i would exect that the correlation would be quite high. once you throw in some less correlated strategy the results could get interesting
i think you are on the right path.

About the cause of curve-fitting, i think the problem is not so much in the optimization method as with what are you optimizing and what data do you use.
I'm not insane; my mother had me tested.
  • #329
  • Edited 12:44pm Aug 31, 2009 12:19pm | Edited 12:44pm
Quoting RoBiK
you say that all of the algos are only a variation on some basic strategy
.. The first random pictures were random variations of a very simple algo, the later ones are not.

I have used parse trees (as in lisp) in the past with some genetic variations but ... They do curvefit a lot So now I have restricted the search to a smaller subset but it's not just a variation of one algo at all.

Quoting RoBiK
About the cause of curve-fitting, i think the problem is not so much in the optimization method as with what are you optimizing and what data do you use.
As I have done so much stuff with gp, I always think optimization as an evolutionary process and overfitting is an evolutionary response to giving specific data if it's possible - algos are very good at finding inefficiencies ... in your code . I have used many different methods in the past, changed periods (including total random steps), generated compound data, different mutation methods, different fitness methods, restricting fitness landscape etc.. I don't even remember what but this time I'm using a bit different approach [restricting algo complexity without restricting overall complexity], I might talk more about that later.
  • #330
  • Aug 31, 2009 2:18pm Aug 31, 2009 2:18pm
Some stuff from today.. Found a few more bugs from the algo generation that affected the results wrong way. So no covariance filter added, pre 2005 is again in-sample and after that out-sample.

I also added some more algorithmic variations to the mix.

I don't think I'm going to post more of these pics until I have something totally unbelievable, I think everyone is getting bored to my curve pics..
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot-AlphaFactory-28.png Size: 114 KB
  • #331
  • Aug 31, 2009 3:08pm Aug 31, 2009 3:08pm
Quoting mikkom
think everyone is getting bored to my curve pics..
You got this wrong, Mikko. Your pics are more than inspiration.

Cheers,
Bluefish
  • #332
  • Aug 31, 2009 3:40pm Aug 31, 2009 3:40pm
I'm also following the progress.
The breaking of a wave cannot explain the whole sea.
  • #333
  • Aug 31, 2009 4:01pm Aug 31, 2009 4:01pm
Quoting mikkom
I don't think I'm going to post more of these pics until I have something totally unbelievable, I think everyone is getting bored to my curve pics..
not at all, it is interesting to see how things are taking shape.

few questions about the data: do you use only one-minute bar data? do you somehow filter and/or post-process the data? do you check for things like errors in the data (for example some weird spikes that didn't happened in the real market), gaps etc.? you know the saying "garbage in, garbage out"
i am using tick data that i harvest in real time from a broker and that data is full of weird things... filtering it in real time and determining where the market actually is and at what prices i could get executed is not a trivial thing.
if you are using one-minute bars than if i calculated correctly it would be about 1,4 millions bars in four years, is that correct?

i don't know how far are you willing to go and what are you willing to reveal, but i will throw in some questions anyway... feel free not to answer if it would go too far

here are some random questions:

you restricted your search to a "smaller subset". care to elaborate on this? how did the whole set looked like and how do you restricted it? i guess it is the same thing that you mean by "restricting fitness landscape"? "restricting algo complexity without restricting overall complexity" sound also to me as making the fitness landscape smaller, but what does "overall complexity" means?

what do you mean by "generated compound data"?

and finally... "algo variations"... will you give some more details about how are you making those variations?

i wish you success

thank you
I'm not insane; my mother had me tested.
  • #334
  • Aug 31, 2009 4:28pm Aug 31, 2009 4:28pm
Quoting RoBiK
few questions about the data: do you use only one-minute bar data? do you somehow filter and/or post-process the data? do you check for things like errors in the data (for example some weird spikes that didn't happened in the real market), gaps etc.? you know the saying "garbage in, garbage out"
Yes, only 1 min data and no, I don't process it. I also always use the most pessimistic version of the possible event, for example if the same candle hits my open and sl, I don't guess, I always close it with loss. The same thing with spikes.

Quote
i am using tick data that i harvest in real time from a broker and that data is full of weird things... filtering it in real time and determining where the market actually is and at what prices i could get executed is not a trivial thing.
if you are using one-minute bars than if i calculated correctly it would be about 1,4 millions bars in four years, is that correct?
It's about 2.5 million candles per pair since 2001. Did I mention I have 6 pairs simultaneously loaded into memory

Tick data would be quite good (book data even better) but at least now 1 min data is sufficient for me. By the way I think dukas offers tick data for download if someone is interested in longer term tick data.

Quote
you restricted your search to a "smaller subset". care to elaborate on this?
This is the core of why I think this is going to work, let's just say that I use stuff that I know works and combine and modify them with different ways. By whole set I mean all possible trading algos that exist. I aim to have lots of simple algos that I then weight by some parameters that I don't even know [haven't decided] yet.

Fitness landscape, compound data and the rest of the stuff refers to the older stuff I did years ago, it's not really relevant to what I do now. Maybe the important thing to say is that fitness algo is *very* important to the end result. I hope everyonw interested in evolutionary computer algos has read this, it's very relevant to every discussion that talks about curve fitting http://neil.fraser.name/writing/tank/ (It also describes pretty well why I dislike neural networks).
  • #335
  • Aug 31, 2009 4:37pm Aug 31, 2009 4:37pm
For those interested 'The Encyclopedia of Trading Strategies' by Katz contains a chapter on GO, they give some basic examples for encoding strategy 'parts' as genomes etc, it would be easy to develop these examples. In the book they also find that GA seems robust out of sample but that the algorithms tend to fixate on rare events, as such don't end up trading much. Obviously these results are specific to their particular fitness metrics etc.
The breaking of a wave cannot explain the whole sea.
  • #336
  • Aug 31, 2009 4:49pm Aug 31, 2009 4:49pm
I would suggest reading this for anyone interested in evolutionary algorithms, also I would recommend reading a few of kozas books or his other research - they are really worth reading for anyone interested in evolutionary algos.

http://www.genetic-programming.com/g...dtutorial.html

I somehow remember reading an article in businessweek years ago that told that he was doing some market research but I didn't find the article again so if anyone has link, it would be really appreciated.
  • #337
  • Sep 1, 2009 10:08am Sep 1, 2009 10:08am
One comment about metatrader tester that I didn't realize before - you get blind to certain things when you work with something too long.

Because the metatrader tester result curve box is so wide and the height is so small, the results look better than they actually are.

In other news, I added a trade profitability analyzer to the right panel - red spike is my fixed risk loss - I test the strategies with the same risk. This pic is a result of about 10 minutes of runtime All the runs are randomized so the curve is always different (and so are algos) each time.

Correlation between algos is still there [no coeffiency filter yet] and it shows, especially in 2008. (pre 2005 is again in-sample). It's really a bit strange that there is no visible difference between in-sample and out-sample - seems that this is working better than I anticipated.

By the way Robik, you mentioned that you have your own frawework that is built from ground up to prevent curve fitting, are you getting good results from your approach?
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot-AlphaFactory-34.png Size: 129 KB
  • #338
  • Sep 1, 2009 10:37am Sep 1, 2009 10:37am
Quoting mikkom
By the way Robik, you mentioned that you have your own frawework that is built from ground up to prevent curve fitting, are you getting good results from your approach?
Right now i have no concrete results, because i am in the middle of a big code rewrite. But from what i can tell until now, i am on a similar route as you are... not too similar, but we have a few similar ideas. Normally i am rather skeptical when talking about backtesting results but from what i have seen until now gives me lot of confidence.

Regarding the genetic algo, until now i used something similar to what Koza proposed... using an array to encode a genetic tree, but with few differences in how the tree is represented and how the genotype is translated into phenotype. The result was a phenotype in a form of a program doing calculations with a stack of values. Now i changed the genetic algo to linear genetic programming, wich allows a bit more flexibility with it's directed graph structure and it also more naturally translates into a computer program which is what i want. What i am doing is: the genotype in form of linear genetic program gets postprocessed into a phenotype that i translate into an intermediate language instructions forming a callable functions. that function generates the trading signals. that intermediate language code is at runtime generated and just-in-time compiled into a native program that is then executed on tick data... number of executions goes into billions... and how fast is it? how fast can you say FAST?
I'm not insane; my mother had me tested.
  • #339
  • Sep 1, 2009 2:36pm Sep 1, 2009 2:36pm
Here is one more pic from today, I added some more algorithmic variations to the mix - this is just 42 algos combined, exponential line (white) uses far too high risk, I'll have to adjust the risk somehow.. (note the crazy high profit and drawdown).
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot-AlphaFactory-36.png Size: 51 KB
  • #340
  • Sep 1, 2009 3:51pm Sep 1, 2009 3:51pm
More curves - I didn't believe this would work, I used just one pair, EURUSD and I still got non-curvefitted results. pre 2005 is in-sample again. There are signs of worse curve after 2005 but not as much as I would have supposed.

This just shows that the fitness algo is not very relevant if you have the correct subset of algos. I didn't expect this at all either. It's great when actual experiment proves you wrong.

After this I just have to try something totally different on larger scale, ie - generating algos per pair, not global algos. edit: doesn't work as well - back to original version.
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot-AlphaFactory-39.png Size: 36 KB
0 traders viewing now