Systematic trading

No I hadn't heard about that, it looks pretty cool. If I was going to parallelize a process I would probably be more inclined to use Boost MPI (I'm using C++), since I'm already heavy leveraging other parts of the Boost library.

Well if you really want a proper framework, then you really need to consider news events plus macroeconomic factors (especially since you use longer TFs). You would need to find a way to quantify speeches, release figures, and other events, short and long term, model them, and use them in your analysis. These are factors that affect price move a lot, and doing the analysis without them just with maths is really half the picture. But this implementation is even harder than the WFO, and I really doubt any single person can do it within a reasonable amount of time alone.

I remember we discussed something similar a while ago, that was about analysing a speech and the impact it can have on the price. What I am talking about here is a bit more complete picture of the market and the world in total. But since this is so hard, I believe your approach about weighting the systems according to recent performance is the best approximation you can get. To this end, it might be helpful to analyse first the persistence of the performance results for each system, i.e. if a system starts losing, statistically, for how long does this last? and the same for when it starts winning. If there is enough persistance on either way, then you can argue that the weighting will help, because your rate of adaptation can be fast enough to actually win more or lose less by adapting, so in a way you will be able to quantify the gain you can have by adapting. On the other hand, if the changes are not persistent and the performance fluctuates between win/loss very frequently, then probably the weighting is futile. Perhaps some form of z-score for the systems performance would help.

In fact I have been thinking this a lot and I would add them to the big picture if I could find reliable source for longer term data - it's really not that hard to quantify numbers and release dates - it's quite easy to abstract them to few different events (first thing that comes into mind is a matrix of low impact-high impact and results estimated under/over). I don't have any plans to do any analysis for textual data, it's quite possible and even easy by bayesian filtering or similar methods, there are lots of frameworks that can do this, see for example http://crm114.sourceforge.net/ - ASSUMING of course there is a pattern there (I would expect companies financial reports / forecasts to be much more easier to parse than FX announcements for example)

Believe or not but news events etc mean less in higher tf's than lower.

The biggest thing I'm interested would be the release times of the news because there are some repeating things before and after every news release (how volume/volatility behaves). The same applies to news events that have very different than predicted numbers.

I actually really would like to model them but as I said I haven't found any good source for 10 years of news data - if anyone knows one, tell me. I don't mind paying for the data if it's somewhere in range of hundreds - under couple of thousand euros.

Hey mikkom,
I've been working on my own GA project in for a couple of weeks now, and I can see why you are thinking about how to get more CPU on-board. As soon as one drops down to tick based processing, the process becomes very slow, I have a number of PCs networked in my house + my servers in the US, so I'm going to try and get the program to 'farm out' the back-test runs using Boost MPI, that's the plan anyway

Haven't posted here for a while.. I have been very busy with other things but this was just so interesting and a bit relevant to this thread so I couldn't resist posting

http://arxiv.org/abs/1002.2284

(just a sidenote: I don't think that anyone who hasn't studied CS understands why I posted this or what he's saying, IMHO paper is both funny and insightful at the same time)

edit: here is other interesting kind of related paper if anyone is interested http://www.ccs.neu.edu/home/viola/papers/HLVreport.pdf

There hasn't been a lot of activity in this thread lately, a pity.

I wanted to go back to something that was mentioned last year in the thread: the implementation of White's Reality Check for a system under development (or Acrary's Edge Test, or the Monte Carlo variations.... ). It seems I am at a point where this has to be the logical next step, and I would welcome any guidance you guys are willing to offer. It is a long story but I think the example I am about to present holds a certain educational value for other people trying to develop mechanical trading systems, so please sit back, relax and bear with me

This all came up when a couple of weeks ago I was backtesting and optimizing a daily system in MT4. Nothing too fancy, just your usual a breakout box trying to profit from the volatility at the opening hours of the markets (systems like this go a dime a dozen in the "systems" section of FF; I just wrote a very generic EA for a variation of one of them). The system showed some promise and I was very thorough with the code and the backtests, trying not to fall for the usual "excessive curve-fitting" traps. In fact, for a system like this there isn't a lot of parameters to optimize (not truly make-or-break ones, anyway): once you have chosen the box boundaries (which are mostly determined by market opening times anyway) it is only a matter of optimizing stop loss and profit targets, which I normally do based on daily volatilities so that the system is adaptive to changes in the ATR through the years. There are a couple more twists, but they are not important at this stage.

The upper graph of the first attachment shows the equity curve for this system with a major pair (GBPUSD) from January 2001 to June 2009. Roughly a yearly profit of 55%, with a maximum DD of 15% through the 8+ years of test (the daily risked amount, slightly above 1% of the account, was adjusted to meet this 15% criterion which I considered the maximum tolerable DD). The profit factor is a very acceptable 1.43. These results look too good, but I can assure you that I was extremely careful in the optimization process: I am conservative and never go for the maximum profits but for the best Profit/DD ratio, and I always make sure that the "optimum" parameters I choose are in the center of a broad interval of valid (profitable) values, so that the system's profits are not a result of serendipity and are more resilient to future changes in market dynamics. In fact, for this particular breakout system I was surprised to see that there were *very* broad ranges of parameters (SL,TP) where the system was profitable (I also try to use decent historical data, but forget about data quality for now, that is not the issue in this post).

Looking good so far! Next step: forward testing with out-of-sample data, for which I had reserved the last six months of 2009 and the beginning of 2010. The lower part of the first attached graph shows the results of this test from July 1st, 2009: the equity starts going down almost from the first day and it hits a maximum DD slightly above 15% by the end of november (later it recovers a little bit). This is more or less the maximum DD exhibited by the system in the previous 8+ years. Even more telling is the fact that it started falling almost from day 1 of the out-of-sample data set. For me this was very unexpected, and you can imagine I got pretty incensed: while a slightly lower performance is to be expected when you forward test an optimized system with out-of-sample data, the optimization had been performed with over 2000 data points (days) extending over more than 8 years, so it is really hard to accept that the system fails from day 1 after the optimization period. Especially when the optimized parameter values were not at isolated "peaks", but at the center of broad ranges of acceptable values

I have tried to look for alternative explanations... for example, last year (starting around summer) was particularly bad for breakout systems, so I might just be witnessing a normal fluctuation of the system's performance during a time of particularly difficult market behavior. Possible? Yes, and in fact the system seems to be recovering from the losses towards the end of the testing period, but I don't know how likely it is. This is probably a combination of a number of factors. Bottom line, I have no choice but to work under the worst-case assumption that curve-fitting, no matter how careful I have been, is playing a role here.

As a counterexample (and to add to my dismay at this whole optimization process), I am going to show you what happens when a similar process (optimization+forward testing) is performed on a second, different system. To keep things as similar as possible to my first example, this system is run by the same EA (it is also a breakout box type of entry) but instead of targeting short boxes at market open it usues long boxes (almost 24 hours). In other words, it is basically a breakout of the prior day's high or low, and it is thus a very different system from the first. I am also running it on a different pair, USDJPY. Using the same EA is just a matter of convenience. Attachment 2 shows -in the upper graph- the 8+ year optimized performance of the system, again adjusted to a maximum DD of 15%. You can see the results are also good, with an average yearly profit of 50% and a profit factor of 1.58. Great. Now we move to the out-of-sample test, spanning (as before) from July 2009 to January 2010 and, surprise surprise... The lower graph in attachment 2 shows that unlike the previous system this one actually would have made money! A 40% profit in those 7 months with an 11% maximum DD (so far), which is within the system's expectations, maybe even a little better.

Do you see my confusion here? I have two different systems which I have developed/tested/optimized following the same pattern, yet one fails basically from day 1 (pointing to curve fitting, maybe?) and the other works fine. The optimization process followed for both of them is exactly the same (the number of parameters, hell, everything. It is the same EA!!), even if the "edges" behind the respective systems are very different. It may simply be the case that both systems works and will work in the future, and I just hit a rough patch with system 1 due to market conditions... or it may be that system 1 will never work because its edge is fictitious, I just fitted noise... or that it will start working tomorrow, and system 2 will fail.

Clearly something else is needed to evaluate the merit of these systems and discern between real edges and fitted noise, and it would appear I am reaching the point where running some sort of Edge/MonteCarlo test can no longer be avoided. However, I am at a loss about how to tackle this. Acrary in ET gives some good descriptions of what he does, and Aronson also goes over this in his book, but their descriptions are very general (and often based on simpler long/short rules) and I am not sure how to apply the procedures to these types of systems. An additional problem is that so far I am constricted to an MT4 environment, and I may need to broaden my "programming horizons" if I want to incorporate these kinds of tests (I am NOT a programmer by trade, I just can code reasonably well in C and some MatLab as a side effect of my job). But that is a different problem.

So, after this long -and I hope somewhat interesting- story... anybody would care to offer some pointers or advice on how to proceed from here? What kinds of statistical tests do you guys perform to evaluate the "edgeness" of your systems once you think you have arrived to something that may work? In case you want to get technical I am acquainted with distributions and statistical jargon in general, btw. Tools of the trade. My "real" trade, that is.

Thanks in advance for the patience to read the long rant...

Thing is, I don't think monte carlo would change your results. First one will still give about as good result with variations as will the second one.

This is one of the hardest problems with the optimization, especially with the kind I'm making where the program is making all the algos so I'm basically curve-fitting random genetic programming algorithms so the fitness algo is the key and it's the one that took me a lot time to make work in a decent way. I also would claim that MT4s optimizer is far too pritive for anything serious.

The main thing might be that you don't really want to find the peaks, you want to find the areas near peaks because peaks are very often curvefitted (google for fitness landscape if you don't understand what I'm talking about). It's also very important to use multimarket optimization because it's much harder to curvefit to multiple markets than one, in fact algos curvefitted to multiple markets might actually be working ones if you monitor correct values.

Also if you use the out-sample result in the fitness formula in any way (even when verifying your results as you do) you are basically optimizing to out-sample too so you must use only in-sample results with large sets of data and basically optimize the fitness algo by hand.

Just my 2 cents, I'm very interested in other comments.

I think mikkom is correct, MC will not help you in this instance, MC is a good non-parametric way of determining the distribution of your results, however it tells you nothing about the future performance.
You have come up against the limitations of MT4 back-tester in terms of where you can go next. The only way of making optimizing work is by applying metric criteria to the optimization that are not available in MT4 and/or combining multiple systems. I have attached a screenshot of my GA project, this is not a back test or an in/out sample test but a walk forward test for ~4000 trades (there are about 16 re-optimizations in the period shown), in other words it is all on unseen data. So AI/optimization can work, there is no infallible way of avoiding curve-fitting, but by applying occam's razor you can skew the odds in your favor. Sorry for being slightly vague, but you are asking the right questions.

Hi Craig,

Nice job.. you use only GA or with SVM/NN/Fuzzy.. ?

Regards.

Thanks for the replies, guys.

Regarding the optimizations, I am taking the basic precautions you mention, Mikko: for example I don't go for the peaks but the broad areas (I am familiar with the fitness landscape and ga concepts in general btw, I have never had to program one but I use them from time to time, they crop up everywhere these days if you have a technical/research job), and I try to see how the systems work on other pairs/markets. I think I am well past the first, greed-driven go-for-the-maximum phase that we all go through when we start looking into mechanical systems.

Given that it doesn't look like there is an easy way to solve the curve-fitting problem, combining multiple systems is not only a good idea, but a must (IMHO). I simply wouldn't trade a system like the ones of my previous post in isolation, even if it has passed the out-of-sample test like the one for USD/JPY. I'd try to get a bunch of systems which are sufficiently uncorrelated (the backtesting results of all the systems by time period, for example monthly, should give a good enough idea of the degree of correlation if the sample is large) and split the total risk among them, to me that is just common sense (and a little bit of risk aversion).

It appears that I am at the limit of what MT4 can offer in terms of optimization, that is clear enough. The problem is that -not being a professional programmer- the task of building a custom-tailored testing enviroment is not a small one. Mind you, I can learn pretty much anything, and as somebody said if you know how to program in a couple of general purpose languages then learning new ones is more of a learning-the-syntax problem, you already know the logic (mostly). But it takes time and effort. For example, I assume your testing frameworks have a "trading engine" at the core (the one that takes care of the basic logic of what to do when price gets here, whether it triggers an order, hits a StopLoss, etc etc...). And then all the optimization built on top of that. It is a BIG project.

Initially, I am tempted to look for an already existing platform that -while not as powerful as a custom-made one- would allow me to do testing/optimization betyond what MT4 can do (that is not too much to ask!), and hopefully can later be extended to do more custom-tailored stuff. Is there such a thing? Somebody has mentioned Openquant/Smartquant in this thread, and probably NinjaTrader would be another alternative. Do you guys have any advice along those lines, or do you think it is just a waste of time and I should go directly for my own stuff?

Thanks again for the very helpful contributions.

GA only.

NinjaTrader is actually quite good, it has a walk forward option and the back testing is customizable in some quiet interesting ways via writing classes in C#, this might be a good way of learning a bit of C#, the problem is going to be (I'm guessing) getting data. I had a real-time trading platform, into which I added real-time simulation and then tick/bar back testing, it was a long process.

Thanks, I'll take a look into it. I have been postponing getting into object-oriented languages for too long anyway (haven't really needed them for the kind of occasional algorithmic programming I do, C always sufficed), so this might be a good excuse. Although I am not as young as I used to be (duh!) and I have to admit this kind of thing -learning new languages- is not as much fun as it once was...

The data... well, yes, that is a problem for all of us. Aside from the usual suspects from where one can get M1 data, like forexite, there is also Oanda, they are now providing tick data upon request (incidentally, it is free for research institutions and I work at one, so a few months ago I requested a data set; took forever but they sent me GBPUSD tick data since 2004, I think; I got it not long ago and haven't really had a chance to take a close look at it). I have also been collecting live ticks from my market maker broker for over a year for a few pairs, I thought I'd come to need it one day.

Tell me about it, I'm just glad I know C++ so well (the day job was good for something!).
If you have a good grasp of C, C# isn't a million miles away, same control structures etc.

This is one of the more interesting posts I have seen in a while

I'm quite interested in how long optimization period did you use or was it "dynamic" based on some triggers? Is this one/multiple algos on one or multiple pairs?

I do undestand why you are vague - IMHO automatic reliable optimization is the holy grail, not some one algo.

And yes, programming the testing framework was not an easy task.

First of all, it's still a work in progress in terms of that I'm not ruling out there being some kind of mistake in the framework (I'm sure we have all been there). What I found with the optimization period is that longer optimizations don't always work best, for example the re-optimization period in the graph is static and less than 1 month. In the example given the algorithm was walked forward on 5 pairs separately, the graph is a composite of the results for each pair. I think the robust nature of the out of sample results may be due to other structural aspects of the optimization and only loosely correlated with the optimization period.

LOL, just found a shocking bug, will repost the result of the same experiment when it is fixed (it takes a couple of days to complete a run).

couple of days? How many data points you evaluate overall and how big in terms of data points is your reoptimalization windows? And how large is your fitness landscape?

anyway... couple of days... seems like you really could benefit from some code optimization

i am using CUDA for my development. Optimization (either brute force, genetic or some other heuristic methods) is usually heavily parallelizable... and that's where CUDA really shines if you can write your code in such ways that it can take advantage of the architecture.

I you or mikkom would like to get your hands on my server with the GeForce card via RDP to play with it, it could be arranged

That is not easy to answer without knowing more about your algorithm. A lot depends on things like your memory access pattern, locality (in memory space) of the accessed data, amount of the data and of coarse the amount and the type of computations...

Ideally the math.operations/memory accesses ratio should be high, but even if it is not the GPU can be a lot faster than CPU if the locality of the data is good. And naturally your computations must be well suited for parallelization.

For example in your case it sounds like your problem will be memory bandwidth bound. If your memory access would allow for coalesced reads/writes you would be probably able to hit close to full memory bandwidth. For comparison:
Nowadays an Intel Core i7 system is commonly equipped with a triple-channel PC1333 DDR3 memory modules which give you a total of 32GB/s of memory bandwidth.
On the other hand a GeForce GTX 285 has a memory bandwidth of around 160GB/s.
Of coarse it depends on your code, if you actually can benefit from that bandwidth. If you can't access the memory in a coalesced way than the actual bandwidth would be significantly lower.

CUDA programming is rather close to metal so you have to consider things like memory bandwidth, memory access patterns, caching, registers and on-chip memory, instruction scheduling and throughput etc.

A lot also depends on the actual hardware that you want to use. For example the new Fermi architecture will bring some big changes that could make a big difference for certain types of problems, especially those that have rather random memory access pattern.

to get you started i would recommend to take a look into this rather lengthy but thorough CUDA Programming Guide:

http://developer.download.nvidia.com...gGuide_3.0.pdf

Hope this helps
cheers
RoBiK

Yeah, I know it could be faster, first I want to get it right!
Some of the slowness is due to the fact that some of the things I am feeding into the GA are quite mathematically complex (matrix inversions etc.).