Systematic trading

I'm using prices without decimal as short arrays. I see no differences with results but operating on integers are simpler. On eSignal, GBPUSD prices are quoted without decimal (I dont know why but it is).

It's already way fast. I'm sure arrays are even faster.

Just remembered that you use c++ (if I remember correctly) - there the difference between objects and arrays should be minimal anyway. 'Java is the language that is full of performance traps.

People in this thread may be interested. I've been interested in forex systems for a while, and finally decided to have a go - using my knowledge of AI and operations research.

I've just started, but I'm blogging about it over at FX.AI.

David

About this performance traps in java... I've found one (for me that was very important) and I would try to explain that. Beginners like me can avoid this trap. When I've made a class storing primitive arrays of data I then wanted to get an exaple highest candle from X candles backward from current point in history. I've made mistake trying to get it from my parent strategy class. Doing the loop through this part of history backward via few other classes was that performance trap. That extends my test time from ~100ms to 3 minutes !!! I then realize that this search should be done inside the history data class. Iterating directly on array (example: high[i] ) instead of iterating through other classes (example: historyGroup.get(0).getHighAt(i)) makes that differences. I'm really newbie and one day get me to realize what I'm doing wrong. Now it came back to previous speed. Other simple indicators I'm going to make the same way. Maybe it will be useful for someone Regards!

Great thread! It's why I decided to sign up. First post...

Over many years I've dabbled in neural networks, genetic programming, computer graphics, etc. C is my preferred language, due to my comfort level with it. In theory, Lisp would rank high on my list, but I'm just not proficient with it.

Regarding your storage choice for bar data, consider using four ints (four bytes each) for storing data in the following manner:

where d1 allocates 24 bits for the open, d2 allocates 24 bits for the high, d3 allocates 24 bits for the low, and d4 allocates 23 bits for the positive difference between the close and the low.

For the day of the month, 5 bits are reserved, yielding 32 days.

For the year and month (combined), 11 bits are reserved yielding 168 years beginning with 1900 and ending with 2067.

For seconds in the day, 17 bits are reserved, covering all the seconds in the day.

The time and date bits are packed into the upper 8 bits of d1, d2 and d3, and the upper 9 bits of d4.

We can store any equity price up 167,000 to a precision of one cent, with a maximum difference between the close and low of 83,000.

Our time and date gives us a range from the beginning of the 20th century until 2068 with a precision of one second. Total storage space per bar is 128 bits, or 16 bytes.

If there are any errors in my above scheme, forgive me. My current program is using the less efficient storage of four doubles for the price data, and further storage space for the date and time. I'm likely going to switch over the above method and use C macros using bitwise operators (and an addition for the close) to quickly access the desired components.

There is so much more to say, especially with regard to the following topics:

- Genetic Programming
- Immune Programming
- Neural Networks
- STDP
- DSP
- Fault tolerance

Also of interest to me is an evaluation of alternative and unique templates for a trade's life cycle, where the dominant paradigm is the following:

1. Identify a set of a patterns given price data which are the trade setup.
2. Identify a price level to trigger entry and a stop level.
3. Determine position size.
4. Enter the trade if it is triggered.
5. Set the stop to break even when a certain price level is reached.
6. Adjust the stop and the take profit continuously until the exit is triggered.

While there are minor variations to the above, I think it might be worth exploring major variations to the above, just to open up the different kinds of analysis that can be applied.

For example, what if step number 1 was turned on its head to be like the following:

1. Always desire to be long, unless you can find patterns or conditions which indicate that you should not be.

Just some thoughts.

Hi,
I'm finishing my master's thesis and I'd like to present some points and problems.

First of all, I'd like to thank everyone who participate in this thread (especially mikkom). You influenced me greatly.

- I downloaded Ducascopy tick data (2007-2011) of six major pairs, checked it, and converted into 1min. Usage of tickdata was probably an overkill
- I computed binary variables, based on technical indicators and BO strategies (ie. RSI of some period break some boundary from below, price breaks from opening range of some length, etc.). I doublechecked everything, visualized computed variables etc. Variable count was around 5000. This stage took VERY long time
- Strategy have been composed of entry rule, exit rule, SL and PT. Rules were logical expressions of binary variables, SL and PT have been picked from predefined set. Logical expressions have been represented as trees. Trees have been modified by crossover and mutation - GP.
- Strategies have been generated in two stages: 1) blind search 2) GP
in 1), almost all strats were rubbish. Ratio between promising strat and rubbish were 1:thousands. Population from promising strats has been composed. In 2), population was driven with respect to final stage-combination of strategies. There were three dimensions to satisfy: high profit/low risk, non-randomness and diversity between strategies. I used concept of Pareto optimality for this multi-criteria problem. Fitness was based on SPEA2 algorithm
- before stage 1), 1000 variables have been picked at random and stored in a memory
- trade execution aims to be little bit worse than reality - double spreads, SL few pips higher to simulate SL hunting. I haven't been trading regularly, don't know if this is enough to simulate bad conditions
- I generated strategies on every pair and every direction (long/short) separately. I'm not sure if this is a good thing, but it makes sense to me. If both direction on directional biased pair (like U/Y) had been generated together, short strategies would have probably prevail. When trend changes in out-of-sample, these short strats will do probably badly
- no position sizing has been applied, all positions was equal
- I picked two strategies of every pair and direction manually. Selection was based on a shape of equity curve in in-sample. Equity was constructed in respect of time, not trades.
- Selected strategies have been weighted in respect of their DD in in-sample and putted together
- Results were disappointing, but not so bad to throw everything away. IMHO, causes of unsucces are follows: 1. lack of experience - I don't know what works and what do not, 2. small population size - only 100 individuals, 3. not so large sample size. I think that fitness algorithm is in general OK and it boosts right behaviour in population (But I can be terribly wrong of course
- Everything is done in Matlab. I'm not a true programmer, Matlab is an environment where I am most efficient. But I am aware of its limits, I will write critical functions in C. Btw, in blind search phase, strategy evaluation took 60ms per strategy on average (i7 920, one thread). I think that 10x-20x speed-up is easily achievable.
- I understand (better than before) that GP is "only" good search algorithm, not magic

Any thoughts?

I think the above is exactly right.

GP (or evolution in general) is just a better method to search complex results than most methods. The real questions are what are the good building blocks for the results and how to tell which of the methods is best.

I would bet that population of size=100 is just too small for GP. I have not much experience with GP at this moment, but people I know they use some GP software already for live trading, they use populations of size at least 10000 or more (up to 50k).

BTW also I wanted to ask - do you all use tree-based GP? Did anyone try linear GP as well?

One thing about GP is it's really important to have really strict data typing. It's not enough to say that arguments and return types are doubles or booleans. Consider an SMA. What does it return? It doesn't just return a number (a double). It returns a price. What does a price plus a price range return? A price. Some examples of various data types in price analysis are: price, price range, time, duration, percentage and specialized indices. One has to answer the question how each of those are allowed in combination with each other, given an operator. Otherwise, you'll end up with nonsense like this:

Hi,
I am probably telling nothing new in this post. I just want to explicitly formulate central question in backtesting: Which properties should strategy have, or, in other words, What determines strategy's success (on OOS/future). Performance alone (ie.calmar) says nothing. Much important is, how was this performance achieved. Observed performance must be achieved in most non-random way. So, another important question is Which metrics support non-randomness. Trades count is obvious. More is better. Another one (mentioned by mikkom), "trade skew" * works also well here. IMO, "equity skew" (trades should be evenly spread in testing period) is next important aspect.

* trade skew, IMO, is tightly linked with bootstrap testing. When trade skew is low (profit is uniformly distributed), then bootstrapped samples won't differ much.

I would love to hear opinions on this topic, because it is (IMO) essential

Long time without posting but somehow I thought this might fit here perfectly.

How I made $500k with machine learning and HFT (high frequency trading)

It's an interesting article, more for other reasons rather than the ML.
But let's face it, you could pound the keyboard with a ham in 08/09 and still make money.
The decay of his edge is fairly stock for scalping strategies as the vola decayed.

Very true. One of the most interesting things is how fast his edge disappeared.

systematik, trade by trade, day by day you have to collect pips and at the end of a week woala-coala
The most important think is to keep your losses as small as possible !!! My maximal value for SL is 12 pips

http://2imgs.com/2i/i/509cd5ea/7914e...867a5083.f.gif

Dear Mikkom,

first - thank you for starting this thread. It is very interesting (took me 12 hours to read it all), and it offers many hidden treasures!!

My background might be a bit similar to yours (ex-CTO & founder, working on a new company now), privately developing a multi-threaded GA/GP system in Java for 8 years, in contrast to yours not profitable yet.

Therefore, I hope you do not mind if I have 2 questions (please feel free to ignore them if they go into too much detail):

your way of doing GP is very clever. I like the way your out of sample curves are created and how they look, and how you create the different animals. How many degrees of freedom do you have in your GP - meaning how many different items/variables/functions are needed to form one single trading algorithm (of which 20 or so form a full tradeable strategy)? How many possible states are available for each degree of freedom?

You devised a very nice trick to compress the bars. You mentioned somewhere that you use indicators also. Do you precalculate and store indicators in the database, and if yes, do you also have a way of compressing and storing the data? I found that if I have massive numbers of animals and work on ETFs, I need to store so much indicator data that it clogs my RAM. Do you store on disc or in memory only or do you use a database to store indicators for different optimisations runs so you do not need to precalculate everything from scratch everytime?

BTW - you seem to like Erlang - I guess you have already heard of Clojure (http://clojure.org) ? Clojure is a Lisp-Dialect that runs nicely in the JVM (the Lisp is compiled into Java-Bytecode internally) and can mix very nicely with Java Classes for stuff that might be not feaseable for Clojure or that is already built in Java ....

Thank you very much again for this thread and kind regards
zapphod

Another interesting link here.. "Buffett's Alpha"

Abstract
Berkshire Hathaway has a higher Sharpe ratio than any stock or mutual fund with a history of more than 30 years and Berkshire has a significant alpha to traditional risk factors. However, we find that the alpha become statistically insignificant when controlling for exposures to Betting-Against-Beta and quality factors. We estimate that Berkshire’s average leverage is about 1.6-to-1 and that it relies on unusually low-cost and stable sources of financing. Berkshire’s returns can thus largely be explained by the use of leverage combined with a focus on cheap, safe, quality stocks. We find that Berkshire’s portfolio of publicly-traded stocks outperform private companies, suggesting that Buffett’s returns are more due to stock selection than to a direct effect on management.

http://www.econ.yale.edu/~af227/pdf/...20Pedersen.pdf

Thanks for the information

I have been thinking of creating an online (web based) version of my trading algo tester so that anyone could test strategies. The language would most likely be simple rule based system or a lisp-like parse tree. I think rule based would be better suited for a layman.. Portfolio testing and multi strategy testing would be included of course. I have a server in a server room in germany with quite a lot capacity sitting without much load that I could use.

I thought this would be a great tool to explore (or more likely disprove ) non-functional systems in system forum for example.

I'm a bit concerned that if I'll do that, I will be thrown to commercial section forever even if the tool would be free. Admins could comment that if they see this. Heck, maybe I'll contact the admins myself and try to create some kind of partnership after the tool is ready. FF would then have a systems testing tool.

Would anyone be interested in this kind of tool? I think it would be pretty easy to generate MQL/other language programs out of the tested rules so the tool would be quite beneficial to people who can't code themselves as well.

Comments? Is this silly idea?

this kind of tools seem redundantly available nowadays over internet. even some are not free, but cheap enough.

google, and you can find them

what we don't have, is a robust tester that can test many imaginations. for example, multi-pairs, basket trading, and so on. now only a few pro software/API can do it. but if you develop this, this will be very costly and ultimately cannot be free.