Systematic trading | Page 34 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Trade another day through Systematic Trading 203 replies

Nonparametric Association Measures in Systematic Trading 57 replies

Anyone use Systematic from Alpari? 1 reply

A Systematic Approach to Markets 68 replies

Systematic Trading vs. Discretionary Trading 14 replies

Attachments: Systematic trading
Exit Attachments

Systematic trading

  • #661
  • Apr 1, 2010 12:04pm Apr 1, 2010 12:04pm
Quoting Craig
QT Creator does look good doesn't it? I was going to port my GUI but Creator seems to be missing a grid component and I also couldn't find a charting component.

I developed my own back-testing software so I could do the GA stuff, my software works though interfaces. For example I have a 'IOrderManager' interface which places/manages orders, from this I derive 'FixOrderManager', 'TWSOrderManager' and 'BacktestOrderManager', by using interfaces I can keep the core logic decoupled from broker specific stuff. Writing your own stuff is a difficult...

What is GA, sorry im still learning about this way of trading.

Mark
Economists have forecast 9 out of the last 5 recessions
  • #662
  • Apr 1, 2010 12:09pm Apr 1, 2010 12:09pm
Quoting markmm
Why not have a column with timeframe and one for instrument, than you only need on table. Depending how long it takes though you might want to generate timeframes at runtime.
I'm aware of normalization. I need fast scanning (range search). If I store everything in one table I would have 60*24*5*53*5*100=190,800,000 rows for 5 years of M1 on 100 instruments. I don't want to go there with Postgres on a non-dedicated DB machine.
  • #663
  • Apr 1, 2010 12:23pm Apr 1, 2010 12:23pm
Quoting markmm
What is GA, sorry im still learning about this way of trading.

Mark
GA = Genetic Algorithm.
The breaking of a wave cannot explain the whole sea.
  • #664
  • Apr 1, 2010 12:26pm Apr 1, 2010 12:26pm
Quoting Craig
GA = Genetic Algorithm.
Great thanks, Craig.

I think this way of trading requires a lot of work upfront, but once everything is in place it (hopefully) becomes easier. I think I have a good 6 months of coding ahead of me lol. But I love this stuff.

Mark
Economists have forecast 9 out of the last 5 recessions
  • #665
  • Apr 1, 2010 12:45pm Apr 1, 2010 12:45pm
Quoting Adal
I'm aware of normalization. I need fast scanning (range search). If I store everything in one table I would have 60*24*5*53*5*100=190,800,000 rows for 5 years of M1 on 100 instruments. I don't want to go there with Postgres on a non-dedicated DB machine.
You might also want to think if you really need sql for that, you might as well use direct HD operations because you know very well what size every row is.
  • #666
  • Apr 1, 2010 12:55pm Apr 1, 2010 12:55pm
Quoting mikkom
You might also want to think if you really need sql for that, you might as well use direct HD operations because you know very well what size every row is.
Yes, I thought about that, using memory mapped files, or even loading all the M1 data for a instrument in memory (around 70 MB, very manageable).

You are right, I don't need SQL at all. All I want is a range-search-able array. I investigated HDF5 and BerkleyDB for this, but they have some issues.

I already spent a lot of time building my tick storage infrastructure, and I want to start building strategies instead of more infrastructure work (bar file format, inserting/deleting, searching, ...). That's why I'm thinking of just going with postgres, and redoing this when I'll have more time.
  • #667
  • Apr 1, 2010 1:07pm Apr 1, 2010 1:07pm
Quoting Adal
Yes, I thought about that, using memory mapped files, or even loading all the M1 data for a instrument in memory (around 70 MB, very manageable).

You are right, I don't need SQL at all. All I want is a range-search-able array. I investigated HDF5 and BerkleyDB for this, but they have some issues.

I already spent a lot of time building my tick storage infrastructure, and I want to start building strategies instead of more infrastructure work (bar file format, inserting/deleting, searching, ...). That's why I'm thinking of just going with postgres,...
In that case I suggest following craigs advice - you really want to normalize your structures.

I personally have a cached system where I have the bars on database but I also have a cached version as a direct stored array on disk. If you really need speed that is the only way to go. Iterating 10 years of minute data from db will be your bottleneck assuming you need really fast stuff (on my GP stuff speed is the essence of the whole thing).

Another totally different way I have been thinking is building a framework based on events on top of the candle data so that way you could skip all the candle data and use only what you need (Also with magnitude of higher speed) but this might be a bit beyond of what you have been thinking..
  • #668
  • Apr 1, 2010 1:29pm Apr 1, 2010 1:29pm
Quoting mikkom
In that case I suggest following craigs advice - you really want to normalize your structures.
Sorry, but I don't get why. I don't have data duplication. It's not like I have the same CustomerName in two different tables. But I'm no expert in DB modeling (for example I never learned the difference between first form and third form).

Quote
I personally have a cached system where I have the bars on database but I also have a cached version as a direct stored array on disk. If you really need speed that is the only way to go. Iterating 10 years of minute data from db will be your bottleneck assuming you need really fast stuff (on my GP stuff speed is the essence of the whole thing).
Actually this is quite close to what I want to do. Store everything in the database, but before testing an instrument, extract the relevant data from the DB into disk-stored arrays, memory map them and work only on that. So I will hit the DB only once. I think that PostgreSQL can dump the binary data directly to disk (using COPY TO), so I wouldn't even need to iterate through a query result.
Quote
Another totally different way I have been thinking is building a framework based on events on top of the candle data so that way you could skip all the candle data and use only what you need (Also with magnitude of higher speed) but this might be a bit beyond of what you have been thinking..
But wouldn't the framework which does this filtering have to read all the data anyway? How can this be an order of magnitude faster?
  • #669
  • Apr 1, 2010 1:35pm Apr 1, 2010 1:35pm
Quoting Adal
But wouldn't the framework which does this filtering have to read all the data anyway? How can this be an order of magnitude faster?
If you know what events you are looking for you only need to find them once and store them somewhere (db for example), after that time series between events are basically irrelevant (if you also exit based on events) and your algo testing would basically be just jumping from event to another - instead of iterating 100000000000 candles, you would be iterating 30 events.

In real time trading you would only need to add the event to the db as it happens and act accordingly.
  • #670
  • Apr 1, 2010 1:40pm Apr 1, 2010 1:40pm
I'm, starting to think that having years upon years of tick data or minute data inst going to be much use, I am thinking just setting up methods to paper trade and monitor them using a meta system. Trying to find an edge to fit markets over 10 years when the technology has changed and players change, and probably will in another 10 years, is a daunting task. I would prefer a more agile system to respond to the market as it changes.
Economists have forecast 9 out of the last 5 recessions
  • #671
  • Apr 1, 2010 1:49pm Apr 1, 2010 1:49pm
Quoting Adal
I'm aware of normalization. I need fast scanning (range search). If I store everything in one table I would have 60*24*5*53*5*100=190,800,000 rows for 5 years of M1 on 100 instruments. I don't want to go there with Postgres on a non-dedicated DB machine.
I see, yeah thats a lot of data, Ive never had to deal with such a large amount, have a look at http://en.wikipedia.org/wiki/Hypertable
Economists have forecast 9 out of the last 5 recessions
  • #672
  • Apr 1, 2010 2:07pm Apr 1, 2010 2:07pm
Quoting markmm
I see, yeah thats a lot of data, Ive never had to deal with such a large amount, have a look at http://en.wikipedia.org/wiki/Hypertable
At my job we're using HBase (very similar to Hypertable). We successfully used it for storing hundreds of billions of rows. But you need a cluster for that. I want to use only my desktop (a beefy one), and without impairing it's normal use

I agree with you, that's why I'm using only the last 3 years of data. And even if I have 50 different instruments, I will realistically only trade about 5 of them, because the others are either too exotic (risk of intervention/geo-political event) or too volatile.

But I keep all the data for other purposes - for example testing data from different brokers against each-other to detect spiky/unreliable ones.

I said it before in this thread. Searching for a system that works over really long periods (3 years+) is a mistake. You will get much more by having a more fitted model (yes, this goes against common knowledge), but you have to monitor it very carefully to detect when it's starting to lose it's fitness.
  • #673
  • Apr 1, 2010 2:12pm Apr 1, 2010 2:12pm
Quoting Adal
At my job we're using HBase (very similar to Hypertable). We successfully used it for storing hundreds of billions of rows. But you need a cluster for that. I want to use only my desktop (a beefy one), and without impairing it's normal use

I agree with you, that's why I'm using only the last 3 years of data. And even if I have 50 different instruments, I will realistically only trade about 5 of them, because the others are either too exotic (risk of intervention/geo-political event) or too volatile.

But I keep all the data for other purposes...
Agreed, how do you find dukascopies data, I used a php script from some guys blog to get 1 minute data. http://eareview.net/tick-data
Economists have forecast 9 out of the last 5 recessions
  • #674
  • Apr 1, 2010 2:20pm Apr 1, 2010 2:20pm
Quoting Adal
I said it before in this thread. Searching for a system that works over really long periods (3 years+) is a mistake. You will get much more by having a more fitted model (yes, this goes against common knowledge), but you have to monitor it very carefully to detect when it's starting to lose it's fitness.
I would really like to see this backed by some solid data or experiments. I'm more than willing to adjust my framework to get better results.'

By the way there are other ways of seeing how well algo works on the current market as well other than just fitting to last X period - for example calculating mar/pf for the last X period of time or ema-style weighted pf/mar for last X period.
  • #675
  • Apr 1, 2010 2:24pm Apr 1, 2010 2:24pm
Quoting mikkom
I would really like to see this backed by some solid data or experiments.
I think for daily and above it is possible to have a system work for many years in a particular market, but intraday I would imagine its a difficult task, just from what I have read.
Economists have forecast 9 out of the last 5 recessions
  • #676
  • Apr 1, 2010 2:35pm Apr 1, 2010 2:35pm
Quoting markmm
Agreed, how do you find dukascopies data, I used a php script from some guys blog to get 1 minute data. http://eareview.net/tick-data
Like I said, I spent so much time on the tick infrastructure that I didn't get to play with the data yet. What I noticed is that they have the highest volume of ticks, around 30% more ticks than FXCM. They also have depth of market data, although I don't know how reliable it is (I found ticks with billion of dollars available on the bid, versus the usual 3-30 million).
  • #677
  • Apr 1, 2010 2:58pm Apr 1, 2010 2:58pm
Quoting mikkom
I would really like to see this backed by some solid data or experiments. I'm more than willing to adjust my framework to get better results.'

By the way there are other ways of seeing how well algo works on the current market as well other than just fitting to last X period - for example calculating mar/pf for the last X period of time or ema-style weighted pf/mar for last X period.
http://i.imgur.com/uwnMZ.png

http://i.imgur.com/BNAH3.png

You don't need a paper to prove this. Just think about it. You need only as much data as required to statistically prove that your system is valid. The more frequent you trade, the less the required test period.

There's also under-fitting, not only over-fitting. Under-fitting is when you extract less profit than possible because you optimized too little.

http://i.imgur.com/YFWJb.png

Me, I want to trade as frequently as possible. First H4, than M5 and then hopefully M1. The more you trade (with positive expectancy) the better the Sharpe ratio will be. HFT algos have Sharpe 40+. Daily trading systems have around 2.
  • #678
  • Apr 1, 2010 3:13pm Apr 1, 2010 3:13pm
Quoting Adal
You don't need a paper to prove this. Just think about it. You need only as much data as required to statistically prove that your system is valid.
When you are ready to do some longer term simulations (on any method you feel best) I'm really interested to see them... until then you are just assuming.

I did my first GP runs over 10 years ago and had many similar thoughts.. most of them have been busted hard. Reality just doesn't always work as you assume, that's why testing is such an important thing to do.

I don't necessarily mean testing single algo on 10 years of data (although that is what I do even though I have tested all kinds of risk variation/strat portfolio algos, usually built inside the strats by genetic algos themselves [that's something to think!]), I mean testing your assumptions by running a rolling test of your assumptions on 10 years of data. There really is no way to get around testing on longer term data - it's not just possible to prove that something is true on longer term by just using short term data.
  • #679
  • Edited 4:50pm Apr 1, 2010 4:22pm | Edited 4:50pm
I believe it is a learning process. You have an idea and you start writing the code. It evalves from there. Bottleneks appear and you resolve them. Other ideas come from evaluating your ideas. The power of writing the code is that you gain in-depth knowledge of what you are doing and you have the ability to change the code and enhance it. I have put stuff in a database only to move it out later for performance reasons but what it is really about do you have an idea that works. You need a "hook" to make it worth continuing and to keep motivated. This process will put you in a different category than the mt4 coders who are "presented" with a lot of short cuts but also with a lot restrictions and insight.

It is a long and tideous process but hopefully worth it in the end.
  • #680
  • Apr 2, 2010 2:10am Apr 2, 2010 2:10am
Quoting spekkiefx
I believe it is a learning process. You have an idea and you start writing the code. It evalves from there. Bottleneks appear and you resolve them. Other ideas come from evaluating your ideas. The power of writing the code is that you gain in-depth knowledge of what you are doing and you have the ability to change the code and enhance it. I have put stuff in a database only to move it out later for performance reasons but what it is really about do you have an idea that works. You need a "hook" to make it worth continuing and to keep motivated....
This is what I have been thinking. I have some seeds of ideas, skills and time to develop the tools I need rather than use a tool that constrains me. If we can use statistics and probability knowledge to analyze our results then we can have more confidence in our systems. To me this way of trading makes the most sense.
Economists have forecast 9 out of the last 5 recessions
3 traders viewing now