Systematic trading | Page 58 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Trade another day through Systematic Trading 203 replies

Nonparametric Association Measures in Systematic Trading 57 replies

Anyone use Systematic from Alpari? 1 reply

A Systematic Approach to Markets 68 replies

Systematic Trading vs. Discretionary Trading 14 replies

Attachments: Systematic trading
Exit Attachments

Systematic trading

  • #1,141
  • Oct 1, 2010 7:04am Oct 1, 2010 7:04am
Found a really powerful charting library for C++/QT4

http://www.int.com/products/2d/carnac/chart_gallery.htm

Looks awesome.
Economists have forecast 9 out of the last 5 recessions
  • #1,142
  • Oct 1, 2010 8:06am Oct 1, 2010 8:06am
Quoting markmm
Found a really powerful charting library for C++/QT4

http://www.int.com/products/2d/carnac/chart_gallery.htm

Looks awesome.
Try loading a 4 million point time series and then scrolling through it. I couldn't find anything that can handle that, so I had to roll my own.

http://dl.dropbox.com/u/190212/ff/veloce_charting.png
  • #1,143
  • Oct 1, 2010 8:36am Oct 1, 2010 8:36am
Quoting Adal
Try loading a 4 million point time series and then scrolling through it. I couldn't find anything that can handle that, so I had to roll my own.

http://dl.dropbox.com/u/190212/ff/veloce_charting.png
Very nice Adal, looks great. I havent even got the library yet, seems like its ot going to be free. Might have to roll my own also, using QT4 Painting. Although I would like to draw some other kinds of charts like bar charts etc. Ah well its fun and you can make it do what you need I suppose if you implement your own.
Economists have forecast 9 out of the last 5 recessions
  • #1,144
  • Oct 1, 2010 8:56am Oct 1, 2010 8:56am
Quoting markmm
Although I would like to draw some other kinds of charts like bar charts etc. Ah well its fun and you can make it do what you need I suppose if you implement your own.
I know what you mean. For non-timeseries, non-performance intensive graphs I will use matplotlib (http://matplotlib.sourceforge.net/gallery.html), although it will be funky using it from IronPython (it's for CPython only).
  • #1,145
  • Oct 5, 2010 4:39am Oct 5, 2010 4:39am
Looks like SEC investigation about the flash crash has been published, quite interesting if true

http://www.sec.gov/news/studies/2010...nts-report.pdf

Some press

http://www.reuters.com/article/idUSTRE6935SA20101004

http://www.zerohedge.com/article/sec...lling-catalyst

http://www.usatoday.com/money/market...ash04_ST_N.htm
  • #1,146
  • Oct 5, 2010 4:41am Oct 5, 2010 4:41am
Also, did you guys see this - some norwegian traders found a flaw in IB marketmaking algo and took advantage. Now they are sued. I personally don't understand what they actually did wrong here, it's interesting to see how this ends.

http://www.ft.com/cms/s/0/edcaf806-a...44feabdc0.html

There was a much better article than FT in finnish that described the method they used better.

edit: here is some info
http://econotwist.wordpress.com/2010...rading-system/
  • #1,147
  • Oct 5, 2010 5:09am Oct 5, 2010 5:09am
Quoting mikkom
Also, did you guys see this - some norwegian traders found a flaw in IB marketmaking algo and took advantage. Now they are sued. I personally don't understand what they actually did wrong here, it's interesting to see how this ends.

http://www.ft.com/cms/s/0/edcaf806-a...44feabdc0.html

There was a much better article than FT in finnish that described the method they used better.

edit: here is some info
http://econotwist.wordpress.com/2010...rading-system/


...
Yeah, they were just exploiting inefficiency, its the problem of IB and the designers of the algo. The guys should not only not be sued but they should keep their profits.
Economists have forecast 9 out of the last 5 recessions
  • #1,148
  • Oct 5, 2010 5:12am Oct 5, 2010 5:12am
Does anyone know what happened to the blogger for puppermaster trading? He hasn't blogged since July!! Ive taken an offline copy btw, just in case his site suddenly disappears, there is so much useful info.
Economists have forecast 9 out of the last 5 recessions
  • #1,149
  • Oct 5, 2010 5:59am Oct 5, 2010 5:59am
On the same theme: http://blogs.reuters.com/felix-salmo...uote-stuffing/
  • #1,150
  • Oct 6, 2010 6:57am Oct 6, 2010 6:57am
Quoting markmm
Ive been implementing my data storage in hdf5, in fact I will probably use it to store all my data, not just back test data. From my inital impression it is extremely fast. I use the C nativc api (from C++ code), but the api is well designed and as OO as you can get in C. Its a bit of a learning curve initally. but once you go it the speed you can pull data into strategies beats any relational database I have used.
I'm thinking of testing hdf5 on my framework (basically because I want to use a larger array of instruments to test). What kind of speed do you get if you want to iterate a moderate size array of price data? For some reference, I have OHLC data for ~3.3 million points per currency pair at the moment. each price data (for example open) consists of one char (8 bit) value if that matters.

Currently iterating the data is lightning fast because I iterate values in memory but reading from disk (raw) takes a bit too much time so I'm thinking of alternatives (or then I jsut go and buy really fast SSD). I'm just not sure if hdf5 can give any boost over raw data reading unless it uses some hardcore precaching stuff.
  • #1,151
  • Oct 6, 2010 7:02am Oct 6, 2010 7:02am
Quoting mikkom
I'm thinking of testing hdf5 on my framework (basically because I want to use a larger array of instruments to test). What kind of speed do you get if you want to iterate a moderate size array of price data (I have OHLC data for ~3.3 million points per pair at the moment)?
I have only tested writing so far, I am implementing the strategy part, so I can actually use the data I have written on a strategy. However from the puppetmaster blog he was getting 1.28 million records per second, so I guess under 2 seconds for a pair for your data set.
Economists have forecast 9 out of the last 5 recessions
  • #1,152
  • Oct 6, 2010 7:04am Oct 6, 2010 7:04am
Quoting mikkom
I'm thinking of testing hdf5 on my framework (basically because I want to use a larger array of instruments to test). What kind of speed do you get if you want to iterate a moderate size array of price data? For some reference, I have OHLC data for ~3.3 million points per currency pair at the moment. each price data (for example open) consists of one char (8 bit) value if that matters.

Currently iterating the data is lightning fast because I iterate values in memory but reading from disk (raw) takes a bit too much time so I'm thinking of alternatives...
Im not sure what you mean, how does the data get into memory in the first place? How long does that take? You could do the same with hdf5 load the data up then iterate over it in memory.
Economists have forecast 9 out of the last 5 recessions
  • #1,153
  • Oct 6, 2010 7:24am Oct 6, 2010 7:24am
Quoting markmm
Im not sure what you mean, how does the data get into memory in the first place? How long does that take? You could do the same with hdf5 load the data up then iterate over it in memory.
I read it directly from disk to memory before iteration. What I'm looking with hdf5 is a scheme where I wouldn't need to do that, instead I could iterate records directly via hd5.

But 1.5 mil records per second is far, far too slow for me.. As silly as it sounds So looks like I'm going to have to optimize my raw loader and do some other tricks (I have already thought about this).
  • #1,154
  • Oct 6, 2010 7:38am Oct 6, 2010 7:38am
Quoting mikkom
I read it directly from disk to memory before iteration. What I'm looking with hdf5 is a scheme where I wouldn't need to do that, instead I could iterate records directly via hd5.

But 1.5 mil records per second is far, far too slow for me.. As silly as it sounds So looks like I'm going to have to optimize my raw loader and do some other tricks (I have already thought about this).
I see, but you are never going to get the same speed as reading from memory, I also dont know the bloggers hardware setup, so maybe it would have been improved greatly with a faster HD like SSD or velociraptor.

Ill have a veliciraptor so will let you know if my results are significantly better, when I get to it.
Economists have forecast 9 out of the last 5 recessions
  • #1,155
  • Oct 6, 2010 7:48am Oct 6, 2010 7:48am
Quoting markmm
I see, but you are never going to get the same speed as reading from memory, I also dont know the bloggers hardware setup, so maybe it would have been improved greatly with a faster HD like SSD or velociraptor.

Ill have a veliciraptor so will let you know if my results are significantly better, when I get to it.
I know that quite well (mem vs io performance). In fact I have optimized some quite large systems in the past

Anyway, I'm probably going to take a little bit of shortcut to get rid of these kind of silly performance issues.
  • #1,156
  • Oct 6, 2010 8:07am Oct 6, 2010 8:07am
Quoting mikkom
I know that quite well (mem vs io performance). In fact I have optimized some quite large systems in the past

Anyway, I'm probably going to take a little bit of shortcut to get rid of these kind of silly performance issues.
Sorry all I was saying is I think that the hdf5 performance is ok considering its reading from disk, as I say maybe with better hardware it could be fast enough.
Economists have forecast 9 out of the last 5 recessions
  • #1,157
  • Oct 6, 2010 8:46am Oct 6, 2010 8:46am
Quoting markmm
Sorry all I was saying is I think that the hdf5 performance is ok considering its reading from disk, as I say maybe with better hardware it could be fast enough.
Okay.. I didn't get your point there sorry. The point being is that I want to have full iteration in very short timeframe and I just don't think hdf5 is the tool for me if that's the magnitude of speed. I think I'll just have to stick with raw io/memory and work on some other scheme than trying to iterate faster as it's not really possible.
  • #1,158
  • Oct 6, 2010 10:22am Oct 6, 2010 10:22am
Quoting markmm
Sorry all I was saying is I think that the hdf5 performance is ok considering its reading from disk, as I say maybe with better hardware it could be fast enough.
Disk performance is cheap. At work we have machines with 16 10K rpm hard drives. They can read 3 GB/s (sequential). If you want to go expensive you can always buy some SSDs.

I think I'm missing something. You have 4 mil candles of 4 bytes each (one byte per value like you said) and maybe also a 4 byte timestamp. That would be 8 bytes per candle. Multiply by 4 millions and you get to 16 MB. That can be read from disk in an instant, even on a single drive (100 ms on my laptop).

That means that you could read all the candles for 10 currencies in one second. I find it hard to imagine what kind of non-HFT strategies are you running that could process the data even faster than that. What I mean is that I don't see how the bottleneck could be the disk reading speed instead of the data crunching.

BTW, what kind of compression/encoding are you using to get to 1 byte per value?
  • #1,159
  • Edited 5:19am Oct 7, 2010 4:03am | Edited 5:19am
Quoting Adal
I think I'm missing something. You have 4 mil candles of 4 bytes each (one byte per value like you said) and maybe also a 4 byte timestamp. That would be 8 bytes per candle. Multiply by 4 millions and you get to 16 MB. That can be read from disk in an instant, even on a single drive (100 ms on my laptop).
I use laptop too [and one other server], I have no idea why it's so slow, I use javas internal libraries so that might be the problem compared to total raw io. I think I'll have to do some more testing to see where all the time goes..

java char is 16-bit btw.

There are also timestamp (java long, 64 bits - overkill, I should just use addition to previous as the values are sorted) and volume (8-bit).
  • #1,160
  • Oct 10, 2010 6:19pm Oct 10, 2010 6:19pm
This might have some interest:

http://papers.ssrn.com/sol3/papers.c...act_id=1686004

http://www.cftc.gov/PressRoom/PressR...pr5920-10.html
0 traders viewing now