Systematic trading | Page 37 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Trade another day through Systematic Trading 203 replies

Nonparametric Association Measures in Systematic Trading 57 replies

Anyone use Systematic from Alpari? 1 reply

A Systematic Approach to Markets 68 replies

Systematic Trading vs. Discretionary Trading 14 replies

Attachments: Systematic trading
Exit Attachments

Systematic trading

  • #721
  • Apr 14, 2010 2:52pm Apr 14, 2010 2:52pm
Quoting Adal
Hello and welcome to the party
What distributed environment are you using? Hadoop? Your own?
Any reason for protobuf? Have you investigated Thrift and Avro?
How much data do you have?
I coded my own distributed environment, but have used Hadoop in the past. Protobuf: mostly out of habit, it's what I used before, but also seems to compare favorably against the others, and I'm using it for a lot more than data storage: it's great for cross-platform and inter-application messaging (no language constraints). I also use it for inter-server communication -- beats the pants off XML/HTTP RPC which is how I used to do it in 'the old days'. My security data currently is mostly equity and bond ~ 180GB aggregated from several commercial sources, then correlation-checked and cleaned before I store my normalized protobuf files (to SSD).
  • #722
  • Apr 14, 2010 2:58pm Apr 14, 2010 2:58pm
Quoting mikkom
Where do you get your data btw? tradestation?
See prev post. I don't use Tradestation for equity/bond data other than tick data, kind of useless for daily data since they don't provide proper total return series, and only go back to ~1968. Still a n00b in the forex world, and would appreciate pointers to good clean data sources, preferably tick. I'm planning to bin tick data into arbitrary size bars (in server RAM), based on statistical analysis of best odds for diff't timeframes, which may differ depending on particular strategy. Any feedback on the merits of this approach versus, say, using bars of pre-determined size like 5-min or whatever?
  • #723
  • Edited 3:19pm Apr 14, 2010 3:06pm | Edited 3:19pm
Here is a problem I have been thinking about for a while and I'm not sure I have solved it in any meaningful way, so I thought I'd throw it out there. As you all know equities (as does forex) has trading sessions, for example, the main NASDAQ hours are 8:30 - 4:00 EST. Now, I don't want to hold overnight, also, I don't want to be trying to close my positions at 3:59 EST. The current way I handle this is to define an 'interval', in this case 8:30 - 3:45 EST, every bar that comes though the strategy I check the closing time of the next bar, if this is outside the interval I close the trades. This all seems very messy. The first thing that comes to mind is 'why don't you just check the real time', the problem there is that I want the code to work in back-testing mode as well, when I'm back-testing I will not have access to the 'real time'.
Edit: The other constraint is that I want my back-tested trades to close exactly at the same time as the live ones do.

Has anybody else looked at this problem?
The breaking of a wave cannot explain the whole sea.
  • #724
  • Apr 14, 2010 3:08pm Apr 14, 2010 3:08pm
Quoting markmm
Do you find having data going back so far useful? I am only really looking at the last few months, given the market changes often.
For my current strategies (equity, bond) I always try to get my hands on all the data I can get. Especially important for strategies that trade on daily signals. For example, gives me much greater confidence that my bond strategy sailed through both the 1930s melt-down as well as the one in 2008. The story changes for intraday systems, one of my equity systems I only tested on 5-min data going back to ~1998 and a universe of only ~1100 stocks. That lead to waaaaaaaay more trades than necessary to give me confidence in the system :-) I imagine that when I start digging into forex strategies, it will be more challenging to find systems that work over such a vast universe and timeframe. From the little reading I've done so far it seems markets, volatility, etc, have fundamentally changed.
  • #725
  • Apr 14, 2010 3:24pm Apr 14, 2010 3:24pm
Quoting Craig
Here is a problem I have been thinking about for a while and I'm not sure I have solved it in any meaningful way [...] Has anybody else looked at this problem?
You may find that depending on your intraday strategy, there is a best time to get into the market, and there are times you want to stay out. For example, my 5-min intraday equity system does not take any trades in the first 30 minutes. It is then free to take trades for the next 2 hours. After that, it just monitors open positions, which are closed ~ 2-5 min before EOD.

Check out JBookTrader for an elegant way to do both backtesting and real time trading within the same framework, by setting a state flag. (I use a similar approach). The key is this: whether you are backtesting or real time trading, your code should feed bars to your strategy 1 tick or bar at a time. The strategy should not know or care whether it is backtesting or real-time trading, other parts of the system take care of that (I believe JBookTrader uses the concept of a 'trading assistant' to take care of that, but your order management and position sizing algos can do it to -- e.g. strategy just submits an order to a 'BackTestOrderManager' which simply adds the stats for the open/closed trades to the backtesting engine, whereas your 'RealTimeOrderManager' would actually submit the order, through the IB API in my case. BackTestOrderManager and RealTimeOrderManager both implement the same interface, lets say OrderManager, and that's what the Strategy uses, etc, you get the idea). In other words, the strategy always thinks it is trading in 'real time' it simply acts on the last bar or tick it has. BTW this is a good way to avoid bugs where your backtested strategy uses data in the future, I've tripped over that one a few times in the past LOL...
  • #726
  • Apr 14, 2010 4:20pm Apr 14, 2010 4:20pm
Thanks for the reply, this is exactly what I do (e.g. the strategy is unaware of if it is in back-test or live mode), the problem is let's say I want to trade up to 3:45 and I'm using 30m bars.

3:30: We get the bar which opened at 3:00 and closed at 3:30.
4:00: We get the bar which opened at 3:30 and closed at 4:00.

As you can see there is no way of knowing we should close @ 3:30 without peaking ahead in terms of time. I suppose one way around this would be to simply make everything close at 3:30 instead of 3:45.
The breaking of a wave cannot explain the whole sea.
  • #727
  • Apr 14, 2010 4:41pm Apr 14, 2010 4:41pm
Quoting charley
Still a n00b in the forex world, and would appreciate pointers to good clean data sources, preferably tick.
This is the best data you can find, but it's very expensive - almost 11.000 euros for 3 years of EUR/USD as you can see in the picture.

http://www.olsendata.com/data_products/historical_data/

They also sell compressed data (candles) much cheaper.

BTW, they only lease the data (yes, you don't buy it) and only institutions can get it.

http://imgur.com/FQqJy.png
  • #728
  • Apr 14, 2010 4:47pm Apr 14, 2010 4:47pm
Quoting charley
That lead to waaaaaaaay more trades than necessary to give me confidence in the system :-)
It should be easy to compute the Sharpe ratio with so many trades. Could you share it with us?

Quote
I imagine that when I start digging into forex strategies, it will be more challenging to find systems that work over such a vast universe and timeframe.
Actually compared to equities, forex is quite small. There are something like 6 major pairs, 40 other important ones, and another 100 of exotics.
  • #729
  • Apr 14, 2010 4:58pm Apr 14, 2010 4:58pm
Quoting Craig
... there is no way of knowing we should close @ 3:30 without peaking ahead in terms of time. I suppose one way around this would be to simply make everything close at 3:30 instead of 3:45.
Yes, or use finer granularity bars (say 5-min) and have your strategy roll them up into whatever periodicity it uses for its logic and signals. For another strategy (still under development), I feed ticks to the Strategy class which it then rolls up into 5-min, which triggers the official 'barClosed' logic. But it uses the ticks to monitor intra-bar stops, profit targets, etc (if you want to keep your stops out of the market), using a secondary 'calendar' for that. Logic gets more complicated but gets the job done. Survey still out how well it works in practice...
  • #730
  • Apr 14, 2010 5:18pm Apr 14, 2010 5:18pm
I'd like to use tick or 1min data, but with the number of markets I'm covering I just don't have the computer power to run the GA in a reasonable amount of time, so I guess it will be a 3:30 close. But thank you for your suggestions.
The breaking of a wave cannot explain the whole sea.
  • #731
  • Apr 14, 2010 5:24pm Apr 14, 2010 5:24pm
Quoting Adal
It should be easy to compute the Sharpe ratio with so many trades. Could you share it with us?
I don't post performance stats in public. (I'm here to learn, and share knowledge, not brag). If you have a particular need to know, or question any aspect of my experience (I'm not saying that you are), please PM me and I may be able to share some more details (I'll reply when the powers that be let me use PM feature on this site ;-)
  • #732
  • Apr 15, 2010 3:38am Apr 15, 2010 3:38am
Quoting charley
I don't post performance stats in public. (I'm here to learn, and share knowledge, not brag). If you have a particular need to know, or question any aspect of my experience (I'm not saying that you are), please PM me and I may be able to share some more details (I'll reply when the powers that be let me use PM feature on this site ;-)
Are your strategies directional or non-directional (or both)?

Just interested, have you noticed any major differences in the data between last 2 years and 20 years before that?
  • #733
  • Apr 15, 2010 4:14am Apr 15, 2010 4:14am
Quoting charley
I don't post performance stats in public. (I'm here to learn, and share knowledge, not brag). If you have a particular need to know, or question any aspect of my experience (I'm not saying that you are), please PM me and I may be able to share some more details (I'll reply when the powers that be let me use PM feature on this site ;-)
I was just curious since you said that your strategies would have worked profitably in the last 80 years, and I wanted to see if you have a "low" Sharpe (<1.2) as I would expect from such a long performing period.
  • #734
  • Apr 15, 2010 11:46am Apr 15, 2010 11:46am
Quoting Craig
Thanks for the reply, this is exactly what I do (e.g. the strategy is unaware of if it is in back-test or live mode), the problem is let's say I want to trade up to 3:45 and I'm using 30m bars.

3:30: We get the bar which opened at 3:00 and closed at 3:30.
4:00: We get the bar which opened at 3:30 and closed at 4:00.

As you can see there is no way of knowing we should close @ 3:30 without peaking ahead in terms of time. I suppose one way around this would be to simply make everything close at 3:30 instead of 3:45.
Simulating real time in a backtest is a non-trivial problem...and the solution to this problem naturally depends a lot on your trade engine implementation.

The way we solve this is to have a "time stamp event" data message type which is sent directly into the generic data stream. If the engine runs in live mode, a timer sends a "time stamp event" data message at regular intervals using system clock time. If the engine runs in simulation mode the data streamer calculates how many "time stamp event" messages is needed to be sent between two regular data messages in order to properly simulate "real time".
  • #735
  • Apr 15, 2010 1:40pm Apr 15, 2010 1:40pm
Quoting BlackMage
The way we solve this is to have a "time stamp event" data message type which is sent directly into the generic data stream.
Sounds not unlike the approach I described here, biggest difference is that I feed (near) real time tick data into the strategy, instead of system clock time. That way, the strategy has access to nrt price data also, to use for stops, profit targets, etc.
  • #736
  • Apr 15, 2010 1:55pm Apr 15, 2010 1:55pm
Quoting mikkom
Are your strategies directional or non-directional (or both)?
Just interested, have you noticed any major differences in the data between last 2 years and 20 years before that?
The intraday equity strategy doesn't care what the overall market direction is, if that's what you mean, e.g., it has profitable Long trades whether we're in a bull market or bear market.

Yes, if you mine any stats about equity and bond markets (and the kind of behavior I study), you'll see that certain market attributes were way off the 'normal' charts in 2008-2009, although some similarities to the 1920s-1930s exist. The key is to design a strategy that thrives during these excesses. In the case of intraday equity systems that's very possible, e.g., higher intraday volatility just means bigger position profit as long as you're on the right side of the trade. On the other hand if your equity or bond system uses daily data, it may have seen some serious drawdowns due to unusual 2008-09 volatility, and you may have just needed some sort of filter to stay on the sidelines during volatility clusters.

This is another reason very long price history is useful. Although we can never predict future volatility, it really helped my bond system to see the market tank in the 30s, which is something the bond market then did not see at such a magnitude for the subsequent 75 or so years, until 2008-2009.
  • #737
  • Apr 15, 2010 2:01pm Apr 15, 2010 2:01pm
Quoting charley
I feed (near) real time tick data into the strategy, instead of system clock time. That way, the strategy has access to nrt price data also, to use for stops, profit targets, etc.
I do the same. The approach I described is mostly only usefull if you for some reason needs to simulate time with a higher granularity than the data you have access to or if you need to check some conditions at very specific time points.
  • #738
  • Apr 15, 2010 2:06pm Apr 15, 2010 2:06pm
But enough of my yappin' ... do you guys have any feedback/input on this ?
  • #739
  • Edited 8:58am Apr 16, 2010 2:17am | Edited 8:58am
Quoting charley
The intraday equity strategy doesn't care what the overall market direction is, if that's what you mean, e.g., it has profitable Long trades whether we're in a bull market or bear market.
I was refering to market direction neutral methods (pairs trading, some volatility strategies etc).

I have been thinking of moving (back) to equities myself too, fx market is a bit too correlated and there are very few instruments. As Craig said earlier, almost every more liquid pair is correlated to USD which can be a good or bad thing.

I'm running my genetic algo stuff on FX now but after I have completed my (major) framework modification I'm probably moving to equities and futures too.
  • #740
  • Apr 16, 2010 1:36pm Apr 16, 2010 1:36pm
Quoting mikkom
I was refering to market direction neutral methods (pairs trading, some volatility strategies etc).

I have been thinking of moving (back) to equities myself too, fx market is a bit too correlated and there are very few instruments. As Craig said earlier, almost every more liquid pair is correlated to USD which can be a good or bad thing.

I'm running my genetic algo stuff on FX now but after I have completed my (major) framework modification I'm probably moving to equities and futures too.
Mikkom,

Could you give us an update on your strategies that you described with your alpha factory, are they profitable.

I have done an interesting test with my strategy explorer (based on 8 months of tick data). I wrote a simple system (on 5 minute time frame) an made 30000 permutations of it. Some of the strategies did ok and some of them bommed. What was quite noticeable though was that in the end all the strategies profitFactor and profit/position declined dramatically. So much so that the cost of the transaction would make it not worth it. I am just wandering if this will happen for any system on the long term. I know "the stop working of system" is mentioned quite often and it looks it may well be true.
0 traders viewing now