Veloce algo trading

#1
First Post: Edited Jul 29, 2015 4:40am Apr 7, 2010 4:27pm | Edited Jul 29, 2015 4:40am

Adal
Joined Mar 2009 | Status: Trader | 770 Posts

#2
Apr 7, 2010 4:42pm Apr 7, 2010 4:42pm

Adal
Joined Mar 2009 | Status: Trader | 770 Posts

My FXCM and Dukascopy tick downloaders working (click for full size):

http://dl.dropbox.com/u/190212/ff/tick.png

The dbFX tick downloader is very similar to the FXCM one from above, since it's using the same API.

Now the GAIN Capital (forex.com) history data .zip downloader:

http://dl.dropbox.com/u/190212/ff/tick_gain.png

And the GAIN Capital .zip to tick convertor:

http://dl.dropbox.com/u/190212/ff/ti..._convertor.png

This concludes my tick handlers.

#3
Apr 7, 2010 5:07pm Apr 7, 2010 5:07pm

Adal
Joined Mar 2009 | Status: Trader | 770 Posts

In the last months I have downloaded tons of tick data. Over 2 billion ticks on 150 instruments (but some, like EUR/USD, are counted multiple times - once for each broker).

The tick data occupies 11 GB in compressed form. If I would convert that to normal .CSV files it would consume 187 GB.

I've decided to generate only M1 candlestick data from the ticks, just like MetaTrader 5. The other time frames will be dynamically generated when needed.

http://dl.dropbox.com/u/190212/ff/generate_m1.png

I prefer working in Python when possible. The tick downloaders are 100% Python. But computing the M1 candles for just one instrument, like in the picture above took 10 minutes. So I wrote a small C++/Python extension which implements the performance-critical candle generation code. With this, I can build the full M1 data for 3 years of EUR/USD in 20 seconds. That's 30 times faster

The compressed M1 candlestick data for all my ticks weighs in at 863 MB. Much easier to handle than the full tick data.

#4
Apr 7, 2010 5:07pm Apr 7, 2010 5:07pm

markmm
| Joined Feb 2009 | Status: Price Stalker | 1,197 Posts

Hi Adal looking forward to your thread, I used a php script to get all the tick and 1 min data from dukascopy, from this guy . Good luck with your trading.

Economists have forecast 9 out of the last 5 recessions

#5
Apr 7, 2010 5:19pm Apr 7, 2010 5:19pm

Adal
Joined Mar 2009 | Status: Trader | 770 Posts

Quoting markmm

Disliked

Hi Adal looking forward to your thread, I used a php script to get all the tick and 1 min data from dukascopy, from this guy . Good luck with your trading.

Ignored

Thanks!

I know that site. In fact if it wasn't for him, I wouldn't have found out that Dukascopy data is so good and so easy to download (the easiest to work with from my 4 tick sources).

#6
Apr 7, 2010 5:44pm Apr 7, 2010 5:44pm

spekkiefx
| Joined May 2009 | Status: Trader | 164 Posts

When testing your strategies are you going to use the tick data or the 1 min bars or both?

Just wandering.

#7
Apr 7, 2010 5:53pm Apr 7, 2010 5:53pm

Adal
Joined Mar 2009 | Status: Trader | 770 Posts

Quoting spekkiefx

Disliked

When testing your strategies are you going to use the tick data or the 1 min bars or both?

Just wandering.

Ignored

Since the M1 data is built from the tick data, I can use M1 for most purposes. I store for each candle the real O H L C values for both the bid and the asks.

So when I test I can truly work on real market prices. I have one more trick here: I added to each candle two more flags, which store what happened first: the low or the high. In this way I know the exact order in which the 4 points were hit.

Obviously, the lower the time frame I use and the smaller the profit target gets, the more important is working with real tick data. But I will limit myself to maximum M5 for start.

This is only when developing my system.

Before deploying the algo for real, I will obviously test it with full tick data.

#8
Apr 7, 2010 6:03pm Apr 7, 2010 6:03pm

spekkiefx
| Joined May 2009 | Status: Trader | 164 Posts

Quoting Adal

Disliked

Since the M1 data is built from the tick data, I can use M1 for most purposes. I store for each candle the real O H L C values for both the bid and the asks.

So when I test I can truly work on real market prices. I have one more trick here: I added to each candle two more flags, which store what happened first: the low or the high. In this way I know the exact order in which the 4 points were hit.

Obviously, the lower the time frame I use and the smaller the profit target gets, the more important is working with real tick data. But I will limit...

Ignored

There is no point to build OHLC for bid and ask, just use the mid. If you want to be as realistic as possible (real market prices) then use the tick data. Obviously this is rather resource hungry when testing but the only thing you don't take into account then is slippage of execution. This is as close as it gets.

#9
Apr 7, 2010 6:22pm Apr 7, 2010 6:22pm

Adal
Joined Mar 2009 | Status: Trader | 770 Posts

Quoting spekkiefx

Disliked

There is no point to build OHLC for bid and ask, just use the mid. If you want to be as realistic as possible (real market prices) then use the tick data. Obviously this is rather resource hungry when testing but the only thing you don't take into account then is slippage of execution. This is as close as it gets.

Ignored

I will use FXCM or dbFX as my broker. They both have variable spread. If I store only the mid point I wouldn't know at what price to buy/sell. Also, before news, the spreads widen dramatically. 30 seconds before NFP/FOMC the spreads go to 20 points. 30 seconds after the news release they come back down, but still 2-3 points wider than usual for another 1 minute. But I probably wouldn't want my system to trade around news anyway.

The first system I want to test, an H4 with average holding period of 1-3 days, has take profit targets of 100 points, and stop losses of 300 with very good winning rates. So I really don't care even for 10 point real price difference/slippage for now.

BTW, I'm on a little known broker at the moment, CMC Markets (very ugly FIFO GUI, but which I've grown to like), and in hundreds of EUR/USD trades I think I was slipped only 3 or 4 times. Unfortunately they don't have any API, and the spreads are not exactly top-notch (3 points on EUR/USD). Hopefully my future broker will be just like that (non slipping)