Systematic trading | Page 75 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Trade another day through Systematic Trading 203 replies

Nonparametric Association Measures in Systematic Trading 57 replies

Anyone use Systematic from Alpari? 1 reply

A Systematic Approach to Markets 68 replies

Systematic Trading vs. Discretionary Trading 14 replies

Attachments: Systematic trading
Exit Attachments

Systematic trading

  • #1,481
  • Feb 13, 2012 7:53am Feb 13, 2012 7:53am
Quoting mikkom
Hint: For speed use arrays of primitives anywhere where you can (no object arrays as they are basically pointers). That way cpu cache starts kicking in. For example my 1 min equity data looks like this:

Inserted Code
    public int barCount;
    public char[] open;
    public char[] close;
    public char[] high;
    public char[] low;
    public int[] volume;
Are you sure that's the code that you are using? Surely you are not keeping numbers in a Java char (which is strictly a Unicode character, unlike in C).
  • #1,482
  • Feb 15, 2012 6:15am Feb 15, 2012 6:15am
  •  rayzeel
  • | Joined Nov 2008 | Status: Automated/DiscretionaryTrader/Coder | 37 Posts
One thing... I've made 1048575 (2^20) combinations in 1h 15min. The csv files was 800kb each (43k lines). This is much faster then I thought ... Now I only need csv files from multiple strategies and markets, TF etc... I'm searching best profit/maxDD (this is Sterling ratio if I remember)... It will be easy now to implement another indicator...
  • #1,483
  • Feb 16, 2012 9:53am Feb 16, 2012 9:53am
Quoting Adal
Are you sure that's the code that you are using? Surely you are not keeping numbers in a Java char (which is strictly a Unicode character, unlike in C).
Yes I am. Java char is a unsigned 16-bit value, nothing more (it's the only unsigned 16 bit value by the way). It can be converted to int just by casting.

http://docs.oracle.com/javase/tutori...datatypes.html
  • #1,484
  • Feb 16, 2012 12:50pm Feb 16, 2012 12:50pm
Quoting mikkom
Yes I am. Java char is a unsigned 16-bit value, nothing more (it's the only unsigned 16 bit value by the way). It can be converted to int just by casting.

http://docs.oracle.com/javase/tutori...datatypes.html
How do you store in a 16 bit char these prices: 1.31052 (EUR/USD), 78.879 (USD/JPY), 103.381 (EUR/JPY), 503.89 (AAPL) or 1354.25 (S&P 500). If you use a fixed point schema, you'll need to also store the exponent somewhere.
  • #1,485
  • Feb 17, 2012 2:55am Feb 17, 2012 2:55am
Quoting Adal
How do you store in a 16 bit char these prices: 1.31052 (EUR/USD), 78.879 (USD/JPY), 103.381 (EUR/JPY), 503.89 (AAPL) or 1354.25 (S&P 500). If you use a fixed point schema, you'll need to also store the exponent somewhere.
I suspect by storing the difference in points from the previous price?
  • #1,486
  • Feb 17, 2012 12:12pm Feb 17, 2012 12:12pm
Quoting realize
I suspect by storing the difference in points from the previous price?
How do you seek then in the series? And that still doesn't account for how you store the first price.
  • #1,487
  • Feb 18, 2012 4:00am Feb 18, 2012 4:00am
I'm curious about the storing of data too.

For a while I've been studying to program in Java, and have hit a little bump when reading data from a CSV.

What I want to do, is read every column of the dataset into a new array, and do this in a generic way - preferably having the name that is in the header correspond to the name of the array. If I understand it correctly, this type of programming is called metaprogramming, which is not really supported in Java.

Right now I'm thinking of creating a class - perhaps with subclasses for containing an arrays of Dates and doubles. Then read the data, and with this data create new instances of these classes. These classes are then to be added to an ArrayList that should finally contain all data. I'm not sure if this would be good practice, and it isn't very efficient for sure, as I'll end up using arrays containing references. Does anyone here have any tips on reading timeseries data?

(the dataset I'm currently working with is from the ECB, and contains columns <date> <eurusd> <eurjpy> <eurgbp> ....)
The nail that sticks out gets hammered back in
  • #1,488
  • Feb 18, 2012 4:39am Feb 18, 2012 4:39am
Quoting Noloqy
(the dataset I'm currently working with is from the ECB, and contains columns <date> <eurusd> <eurjpy> <eurgbp> ....)
I just learned about the functionality of a map. This may do the trick, as it stores both keys and values.
The nail that sticks out gets hammered back in
  • #1,489
  • Feb 23, 2012 1:17am Feb 23, 2012 1:17am
  •  rayzeel
  • | Joined Nov 2008 | Status: Automated/DiscretionaryTrader/Coder | 37 Posts
I've just started to build my own tester. I've loaded the data to primitives arrays and one question I have. Do I need to use data and time. You Mikkom wrote that you are just counting bars. Is that all ?

I'll post some info about the results of my work when I finished
  • #1,490
  • Edited 4:58am Feb 24, 2012 3:11am | Edited 4:58am
  •  rayzeel
  • | Joined Nov 2008 | Status: Automated/DiscretionaryTrader/Coder | 37 Posts
I,ve just run my first test on 1m bars. Simple buy if orderstotal=0 and close after tp or sl. Just to check if all working good. One more question here I have if someone can answer. When tp and sl are in the same candle Im taking sl (same as mikkom) but is it will be correct if all my positions long will be opening on high of the candle +spread and all long positions will close on the low val of candle (opposite for short). It is the most negative scenerio I can assume and I wonder if it will be correct to test like this... In theory if the strategy will pass test well I can expect that strategy is real robust... But maybe I'm wrong...

Example of closing long:

EDIT: In the test tp = 100, sl=50 and the results from logs:

Order opened: 0,BUY,Fri Jan 06 18:37:00 CET 2012,1.2721,1.2669,1.2819,0.1
Order closed: -0.0054
Order opened: 1,BUY,Sun Jan 08 23:23:00 CET 2012,1.2671,1.2619,1.2769,0.1
Order closed: 0.0094
Order opened: 2,BUY,Mon Jan 09 08:37:00 CET 2012,1.2773,1.2721,1.2871,0.1
Order closed: -0.0053
Order opened: 3,BUY,Mon Jan 09 16:17:00 CET 2012,1.2726,1.2674,1.2824,0.1
Balance: -0.0013 Equity: 0.0

Loses are few pips bigger and profits are smaller.
  • #1,491
  • Mar 12, 2012 12:26am Mar 12, 2012 12:26am
Quoting Noloqy
I just learned about the functionality of a map. This may do the trick, as it stores both keys and values.
For what it's worth, I also use maps to store data in time order, the key is the time and the value is a bar object. I've found that correct and concise code is much more important than optimizing code to be able to crunch millions of combinations. i7s are such studs that I'd rather be churning out good trading ideas than optimizing cache hits. The best optimizations are the ones that you do with your brain.
The breaking of a wave cannot explain the whole sea.
  • #1,492
  • Mar 12, 2012 12:37am Mar 12, 2012 12:37am
Quoting rayzeel
When tp and sl are in the same candle Im taking sl (same as mikkom) but is it will be correct if all my positions long will be opening on high of the candle +spread and all long positions will close on the low val of candle (opposite for short).
IMO, if your tp & sl are within the bounds of a 1 min bar then you're trying to test something that exceeds the resolution of your data, trying to optimize on stuff like that is a recipe for disaster. It also means you're probably not really setting realistic stops, instead of setting an arbitrary tp you may want to consider coming up with a exit condition instead. It's easy to define profit exit points for RTM strats, trend following strats don't tend to have take profits and the stops will be wide.
The breaking of a wave cannot explain the whole sea.
  • #1,493
  • Mar 13, 2012 3:09am Mar 13, 2012 3:09am
  •  rayzeel
  • | Joined Nov 2008 | Status: Automated/DiscretionaryTrader/Coder | 37 Posts
Thanks Craig for the answer. Ill be really careful with this approach and also prepare my tester for easy to setup changes. Im near to end first part of my work and I would like to compare results between my tester and MT4 tester. Maybe it will fail but I have to see it Ill tell how it ends.
  • #1,494
  • Mar 14, 2012 7:07am Mar 14, 2012 7:07am
Quoting Adal
How do you store in a 16 bit char these prices: 1.31052 (EUR/USD), 78.879 (USD/JPY), 103.381 (EUR/JPY), 503.89 (AAPL) or 1354.25 (S&P 500). If you use a fixed point schema, you'll need to also store the exponent somewhere.
Yes, I store the exponent to the equity type.
  • #1,495
  • Mar 14, 2012 7:09am Mar 14, 2012 7:09am
Quoting Craig
For what it's worth, I also use maps to store data in time order, the key is the time and the value is a bar object. I've found that correct and concise code is much more important than optimizing code to be able to crunch millions of combinations. i7s are such studs that I'd rather be churning out good trading ideas than optimizing cache hits. The best optimizations are the ones that you do with your brain.
If you are not aiming to speed then that is totally okay. If you want speed some day, benchmark the array version and you'll be surprised how big the difference is.
  • #1,496
  • Mar 14, 2012 11:17am Mar 14, 2012 11:17am
Quoting mikkom
Yes, I store the exponent to the equity type.
And what do you do when there is a split? Create a new equity type?

This is funny, how you keep on tying to dig yourself out of a hole instead of admitting that you made a mistake and that structure is not the real one you are using.

Anyway, you also have a "short" type in Java in case you wanted to store integer values instead of torturing them into Unicode chars. Or maybe you benchmarked both types and found out that the Unicode char is faster for storing values.
  • #1,497
  • Edited 5:40pm Mar 14, 2012 5:05pm | Edited 5:40pm
Quoting Adal
And what do you do when there is a split? Create a new equity type?

This is funny, how you keep on tying to dig yourself out of a hole instead of admitting that you made a mistake and that structure is not the real one you are using.

Anyway, you also have a "short" type in Java in case you wanted to store integer values instead of torturing them into Unicode chars. Or maybe you benchmarked both types and found out that the Unicode char is faster for storing values.
I'm not sure if I understood correctly, by exponent I meant the amount of decimals and no, I can't store numbers with larger amounts than 65535. That has been enough for most meaning >99% of stocks.

Short is signed and you don't need negative values for prices. You basically waste one bit if you use shorts and if we are as low as 15 bits, there will be much more prices that don't fit to 15 bits than 16 bits. I have no idea what you mean by torturing, char is just a value with 16 bits. You can convert char to int with

int value = (int) charValue;

and vice versa just as easily. That has nothing to do with unicode, short and char are basically equal in bit size but char is unsigned - java does not have other 16 bit types that are unsigned. Only place where there will be any kind of problems is when you print a char - you have to convert it to int (or other numeric type) before printing or it will print a character.

About splits, my database is split adjusted.

I actually couldn't care less if someone on internet believes me or not but check your facts the clause below is false - I just want to point that out so that nobody else gets confused. If you want, we can argue about this but please post the facts. Why do you think java char is different from c (except that in c char is usually 8 bits) and what *exactly* do you think is wrong with my approach?

Quote
Surely you are not keeping numbers in a Java char (which is strictly a Unicode character, unlike in C).
I usually don't even need to decimalize the prices as I know the purchase price and sell price and the (sell price / buy price) * trade size is what matters, not how many decimals there are.
  • #1,498
  • Mar 14, 2012 6:49pm Mar 14, 2012 6:49pm
I assumed that you wanted to be able to store any stock, no matter the price. If you are ok with being able to handle only 99% of them, fine. But you are missing some important stocks, like AIG, which went from 1500 to 26. With two decimal digits, 1500 doesn't fit into a Java char (150000).

BTW, this scheme kind of works for stocks, since most of them are between 1-654 and with two decimal digits. But it wouldn't work so well for other assets, like currencies (some of which are of the XX.XXXXX format or XXX.XXX), unless you throw away some data (like the last digit).

Since you asked, I believe that another flaw of your approach is that you don't store for each candle what was first - the high or the low. You could just add a new boolean array for that. Of course, if you do that you'll need to build the M1 data yourself, since most data providers don't deliver this field.

That's what I did before moving to working mainly with tick-data. For higher "time-frames" I use a different data reduction scheme, where I just decimate the tick-data stream, while preserving the high and the low (in the correct order). This means that the same code can work with both tick data or "M1" data, since they are stored in the same way. As a side benefit, I also know the exact time the high and the low were hit.

Inserted Code
// classic way
bar {
  time: 1
  O: 1.4
  H: 1.5
  L: 1.3
  C: 1.4
}

// my way
point { time: 1  value: 1.4 }
point { time: 2  value: 1.5 }
point { time: 6  value: 1.3 }
point { time: 9  value: 1.4 }
  • #1,499
  • Mar 15, 2012 3:06am Mar 15, 2012 3:06am
Yes I'm ok with it, I get so low memory usage and so much speed that I can exclude some stocks easily. I try to find strategies that work with hundreds of stocks so eliminating a few doesn't matter.

Quoting Adal
Since you asked, I believe that another flaw of your approach is that you don't store for each candle what was first - the high or the low. You could just add a new boolean array for that. Of course, if you do that you'll need to build the M1 data yourself, since most data providers don't deliver this field.
I use "the worst possible" approach so I don't care about which one hit first, if "sl" hit on the same bar as open, I just assume it was a loss. On fast moving markets this eliminates some strategies but it's good enough and will be at least as bad as real market.

Just something to think about adding a boolean field in java:

I have 100-500 equities with ~1 million data points each. As java boolean is not really one bit in size (this depends on vm implementation but it can be as high as integer (!)), the consequences to the memory usage will be huge. Not worth the extra precision IMHO.
  • #1,500
  • Mar 15, 2012 3:16am Mar 15, 2012 3:16am
By the way currently I'm experimenting with a new market, I don't want to post anything yet but it's worth mentioning that on this market I found couple of strategies by hand that give solid results on 20 years of data and this market

a) Has no trends
b) Doesn't change basically at all
c) Almost all other participants are overall losers
d) There are almost no big fishes
e) Basically almost all data relevant to price is available

Oh and did I mention that the profits are not taxed? (This probably reveals the nature of the market I'm talking about).

Sounds too good to be true? That's what I thought
0 traders viewing now