Trading Prerequisites - Doing it Right | Page 2 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Right Wrong Right Wrong Right Wrong 13 replies

Why Is Price Doing What It's Doing? 65 replies

Converting commissions to pips: I am doing this right? 14 replies

4105 No order selected? Em i doing this right? 1 reply

Newb at the right place and the right time 1 reply

Attachments: Trading Prerequisites - Doing it Right
Exit Attachments

Trading Prerequisites - Doing it Right

  • #21
  • Jan 5, 2011 1:48pm Jan 5, 2011 1:48pm
Quoting Sauron
Adal, Scotty uses Windows. And for programming on Windows there's nothing better than VC++ or VC#. Sooner or later he will want to make a user interface and I don't think you can come even remotely close with R or Python to what Visual Studio can offer.
Even if R has some predefined functions like EMA you still need to write your own functions and algorithms. And why would someone learn two languages when one (C++ or C#) is enough?
WPF (the new Microsoft GUI framework) is much easier to use from IronPython than from C# (my opinion). This GUI (the Veloce window) is written by me. 95% it's (Iron)Python. You can see that it has integrated R and Python interpreters. The only C# code is for drawing the actual chart lines (for speed reasons). Do you think it's good enough?

http://dl.dropbox.com/u/190212/ff/veloce_r.png

How about this audio editor I wrote 5 years ago, 80% in Python (using MFC). Doesn't look like a Python application, right?

http://nuotex.com/audiopencil/AudioPencil.png

C++ or C# can be enough, but they are not productive. What I can do in one day in Python I need one week to do in C++. And it's not because I don't know C++. I learned C++ years before Python. But I don't use it anymore if I can avoid it, since my time is more precious than CPU time.
  • #22
  • Jan 5, 2011 1:49pm Jan 5, 2011 1:49pm
Quoting Sauron
Maybe it's semantics but SQL is a language designed to querying relational databases. SQL Server or MySQL are applications whose goal is to manage relational databases and allow access to their data via SQL (Structured Query Language). So SQL is useless per se.
Yeah yeah yeah, I should have been clearer, that note was meant for Scotty just in case he wasn't familiar with terminology. I'm aware of the distinction, believe it or not.
The breaking of a wave cannot explain the whole sea.
  • #23
  • Jan 5, 2011 1:51pm Jan 5, 2011 1:51pm
Adal: Are you using RCpp?
The breaking of a wave cannot explain the whole sea.
  • #24
  • Jan 5, 2011 2:10pm Jan 5, 2011 2:10pm
Quoting Craig
Adal: Are you using RCpp?
I wish. No, R code (thus RCpp) doesn't compile under Visual Studio - it has too many GCC dependencies, so it's built on Windows using MinGW.

I load the R.dll dynamically with LoadLibrary (think dlopen) and get the pointer to the required functions with GetProcAddress (think dlsym). I only needed to access around 100 functions from the R C interface so this was not a major pain. I looked into the R source code for inspiration on how to implement an evaluation loop.

I know that I'm not doing C++ side reference counting for R objects 100% correctly at the moment, but it works for now with no crashes (but I think I leak some memory).

In an earlier iteration I communicated with a Rterm process through stdin/stdout/stderr pipes, but I encountered a major block (related to receiving syntax errors) and had to move to the current solution.
  • #25
  • Jan 5, 2011 2:42pm Jan 5, 2011 2:42pm
Quoting Adal
I wish. No, R code (thus RCpp) doesn't compile under Visual Studio - it has too many GCC dependencies, so it's built on Windows using MinGW.
Oh, that is a bummer, I didn't realize that.
That was my next project, you might have saved me a lot of trouble.
The breaking of a wave cannot explain the whole sea.
  • #26
  • Jan 5, 2011 4:25pm Jan 5, 2011 4:25pm
Thanks for the discussion, gentlemen. Most of it is whizzing over the top of my head at close to the speed of light, but when I know more I'll understand what in the heck you guys are talking about.

I've been studying and experimenting with R, all day today. I have to say I'm impressed with all the quick and easy stat solutions it has to offer. I'm so used to excel that I'm accidentally returning plenty of errors because of incorrect syntax, but I appreciate the fact that all I get is an silent error instead of MS's horrific sound that's produced when I mess up in excel.

Instead of having a huge collection or long list of price data cascading down my command prompt screen, can I draw off of a file with price data neatly contained in it? How do you guys input price data into R?

Also, will R work like a spreadsheet and allow me to test conditional relationships between what used to be cells in excel? Sorry for all the elementary questions.
  • #27
  • Jan 5, 2011 4:47pm Jan 5, 2011 4:47pm
Quoting Scotty B
Instead of having a huge collection or long list of price data cascading down my command prompt screen, can I draw off of a file with price data neatly contained in it? How do you guys input price data into R?
I pull data out of a db using the RODBC package.

Quoting Scotty B
Also, will R work like a spreadsheet and allow me to test conditional relationships between what used to be cells in excel? Sorry for all the elementary questions.
It does, but it's probably best to work though a tutorial on the basics of vectors and matrices. I know I tried to jump in and got all confused, it wasn't until I went back and wired the basics that stuff started to fall into place. I you want to post some R code, I'm sure others and myself can take a look.
The breaking of a wave cannot explain the whole sea.
  • #28
  • Edited 5:49pm Jan 5, 2011 5:05pm | Edited 5:49pm
I read data from my own tick/candle data store in Python and then convert the data to a R vector and continue processing it in R. The processed result (for example an EMA(50) from the data) I can convert back to a Python vector (actually .NET to be precise) which I can either save back to my data store or use it for example as input to a strategy.

Here is some info on how to read data from files in R:
http://cran.r-project.org/doc/manual...ata-from-files
http://cran.r-project.org/doc/manuals/R-data.html

You might also want to print this PDF and go through it: http://cran.r-project.org/doc/manuals/R-intro.pdf I found it very useful.

It did take a few hours of grinding through for me to get a basic understanding of R. It's not very beginner friendly. I also got confused like Craig the first time I tried to use it, and I gave up. A few months later I tried again.

I never used Excel, so I don't know how you would translate the Excel workflow into R. But check this out: http://chartsgraphs.wordpress.com/20...miliar-with-r/

EDIT: Here's a simple data processing session using both Python and R in my app:

http://dl.dropbox.com/u/190212/ff/veloce_r_session.png
  • #29
  • Jan 6, 2011 12:54am Jan 6, 2011 12:54am
Scotty if you need to design & test mechanical trading systems I'd recommend Amibroker. It is cheap and really does the job.
Trading = a mirror to your human flaws. Fix them or be fixed.
  • #30
  • Jan 6, 2011 2:41am Jan 6, 2011 2:41am
Adal, what can I say, it is a nice interface and it's great that you can do it combining so many languages. I personally like to stick to VC++ and C#, I developed my libraries to read, analyze and display data fast and now hardly I need something else.
Anyway, I'm intrigued, what does it take so long to do in C++ and very fast in Python, maybe it's the time to take a deeper look aty Python, can you give me an example?
  • #31
  • Jan 6, 2011 5:59am Jan 6, 2011 5:59am
Quoting Sauron
Anyway, I'm intrigued, what does it take so long to do in C++ and very fast in Python, maybe it's the time to take a deeper look aty Python, can you give me an example?
It's hard to explain in a few lines. I'm sure you've already experienced something similar. I bet that you are more productive in C#, am I right? The .NET Framework has a lot of functionality with a common interface, which is not true in C++. For example, in C# you can connect to a website and read it's content much easier than in C++. You could download a C++ library which makes this job possible, but the .NET version is already there and you'll be able to inter-operate with it in an easier way. For example, in C++ you might need to do UTF8 -> wchar_t conversions because the library that you downloaded uses UTF8, but you work with wchar_t internally. What I'm trying to say is that Python, like C# has a lot of stuff already builtin in it's library. So this is one aspect of the productivity - you can do a lot of stuff with the builtin library, without having to download additional libraries.

But this is also true for C#. So what else does Python bring to the table? I'll assume that you agree that C# is more productive than C++, so I'll only compare Python and C# from a productivity view from now on.

The language is much less verbose, and doesn't need as much hand-holding as C#. Less casting, less custom datatypes. BTW, C# it's moving in the right direction with it's new features, like the "dynamic" keyword. No doubt because of the pressure coming from languages like Python.

It's fun to work with Python. Let's say you need to store the digit count and the minimum order size for the currencies offered by you broker. You could do something like this in Python:

Inserted Code
CURRENCY = {
    "EUR/USD": (4, 10000),
    "USD/JPY": (2, 10000),
}
Then, you add gold to it, but you learn that gold, unlike normal pairs, has a minimum stop distance at your broker. So you add a third field, just for gold:
Inserted Code
CURRENCY = {
    "EUR/USD": (4, 10000),
    "USD/JPY": (2, 10000),
    "XAU/USD": (2, 1000, 5),
}
In your code, it will be trivial to get the third member only if it's there:
Inserted Code
pair = CURRENCY["XAU/USD"]
minimum_distance = pair[2] if len(pair) == 3 else 0
There are many ways to do this in C++/C#, but none is as simple as this.

You can see that there are no "typedef std:: pair<int, int> CurrencyPair" definitions around. You don't need boilerplate like that in Python.

Now this may seem very messy and low quality code to you, if you have a static language programming background. It felt the same way to me at first. I though all the time that there will be horrible bugs cause by such code, by the lack of static type checking. But this it's not true. Everything will be fine, the world won't end because you have different data types in your hashmap (2-field versus 3-field tuples). And you'll appreciate not having to write tons of typedefs and other boilerplate over and over again.

Another thing that Python is great at is string processing: reading values from (badly formatted) CSV file it's a breeze. After working with Python, I implemented my own little C++ library with Python like string methods (split, join, startswith, endswith).

Like I said, it's hard to describe in a few lines the power of Python. So I'll talk about some results: a script which can connect to the Oanda tick server (used by theirs Java apps) with sockets and download the raw ticks. 121 lines. Yes, under 200 lines. This can just connect to the server, get the data and dump it to a daily file. With auto-connect on failure, twitter alerts and some statistics displayed on screen. I estimate that an equivalent C# application will be over 500 lines.

Then, the ticks are stored in some sort of delta-compressed format. The Python decoder, which works at bit level (yes, this is possible in Python) has only 153 bytes. This would be another 500 lines of C# code.

Some code from the decoder (I've obfuscated a little bit):
Inserted Code
UINT32            = struct.Struct( "I")
UINT32_BIG_ENDIAN = struct.Struct(">I")
UINT64_BIG_ENDIAN = struct.Struct(">Q")

# ...

INSTRUMENT = {}

symbols = open("symbols.csv", "rb").read()
symbols = symbols.strip()
lines = symbols.split("n")
for line in lines[1:]:
    index, instrument, rest = line.split(",", 2)
    index = int(index)
    INSTRUMENT[index] = instrument

# ...

xxx_a =  value >> 21
xxx_b = (value >> 10) & ((1 << 11) - 1)
xxx_c =  value        & ((1 << 10) - 1)

xxx_b |= last_server_xxx_b & 0b11111111111111111111100000000000  # last 11 bits unset
delta_xxx_b = xxx_b - last_server_xxx_b
if abs(delta_xxx_b) > 1023:
    if delta_xxx_b < 0:
        xxx_b += 2048
    else:
        xxx_b -= 2048

# ...

scale  = 10 ** (exponent_base - digit_count)
bid    =  bid_mantissa                   * scale
spread = (spread_mantissa + spread_base) * scale

# ...

if marker in (9, 17, 25, 33, 41, 49):
    # ...
elif marker == 98:
    # ...
else:
    raise RuntimeError("unsupported marker (%s)" % marker)

# ...

value_size = ord(data[0])
assert 5 <= value_size <= 8
value_data = data[1:1 + value_size]
padding = "\x00" * (8 - value_size)
value_data = padding + value_data
value, = UINT64_BIG_ENDIAN.unpack(value_data)

marker = ord(data[1 + value_size])
assert marker == 109

bid, spread = decode_bid_and_spread(value, value_size, *BID_SPREAD_ENCODING_B[value_size])
ask = bid + spread
What you should observe is that almost all the code is about doing something, not about hand-holding the compiler with type declarations and casts.

And remember that for these two Python applications you have exactly two files on disk. For C# you would also need project folders, project files, a solution, a compile cycle, resulting binary files. It takes me a few seconds to open a small text editor on one of this files and fix an error. It would take much longer to fire up Visual Studio and do the same for the C# version.

I have around 20 scripts like these which do various jobs, like connecting to brokers to retrieve tick data, parsing the tick data and converting it into a common format, building candles from the tick data, downloading interesting web content (like Oanda positioning info). That's 20 Visual Studio projects that I don't need to manage.

Anyway, the simplest thing is to just try it. Parse a CSV tick file in Python and build minute candles from it and compare the resulting code with a C#/C++ version. There is a small problem here: you'll just write C++ code with a Python syntax, you won't use the usual Python idioms. Happened to me at first. It took me around 6 months to stop writing C++ like code in Python and start thinking the Python way.
  • #32
  • Jan 6, 2011 8:02am Jan 6, 2011 8:02am
You certainly need a lot of stuff and you're probably faster with Python. As my approach doesn't use smaller TF's than 5 minutes and their exactly boundaries are not important I can still live with my VC++ libraries and interfaces but in the future I'll have to find time for Python.
  • #33
  • Jan 6, 2011 11:42am Jan 6, 2011 11:42am
Quoting Adal
Another thing that Python is great at is string processing: reading values from (badly formatted) CSV file it's a breeze. After working with Python, I implemented my own little C++ library with Python like string methods (split, join, startswith, endswith).
All that C++ string stuff is already here...
http://www.boost.org/doc/libs/1_45_0...ring_algo.html

Not wanting to start some kind of shit-fight here as Python has its place, but I when I recommended C# I was trying to come at it from a rank beginner who wants to learn enough to automate POV, C# allows somebody to drop a button on a form and then write code to handle the event. I'm not sure there is any equivalent in Python, also the debugger is an order of magnitude easier to use. It's all well and good for us to throw around wizz-bang technology and what we have done with it, but this probably isn't helping Scotty decide what to do.
The breaking of a wave cannot explain the whole sea.
  • #34
  • Jan 6, 2011 12:38pm Jan 6, 2011 12:38pm
Quoting Craig
All that C++ string stuff is already here...
http://www.boost.org/doc/libs/1_45_0...ring_algo.html
More or less, couldn't help notice that startswith and endswith which I mentioned are missing

Quote
Not wanting to start some kind of shit-fight here as Python has its place, but I when I recommended C# I was trying to come at it from a rank beginner who wants to learn enough to automate POV, C# allows somebody to drop a button on a form and then write code to handle the event. I'm not sure there is any equivalent in Python, also the debugger is an order of magnitude easier to use. It's all well and good for us to throw around wizz-bang technology and what we have done with it, but this probably isn't helping Scotty decide what to do.
I understand your point and agree partially. The Microsoft eco-system is wonderfully integrated. And it's true, the Python debuggers are not comparable to the C++/C# ones. But they do exist. For example you can add a plugin to Visual Studio which adds support for the (Iron)Python language, including IntelliSense and a debugger - http://ironpython.net/tools/

There are also good ways of making a simple GUI for Python. EasyGUI comes in mind. But you're right, it's not a point&click affair like in Visual Studio.

I also remembered that MIT recently switched it's undergraduate programming language that it teaches from Scheme to Python.

Anyway, enough with my Python advocacy
  • #35
  • Jan 6, 2011 12:48pm Jan 6, 2011 12:48pm
Quoting Adal
More or less, couldn't help notice that startswith and endswith which I mentioned are missing
You didn't look hard enough, look at the 'predicate' examples and header
Inserted Code
bool is_executable( string& filename )
    {
        return 
            iends_with(filename, ".exe") ||
            iends_with(filename, ".com");
    }

In the end it's each to his own I guess in terms of tools, all the things suggested so far have validity. A lot of guys I know are hardcore unix command line guys, they think I'm a total pussy for using a GUI
The breaking of a wave cannot explain the whole sea.
  • #36
  • Jan 6, 2011 1:08pm Jan 6, 2011 1:08pm
Quote
You didn't look hard enough, look at the 'predicate' examples and heade
Busted

Quoting Craig
A lot of guys I know are hardcore unix command line guys, they think I'm a total pussy for using a GUI
Yeah, I know the kind. Vim lovers. I love making fun of them when I catch them using GUIs like GitX instead of the lovely git command line.
  • #37
  • Jan 9, 2011 5:23pm Jan 9, 2011 5:23pm
Scotty, here's a full tutorial series by a former Excel user for those wanting to learn R: http://chartsgraphs.wordpress.com/20...-move-up-to-r/

I think I'll watch it myself.
  • #38
  • Jan 9, 2011 6:05pm Jan 9, 2011 6:05pm
Thanks, Adal.

Looks interesting. One of the ironies in learning R is that I've gotten so far away from charts in the last few months, while R is heralded for the powerful graphics. I've continued studying R, and have a long way to go, but so far It doesn't compare to excel for testing purposes.

For instance, in Excel when I'm testing for a special relationship I just have excel return 1s or 0s and I then can simply do a sum and have the stats I'm looking for fairly quickly. I haven't found a quick way to carry this sort of thing out with R. I'm sure it can be done, but Craig is right when he says the syntax is fairly difficult to learn.

One thing I've seen done, as with you, Adal, is that traders will code a custom dashboard--charts, graphs, trade reports, ect, that return all the pertinent trade information, but they are no longer using their eyes i.e. manual trading anymore. The strategy is all automated and the dashboard conveniently and quickly informs, I see R being invaluable in creating this visual interface.

Thanks again for the link.

Jim
  • #39
  • Jan 9, 2011 6:20pm Jan 9, 2011 6:20pm
Quoting Scotty B
For instance, in Excel when I'm testing for a special relationship I just have excel return 1s or 0s and I then can simply do a sum and have the stats I'm looking for fairly quickly. I haven't found a quick way to carry this sort of thing out with R. I'm sure it can be done, but Craig is right when he says the syntax is fairly difficult to learn.
Do you mean like 'count the number of times high < low, given the prev. bar was up', or something like that?
The breaking of a wave cannot explain the whole sea.
  • #40
  • Jan 9, 2011 7:42pm Jan 9, 2011 7:42pm
Quoting Craig
Do you mean like 'count the number of times high < low, given the prev. bar was up', or something like that?
Yes.

For example, =if(and(today's high> yesterdays high, OR(tomorrows high is> than today's high, the following day's high is >today's high, ect...)), 1,"")

I can test this sort of thing several days into the future to see what I think of as "latent responses" in the data. If you have a signal for example that won't be effective on average for two to three weeks, it's hard to see that manually without the help of a program. This boils down to a momentum style study, or one that identifies when trends continue in their original direction. This again is just an example of something complicated that excel does easily. Many times there is no pattern that leaves the 50/50 range, but as I've mentioned, some happen >90% of the time.

Because all the data points have specific names or identifiers (cells) in excel, one can carry out fairly complex tests easily. I know that in R when you have a data series you have a "vector," where each data point has a number assigned to it, but I haven't figured out how to ask R to run a test down the whole vector, like the auto fill feature in excel. I'm sure R can do the same thing, I just haven't learned that much yet.

I'm still figuring out how to get R to read a csv. The "read.csv" feature is not working for me. I downloaded tinn-R, and have been having a little more luck in that environment, but it's a steep learning curve for a newbie like me. The only programming experience I have is html, which maybe shouldn't even be compared to programming.

I have a heck of a lot to learn still, but I'll get there eventually.
0 traders viewing now