Trading Prerequisites - Doing it Right

Hello FF community,


As I press ahead on my journey, as I continue to learn, I am continually humbled in that I realize how much there is that I don’t know. I’ve studied order flow analysis (OFA) for the last year and a half and have come to the following conclusion: I need to become a programmer. Even if I want to trade statistical occurrences within price data i.e. chart patterns, I need to have a robot trading for me. I can put in hundreds of hours trading manually, making the inevitable mistakes and burning my retinas or I can put in a fraction of the time and put a 100% manual strategy into code.

OFA for example, when done “right,” (I define “right” as maximum efficiency) is done via a robot. I am still pursuing this end, and plan on eventually trading FX futures on the CME. You can’t trade order flow with no information available about orders. Enough about OFA for now…

One other huge barricade I’ve run into is this: I need to learn statistics and probability theory. Haha. Many of you are laughing now, well, I’ll laugh with you. For those of you who think you can trade without being proficient in statistics while profitable at the same time are either liars or criminals—Google “Martha Stewart inside trading.” You just can’t find success in trading without knowing what the odds are. Most novice traders are trading patterns that lose on average. You can’t look back and forth on a chart, see a few “winners” and then assume you have a winning strategy. Any system that requires trading regularly needs to be backed up by solid statistical evidence.

I’ve been spending the majority of my time lately in statistics textbooks and in Microsoft Excel. It’s amazing what can hide right under your nose in everyday price data. For learning’s sake, I’ve been sifting through daily EU data and have found several specific occurrences that happen greater than 90% of the time. With my eyes beginning to open, I’m realizing that learning to trade the right way is spilling over into other aspects of my life. Take all the math and programming for example-- that’s something you can carry with you and use in many other areas of life. My point in all of this is mainly for beginners and those who have some trading experience but are stuck adrift in the mighty trading waters without any hope of success:

If you are not profitable, you need to stop throwing money at the hungry monster. If you insist on throwing your money still, throw it at some books. Go down to your local college book store, or better yet, abebooks.com (cheap textbooks, I have no association with them), and get your hands on some used statistics textbooks. . Spend time with your new books like you used to spend time playing around on price charts. Familiarize yourself with all the concepts within. If you want to mess around with trading, play with a demo account. One good thing about an MT4 account is the access to free .csv data for you to apply your new knowledge on. If you will do this you might start to enjoy the prospect of trading again.

To prevent this thread from going in several different directions, I’d like to keep the topic this time focused on advice from those who have experience with the prerequisites of trading, and the resources and links to those resources. I’m talking about the mathematics that you feel are essential, the programming resources you feel are important, the best types of programming for trading, and anything else I’m obviously missing.



Here are some online resources I have found helpful for those who might not have the means to buy books:


Statistics:
http://www.statsoft.com/textbook/basic-statistics/

Thanks and good luck,
Jim

Excellent post, the problem with programming and trading is that building ATS software is a giant leap from build little test and learning programs, ATS software has to be robust and scalable, I would recommend this book to any programmer.

http://www.amazon.com/Design-Pattern...3735406&sr=8-1

As for statistics, I would also recommend learning R, this is rapidly becoming the tool de Jour. Much more powerful than Excel.

http://www.r-project.org/

Also in the maths line, I have found this book useful.

http://www.amazon.com/Machine-Learni...3736185&sr=8-1

For trading, the best book I have read is

http://www.amazon.com/Option-Trading...735580&sr=8-12

this is not directly related to forex, but has the best discussion on general aspects of trading that I have seen. If you goto this book looking for systems, you're going to be disappointed, but it does contain the elements of how professional traders actually go about their business.

I also keep an eye one this

http://arxiv.org/list/q-fin/recent

On 'order flow', I've recently been discussing this subject with a professional prop trader, he really opened my eyes to some of the shenanigans that happen on a daily basis, I'm not sure that this is the best first target for any trader as it requires a lot knowledge of market structure. But here is an example which may give you some ideas.

http://arxiv.org/abs/1003.2981

Not Trading related, but I consider an excellent book on gaming mathematics and probabilities.

Regards,

Nader

Craig,

Thanks so much for the suggestions you gave. You were one of the top three traders I was hoping to hear from. I'm looking at downloading the R software but am a bit confused which version I need running Windows 7 64b. I'll figure it out though.

Through time I will go through all of your suggestions. I really appreciate it.

Nafara,

Thank you sir for your suggestion, I follow through on it also. Let me know if you think of anything else.

No problems, with R, the syntax is a little hairy to begin but perseverance will pay. There are plenty of basic tutorials and references available on-line.

I agree with you Jim. I took a course in statistics 2 years ago, strictly for trading purpose and it turned out to be a great investment even though it cost more than 1000$.

I came to the same conclusion. I started to learn java but not sure it is the right language for ATS programming in the future.

I would be wary, given the recent shenanigans with Oracle.

Nice post.

I specifically agree with your comment about 'spill-over in to other areas of life' and came to the same conclusion for my self a while ago. I'm one of the people who came to forex factory looking for the easy money, the money I didn't need to be educated to get my hands on. I came to realise that the same things are that are necessary to succeed here are the same ones I need in the rest of my life. A profitable trader needs to have skills such as critical thinking, be numerate to a good standard and have a good knowledge of real world factors that affect currency prices... just like any professional needs to be able to make good decisions and have an excellent knowledge of the area in which they draw their livelihood.

I've been topping up my Maths and stats for a while now too, as well as studying economics so that I can properly understand the larger driving forces behind the markets, and the way that our financial systems and economies at large work. I have to say that, regardless of whether I ever make a living trading, being able to finally somewhat understand how the system works gives me great satisfaction and the world now seems like a much smaller place.

I'm going back to college, again, this time to take Maths and Economics. My eyes are set on studying the Bachelor of Economic Science degree at Manchester.

What I'd like anyone reading this post to take from it is this: Trading is hard work, just like being successful at anything worthwhile is. In the markets, as in society, it is the hardest working, most skilled and realistic people who succeed. We are all in competition, so don't underestimate how hard the other guys are willing to work. Always focus on improving your self.

Here's a (probably slightly incorrect) quote of Einstein that I read recently...

"Rather than trying to become a man of success, strive to become a man of value".

A man of value knows things, he can do things that others can't; this is what makes him valuable. How can anyone expect to become successful without first becoming a man of value?

That's my input.

Enjoying the posts guys.

Wow, Leon, Bravo! What an honest, humble post. I agree and can empathize with everything you said. I also came into the trading world thinking I could easily turn a profit, "how hard could it be?" I was asking myself. Well...

I'm somewhat of a stubborn person, coming to these conclusions was a good life lesson for me. The truth is that engineering a profitable trading system is very difficult and takes a lot of time and specialized tools that come only though dilligent study--for the reasons you cited.

I think you will inevitably find success in life given your current attitude, outlook, and willingness to work hard.

I've been sort of obsessed lately with the Amish people we have here in America, I'm not sure everyone around here will know what I'm talking about. The Amish are simple people, they choose to live their lives without all of the modern convieniances of the modern day. No electricity, no cars, farming done by hand, ect. I've had the privledge to get to know some of these people and one of their viewpoints really struck me. It's the way they look at work. Basically, they find their joy IN their work, not away from it. I thought a lot about that and realized that I'm happiest when I have a project going on. I've been to a lot of nice places and done some cool things, but that sort of frivelous novelty cannot compare to that comes after studying something for a week and finally "getting it," or working hard on some physical project and having the satisfaction of looking at the product of all the effort.

The mistake most new traders make is they come into trading for the end goal and take the means for granted. Sadly, some really make a mess of their lives and their finances because of this attitude. Their are enough old posts around here to prove that, I still wonder about the kid who squandered all his college money on forex. I don't know where the post is, but many of you will know what I'm talking about.

Anyways, thanks again, Leon, for a great post. I'll email you here soon and show you some of the things I've been working on.

Take care,

Jim

Hey Craig,

I have some questions for you. I was talking with a programmer friend of mine yesterday who writes programs for banks, he's not a trader. He had a general knowledge of trading and he suggested that I download Microsoft SQL 2008, as well as Microsoft Visual Studio 2010. My question for you is, is either of these applications going to be of any use to me on the programming side of things, even just for getting aquanted with programming for learnings sake?

We've talked before and I recall you telling me that you work with C++, is that something I need to learn? I guess a better question would simply be, I want to learn programming that will eventually relate to trading, where should I start?

Thanks,

Jim

Craig,

One more thing. I wanted to thank you for the links to arxiv.org. I have observed what the second link explains-- in a simplistic sense. I've seen non-institutional professional traders write programs to detect where large orders have been split, they do it in a non-statistical way, but they are right on the money--pardon the pun. When it comes down to it, OFA is all about seeing professional demand i.e. unusually high volume traded per small packets of time. This can be done by simply ovserving huge single orders, or with a volume/time indicator. I've also read about looking at the rythm of trade to identify split orders. This same group I'm speaking of works with Algo traders to better mask what they are doing. Nevertheless, when it's time to trade its time to trade, there is only so much one can do to hide their trading.

I look forward to learning the programming side so I can apply what I know in principle to actual data. Thanks again.

Scotty is going to be a C++/SQL programmer, just like that. It takes many years for people to become a programmer just like it take years to become a trader. And just like trading a lot of people will never get it. I think you are going to fit in that category for both trading and programming

Well, I apologize if I gave the impression that I thought programming was easy or wouldn't take long. We've been talking about the willingness to work hard. Maybe to save time you skipped to the end of the thread to see what all the buzz was about and missed that part. I think you are going to fit into that catagory of forum visitor. You've proved your own point though, it's hard to "get it" when you blindly skip ahead. Thanks for the post.

This is a big question, with probably no one correct answer.
I'll assume you're a windows guy.

I would start by using this http://www.microsoft.com/express/Dow...2010-Visual-CS and a good tutorial book to learn C#, once you have a medium grasp of C# you will be able to automate with Ninjatrader (or this http://www.dinosaurtech.com/utilities/).

The next step would be to learn how relational dbs work, they are all pretty much the same you could use http://en.wikipedia.org/wiki/MSDE, I use http://www.postgresql.org/.
C# will let you connect to either and on-line tutorials are everywhere.

Once you have a grasp of these subjects, you should be able to throw together something basic.
You can go pretty far with C#, you get a lot of bang out of the box.

C++ is hard and is only really needed if you want to go high end, but if you're really keen I would start by downloading http://www.microsoft.com/express/Dow...010-Visual-CPP and working you way though this book http://www.amazon.com/Accelerated-Pr...8013564&sr=8-7.


Learning programming from scratch will be tough, but super worthwhile.
You don't need to be a programming jock, you just need to know enough to express your ideas in code.

Forget about relational databases, SQL or any other related stuff, you don't need them. It only adds a new abstraction level.

The data that you need to test in this business is a simple combination of time and price and a database makes your life harder. You must be able to develop a function which reads from a structured text file, very easy to do and faster than querying databases.

In the end you'll want to draw your own interface so go with VC++ or VC#. The easiest way is to take a programming course, the books can help if you already have a grasp of programming.

Note: SQL is slang for any relational db (SQL Server, MySQL, Postgres etc).

This is a blanket comment and as such is not very helpful, what Scotty needs to do is explore the options, I don't just use SQL for bar data (duh), it also has my setups, results, executions and positions, my code is 'data driven'. Reading stuff out of a DB is no harder than reading it out of a file. I have 1min data for all of the S&P500 going back to 2007, SQL handles this no problem. The only reason to go beyond SQL is when the data demands become too great, when people are dealing with tick data from US equities (10s of Gigs per day) then SQL is useless, at this point people start developing a custom file formats or use something like kdb (http://kx.com/Products/kdb+.php).

Scotty, I would recommend that you first learn R, then if that is not enough Python. My advice is against learning something like Java/C#/C++.

Start with R. Get a GUI. The default one (RGui) is a joke - not even syntax highlighting. Unfortunately, I failed to find a good GUI for Windows. The ones which were ok, weren't updated in the last 2 years. The newer ones I didn't like. But it depends on your taste. Some GUIs are listed here: http://sciviews.org/_rgui/projects/Editors.html

R is very powerful for numerical work. You already have packages for working with time series, candles, you can pull data from Yahoo Finance, you have indicators (EMA, RSI, ...).

Python is very powerful in general, but weak on the numerical side (as weak as C++/Java compared to R). You can add the NumPy extension to Python which allows you to work with vectors in a way resembling R.

The only place where you would need Java/C#/C++ is for speed. But unless you are doing high frequency trading or processing tons of data, speed is not an issue. I know C++ very well, but I prefer waiting two hours to finish some heavier processing in Python than rewriting the algorithm in C++ and waiting only two minutes. The Python version of the code is much easier to enhance and update. And stuff which is piece of cake in Python can sometimes be incredibly hard to do in Java/C++.

Python is extremely easy to use in general. Everything is obvious (or as obvious as a programming language can be). For example, compare the code required to read the data from a file in Python vs. Java or C++: http://www.codecodex.com/wiki/Read_a...o_a_byte_array C# (.NET) in this case has a utility function, which solves the problem also in one line, but this is not true in general.

And it's not a toy language as many think. There are many C++ finance HFT jobs I've seen that mention Python familiarity as a bonus, since Python is perfect for "glue work", like downloading exchange CSV data and prepare it in a form which then you can fast analyze with C++.

I would use R as much as possible. The only place where it's not so useful is for general processing - for example you want to download from the forex factory it's calendar for the last month, and extract the dates of all news events. It is possible to do this in R, but it's much easier in Python.

So in general, anything else besides working with number vectors it's easier and more fun in Python.

I would start with some R and Python tutorials to get a feel for the languages. Then try some real problems that interest you - for example download some EUR/USD ticks from http://www.truefx.com, import them in R and plot them. Then add to the plot a EMA(50).

A very good place to ask programming questions for any language, including R is here: http://stackoverflow.com

On an side note: there is also a package for interfacing R to IB.
http://blog.fosstrading.com/2010/10/...-ibrokers.html

Maybe it's semantics but SQL is a language designed to querying relational databases. SQL Server or MySQL are applications whose goal is to manage relational databases and allow access to their data via SQL (Structured Query Language). So SQL is useless per se.

Adal, Scotty uses Windows. And for programming on Windows there's nothing better than VC++ or VC#. Sooner or later he will want to make a user interface and I don't think you can come even remotely close with R or Python to what Visual Studio can offer.
Even if R has some predefined functions like EMA you still need to write your own functions and algorithms. And why would someone learn two languages when one (C++ or C#) is enough?