Automated Trading & Backtesting Course - Part 1: Programming

Many people have posted their trade systems on this forum, but most systems are missing something essential: they have not been backtested. So there's no way to know if the system is profitable or not. A "manual backtest" by eyeballing some charts or trying some demo trades can not give any useful info. Consequently, you can see in five minutes with a simple 4-year backtest that most of the systems posted here will lose money. On the other hand, a few of them can generate remarkable profit. But how do you know which?

I think no one can develop a successful system as long as he has no method for testing it. That however requires to program the system as an EA, script, strategy, whatever it is called. Here's the problem: most traders think programming is too difficult for them. But it's not. Anyone can learn EA or strategy programming in a few days, and can then easily backtest his systems or even trade them automatically. I'm in the process of writing a course for this. I hope from posting the course here that I get some feedback about what is easy to understand and what not, so that I can improve the material.

I'll post the first part here over the next days. The first part is just about learning programming in C, and covers variables, functions, branches, and loops. The second part will be about programming trade systems, and will cover trend trading, counter trend trading, optimizing, walk forward analysis, portfolio strategies, and money management. It uses some new trading algorithms such as frequency filters.

The C language has the advantage that you can use C knowledge for writing EAs or strategies on many platforms such as Ninja Trader, MT4, even Tradestation etc. because all those languages are rather similar.

The course will go over about 2 weeks. The goal is to develop a portfolio strategy with a solid annual return on capital. There might be a third part about using artificial intelligence, such a perceptrons and decision trees, but it's not finished yet and comes at a later time.

For the course you'll need some program for running the script examples and testing the strategies. It's called "Zorro" and you can download it for free from zorro-trader.com. We won't use MT4 for this course, although MT4 also uses C for its EAs. There are several reasons: MT4 requires some extra code stuff that is unnecessarily hard to learn for a beginner. MT4 lacks also essential features for properly testing a strategy. The third reason is that MT4 has a hidden feature called "Virtual Dealer Plugin", which can be used by your broker to remove a part of your profits. Therefore I tend to avoid MT4. However when you're through this course you'll also have the basic knowledge to write an EA for MT4 if you want.

Let's start working. At first a little theory. The purpose of a script (or EA, or program, or whatever it's called) is telling the computer what to do under which circumstances. A script consists of variables and functions - in this first lesson let me explain variables. A variable is a place in your computer's memory (just like a container) that is used to store numbers, text, or other information. Because you don't want to have to remember where in the computer which variable is stored, any variable has a unique name that's used in the script. A few example script lines that define variables:

var Price;
var PercentPerMonth = 1.5; // the monthly interest
int Days = 7;
string Wealth = "I am rich!";
bool Winning = true;

These are a few short lines of code but we can use them to learn many new things:

► Every variable must be defined (programmers also say 'declared') before being used. We have different variable types: var for variables with decimals, like prices or rates; int for variables that have no decimals, such as for counting something; string for text; and bool for a sort of 'toggle switch' that is either true or false. There are even more basic variable types, but in trading scripts you'll normally encounter only these four. If you write this line in your script:

Days = 3;

and you haven't defined the variable named Days before, you'll get an error message. The only exception are variables that Zorro already knows because they are pre-defined in the compiler.

► Any variable can receive an initial value at start, but we aren't forced to do that when the initial value does not matter. Example:

int num_bars = 7;
var limit = 3.5;

► We can add our own comments to the code. Every time it encounters two slashes //, Zorro will ignore the words that follow it, up to the end of the line. This way we can add useful information to our code:

int bar_width; // the width of one bar in minutes

or we can temporarily disable a line in the script by placing two slashes in front of it. This is called "commenting out" a line, and while programming it is used so frequently that the script editor has two extra buttons for commenting out and commenting in.

► Every definition or any command in C needs to end with a semicolon. Many beginners forget to add ";" at the end of their lines of code, and this also leads to an error message - not in the line with the missing semicolon, but in the following line!

► Every variable name must start with either a letter or with an underscore _. Here are some valid variable names:

var AlohA;
var _me_too;
var go42;
var Iamb19;
var _12345;

Now let's take a look at some bad examples:

var #ItoldYou;
var 1_for_all;
var 12345;

I'll let you discover what is wrong in the definitions above.

► Variable names are case sensitive. This means that if we define an int this way:

int MyTradePositions;

and then we use it later in our code this way:

mytradepositions = 5; // or
Mytradepositions = 5; // or
MYTRADEPOSITIONS = 5;

Zorro will not accept it.

► You can define several variables in one line. This saves lines and keeps your code short. Example:

var momentum, strength, score;
int width = 7, height = 14, depth = 20;

► Finally, the variables should have significant names. While it is possible to define a pile of variables that look like this:

var x32;
var a125;
var h_34_5;
var _z34187;

it isn't a good idea to do it this way. You will have problems trying to remember what these variables do if you look at your script a few weeks later. If you want to show your script to other people, they will have a hard time trying to figure what you wanted to do with your code.

Tomorrow we'll write a little example script that does something with variables.

Please don't hesitate to ask here if something is unclear or bad explained.

Today we'll write our first script. You'll need to download Zorro now if you haven't done so already. After installation, start it and you'll see this:

http://zorro-trader.com/images/zorro.png

Click the [Script] scrollbox, go all the way down and select [New Script]. A text editor will now open. Type in the following:


Now, select [Save As] in the [File] menu of the text editor, and save it under a name like "myfirstscript.c" in the "Strategy" subfolder of your Zorro installation. When you now click Zorro's [Script] scrollbox again, the name "myfirstscript" should appear among the other scripts. Select it, press [Test] on Zorro's panel, and watch what happens in its message window.

If you did anything right, you should now see a message like this:

http://zorro-trader.com/manual/images/work1_1.png

If you see something different in the "Result =" line, please complain here.

Now, edit the numbers in the a and b lines and replace them with

a = 5;
b = 12;

Save the edited script (File / Save or [Ctrl-S]), then press [Test] again. Tell us if your result is different than 17.

That was our first program. Starting to make sense of this, isn't it? It looks like c is the sum of a and b; try to type several values for a and b and you will convince yourself. Now let's take a look at the piece of script that transforms Zorro in a sort of simple calculator.

function main()
{
...
}

The script starts with a function named main - everything that happens in a program is inside the winged brackets { } of a function. But we'll come to functions tomorrow. Here we concentrate on variables, and the next lines should already be familiar from what I said above:

var a,b,c;

We can see that the 3 variables a, b, c are defined, just as described above. The following lines are commands:

a = 5;
b = 12;

The variables here get a content; they are set to 5 and 12. Now the following line is the core of our script:

c = a + b;

This line of C code appears to be simple too; it makes c equal to the sum of a and b. It is a command to the computer to add the content of the variables a and b and store the result in the variable c. Commands are lines in the code that do something, usually with variables.

The last line is also a command, used for displaying c in the message window:

printf("Result = %.f",c);

Let's make a small experiment. Find the line of code c = a + b; in the editor, and then replace the "+" by a "*", the 'times' character, so that the line now reads:

c = a * b;

Save the script, then press the [Test] button again. Does the result make sense?

You have now mastered the basics of lite-C. Zorro has multiplied 5 by 12, displaying the correct result: 60. I know it's not yet a trade strategy what we're doing here, but we're going somewhere! So now we know how to add and multiply values; we can use "-" to subtract two numbers or "/" to divide them. We could replace the line that does the c = a * b; calculation with much more complex expressions...

But we'll save that for the next lessons. Enough for today.

Please let me know if something was unclear or hard to understand.

I think I run into zorro site a few month ago when doing a search on c related struct or functions to write my own trading backtest, at that time it was not ready for download yet, but seem like a sleek design, with nice tutorial and the c like language. At that time I run into the site, it looked like a established software already, even though I was wondering why the site started years ago, but still not yet ready for deployment.
It look a lot like amibroker script. is it array based language?

Are you the developer of the platform? wish you success on the this app. If the backtest and strategy testing is on par with amibroker, maybe I will pick up another c style language for the backtest.

The Zorro website is online since a few weeks only, but we had the Zorro manual on that web address since several months for the beta testers.

Zorro was developed by a team from game programming, but I was involved in designing its strategies and writing its manual and tutorial. Yes, the language supports arrays and trading variables are array based.

Back to work. This lesson will introduce functions. Once you got variables and functions, you have a basic understanding of how a script works.

What are these functions, anyway? Let's see a small example:


Do you see what I see? A function is nothing more than a collection of C commands that are executed by the computer one after the other. Let's see some properties for these functions:

► A function is normally defined using the word function followed by the name of the function and a pair of parentheses (). The parentheses are used to pass additional variables to the function; we'll learn about that later. In our case we don't pass any variables, so they are empty.

► The body of the function (its list of commands) must be written inside a pair of winged brackets { }. The body consists of one or more lines of lite-C code that end with a semicolon. For clarity, programmers usually indent the code in the function body by some spaces or a tab, for making clear that it is inside something.

► The names used for the functions follow the same naming convention as for variables. You shouldn't use the same name for a variable and a function; this will lead to errors.

If you can read this tutorial I hope that you know your age, too. Not in years, but in days! What, you don't know it? Ok, so let's try to write a function that computes the number of days spent by me (or you) on Earth.

I know how to start! I write the keyword function and then the name of the function; let's name it compute_days:

function compute_days()
{

I haven't forgotten the parenthesis after the name of the function and I have added the first curly bracket!

We will use some variables, so we'd better define them now:

var my_age = 33; // your age (in years) goes here
var days_a_year = 365.25;

Nothing new so far, right? We have defined two var variables and they have received initial values, because I know my age in years and I also know that every year has about 365.25 days. Now comes the scary part: how will I be able to tell the computer to compute the number of days? How would I do it with a pocket calculator? I would enter something like this:

33 * 365.25 =

Now let's take a look at our variables; if I replace 33 with my_age and 365.25 with days_a_year, I will get something like this:

number_of_days = my_age * days_a_year;

Ok, so our function should end like this:

var number_of_days = my_age * days_a_year;
printf("I am %.f days old!",number_of_days);
}

I have remembered to add the second curly bracket, so now the body of the function is enclosed by the two required curly brackets. I am really curious to see if this function works, so let's test it. Fire up Zorro, and then select [New Script] in the Script list. Wait until the editor opens. Then copy and paste the lines below into the editor window. Select the entire script below with your mouse, right click and choose Copy (or hit [Ctrl-C]), switch to the editor, right click into the empty window named "script1", then choose Paste:


The code looks pretty simple, doesn't it? We already know how to work with those variables, we know how to add comments... So let's save it (File / Save As) into the Strategy folder of the Zorro installation, under a name like myscript2.c. Don't forget the ".c" at the end - it means that this file contains C code.

If you did everything right, you should now find myscript2 in the Script list. Select it. Time to [Test] our script:

http://zorro-trader.com/manual/images/work2_1.png

Does this error message mean that a script always need a main() or run() function? Yes, main is a predefined function name. If a function is named main, it will automatically run when we start our script. The function named run is special to Zorro; it contains our trade strategy and is automatically run once for every time period. If a script has neither a main nor a run function, Zorro assumes that you made a mistake and will give you this error message.

Now, let's enter a main function at the end of the script:


The way I see it, this code 'calls' (meaning it starts) our compute_days function. Ok, now that we are here let's see how we call a function: we write its name followed by a pair of parenthesis and then we end the line of code with a semicolon. Sounds logical, doesn't it?

http://zorro-trader.com/manual/images/work2_2.png

Important tip: write the lines of code for your functions first and call them later. The computer reads the code the same way you read a book: it starts with the top of the script page and goes down to the bottom, reading the code line by line. If I would write my script the other way around, like this:


the computer will say: oh, that's function main. I know function main; I need to run it every time. What does it say now? compute_days(). What's with this function? I don't know it yet! I don't know what it wants from me. I'm going to display an error message and I will take the rest of the day off:

http://zorro-trader.com/manual/images/work2_3.png

Don't forget to define your functions first, otherwise the computer will complain when you try to use it.

Tomorrow we'll look into functions more closely. Please complain here if something didn't work or was unclear.

A short summary of the things we've learned yesterday :

► We define functions by writing "function name(...) { ... }" in the script.

► If a function has the name main or run, it is automatically executed. All other functions must be called from an already-running function to be executed.

But that's not all what a a function can do. It can also get variables from the calling function, and give the value of a variable back in return. Let's see an example of a function that gets and returns variables.


For a function to return the value of a variable, just write the variable - or the arithmetic expression that calculates the value - behind a return command. The function then gives that value back to from where it was called.

You did notice that we defined this function not with the keyword "function", but with "var"? But is var not a variable definition? Yes, but when a function is expected to return something, it is just defined with the type of the returned variable. So if a function returns a variable of type int, define it with int; if it returns a var, define it with var. Still, the computer knows from the (..) parentheses that this is a function definition and no variable definition.

If a function expects variables, place them together with their type in the function definition between those parentheses. If there are several variables, separate them with commas. When you then call that function, just put the values of the variables you want to pass to the function between the parentheses. The function will then use those variables just as any other variable for computing a result.

If a function returns something, you can just place a call to that function instead of the variable that it returns. This sounds sort of complicated? Let's try it right away with our new compute_days function. This is our new script:


We've just set our number_of_days variable direct from the returned value by compute_days(my_age). This makes our code shorter and more elegant! Still, the result is the same:

http://zorro-trader.com/manual/images/work2_2.png


Now what is this mysterious printf(..)? It has parentheses attached, so it's obviously also a function that we call for displaying our result. However we have nowhere defined this function; it is a function that is already "built-in" in C. Just as the built-in variables that we mentioned in the last workshop, there are also many functions already built in the script language.

The "I am %.f days old!",number_of_days are the two variables that we pass to the printf function. The first variable is a string, used for displaying some text: "I am %.f days old!". The second variable is a var: number_of_days. Variables passed to functions are separated by commas. You can read details about the printf function in the Zorro manual: Click [Help], then navigate to "Script functions / Input/Output / printf".

For the moment we just need to know that the strange "%.f" in the string is a placeholder. It means: the function inserts here the value - with no decimals - of the var that is passed to the function. So, if number_of_days has the value 12045, our printf function will print "I am 12045 days old!".

We can make our code even shorter. Remember, if a function returns a var, we can just place a call of this function in stead of the var itself - even inside the parentheses of another function. We'll save one variable and one line of script this way. Programmers do such shortcuts all the time because they are lazy and prefer to type as less code as possible:


Enough for today. The next workshop will teach us how a script can make decisions (called 'branches' in computerish). Being able to decide something is important for trading. So we're now going in huge steps towards writing our own trade strategy.

Please let me know if something was unclear with passing variables to and from functions.

Ok, I'm interested but who's data are you using? If it's fxcm demo then where is the benefit over using fxcm's metatrader? My biggest problem with back testing on mt4 is the reliability of the data as you can run the same ea with the same settings on different brokers feeds and get a different result. I'm using daily charts too so you would not have thought there would be that much difference.

I use data from all possible sources, but you're right - there can be a huge difference and it can have several reasons.

The first problem is using MT4, as MT4 is not suited for backtesting. Simulating a system over a couple years alone won't tell you much about it's performance; a serious backtester is combined with a parameter optimizer and walk forward analysis so that you see how your system reacts on parameter modifications. Only then you can judge if its performance is stable or not. There are also several other features that a backtester needs. The backtesters of most professional platforms, such as Ninja, Tradestation, or Zorro of course, are ok, but MT4 is no professional tool for automated trading and backtesting in my opinion.

Second problem is when your daily bars start. For comparable results they must start at the same time, f.i. at midnight, and you should be able to set up that start time in your test platform.

Third problem is that slippage, commission, spread and rollover must be simulated in your platform, and they depend on the broker. Spread and slippage have strong influence on short term trading performance and rollover has also noticeable influence on long term trades.

Ok thanks but how does using Zorro make any difference if the data you are using is the same? I wouldn't want to just rely on the data from one provider?

With my expert advisor testing on mt4 now all I do is test the different parameters on 3 or 4 different mt4 brokers and see if they come out similar. This is not a perfect solution but it gives me more confidence in my positive expectancy.

I have used so far data from several providers with Zorro, but have never seen any noticeable differences in the backtest results when the broker parameters were the same.

The only difference I've noticed, aside from the usual minimal price differences, is the data quality, i.e. gaps or outliers. For instance, FXCM data is sometimes missing intraday bars or has price spikes. But they normally won't affect the result much as they should be filtered by the platform. When you get definitely different results, there must be a certain reason, such as different bar start times, and you should look into it. Otherwise you can't trust your results.

On the other hand, getting the same results with data from different source is just natural, and constitutes no stress test of your system. Later in this course we'll deal with how to do a real stress test.

Ok thanks, I am downloading the platform now and am watching your tutorials with interest.

To avoid misunderstandings, my warning against using MT4 for backtesting was only meant for finding a profitable system. MT4 is very well able to find most unprofitable systems. A positive MT4 backtest does not mean that the system is profitable, but a negative backtest guarantees that the system will lose money. Thus, a backtest even with MT4 is better than nothing.

Got you... well hedged!

So what is your opinion on my method of testing the same ea with the same settings on multiple brokers to get an average result?

Well... if the results differ by a large amount, then either the system is hopelessly inconsistent or something is wrong with the price data. The average is not much worth in such a case.

If however the results are very close together and all positive, then you should get a better platform (I think you know which I mean ) and continue with some serious tests - oversampling, walk forward, and portfolio analysis. The system might be worth it.

The system can't be hopelessly inconsistent if you are trading it over the same period and the only thing that is changing is the data provider. It must always be a problem with the data.

I think that if the system tests positively on all 4 brokers then it should be ok to trade going forwards.

My problem with your assertion that your platform is better (I have no doubt it is better than mt4 strategy tester, no doubt at all and in fact is probably on a par with the best strategy testers available from what I have seen so far) is that you are still at the mercy of the data provided to you by your provider. Who is to say that fxcm data is any better than anyone elses?

Maybe you could include in your tutorial instructions on how to access other data providers for the purpose of cross testing?

Sorry to have taken over your thread a little but I think this discussion will be useful for people coming behind us.

Gaussian series, fisher transformation? My head hurts. Couldn't we have started with ma cross overs?

I'm really liking the look of the walk forward optimisation function. All in all this looks really excellent. Are you planning on making the tutorial here any gentler for beginners?

In the Zorro manual under "import/export" you'll find a brief description how to import price data from other providers. (http://zorro-trader.com/manual/en/export.htm)

I didn't want to start the tutorial with a simple MA crossover because it is normally not profitable, thus Alice could not honestly develop such a strategy and demand money for it . But don't worry, anything will be explained. That is the reason why I post the course here: You can ask specific questions if something is unclear, and I'll learn where the tutorial can be improved, and can make it gentler.

-------------------------------------------

- Back to work. We're not yet at trade systems; first we'll have to learn how a program makes decisions. If my bills grow bigger than $3000, I need to find a new job, else I will keep my existing job.

Yes, my brain is still working ok, thank you for asking. That was just an example of an if - else branch; its associated code would look like this:


You will use "if" statements when you want your script to make decisions - meaning that it behaves differently depending on some conditions, like user input, a random number, the result of a mathematical operation, a crossing of two indicators, etc. Here's the basic form of the "if" statement:

if(some condition is true)
do_something(); // execute this command (a single command!)

or

if(some condition is true)
{
// execute one or several commands that are placed inside the curly brackets
}

A more complex form of the "if" instruction is listed below:

if(some condition is true)
{
// execute one or several commands that are placed inside the curly brackets
} else {
// execute one or several commands that are placed inside these curly brackets
}

The instructions placed inside the "else" part are executed only if "some condition" is not true. Here's a practical example:


It is pretty clear that income can be either 2000 or 3000 because only one of the branches will be executed (income = 2000 or income = 3000, not both). The conditional parts of the code are called "branches" because several nested if instructions can look like a tree with the root at the first "if" and many branches of which only one is executed.

Let's draw some conclusions:

► "if" branching statements start with the if keyword followed by a pair of parentheses;
► the parentheses contain a comparison, or any other expression ("some condition") that can be true or false;
► if the expression is true, the following instruction or the set of instructions placed inside the first pair of curly brackets is executed;
► if the expression is false and we don't use "else", the set of instructions placed between the curly brackets is skipped (it isn't executed);
► if the expression is false and we are using the "else" branch as well, the set of instructions placed inside the first pair of curly brackets is skipped, and the set of instructions placed inside the second pair of curly brackets is executed.

You'll be mastering these branching techniques in no time, trust me! Let's write the following script :


The code doesn't look that complicated; we have defined a var named profit which receives an initial value of 50, and an if statement. If profit is greater than 100, we have enough, otherwise not. We can omit the if / else pairs of curly brackets mentioned above when their content consists of a single line of code.

Create a new script - you have learned in the last workshops how to do that - copy/paste the content from the code box above, save it as "myscript3.c" in the Strategy folder, select and [Test] it:

http://zorro-trader.com/manual/images/work3_1.png

Now let's try something else. Modify the code by editing the marked line:


When you now click [Test] to run the script again, you'll notice that the bottom slider gets the label "Profit". Move it all the way to the right, so that 200 appears in the small window, and click [Test] again:

http://zorro-trader.com/manual/images/work3_2.png

What happened? The slider() function put its return value - which is the value from the bottom slider - into the profit variable, and thus the if(..) condition became true as the value was now bigger than 100. We can make an educated guess how the slider function works: It gets six variables - the slider number (3), the initial value (50), the right and left borders (0 and 200), the name of the slider ("Profit"), and a tooltip (here 0, i.e. not used). You can find a detailed description of this function in the Zorro help file. Put the slider to the right again and verify that the program now prints "Not enough!" when you click [Test] with the slider value at 100 or below. You can now imagine how we can use the sliders for adjusting variables for our strategies.

We're now almost through with the programming part. One last lesson tomorrow and then we'll start writing our first trade robot. Please mention here if something was unclear so far.

Jcl, I sent you an email last night. If you didn't get it could you please contact me at my address in my signature. I have a few ideas for you. Thanks.

A little more about if statements. Let's assume that you want to do something only when two different conditions are fulfilled. Try the following program:


Now two sliders are involved. How do you need to set them for the "Enough!" condition? We leave this as a little puzzle to the reader... but we've learned that we can combine two conditions using the "and" keyword. There is also an "or" keyword when one or the other condition needs be true. (C/C#C++ programmers might wonder why they have to use and and or instead of the familiar operators && and ||. Don't worry, all C operators work as before, we're just using easier-to-memorize operators in lite-C for beginner's sake.)

Now, I'm giving you three important tips for avoiding coding mistakes. Here's the first one. Have you noted the use of parentheses around (profit > 50) and (loss == 0)? We know from school mathematics that in a mathematical equation, the expressions in the parentheses are solved first. (1+2)*3 is not the same as 1+(2*3) - this is true in mathematics and also in a programming language. Always use parentheses to make sure that the program calculates in the same order that we want... and make sure that you have as many opening as closing parentheses! A missing parentheses at the end of a line is one of the most frequent reasons of compiler error messages. The computer will usually complain about an error in the following line because it's there looking for the missing parenthesis.

What's with that "loss == 0" in the first new line of code? Is the double equals sign a typing error? No, it isn't. Whenever you compare two expressions (loss and 0 in the example above) you have to use "==" instead of "=", because a line of code that looks like this:

if(loss = 0)
{
// do some stuff
}

will set loss to zero instead of comparing loss with zero! This is one of the most frequent mistakes; even an experienced programmer might set a variable to a certain value by mistake, instead of comparing it with that value. Using one instead of two equality signs for comparing two expressions is a very frequent mistake. Don't forget this!

if (a == 3) // correct
{
// do_some_stuff
}

if (a = 3) // wrong!
{
// do_some_stuff
}

You can avoid this mistake if you make it a habit to put the constant value on the left side of a comparison and the variable on the right side. The loss comparison statement would look this way:

if(0 == loss) // correct
...


If you then accidentally put a single '=' instead of '==', Zorro will report an error because it knows that 0 can't be set to a different value. That was the second tip.

The third tip is that you should use the '==' comparison with care. When you calculate a var variable in a complicated mathematical expression, the result is usually inaccurate. Instead of 2, it might be 2.00001 or 1.99997. This is due to the limited precision of variables in a computer. When you then compare it with 2, the comparison always comes out false - so better use 'greater' or 'smaller' comparisons (< or >) when working with var variables. This problem does not affect int variables.