1min Historical - Strategy Test

Hi there,

I am a discretional trader, and cannot complain, I've managed to double my account few times by now. After a while trading currencies, I have been working in the automation of my strategy in order to open a PAMM account (kind of tired of being glued to the monitor).

I have past few months coding (quite a bit) this strategy and finally came to a very analytical study of price action. The results were very good for the last year (2011), so I tried a couple years back, and got the picture you see bellow from 2009 to 2012. After that, I went to 2006 and then 2000 to see what happens. Well, I found that in fact, the strategy is very consistent and survived the internet crash of 2000 and the mortgage crash in 2008 with minor scratches. And as you can see, the worst drawdown was in 2000 where it experienced a 59% DD; right after the internet crash, the system recovered the losses and resumed its way.

As everything, there is only one detail, and is that I still have not managed to test it for this year 2012 in 1 minute timeframe, since Metatrader only downloads historical data until 1/13/2012 for 1 minute. And the current data you see on charts, are only 1 month back for that timeframe, so I have no data from 1/13 to 8/1/2012.

Now, if there is anyone who could please find for me historical data of 1 min for this year 2012 (from live ECN account, as you may know, I cannot use demo data nor bucket shops data), I would really appreciate it. I have tried to find historical data out there, but with not too much luck, most of it are corrupted, incomplete or not accurate enough. So, I decided to write a post here to see if there is anybody who has this data available.

Well, I think that's all by now.

Thanks in advance guys.

Hey Joe I'm not quite sure if this is what your after it's the closest thing to what your asking that I can remember coming across,see if you can make this work for your task at hand

http://www.forex-historical.com


P.S I promise I will never sell you nothing and in no way am I an affiliate for this site I just remember it and comes close to coincide with what you ask for so check it out,it may assist you in what you seek it may not,let me know if it's usable for you

Thanks, PipLegend, but I'm afraid they only offer 5 minute data; I need is 1 minute data only.

BTW, these guys from forex-historical are selling data we already have, everyone with MT4 live account have access to that data until 1987 and older I guess, depending on the currency pair.

Interesting results
To make sure your back testing to be more authentic, I would suggest to visit

www.eareview.net
There you will find the way to back test up to 99% quality as well there is a method explain how to download and prepare tick by tick data from Dukascopy.

Hope it helps

Good luck with your success :-)

Yeap... I agree with ForexZak.

If you do not have tick data back tests, the results could be immensely misleading.

J.

Thanks forexzak, nice website! I see there you can get a 99% quality on your results . When I was running the first tests, I always wondered why it only gets to 90, because I have all data in all timeframes availables from MT4.

Eventhought, I see I can get tick data with dukascopy, oanda, etc, but until what date? will it contain tick data until 2000?. Don't know, I hope to at least find for this year 2012

Well, I'm really not worried about restesting again these results, since the strategy is not pip hunting or doing very small scalping. I have results with larger tp (eg 25, 32 pips) also, and the system shows the same growing pattern, but obviously with a net profit 2-3 times higher than these you see here. But, I will do the work and run the tests with the tick data over the same period (if possible). 99 is always better than 90, thanks guys.

Hey forexzack, after checking that website, I found that there are systems that I really don't know how to qualify. Check these results http://cdn7.eareview.net/wp-content/...spread-1.5.htm

It's just obscene!, and they're selling it for 99$!!! And the guy says, he is taking the money to invest it in that same system... Sure, you are bro!!!, because having 1 single year with profits over 100M and you need the 99 bucks to pay the subway ticket!!!

These guys think programmers are just kids with glasses. They can keep their harry potter system.

I'll try to find the historical data I need at dukascopy I guess, eventhought it will be from a demo account, but will be the best thing until I find data from real account.

Well, thanks anyway zack.

Hi Joe.
All historical data "stored" in any broker is only 1. What gets stored are the "quoted" prices. All Brokers "quote" in real and demo the same price and that is what get stored.(well, 2 of course ask and bid).

That is the real and "only" data available.

You need to think in terms that a Broker quotes a price... then you place an order.. then the Brokers accepted or not accept your order at the price quoted.

Say that in a real account, in the precise moment you place an order, an spike of 5 pips occurs. The spike of 5 pips will show in a real account an in demo because that "was" the price and what they quoted. Now, maybe your order was "filled" at the quoted price, maybe not... maybe somewhere in between.

See, will be impossible for anybody to store prices of filled orders. I mean, say they had 100 orders at that price, it is a big chance they filled everybody at a different price, perhaps in a sequence of orders received in intervals of 1ms. So, they store the quoted price.

When you back-test with tick data, you still have to make provision for that slippage. I use 2 pips per order and that may be overkill as an average, but I rather be conservative.
For example: If I run a back-test and have 200 transaction. From whatever result I deduct 400 pips.

Now, the use of tick data for back-testing is to avoid a different and greater type of distortion, one that is impossible to assign a value to.

1m bars (non-tick) run this way.. say it is an up bar. First, will go to the lowest point, then will go to the high point and then will go to the close point. That can be a big distortion of the real price action.

For example, say the price start at 1.2800, spike to 1.2820, then comes down to 1.2802, then goes back to 1.2845, then comes back to 1.2804 and then goes up to 1.2865... all that in 60 second.

If you run that non-tick bar in visual in mt4, will show open at 1.2800, will go to 1.2802 and stop a little, then to 1.2804, then to 1.2820, then to 1.2845 and will close at 1.2865. Absolute deviation from the real past price action.

Now, if you run a back-test in 5m with 1m distorted bar... how accrued you think the back-test could be.

1m bars (non-tick) do not reflect the real past price action, the bigger the 1m bar the bigger possible distortion.

J.

Joe
I did not find any other reliable source and/or ways to backtest what I have seen and experienced with "EAreview site", It does not mean that I am promoting them I believe people who have skills and devote their time have right to ask to be compensated, probably this is why they charge, also they have an additional monthly fee of $10/- so u can download their utility to backtest on new MT4 versions, whenever new MT build is launched

Joe, steps to back test on tick data what you need ?

1. You need Tick Data, (Its Free from Dukascopy open an account and download
2. You need a script to convert the Tick Data and prepare it to be used with MT4 format (these scripts are Free too, you can find it even at MQL4.com or FF or other forum)
3. You need a tool that can allow you test your strategy on ur Data, (NOT FREE), this is what you pay to a 3rd Party to provide u a tool so when u run ur back test, the MT4 skip the creation of data and use the data that you have prepared. Their utility does this process by starting MT4 in a way that when you start your back test it does not create the data. Also the utility lift the restriction of MT4 of 2GB file and u can perform testing on larger data sets.

Read the following link for better understanding : http://eareview.net/tick-data

The other alternate could be:
1. You trade on ur strategy on a demo account for 3 months.
2. Open an account with small investment $50/- and start forward testing for 3-months.
If strategy survive 3 months in demo and 3 months in forward test, then you don't bother about back test, u just keep growing :-)


Hope it helps, good luck

Hi jeuro,

Yes, you have a point, in fact if you see the second test, there is a larger profit trade of 23.96, when the tp is only 11p. So, no doubt that was a spike that caused that profit distortion.

Regarding your text "All Brokers "quote" in real and demo the same price...", mmhh..., I think I have to disagree with you here, because ehh..., if you have a live account and a demo account you'll notice that really, the two accounts are not quoting the same price (at least not in FXOpen). I have noticed differences of several pips (from 1 to 5-6 pips) in 1 minute date in both accounts. So, demo account data, is data I really wouldn't use.

I'll try to make an effort in finishing the back-testing with that tick data (if I find it). At least the system has a tendency to a profitable territory so far.

forexzak, yes I know ur intentions are not to promote the site, I was just quoting how surprised I was after reading the content of that site, definetly... I can't help it. It's like the story of a guy who finds a 100$ bill on the street, and then finds you wlaking on the same street and tells you: "hey!, you're a nice guy take this 100$ bill I just found, for you...". Please, nobody puts money in your pocket for nothing :s

I tell you, I have 16+ yrs doing code and have few yrs working at freelancer dot com, writing code is the best thing I know to do, and if I manage myself to write a good code for the markets, why would I sell it? and offer subscriptions for a couple of burgers??. My eyes bled after seeing those offers....

Well, let's go back to your question. Ehh, not exactly, I was just looking for 1 minute data only, but obviously if I can get ticks and 99% accuracy then better. But again, if that's too much trouble then I prefer not to bother with that, I thought that may be somebody with a live account for a year could have it. This strategy has worked for me for almost a couple of years, the only difference is that it now trades more often than I use to do manually, and it trades in moments where I (clearly) would have been out of the markets. Yes, it generates more profits, but also more loses.

Yes, I have been thinking in put it in to play with real money. I will evaluate the options. And I still have to code the "volume optimizer", that surely will take me a month or so.

I'll continue looking for a couple days more.

Well, it seem that with FXopen anything can happens.
They claim to be an ECN Broker, but if they feed ECN price to the demo, and they quote different in real account, it does not sound the broker gives you the ECN. But because I do not trade with them I can not say anything for sure. Maybe they have different account like most Brokers and may add the mark-up spread for min-accounts, but the price would still have same moves but just a bit different marked up price.

Anyway, you are still missing the point about back-testing .
Even if you have tick data from a real account, you still have to consider some pips per trade for slippage....because still is a "quoted" price. Not execution price. Back-tests execute instantly at the price the data shows... "live" real account do not.

Like I said, the main point of using tick data is to avoid the way mt4 interpolate the prices to come up with a 1m bar. Not the slippage.

J.

PS... if the prices are 5-6 pips different as you said, I would be scalping the crap out of the latency (arbitrage). I have the best software for it. I could test real/demo if you want... not to execute the trades... but just to see how much different the prices are.

Ye I know, quoted price, execution price are not usually the same, that's using an ECN pool for transactions can make a difference, there you do not get re-quotes. But there's still the slippage, and as the system uses market orders and not limit orders, then the slippage can go as far as it wants.

That's another difference between my manual execution and this automated, I never use market orders.

Regarding the difference in pips from demo/live accounts, I think you are confused, yes in fact sometimes there is a difference of several pips between the two accounts, but because the demo account do not receive ALL ticks quoted to completely form the candle as the live account does, so you'll see usually in live account larger candles with several pips of difference in coparison to demo account (while live accounts receive 10 quotes, demo accounts receive (let's say) 5-6 quotes for the same period of time). But, it does not mean that one price follow the other with a DELAY between the two. Have to be carefull with that understanding.

BTW, I think I found the 1 minute data in Alpari, that platform download everything.

[quote=joecn;6075355]

Regarding the difference in pips from demo/live accounts, I think you are confused, yes in fact sometimes there is a difference of several pips between the two accounts, but because the demo account do not receive ALL ticks quoted to completely form the candle as the live account does, so you'll see usually in live account larger candles with several pips of difference in coparison to demo account (while live accounts receive 10 quotes, demo accounts receive (let's say) 5-6 quotes for the same period of time). But, it does not mean that one price follow the other with a DELAY between the two. Have to be carefull with that understanding.

No I am not confused but I think you have it backwards and all wrong. Demo accounts most of the time is the real price feed that Brokers get from their liquidity providers. From there on, most Brokers make some extra ticks to "some" of their live accounts, specially micro and non-ecn accounts. ( that is from were the term "Market makers" come from. They make prices that do not exist in between quotes. Usually to match orders or any other of those tricks we know about. Anyway, for back testing this part is not as relevant as the use of 1m bars constructed with ticks instead then the "made up" 1m bars you get via most mt4 brokers.



BTW, I think I found the 1 minute data in Alpari, that platform download everything.

Alpari 1m data is full of holes. Sometimes hours are missing, sometimes day or weeks. Make sure you review the data.

But you will still have regular mt4 1m bars.... those are interpolated prices that do not follow real price action. If you are planning to sell the EA you are working on, it does not matter, if you are planing to use it with your real money (and with other people money) it is better that you take the time and get the tick data from Dukascopy and make your own "correct" 1m bars.

If you have something good, you have something good , and you should enjoy the fruit of your work, but please do not cut yourself short of doing the best possible and burn your reputation because you did not do a bit of extra work in the back test.


J.
quote]

Hi jeuro,

Nice theory about market makers. I thought that real price was on live ECN accounts.

A friend of mine who has a relative successful PAMM account gave me the recommendation to use Alpari meanwhile, and to consider the back-testing as an idea of feasible results. He even offered some help to get the 99% quality test. I'm also downloading the same data from dukascopy to compare the results on both providers (Alpari and Dukascopy [Demo]).

Sure, that could be possible, I'll be alert in checking that the data is complete and with no holes. With a simple routine in java it can be verified. Anyhow, someone yesterday also offered to send me later historical data from his live account.

I understand your worries, but don't have to; I'm alread doing my homework in back-testing with the best data available and forward-testing. The system still needs several months to get a conservative conclusion about success or failure.

Thanks for the concern.

Super... sound good, it seem you are in your way.

Regarding the ECN, who knows anymore. I mean every brokers is connected to it and "they get" the "multiple" pricing found there.

It is, sortof speak, one line.

Obviously, all brokers's software finds the best between those multiple and that is what they quote to us. Now, from that one line, they decide what they feed to what. Most Brokers that Provide true ECN accounts, feed their demo straight true, so it is the same price. Some others may get the ECN feed, pass it trough their order books, make their own pricing for their clients..and then from there feed the demo.

Not many brokers bother to have different demo pricing. I guess is complicated and costly( I know alpari uk does have a demo-pro and a regular demo). So most brokers feed their demos, either straight true from the ECN or from 1 of their quotes to clients. At least one of their accounts will match their demos. No point of making up a different pricing just for demo.

J.

hi! nice backtesting there, can you please tell us about your strategy in this EA?

Best regards and success

I don't know if you've got this resolved already, but I recently used this tutorial in order to download tick data:

http://eareview.net/tick-data

And then used this one to convert it for use:

http://eareview.net/tick-data/convert-tick-data

Then used this one for the backtesting:

http://eareview.net/tick-data/tick-data-backtesting

Thanks jeuro, I'll keep you posted on new results.

Peepo, as many other traders the system detects when the market is inefficient and place trades based on that information that's all. The results are not definitive, lately I have made few changes to the code, and has shown some improvement, I'm still tuning the code.

Thanks FXAssasin, yes I finally resolved the problem with the data; I'll try later to get the same data from other brokers and live ECN accounts.

Hello Joecn

Are you telling that watch when the market have low volume and movement, then place pending orders ?

How you determinate is a BUY pending or SELL pending order?

Nice try Peepo, but it'd be useless to include volume in your formula, remember Forex is not centralized. Just learn how to distinguish an efficient and inefficient market. Inefficiencies only puts odds in your favor, that does not mean you are going to beat the markets any time you want, it increases your probability of success that's all.