EAs and historical data

Hi everyone,

I'm experimenting an issue while running an EA, the underlying candle values (for instance Close[0]) are different from the ones that are displayed on the opened chart and the generated offline chart. Where is the data read when an EA is running ?

To reproduce the problem, simply run an EA that prints out Close[0] for example. All I want is backtest an EA on the historical data I've loaded into the archives.

Thanks in advance,
Mark

Post you code...
You should not have any issue....
Online or offline the data is the same...
If your online and you missed some data(Because the computer was close) then the application will update the data, but after that, is the same.
Maybe its the way you use your Close...
Post your code and will see...

Hi Zeuzere,

Thanks for your reply. Here is the code of the testing EA :

int start()
{
if (Time[0] < D'2004.12.29 15:00' || Time[0] > D'2004.12.29 16:00')
return (0);
Print(NormalizeDouble(Close[0], Digits));
}

I just test the Close value in a range picked at random. Anyway, I'm not pinpointing a single small divergence, the online and offline charts are VERY different. If you want, I'll send you 2 screencopies. I discovered that synchronization problem when running a true EA. The indicators didn't react normally on the screen data...

So please explain me : why are there 2 charts : online and offline ? Why isn't historical data (in a HST file) used while running an EA ? I've downloaded M1 data from alpari-idc, and I'd like my EA should use this data.

Have a nice day,
Mark

Many threads on backtesting but in essence, backtesting data is cached from the hst folder and makes a new folder called Tester and inside that is a file called .fst

This file is remade after each time the recalc button is pressed and a fresh scan is made of the hst file file. Now what happens is this. When you download alpari ticks and i assume you know how to make all the other time frames to get 90% which is essential then this file .hst is used to make .fst Now the twist is this. When you go online to a broker that broker has its OWN 60 odd days worth of history data , MT4 compares that data to the hst file and makes the server or broker data the master. If you look in the journal you can see things like syncing 5 mins, bad tick replaced etc and this is making broker data master for whatever period they are holding on their server and every broker is different.!!

So if you want to use Alpari then do NOT connect to a broker after you have downloaded the 1 min ticks. Stay offline. Anyway i personally see no advantage here for backtesting improvements unless Alpari IS your broker your better off using the new download feature when gets metaquotes data instead and makes the other time periods much simplier. Whatever data you use you cant get more then 90% anyway and the smallest timeframe MT will hold or process is 1 min regardless even if you try and force in real ticks....wont work.

For practical purposes when all the correct procedure are followed the backtest might miss a trade or get a stop hit when it did or didnt in real life about 1 in 20 trades it will mess up. So its close but not perfect. I can run real trades for 1 or 2 weeks and sometimes see just one mistake when re backtested over the same live period even on 5 min scalper systems. Naturally the errors get smaller the less frequently your EA trades and the larger the time frame it runs on. So the real question is this. If i make a good EA which means a properly designed one that meets all the money man rules and correctly checks trading status that makes a profit on backtesting can it make money in realtime? The answer is YES it will do so long as you did not curve fit too hard to recent history. The winning contest EA's were all results of proper backtesting and carefully designed with MT4 backtester so thats the proof.

Mark53...
Maybe we have the same problem...
this morning i was writing a EA base on my new indicator...

But i found out that the iRSI(Well maybe other too) function does not give me the same value as the
custom indicator with the same algorithm...
You can see here.

Hi Bolt,

Thanks alot for your detailed reply. So, the HST file data is copied to a FST file in order to replace values with ones from the current server.

I precise I run my EA with the "complete bar only" model so the modelling quality is not applicable. But of course, I downloaded M1 data and adapted it for smaller timeframes.

Well, since my account has expired and I didn't reactivate it, I only run my EA offline. Plus, you talked about 60 days old data from the server, but I run my tests in 2004, so I shouldn't experience this synchronizing issue... So, what can explain the difference between offline and online data ?

Ok for the backtesting issues.

Thanks again and merry Christmas,
Mark

Hi Zeuzere,

Well, I've already posted the issue you mentioned, perhaps in another forum. Sometimes, calling iRSI doesn't work the same way as calling Custom(NULL, 0, "RSI", ...). It must be due to the same problem I pinpointed in this thread, the fact that MT4 doesn't use the same history when calling an EA as when displaying the chart.

Here comes a new question for Bolt : where does MT4 read history to display the charts ?

Merry Christmas to you,
Mark

"I pinpointed in this thread, the fact that MT4 doesn't use the same history when calling an EA as when displaying the chart."

Mmm i think your missing the point here the data you get from your broker live and the most recent trail of that data is stored on the hst file. Using a sync process the broker data is the master over your stored pc data. It HAS to be in order to make sure you have the most accurate recent history for when you are offline or have an internet outage and miss ticks. BUT to prevent the broker server having to download 4 years worth everytime you log on its limited to roughly 60 day of ticks. Your charts AND EA's use this data for writing indicators and processing forward trades. So earlier data that you download from elsewhere is over written from the last 60 days ONLY when you go oline. Is that a bad thing? NO of course not you are dealing on your broker feed and you need their data ideally but you can only get 60 days worth so when you press the download button on the history box to get 4 years of 1 min ticks that data comes from Metaquotes now instead. IMO there is no practical qaulity advantage compared to Alpari data its just far easier now to get it.

If it anytime you want to be very sure you have the best and most current data for backtesting then delete the TESTER folder close MT and reboot it. This will make a new TESTER folder (yes its smart enough to rebuild itself) and grab all the history it needs for backtesting from the HST files.

Ok, Bolt, you explained how things should work in MT4 and that was interesting.

But that didn't explain why -and this is the topic of my thread- candles characteristics (open, high, low, close) are different between what is read by the EA and what is displayed on the chart.

I repeat briefly the operations. So, I deleted the "tester" folder before loading MT4. Then I ran the EA whose small code is mentioned in a message below on M1 timeframe with "completed bar only" model. The Close values displayed in the journal tab are different from the corresponding Close values on the chart. And of course that makes my indicators react differently ! That's an important point... I can't continue developing my EA while I haven't solved this problem. I have contacted MT4 staff about this problem but they didn't seem to feel like helping me and I have no news about them.

Thanks,
Mark

Yea i confirm, thats exactly the problem i have...
Chart value do not match the indicator...

I'm unable to test my EA until i know the problem.
It is base on zero line...

Hello Mark53,

You cannot set Close[0] for backtesting and even use it for live - the reason is very simple - the "0" (current) bar that is calling for is "never" closed. Therefore the EA will call for the "current tick" in live mode and call for whatever value in the bar for the backtesting (because it doesn't have ticks).

You can use Open[0] with reliable backtesting and running in live mode. If you want to use high, low or close. You should use High[1], Low[1] or Close[1].

Hope that helps

Hum good one...Is logic,
And probably, its my problem too with the RSI
Ill give a try

yes that right close 0 will act on the filled current 0 which of course is futuristic for live use. In other words the EA will act faithfully on something which hasnt actually happened yet but is true for the history. This will produce an EA with outstanding perfomance of 99% win rate and $10000000 profit a year


This is a common newbie mistake and in any case you should always backtest in 1 min ticks turned on. I seen no practicle use for full bar only except for high speed optimising on some unknown varibles which is later verified in per tick mode. I suspect this is the reason for different chart views.

I was trying to retreive 4H RSI value on a 15 minutes charts
i notice that only every 4H i did have the good value, the in between value is not good(The good value was at the end, just before the open of the new 4h period) ...
So maybe this the same thing...
Thanks

Ok, the key to my issue is simply that the start() method is called at the OPENING of each bar. It implies that one shouldn't call indicators operating on close price with a 0 shift, it would have no sense.

Bolt, the strange thing is that the displayed close value isn't the real close value of the candle (when you run your EA on historical data of course) and that's why I have created this thread. But now I know that I shouldn't use it. Maybe it's an interpolated value.

Zeuzere, I noticed that when the "every tick" model is activated, it's the last tick's close value that gives the global close value for the current bar, which seems quite logical. That's why the last tick is the only moment where you can call RSI applied on close price. All the other times will give wrong results because of the rule given above.

You all have a nice day,
Mark

Thanks all...
I did a test with a 20 bar delay(So that i dont ask the current 4 h bar) and it work great...

You can make the EA so all your indicators have bar 0 to bar 10 as a varaible. I call it extern int BarShift = x; then use the GA optimiser to find the best values of all your indicators for opening and closing a trade. More often then not the best values are two bars back from current price is the best signal NOT bar 0. In fact i often see bar 4 for opening a trade and bar 1 for closing. This means we are very fussy going into a trade but get out very fast.

In fact you only really need 2 or 3 standard indicators like CCI MACD RSI plus stop and fixed tp's then optimise everything on a year of ticks and most simple 1H swing systems will take 10k to 100k over the year quite easily when use compounding. Risk 30% drawdown 35%. I call that a standard to work to at least on 1 year backtests.

Question is will that system work for 2007? mmm maybe maybe not

I don't understand at all why you don't use the current bar to make your orders... Plus, you can't open a trade with a past price, you must use current Bid/Ask. Well, I don't see what you're driving at.

I've sent you a private message about shifting.

Mark