Plot News / Downloader version 4.00

I'm replying to the quoted post, but also to anybody else who's totally bored interested..........

I'll leave post #320 intact, as it contains some good info, but this new post effectively supersedes it.

I've updated the 'Convert news to GMT' app. Download it from post #1 (latest version is in the link in the green typeface, and you'll find the app in the .../MQL4/Files folder in the zip file). Windows 10 Defender will probably display a virus warning when you try to run it, then it's up to you whether or not you want to take the risk.

Please read the following instructions carefully. The entry screen now looks like the one in the screenshot above. The calculations for FF calendar (New York) timezone (EST or EDT) continue to be automatic. The settings that you enter determine the shift for your LOCAL timezone, for when daylight saving starts and ends. This is perhaps best explained using an example:

Suppose your local daylight savings begins on the first Sunday in April and runs through to the second Sunday in October. Then you would enter 4 (for April) in the 'From Month' and 0 in the 'First Sunday + days', because it is 0 days from the First Sunday.
 If DST began 7 days later, i.e. on the second Sunday, you would enter 7 for 'First Sunday + days'.
 If DST began 14 days later, i.e. on the third Sunday, you would enter 14 for 'First Sunday + days', and so on.
 If DST began 7 days EARLIER than the first Sunday, i.e. on the LAST Sunday in March, then you would still enter 4 for the 'Start Month', and -7 for the 'First Sunday + days'.
So you are ALWAYS working from the first Sunday in your nominated month, and then adding or subtracting a multiple of 7 days to advance or retreat it to a later or earlier Sunday.

The 'To Month' works the same way. Hence if your local cutoff date for DST was the second Sunday in October, you'd set 'To Month' = 10 and 'First Sunday + days' = 7
NOTE: The 'To Month' MUST always be a higher value than the 'From Month'. Failure to do this will cause unpredictable results.

Not confused yet? Read on.

The FIRST 'GMT/UTC + hours' field represents the number of HOURS ahead (enter a positive number) or behind (enter a negative number) GMT/UTC your local time is, during the period of the year that is between the 'From month and the 'To month' (i.e. during the MIDDLE part of the year).
The SECOND 'GMT/UTC + hours' field is the number of hours ahead or behind GMT/UTC your local time is, OUTSIDE of the 'From month and the 'To month', i.e. at the START and END of the year.
 Hence if your daylight saving (summer) happens during the middle of the year (in the NORTHERN hemisphere), the value in the first field should probably be 1 hour GREATER than in the value in the second field.
 But if your daylight saving (summer) happens at the end of the year (in the SOUTHERN hemisphere), the value in the first field should probably be 1 hour LESS than in the value in the second field.
These values are NOT the number of hours your local time differs from FF calendar (New York), they are the number of hours your local time is ahead or behind GMT/UTC.

NOTE: In ALL cases, to input a positive number, just type a number WITHOUT the + sign, e.g. 9; to input a negative number, type the number with a LEADING minus sign, e.g. -9
The 'GMT/UTC + hours' field may contain a decimal entry, e.g. for 5 hours and 30 minutes behind GMT/UTC, type -5.5 (because 30 minutes is 0.5 hours)

 The default values of 4,-7,2,11,-7,1 should work correctly for any location that shares the same daylight saving rules and times with Oslo, Norway.
 What if your locality does not have DST? Example: if you live in Hawaii, which is GMT/UTC -10 hours all year round, then your inputs should be 4,0,-10,11,0,-10 (the 4,0 and 11,0 are 'dummy' values that meet the entry requirements)
 If you live in New Zealand, as I do, then your inputs should be 4,0,12,10,-7,13 [so, just to recap, these six values mean that from 0 days after the first Sunday in he 4th month (April) through until 7 days before the first Sunday in the 10th month (i.e. this equates to the last Sunday in September), NZ's timezone is GMT/UTC+12 hours; but for the rest of the year, it is GMT/UTC+13 hours. NZ is in the southern hemisphere, hence its DST occurs at the END of the year.]
 FYI, the values for FF calendar (New York), which are hardcoded into the software are 3,7,-4,11,0,-5

Hopefully that's enough examples.

To find out your local daylight saving rules, go to World Clock, click on a city that's in the same timezone, and choose the 'Time Zone' tab (see screenshot below). You should be able to work out which Sundays daylight saving starts/ends, and how many hours you're ahead/behind GMT/UTC.
If the above instructions aren't clear, then if you let me know your city/location, I can calculate the six values for you.

The entire FF calendar (news events dating back to 2007) can be downloaded from post #1. You can now use the 'Convert news to GMT' app to convert all of the event times to your local timezone (or any other timezone that you wish).

hello.
this inicator is amazing thanks to all involved in supplying it to FF.

i have just noticed today that newsdownloader has managed to scrape some news events that dont actually appear on the FF website calander.
for example today 28/1/2020- GBP nationwide hpi and CNY foreign direct investment are shown on my mt4 charts at 1:01 (NY) but are not shown on FF website calendar.
Why is there news events that dont show up on FF but newsdownloader still manages to find them somewhere?

sorry for the random question but its actually quite important for me to know the answer.
thanks

ForexFactory has some internal database of news events, including times that are a combination of Expected (most); Tentative (a few); "All Day" (i.e. Holidays); and then Actual times, after the news event occurs.

They generate the Web calendar in the order of news events that their *internal* database knows about, but they don't always *say* (in the HTML file) what exact time of day is their own estimate. When it says "Tentative" or "All Day", that's all that the News Downloader knows, not any specific time of day, nor even the rough estimate of when that may be.

So the NewsDownloader / Plot News assumes "00:01" (12:01 AM) for what would appear to be the same day, but the day is (likely) measured as the mm-dd-yyyy in New York (but always as Standard time; that is, UTC-5). Worst case, it might be off up to <24 hours.

On the other hand, the XML file usually has a better time estimate in the database. The XML files have different offsets from the UTC-5 HTML file. Weirdly, it seems to me to be NY+5, which follows the US DST schedule. That means it's "UTC" in US standard time, "UTC+1" in US DST time.

The downside of the XML is that it has no "Actual" news results. It has only "Previous" and "Forecast". So it is probably better for knowing what news is coming up, but worse for knowing what Actual values came in when the news is over.

I suggest to use unique filenames for the two styles, and as already mentioned, it requires different offsets. (Likely the offsets need to change, depending on your broker data, what DST it follows; and depending on the USA ST/DST schedule). You could display the HTML source in a sub-window, and the XML source in the main window so you can compare (or vice-versa).

One other fact: Both the XML and Download News HTML file get updated throughout the week with new actual *times*, once the news is past, or even for a few events which earlier in the day are marked as "Tentative", but later on are marked as some specific time-of-day. I have no idea how often this occurs.

I notice that in an older Download News news.csv file, the GBP nationwide hpi was on 1/28/2020 7:00 UTC (It *may* have been marked as "Tentative", but I'm not sure). But by now, it's on the calendar still at 7:00 UTC but a day later, 1/29/2020. (~2 hours from now). So either it was an estimated time, or FF made a correction to the date&time.

Hanover has done the best he can with the HTML file. It simply doesn't have enough details to know better estimated times. Use the XML method instead if you want better estimated upcoming times, but there is no clear definition for "All Day", or other "Tentative" times, so even the XML is not perfect either.

Just roll with it...

Kent (pips4life)

P.S. What I think are the timezones of the HTML and XML news files might be wrong. I'll have to double-check after March 8th when USA resumes DST.

They appear on mine (see screenshot). Check the filters you've applied to the calendar.

Also, note that FF are continuously revising their calendar; news events are getting added, deleted, rescheduled all the time, supposedly as FF's source of news data reports these changes. You can see the effect when, after running the downloader, you refresh the Plot News indicator on your chart: some little circles appear, disappear, move around.

The News Downloader app merely grabs whatever data is in the FF calendar HTML at the time it's run.

Your FF profile says that you live in NZ. I've lived in Hamilton since 1972.

I forgot to mention that I fixed a bug (thanks to info provided by pips4life) in the 'Convert news to GMT' app. You can download the latest version (v 4.16) from my github page (see the green typeface in post #1). This app will convert data extracted by the News Downloader to any timezone you wish, including an auto adjustment for daylight saving. Settings that you enter are now saved so that they remain 'sticky' from one run to the next. I've also (hopefully) made the entry screen more user-friendly:

Awesome guys thankyou for your replies they are very helpful.
@Hanover , Hamilton is a great place to live ,, I live on the border of Pokeno so im practically an Aucklander .
Great to see another kiwi on here though..

Hi Hanover,

This indicator, like the others I've recently found of yours, is fantastic. I am very grateful for finding your generous work! I spent the better part of today reading up and getting it up and running on my platform. I'm near the tail end of what I'm trying to do but coming up short.

I am building a UI trade panel that allows me (among other things) to enable/disable custom indicators on demand. I would like to have two check boxes on my panel dedicated to the Plot News indicator. For one, I just want to print the circles on the chart. For the other, I want the bubbles as well as the text feed on the rectangular background. (I don't want the second part there all the time because it covers up too much space on my chart and can be a distraction.) I know how to make the indicator do either of these things with your provided parameters as well as by parameter file.

My issue is that I need to have two instances of the compiled indicator .ex4, each one with a different ParameterFile parameter hard-coded (e.g. 01 in one copy of the indicator, 02 in the next, etc). This would be trivial if I could edit the code and recompile, but I gave that a shot and it's not working. (I can get it to compile after changing a handful of 'char' references to 'sChar', but the new compiled indicator fails to plot anything. No errors on loading into the chart, it just doesn't do anything.)

Do you happen to have updated source that compiles easily in modern builds of MT4? Or, as a workaround, could I possibly trouble you to pass me a few compiled files, each with a different "ParameterFile" param hardcoded (e.g. maybe something like 01, 02, thru 05?), so that I can use them to point to various parameter text files of my own? If you are willing to do this, can you also make the indicator's "shortname" a simple constant which has the parameter file code appended (for example, resolves to "Plot News v4-00_01", "Plot News v4-00_02", etc, to match the parameterfile code)? I am open to any other ideas you have as well, as I hate asking for you to do something special for me as a one-off, but that was the only idea I had.

Extremely grateful for your consideration!

Best,
FtN

@hanover, gentle ping just in case you didn't see my post 327. Thank you for your consideration.

Some time ago I attempted to create a version that would compile in MQL4 b600+, but wasn't successful. The main issue was that the ReadWebPage() function (which is needed to get timezone offsets from http://www.worldtimezone.com/time/wtzstandard.php) that worked ok in build 509 now fails. In the end I gave up.

I looked at the MQL website to find an alternative to ReadWebPage(). Apparently the solution is WebRequest(), but this can be used only in EAs and scripts, NOT indicators. Hence I would need to convert Plot News from an indicator to an EA.

Thinking more laterally, another potential approach would be to get the downloader to scrape the relevant webpages into a file for the Plot News indicator to read. I'm not sure how much work would be involved. This would mean that the downloader would need to be run, even if SourceDataFromXML is set to true, which is somewhat self-defeating.

I ran some quick tests and it seems that there are other issues. For starters, MQL4's StringReplace() function in b600+ is at worst buggy, and at best doesn't work the same way as it did in b509. I would need to write my own function to replace MQL4's.

Hence it would seem that there is a significant amount of work involved. I'm busy this weekend but I'll look into it further if/when I next get some spare time.

Plot News for MQL4 build 600+ is now ready for beta testing. This necessitated a major overhaul of the b509 code.

Plot News v4-00 b600.mq4 is an EA, not an indicator, hence it must be installed from the attached zip into the /MQL4/Experts folder. (I had to make it an EA because the WebRequest() function returns error 4060 if it's run as an indicator).
Install the Plot News subwindow.mq4 indicator in the /MQL4/Indicators folder.
Install the Presets---Plot News.600 file in the /MQL4/Files folder.

Note: before running the EA, you'll need to type Ctrl-O and then add the following two webpage URLs as shown in the screenshot (copy and paste them to make sure you get the spelling exactly correct):
https://cdn-nfs.faireconomy.media/ff_calendar_thisweek.xml
http://www.worldtimezone.com/time/wtzstandard.php?sorttb=CountryID&listsw=capitals&forma=24h

You must also have the 'smiley' face in the top right of the main chart window, for the EA to run correctly (more info here).
Parameter Changes

 If you want Plot News to plot the symbols and upcoming events table in the main chart, set the new Subwindow parameter to 0.
 If you want PlotNews to plot the symbols and upcoming events table in a subwindow, you must first drag the Plot News subwindow indicator onto the chart to create a 'dummy' subwindow. Then set Subwindow (in the EA settings) to 1,2,3 etc, which is the number of the subwindow underneath the main chart (the screenshot shows plotting in subwindow 1).

 If you want Plot News to source its data from the FF calendar html (which allows plotting of multiple weeks of past data, and also historical 'actual' values), set SourceDataFromXML to false. You must also run the News Downloader.exe app periodically, to download any new or updated events on the FF calendar, since it was last run. Then press F7 and click OK to refresh the plot.
 If you want Plot News to source its data from FF's xml file, set SourceDataFromXML to true, and the first entry in the TimeOffsets to 0 (because events in the xml file are timestamped GMT/UTC+0).

You can try setting the experimental feature TimeOffsets to auto, which uses data from the webpage http://www.worldtimezone.com/time/wtzstandard.php to calculate the offsets, but if you can't get it to work correctly, just enter the three values (New York timezone, broker timezone, local timezone) as before.

Everything else should work the same as the b509 version. For more info, read the user manual pdf.

If you compile the Plot News v4-00 b600.mq4 EA, you'll likely get something like 41 warning (yellow) errors. These can be safely ignored.

I'm unfamiliar with MQL5, but the source code is there if anybody else wants to re-code it.

If no bugs are reported over the next couple of weeks, I will include these new/updated files in the zip in my github repository (link in post #1).

If you wish to report a possible bug, be sure to include a screenshot, a comprehensive description of the problem, the MT4 broker/build, and a list of all of the parameters that you were using. That's the minimum info I need to troubleshoot.

Attached File(s)

Plot News build 600.zip   454 KB | 488 downloads

Attached File(s)

Plot News v4 - operating instructions.pdf   400 KB | 3,105 downloads

____________________________

[EDIT] Further notes:
 Because Plot News is an EA, to remove it you must right-click on the chart, then select Expert Advisors => Remove.
 Instead of using parameter files, you now have the option of setting ParameterFile = NONE, and using MT4's Load/Save buttons to save and recall your settings. I've updated the attached zip with a plotnews.set file (in the /MQL4/Presets folder).
 You can have Plot News overlay other indicator(s) in any subwindow, if you want to save screen space. For example, if you have RSI loaded in subwindow 1, then you can set Subwindow = 1 in the Plot News v4-00 b600.mq4 EA, and it will share the subwindow with RSI. In which case there's no need to run the Plot News subwindow.mq4 indicator.
 If you prefer, you can still use the old Plot News v4-00 (b509) indicator. It works exactly as it did before.
____________________________

@FtNTrader: sorry I didn't see your post earlier. Instead of bookmarking threads, I do a search for the keyword 'hanover'. Hence if you include 'hanover' somewhere in your post, I'm likely to find it more easily. Also, posts in Commercial Content and Recycle Bin threads don't seem to create an alert in the Notifications area on my profile page. Having said all that, I spend less time on trading forums nowadays. My wife tells me that I spend too much unproductive time here, and as usual, she's absolutely right.

@hanover, this is awesome. I have confirmed that (A) the compiled .ex4 works in my modern MT4, and (B) the .mq4 compiles and the resulting .ex4 still works! I'll plan to dig in this weekend and see if I can get it setup exactly how I'm looking for. Moving the code to an EA gives me a new challenge to look into but will see if I can integrate it with my current EA or just run it in a different chart window effectively for my needs. Will reach out with an update as I get further along. Thank you so much!

[UPDATE] @hanover, again, thank you. I integrated your code into my existing EA and it's behaving exactly the way I was looking for. Thanks so much again, and please thank your wife for me as well. I don't know whether your time on this was productive for you, but it was very productive to me, and I am grateful and in your debt. Thank you again!

@hanover, a couple of issues, perhaps for user error or enhancement that proper bugs, but for #1 in particular I am not sure:

1. First screenshot of param file, see highlighted portion. I am expecting to see the time of the event in my news info screen, but there is a 3-minute offset. On the broker's install of MT4 from which these screenshots come, it is 3 mins late, on another it is 2 mins late. The 'countdown' field is correct (parameter value T, 2nd column in screenshot2), but the first field for time of event is incorrect (parameter value S, 1st column in screenshot2). Furthermore, I see single quotes following the first column in screenshot2. (Note, I had the same issue with the 2/3-min time offset in the prior indicator version, but I did not have the single quotes problem there)
2. Is it possible to add labels to the top of the columns?
3. When I add the "A" value to the UEItemsDisplay parameter name, I never actually see Actuals appear on my news feed. My guess is that it is not designed to support this because the news item disappears as soon as it is no longer "upcoming," but it would be nice if I could have recent events linger for some amount of specified time so that I can see and digest the actuals vs forecast.

Best, FtN

In reply to your numbered points:

1. I'm not aware of the '3 minutes late' issue. Will attempt to troubleshoot after market opens. Check your RefreshPeriod setting. The Upcoming Events table and alerts are based on your computer's clock time, not MT4 or FF calendar time. Entry in UEItemsToDisplay should be S=H:IA (M=month, I=minute).

2. I have included headings in the next release (see screenshot).

3. You're correct, the rolling table is designed to show upcoming events only. That means that options A ('Actual') and R ('Revised From') are actually pointless -- what was I was thinking?!? I guess you already know that you can hover the mouse cursor over the little circles to show Actual/Forecast/Previous/RevisedFrom values (after running the downloader to update with the latest values from the FF calendar). Sometimes FF doesn't update the values on their calendar until 5-10 minutes after the event (there's a little green circle on the calendar instead).

Please remember that I don't get tangible reward for any of this. More than 300 hours' work has gone into the development of this software, over 4 revisions and almost 13 years (version 1.0 was initially released in 2007, 2.0, 3.0). Hence software is provided on a "take it as it is, or leave it" basis. mq4 source code is provided if anybody wants to make their own enhancements.

Finally. . . . . . . I've built a downloader in MQL4 (build 600) into the Plot News v4-00 b600 EA. Hence the whole process of scraping the news from the FF calendar html, right through to the plotting of the little colored circles, the alerts for upcoming news, and customized outputs to files for further analysis, is now all self-contained within one EA. The downloader function is not as feature rich as the trusty old NewsDownloader.exe; it can only download only up to 6 weeks' news events at a time, each week being Sunday thru Saturday. But you can run it repeatedly to obtain and plot several sets of weeks' news, and better still, the 'actual' and 'revised from' values are available because this is the only news reporter that reads the actual FF calendar html, rather than the ff_calendar_thisweek.xml file.

The EA is made available here for beta testing. I'm undecided as to whether I'll release the .mq4 source, as I've spent several hours working on it, and I don't want to run the risk of my efforts being sold on ebay (see this). Once source is released, you can never take it back, and you have no control over how it will be distributed and abused.

Following on from the enhancements in post #330, the new parameters work as follows:

DownloadNewsFromCalendar defaults to false. If true, the EA scrapes data from the Sunday that's most recent to the date you enter in StartSunday.
NumberOfWeeks is the number of weeks to download, moving forward in time from (and including) the week of the StartSunday. Valid entries are between 1 and 6 (inclusive); if you enter a number > 6, only the maximum of 6 weeks will be downloaded (to avoid risk of overloading FF's servers). Depending on how busy your charts and FF calendar are, downloading takes anything from 5-30 seconds per week's worth of data, during which time the plots disappear and the comment "Downloading news. . . ." appears in the top left of the main chart. (I just downloaded 3 weeks' worth of data while the markets were closed, and it took 19 seconds). Everything gets re-calculated and re-plotted immediately the download finishes. If you simply want to change some parameter settings, you can reset DownloadNewsFromCalendar to false, and save yourself an unnecessary wait; there is also a confirmation message that lets you bail out.

You can also run the process several times to gather data for more than 6 weeks, by setting AppendToFile to true for the subsequent sets of weeks. If you want to start over, set AppendToFile to false, and the NewsInputFile (into which the calendar events are downloaded) will get cleared at the start of the process. It doesn't matter in what order the weeks' data is downloaded, although it's probably both safer and more logical to proceed from earliest to latest. If you download the same weeks repeatedly with AppendToFile = true, the data gets repeated in the news.csv file (it's not as smart as NewsDownloader.exe). Currently the only remedy is to either clear the file (set AppendToFile = false) and start over, or edit the news.csv file using a text editor like Notepad.

ReplacementTime and PreviousDay fulfil the same function as 'Set time for untimed events to' did in the inputs screen for NewsDownloader.exe. Input the ReplacementTime in the form hh:mm, and set PreviousDay to true if you want the plotted circle to appear at a certain time on the previous day (usually midnight MT4 time).

All of the Include…..Events filters work the same as in the NewsDownloader.exe, and should be self explanatory. Excluding types of events from the download keeps the news.csv file smaller, making both the download and the plotting faster and more efficient. However, you can still use the PlotTheseCurrencies and PlotTheseImpacts to exclude downloaded events from plotting, if you prefer to do it that way.

The download process sends output to both NewsInputFile (which defaults to news.csv) and also debug.txt, for troubleshooting purposes. If you want me to troubleshoot a possible problem, you must provide both a screenshot and the debug.txt file as a bare minimum, to give me a starting point to work from. If you can't get the news download facility to work, you can always restore the old Plot News v4-00 b600 EA from the zip in post #330.

There is also a new setting UEHeadingsText. If non-blank it will print headings above the rolling Upcoming Events table. It is free format and will print up to 63 characters. You might need to experiment to get the headings to align perfectly over your columns, but I did it this way to allow for maximum flexibility (you might want to abbreviate some of the text to fit the columns, for example). Font name/size/color is determined by the UEFontNameSizeColor setting.

I have also added SendPushNotification, as build 600 allows this option. You must complete the 'Notifications' tab fields for this to work correctly (type Ctrl-O for MT4's options entry window).

Parameter and preset files have been updated accordingly, and are included in the correct folders in the zip.

Everything described in post #330 continues to work as it did before. Read post #330 and the instruction manual for more information. I will update my github repository (see link in green typeface in post #1) with the latest version in a couple of weeks.

@hanover Understand 1000% and only have hopes, never expectations, with regard to any question I ask or request I make. Extremely grateful for your ongoing sharing of your incredible work.
Also totally understand with regard to your wanting to be cautious on releasing code on an ongoing basis, however, for me personally, I can't use it without the code because of how I've had to make tweaks to integrate with my own custom trade panel. After many hours, I finally have your download and chart-plot functions (from initial build 600) entirely integrated with my panel. While I would love the chart header and time-fix enhancements, I won't be able to use them without the code. And I totally understand if that's the case and am no less entirely grateful for all your sharing.

Best and thank you again,
FtN

Not sure if you are busier now because of the Covid-19 situation. But if you did have some free time and wanted to experiment, you could consider seeing if your newsCalendar.exe could be repurposed to scrape EnergyExchange and MetalsMine calendars (CryptoCraft does not yet have a calendar)

Likely you would have separate html templates (.FMT files) for each calendar.
And the ini file would have some new configurations, like

CalendarUrlLink=https://www.forexfactory.com/calendar
FMTLocation=[\path\to\.fmt file]
ScrapePattern=01 [to distinguish from ff scrape vs EE/MM scrape pattern]
CountryToCurrencyList=[\path\to\.txt or .csv file]


And of course, eliminate the expiration date. I think the exe has some sort of hard coded scrape until date for the calendar. Or set that date year to a large # like 2099.

We discussed this a little while ago (see post #303). It appears that there would be a significant amount of work involved.

Meanwhile, it looks like Plot News downloader is creating an empty news.csv file. It started happening around last Thursday. I see that FF have changed their calendar URL from https://www.forexfactory.com/calendar.php to https://www.forexfactory.com/calendar
I changed the code to accommodate this, but it still created an empty news.csv

I'm still troubleshooting. It is bizarre: autohotkey's URLDownloadToFile command works perfectly when downloading other sites, but no longer for FF calendar, and I don't know why. Perhaps FF have changed the html or its format.

In bygone days, the downloader would call curl.exe to obtain the html (and prior to that it used wget.exe). I switched back to curl and got a successful run!!

Not sure if anybody else has been experiencing problems like this. Nobody has reported anything here. Makes me wonder if anybody is still using Plot News.

Anyway, I'm continuing to try to get to the bottom of the problem. Will report again if/when I discover anything significant.

Take care and stay safe.

Hi Hanover,

This Jan post of mine was regarding my observations of the timezone offsets for the Download News (news.csv) vs. the weekly XML file.
What I had concluded was various offsets and ST/DST schedules for each file, but I was *wrong* with my conclusions. It turns out the real story is much more complex.

My present observations are:
* The XML source follows UTC year round. (Previously I had thought it was UTC in winter, UTC+1 in summer, but it's not).
* The news.csv (Download News from HTML) follows NY time, based on *when* the news is downloaded!! (I did not anticipate this).

If downloaded during winter, the entire news.csv follows NY ST.
If downloaded during USA** summer, the entire news.csv follows NY DST.

**Note: Unfortunately, I didn't do any tests right after March 8th but before Mar 29th when Europe changed to DST. It's likely that FF follows the USA schedule, but I guess it's possible they could follow the Europe DST schedule. Regardless of which, the entire data downloaded follows whatever the ST/DST is, either on or off.

Unfortunately, this strange variation makes a bigger mess of the news_archive.csv file, which has a combination of what was previously downloaded, plus what the user just downloaded.

Example: Looking at your archived news events you have in post#1, use a grep tool (e.g. AstroGrep) to search for all the lines with "Unemployment Claims". (See picture). Handily, this matches a regular USD event that occurs every Thursday at 8:30A NY (local) time.

You will find that from 2007 - late 2018, the times given are either 9:30 or 8:30. I conclude that you downloaded the bulk of these during USA summer. The "9:30" would be a NY DST time but on days during the *winter*, which does not occur.

Then in 2018-2019, your archive gets real messy. You have times of 7:30, 8:30, or 9:30. I think you may have archived some period during USA winter, some during summer. The file, therefore, has an inconsistent timezone.

Today I downloaded all the news from 1/1/2007 to present. The entire file has 9:30 and 8:30 times, which is NY with year-round DST. I'm sure that if I were to do the download during the winter, the times would instead be 8:30 and 7:30, NY with year-round ST.

The way in which you append/merge the news_archive.csv file now gets very tricky, and the way in which a program must determine the timezone also gets a lot harder.

One idea is for Plot News to calibrate each week's TimeZone based on when the Thursday 8:30AM "Unemployment Claims" event occurs, which in the .csv is going to always be either 7:30, 8:30, or 9:30. From that info, one can assume that the entire *same*week* follows the same schedule, and offset all news events for *that* week accordingly. (What a pain to code, and this is really a hack to base it on a specific weekly data event. Yuk).

Another (better?) approach would be for the Download News program (or MT4 equivalent) to do a test for each week of data, and convert all the times to something consistent, e.g. to the true UTC (to match the XML!). Or, to the true NY time (e.g. all USD "Unemployment Claims" would occur at 8:30am each Thursday). The downloader can use a Windows DLL with time functions to check whether the USA is -- at that moment-- either ST or DST, and adjust the newly downloaded events accordingly.


I've attached a picture with the inconsistent times for USD "Unemployment Claims" in my news_archive.csv, which started with your original archive source and was later added to by me when I ran the Download News.


Just bringing this to your attention for your awareness, and to ask what you think may be possible for the downloader(s) to do to fix this?

Thanks for all you do!

Kent

Hi Kent,

Many thanks for your post.

The anomalies in the output data don't surprise me; daylight saving is a labyrinth. However I'm not looking to fix the downloader at this point; there are plenty of workarounds allowing users to adjust the news times. The system works well enough for me the way that it is, and it would take some time to fix and test, especially as I find the whole daylight saving business something of a headache. I sent you the source code (privately) for Plot News v4-00 b600, which includes a download facility. Please feel welcome to make any changes that you feel are necessary (and also share the .ex4 file publicly, if you wish).

I don't use the https://cdn-nfs.faireconomy.media/ff...r_thisweek.xml file. I believe you're correct: the times are GMT/UTC regardless of whether New York is on summer time or not. I prefer to get data from the https://www.forexfactory.com/calendar.php html source (Plot News v4-00 b600.mq4 includes the code to achieve this).

I've just uploaded a new version of the Plot News zip to github. All existing files are unchanged — I haven't yet found time to address the issues in post #337 — except that there are updated versions of MQL4\Files\Convert news to GMT.exe (see posts #321, #325 for instructions) and a new app named MQL4\Files\Adjust news times.exe, which simply adds (or subtracts) any specified number of hours from all news events between the dates that you specify. These apps allow you to make block changes to times within news.csv (or any news file that you specify), however you wish. Beyond which there's always Excel, or a text editor.

Because I can't possibly know the timezones of every user and their broker's MT4, Plot News has always been flexible enough to allow users to enter their own TimeOffsets into its parameters, to get the news symbols to plot under the correct candles, and hence fix any daylight savings issues in the process. MQL4\Files\News Downloader.exe also has a '+/- hours shift' parameter.

Hope that you're enjoying the Easter break.

David

Can someone help me please, when i try to download forex calender i got this error