Plot News / Downloader version 4.00

WUPS guess that's it! There's no NewsDownloader anywhere.

Try https://github.com/4evermaat/news_do...News.v4-15.zip
dropbox seems to limit downloads, especially on free accounts.

----------------------------------------------------------------------------------------------------------------------

@Hanover
Since FF's parent company now has two more sister sites, MetalsMine and EnergyExchange Calendars that appear to be formatted identically, is it possible to put the calendar url(s) in the ini file? This way, the scraper can be repurposed for the other calendars.

I suppose the plot news MT4 indicator could also use a similar update.

And of course update the pdf or provide a post/text to explain how to scrape the other sites.

Hopefully you can share your thoughts before the new year. Thanks.

Imo your suggestion re FF's sister sites' calendars is a good one, but there could be more work involved than merely alternative URLs. Just for starters the entries in the Country column are different, and that would mean new mapping and wingdings symbols for Plot News. Also, it's possible that there might be subtle variations in their html format. Also, some of the announcements are repeated on different calendars, hence (assuming duplication of plots is undresirable) each attached instance of Plot News would be able to display events for one calendar only.

I don't trade metals or energy myself, so an enhancement wouldn't be useful to me personally. However, I'll investigate further in the new year if/when I get some spare time....... but no promises.
___________________

[EDIT] I changed the URL in the downloader program code and it downloaded both the metals and energy announcements correctly, but the news.csv file was empty. Hence there are compatibility issues somewhere within the processing. I'll investigate later.

@hanover

Merry Christmas and Happy New Year mate!

it looks like the News Downloader is have the "Unable to Process Calendar" issue again.

If you want to look at it when you have some time to check it out and let me and @forevermaat know, we can update our repos?

S O S to David.


Season Greeting my good man.


Pls help when u have the time.



News Downloader has the usual error message when FF changes format.

Not a good start to 2020.

The year is not obtainable from the body of the FF calendar, and at the start of each new year I have to manually update the code. I will patch the code temporarily** once again, and then fix it permanently later, when I get some time.

** I've attached the ZIP file to post #1, because I can't email EXEs, nor ZIPs that contain EXEs; and will remove it after a couple of days, before FF discovers the (non-existent) "virus" again, and removes it themselves.

I have a programmer friend who is willing to upload the ZIP files to his Google drive. Let me know if this is helpful to anybody, and I'll contact him. You would need to create an account with Google, and also email him so that he can grant you access to the relevant folder.

It seems that all of the code I've written using AHK (this also) gives a false positive virus warning. If anybody knows which virus checker is causing this problem, please let me know, so that I can run some tests myself to establish which part of my code is triggering it (although if the fault lies within the AHK compiler itself, there's probably nothing I can do about it). Thanks.

Happy New Year folks! I see requests in the other news reader threads for the plotting of past 'actual' announcement results, but this can't happen as long as the code continues to obtain its data from the xml. Plot News avoids this shortcoming because it reads the calendar html. I'm surprised that nobody else has attempted something like this.
________________________________________

Many thanks for the info, and for your kind words. I have a Win10 laptop, I will try running the EXEs on it and see what happens.

Best wishes and many pips to you also.

Hi Hanover, for me it was Windows Defender (with Windows 10). When I updated Windows Defender it then passed ok, if my memory serves. Plus ok when ran though other checkers. If you look at your myfxbook indi thread, the more recent post on this again mentions Windows Defender (& Windows 10) as generating the alert.

Best wishes for 2020. Thanks for all your great posts

Hey @hanover

Thanks for the update mate. If you want you can update my repo on Github whenever you like? if you want to do that i would be more than happy to have it there.

I've just downloaded the upcoming weeks' news on my computer without any obvious issues (see news plotted for Jan 3 in screenshot). However, I've received a couple of emails saying that the downloader is still failing. Can anybody confirm whether the version that I uploaded to post #1 recently is working ok?

Thanks,
David

No, Not working.
it still loads last years news but nothing shows up on chart for jan 2 and jan 3.

(with a few new minor edits)

Hi David,

I have been using an older version of Recent News that I modified. I almost only use it to read the XLM file, which means I see only the current week of news events. (I do like to study past events, so the downloader -- or your archive -- does come in handy).

I already have some of your news files on all my MT4 installations (some I've customized), but I did just overlay your latest files into one of my MT4 folders.

Observations:

1. Your "Plot News v4 - operating instructions.pdf" says nothing about what to do with the Consolas Font you provide. Maybe I already did it long ago (or it already exists on my computer), because I can display it fine. But... the installation instructions have been lost, so you may want to comment in the PDF file -- and here in the thread -- what users should do with that file.
2. Fyi, I have Windows 10.
3. When I dragged the script "Download News v4-00" onto a chart, nothing happened. No popup, no action, nothing. Initially, it did not work.
4. I then went to the MQL4\Files\ folder, and double-clicked on "News Downloader.exe".
5. IMPORTANT: Running it was initially blocked by Windows Defender! I allowed it to run and got the downloader popup. After the first time, I can re-run it again without it being blocked.
6. NOW I can run the script to download news. It now gives the same popup to download however many weeks I specify.
7. I added "Plot News v4-00" to my chart. The most obvious problem is that the ThruDate default is "2019.12.31 23:59". This is likely the problem for people not seeing news for >= Jan 1st.
   
   I suggest to make it "2038.01.17 23:59" (which is within hours of the max allowed, 2038.01.18 ??:??). No point to have a default that will expire and fail to show current news.
8. NOTE for users of templates: You will need to update your template to fix the 2019.12.31 expiration! Even if David provides an update with a new default ThruDate for the indicator, it still won't fix your templates, so you'll have to do that yourself.
9. I noticed I need to change the TimeOffsets to display the news at the right spot on my chart. (For me, at the moment non-DST: -5,0,-8).
10. For the above, it displays the events on my chart just fine.
11. When I download news, there is a minor problem with the popup window that gives instructions to use Ctrl-J. In fact, my monitor is set to 125% instead of 100%. This scales fonts larger, but when it does, the width of your display box with the message is too narrow. I only see:
    "Now go to MT4, select a chart with Plot News, and then press" Uh, press what??
    
    What I should see is the full text:
    "Now go to MT4, select a chart with Plot News, and then press Ctrl-J"
    
    There are now ways to detect the monitor scaling setting, and then scale your stuff accordingly. It's a lot of work, but an easier workaround would be to just make your text box wider than necessary for the 100%, so that it works for 125% (or more??).
12. Unfortunately, this Ctrl-J does nothing for me. It doesn't refresh my chart. I have to change Timeframes, for example, then it works. Am I missing something??

1. I next attempted to use the XML file by setting SourceDataFromXLM=true (default is false), but it did not work! The URL is bad. "Unable to find requested URL"
2. The PDF says the URL is (was) http://cdn.forexfactory.com/ffcal_week_this.xml
   
   Since you don't provide .mq4 source to all, I suggest you make the URL an extern string variable. The current value is:
   https://cdn-nfs.faireconomy.media/ff...r_thisweek.xml
   
   I am aware of the Tools -> Options -> Expert Advisors tab -> CHECKBOX to "Allow WebRequest for listed URL", under which one can specify specific URL's to allow... but I haven't customized this, and yet my own version that displays the XML file works just fine. But this is something for users to be aware of if they have problems downloading the URL.
   
   BY THE WAY... in my version, I added a 2nd independent filename var to use when XML=true. I use "newsXML.csv". That way when I toggle between the methods, it doesn't override the manually downloaded "news.csv" file, which I want to keep. In fact, when I attempted to use yours to download the XML, not only did it fail, but I was forced to re-run the download script again to show the news with XML=false.
   
   In my version, I also separately can specify TimeOffsetsXML when the two different methods have different offsets. (Actually, I've more or less automated the offsets, though I intend to do some additional work on the method).

I am still using your News indicator after all these years, so thank you very much!

Eventually, I'll get around to implementing a few more ideas I have on the topic, but for now, it's a lower priority to other work. I would point out that some other news plot indicators have apparently figured out a way to avoid any separate news downloader script. It's all built-in to the single indicator. I have no recommendations, especially since some cost money (and I won't be able to customize it as I want without source), but one can search the mql5 Market, for MT4 indicators, with keywords "news on chart".

Thank you, David,

Kent (pips4life)

Many thanks for your reply, I don't have time to investigate at the moment. See my reply to pips4life below for more info.

Hi Kent,

Your post appears daunting for a tired old man like me. So many items to investigate and fix! Recent News has been obsolete for years, it's a bit like asking Microsoft to overhaul Windows XP. The entire history behind Plot News/Recent News — how/why/when it was rewritten — is detailed in an ealier reply to you here (and answers to your other points here). Since then, the FF admins claimed that the AHK-written downloader contained a virus, and removed it.

David

_________________________________________

[EDIT]
Some (belated, sorry) responses to your points:

Your "Plot News v4 - operating instructions.pdf" says nothing about what to do with the Consolas Font you provide.
I prefer this to Courier New for displaying the upcoming items in the rolling window, but you can use any (non-proportional) font that you like.

When I dragged the script "Download News v4-00" onto a chart, nothing happened. No popup, no action, nothing. Initially, it did not work. I then went to the MQL4\Files\ folder, and double-clicked on "News Downloader.exe". IMPORTANT: Running it was initially blocked by Windows Defender! I allowed it to run and got the downloader popup. After the first time, I can re-run it again without it being blocked. NOW I can run the script to download news.
See the manual, page 3, 'Installation' point 3.

It now gives the same popup to download however many weeks I specify. I added "Plot News v4-00" to my chart. The most obvious problem is that the ThruDate default is "2019.12.31 23:59". This is likely the problem for people not seeing news for >= Jan 1st. I suggest to make it "2038.01.17 23:59" (which is within hours of the max allowed, 2038.01.18 ??:??).
Thanks for pointing this out. Please feel welcome to change the date and update your template. I have updated the default ThruDate in the .mqh to '2029.12.31 23:59:59' hence it should run for another 10 years. MT4 should be defunct by then, but we were saying the same thing 10 years ago!

I noticed I need to change the TimeOffsets to display the news at the right spot on my chart. (For me, at the moment non-DST: -5,0,-8). For the above, it displays the events on my chart just fine.
See 'Time Offests' on page 9 of the manual for more info. Note that there is only one triplet of settings available, even if daylight saving cuts in or out during the period being plotted on your chart. In which case it means that some symbols may be plotted inaccurate by 1 hour, a known limitation.

When I download news, there is a minor problem with the popup window that gives instructions to use Ctrl-J. In fact, my monitor is set to 125% instead of 100%. This scales fonts larger, but when it does, the width of your display box with the message is too narrow.
I think Botan626 pointed this out a while ago. I guess somebody could potentially set their monitor scale to any value, hence there's never going to be a width that's guaranteed to cover every situation.

Unfortunately, this Ctrl-J does nothing for me. It doesn't refresh my chart. I have to change Timeframes, for example, then it works. Am I missing something??
Please see the manual, page 7 'Macro for indicator refresh' for more info.

I next attempted to use the XML file by setting SourceDataFromXLM=true (default is false), but it did not work! The URL is bad. "Unable to find requested URL" The PDF says the URL is (was) http://cdn.forexfactory.com/ffcal_week_this.xml Since you don't provide .mq4 source to all, I suggest you make the URL an extern string variable. The current value is: https://cdn-nfs.faireconomy.media/ff...r_thisweek.xml
Looks like FF have changed their URL. Of course I have no control over this. Agreed, changing it to an input setting is the logical solution. New parameter is URLforXMLdata.
_________________________________________

[EDIT 2]
Sorry Kent, I was pretty stressed out last night, troubleshooting another MQL4 app (which I hope to release later this year). Anyway, I've made your two suggested changes (highlighted in red typeface above), and have uploaded everything in a new zip file to post #1, which I will leave there for another 48 hours (and then remove it before the FF admins do their Win10 Defender virus check). There is a new folder in the zip 'Source code (for programmers)' which contains all of the (b509) source for Plot News and Recent News. The downloader should work for at least the next 12 months; I will attempt a permanent fix when I have time. Many thanks for your feedback, as always, and all the best for 2020.

Hi David,

Sorry to bum you out! I will look at any source you post and will make the easy changes myself and re-post them when I get a chance.

However, just changing the ThruDate and the bad XML URL would fix the main issues affecting, for example, moron and others who don't see events >= Jan 1. Recompile with b509. No research necessary! (The b600+ might require a little more, but I've done that before with an older v4 mq4 and it works; I can repeat the same).

Thanks,
Kent

For some reason I was unsubscribed from this thread and forgot to check it beforehand. For me, the scraper continued to work until 05 Jan 2020. I only use the News Downloader.exe to re-scrape the latest calendar info daily.

I can confirm the latest quick fix to the News Downloader.exe (uploaded 31 Dec 2019 from hanover) works on windows 10 x64 and win 2012 r2 x64 without any problems. and I don't receive the "Unable to download news" error anymore.

You may have to whitelist the News Downloader.exe from Windows Defender / Antivirus program first. BTW, do this at your own risk.

I do hope another fix is provided soon so that a yearly exe edit isn't necessary. Using the start date only for the week/month/year should be possible to retrieve the calendar html pages and scrape them without error. And it will be future proof the exe. Placing the url in the ini might also be a good idea. No rush though, since the main News Downloader.exe is working now.

Also, the new backup download link from me is here: https://github.com/4evermaat/news_do...ctory/releases .

Happy new year Hanover and thanks again.

David,

I think its just a formatting issue in the news file.

It downloads the info, but when I use Excel to look at file all info is in one row, not a list.

If I use notepad I see the data formatted correctly.

Plot News was displaying events up to 31 Dec 2019 OK but events in 2020 did not display so I think it may be a format issue in news file.

My 2 cents/observations

Hope u can look at soon or Pips4life can help out. Would be appreciated.

thks

It now gives the same popup to download however many weeks I specify. I added "Plot News v4-00" to my chart. The most obvious problem is that the ThruDate default is "2019.12.31 23:59". This is likely the problem for people not seeing news for >= Jan 1st. I suggest to make it "2038.01.17 23:59" (which is within hours of the max allowed, 2038.01.18 ??:??).
Thanks for pointing this out. Please feel welcome to change the date and update your template.



Where is this ThurDate ? .....looking for it in a config / Template file or INI file ....cant find it.

Hope its in a template and not in source code. Any guidance would be helpful.

Found the ThurDate.....its in the Plot News Presets.

Got it man

Thks.

Please see the two edits I've made to post #312 (if you haven't already done so).

Sorry about the confusion. I've uploaded another version of the zip file to post #1 which should fix everything, as explained in '[EDIT 2]' in post #312. Many thanks for your feedback, as always, and all the best for 2020.

I've uploaded another version of the zip file to post #1 which should fix everything, as explained in '[EDIT 2]' in post #312. I will attempt a permanent fix to the date issue in the downloader when I get some time, and have also updated the link to your github URL in post #1. Please note: the latest version is still 4.15, I will update it to 4.16 when I fix the downloader properly. Many thanks for your feedback, as always, and for hosting the software on github. Happy New Year, all the best for 2020.

David

Hi David.

Did you tested your software(downloaded data) with Eastern European timezone ?
I've downloaded three years data with +7 hours shift but I think there is some daylight saving shift hours in March and October for this particular timezone..

Can you fix timezone issues ?

Thanks

Hi,

This is a known limitation with the software. There are hundreds of countries in the world, all with different daylight saving rules. Some countries, like USA and Australia, span multiple timezones, and different regions have different rules; while in other countries, the daylight saving on/off dates have been changed from year to year. Sorry, but it would take many, many hours for me to process every item in a list like this one, and build every possible contingency into the software. Please understand that I don't get paid to do any of this.

What you could try is to run the 'Convert news to GMT' app twice, once with shift (for example) +1 hours and once with +2, sending output to 2 different files. Then use a text editor to copy/paste relevant sections from one file into the other (say copy from March XX to October XX in one file, and overwrite the same news events in the other**). You can click on your country/region here, and it will give you the values for the exact day (XX) that daylight saving started/stopped for each year (see screenshot). It requires some work, but at least you'd need to do it only once, and it's the best solution I can think of.
____________________________

**[EDIT] If your timezone is the same as Oslo (Norway), then you are GMT+2 from the last Sunday in March to the last Sunday in October ('file #2'), and GMT+1 otherwise ('file #1'). The 'Convert news to GMT' app automatically takes care of FF calendar (New York) time. So you would run it with a setting of +1 to generate file #1, and +2 for file #2. Then copy each section from March XX thru October XX in file #2 (where XX is the date of the last Sunday for a given year), overwriting the matching news events in file #1.