Plot News / Downloader version 4.00

Sorry, this is not possible, the indicator plots wingdings symbols only. I intentionally avoided using vertical lines, because when you get multiple news events occurring at the same time, the lines plot over top of each other, allowing you to see only one of the events.

If you want to align the symbols with the price candles, click your mouse wheel to convert the MT4 cursor into a crosshair (and then click the left mouse button to convert it back into an arrow).

However, if you specifically want vertical lines plotted, you will need to find another indicator.

I just want to say, thank you so much for all this work, and for helping others like this. Recently discovering this indi was a godsend, does exactly what I want in amongst so many that don't. And thank you so much for the very well written and detailed PDF instructions. I have gone away and read it closely and implemented what I want.... I have now come back to thank you and see that you have left the forum...poor timing on my part! Anyway, hope you might pop back at some point in the future and read this.
All v best to you sir. Bless you.

First of all, huge thank you Hanover for this free tool. I know someone who paid ≈$700 for a c# tool that scrapes fxstreet csv files into local csv files, and then splits downloaded calendar up into individual days in tsv format.

Is it possible to run the "newsdownloader.exe" using command line? I see that if you add a capital A at the end of the input for Weeks or Months to download (or edit the newsdownloader.ini Weeks_or_Months input to add A at the end), you can skip the dialog box and download news semi-silently.
Although maybe there are still some things that can be done via arguments?

What about running "convert news to GMT.exe" using command line? or "silently"?

PS: minor bug: you don't save the untimed event start time in the newsdownloader.ini . So the user would have to manually re-adjust the time input each time he loads the newsdownloader.exe

Looks like Hanover fixed this bug. That was fast The OP was updated with "News Downloader.exe"
Now the ini file will store the Untimed_events_time parameter. The changelog was not updated though.

I tried to send a PM, but @hanover 's profile settings restricted me. Thanks again.

----
One more thing I noticed. This may be a little bit more serious :

+/- hours shift input. If I put +4 now to convert csv event times to GMT, what happens in the winter? The user would have to manually re-enter the input as +5....and remember to do this at exactly 02:00 that Sunday. And manually re-adjust again to +4 during the next summer. This can lead to mistakes in the news.csv and news_archive.csv calendar event times, correct?

> Either a checkbox or an input for "adjust for DST/EDT automatically" might fix this. So downloader can continue to download events unattended and the user will not worry about DST changes.

The "Convert news to GMT.exe" is only designed for one-time use to change already downloaded calendar offset once. The DST logic needs to also be in main news downloader.exe offset function so that regardless of the offset DST will be accounted for to parse the proper event times in the csv.

I hope I got this right. The people who use this in mt4 with Plot News.ex4 may not have to worry because that indicator may have DST consideration already. So primarily this fix is primarily for people using csv directly for external study.

I've thought about this ST/DST problem a lot.

Very few people have addressed ST/DST properly, and it gets very complicated. I've not yet dived into the Plot News code to figure it all out, but what I can say is...

1. You must consider the timestamp of the source data, and whether it follows any ST/DST schedule, and *what* schedule does it follow.
I'm under the impression that the FF XML data source follows the USA ST/DST schedule. If it is converted to what is called "GMT", is it truly "UTC" ? (UTC is Universal Time Coordinated, which absolutely does not follow any ST/DST schedule). Some people say "GMT" when what they really mean is it's London time... and, do they really mean actual London time that follows the European ST/DST schedule?? Or does it follow USA ST/DST schedule??

I think the FF data is simply NY time***. If it was mapped to so-called GMT, it might really be true "UTC", or it might instead be a variation that is UTC in winter, and UTC+1 in summer, according to USA schedule. ***I may be wrong! I'll have to dive into the details to figure it out for the source data.

2. NEXT: You must consider what ST/DST schedule does your broker use! Note, as far as I can find, there is no function or MT4 property that will tell you the answer! (I'm well aware of functions TimeDaylightSavings() and TimeGMTOffset(), which are for local-vs-UTC, but do not help you with *broker* schedules!). Rather, you must analyze a chart and infer from various dates (USA, EUR, AUS, etc. ST/DST schedules), check the weekly Open Hour, and weekly Close Hour, and watch for the number-of-bars-per-week to either be the same or change. (120 is ideal; I sometimes see 119 and 121 during changeover weeks). I've pretty much figured out how to analyze an H1 chart data feed and prove what ST/DST is it following, and what is the UTC offset.

Some brokers are True UTC feeds, that follow no ST/DST schedule itself. The broker 00:00 is truly UTC 00:00, year round. Counter-intuitively, you will see the weekly Open hour changes during the spring and fall when the USA ST/DST changes occur, because the brokers follow the USA NY 17:00 close. In other words, NY 17:00 is UTC 22:00 in the winter, but UTC 21:00 in the summer!

Some brokers use the USA ST/DST schedule. These might be NY+0 (actual NY time), or NY+5 (which is UTC winter, UTC-1 summer, per USA schedule), or other variations (e.g. NY+7 => UTC+2 winter, UTC+1 summer). This schedule of broker would be best for Plot News, assuming that both the data and the broker are following the same USA schedule! The offset between them never changes.

Some brokers use the Europe ST/DST schedule. Some of these feeds are so called "GMT+2" (which I like, for 5-daily-bars/week). I now refer to this as "LON+2", because it follows London's (=Europe's) ST/DST schedule, and is different from "UTC+2" (which by UTC definition, does not change ST/DST ever). This is the hardest offset to map to USA data, because there is a common offset for *most* of winter AND summer, but has an offset-1 during the brief 1-2 week windows that USA is on DST when Europe is still ST time, both in spring and in fall.

Tip: This is my preferred URL to view world times: https://www.worldtimezone.com/index24.php

guys someone knows why my news on chart started on august 2018, when i already download the history news from 2015?

Hey Exidez,

i am trying out your script and i am getting an error:

List index out of range.

is this script still functional?

Hello
Never had this before...this morning my windows computer (windows 10 1809) has flagged up virus alerts and quarantined the file:
\MQL4\Files\Refresh News.exe
for a Trojan called Zpevdo.A

My 4.08 version of this indi was originally downloaded April, so I downloaded the most recently uploaded 4.11 version from post#1, but the downloading process again found the same 'Trojan' and deleted the file. Please see attached screenshot.

Has anyone else encountered this? Thanks for any help.

Hey Guido9,

dont worry about the message. It’s a false positive on that file. Because it emulates keystrokes, windows deems it a Trojan. It was made with a program called AutoHotKey. No big deal. I have made my own version and it gave my one a false positive.

Thank you v much frankymcgee for replying

I was getting an error "unable to process calendar" with NewsCalendar.exe Just started happening from the 29th Dec 2018 I think. was working for months without a hitch, set to run 1 time each day. 2 different machines: win8.1 /win 2008 64 bit.



Was using v4.09 But the latest v4.12 i'm not sure if it would help with a scraping format change. in any case i will test it.

Anyone else having this issue?

Update: v4.12 seems to be scraping ok (the downloading part ) But I have additional question/bug reports.

Bugs v4.12:

1) Download failure results in deleted events in news_archive.csv. If there is no data connection (or server is busy), downloading of the data will be skipped. This will make the news.csv file blank. But I noticed that the news_archive.csv gets modified also. The merge function will start, and it will literally wipe all events from the download start week forward, and replace them with nothing. In case of some failure of news downloader (downloading part), an option to cancel the merge should be there.

You can reproduce this by:

1. temporarily blocking internet access to NewsDownloader.exe via firewall app
2. [make a backup copy of news_archive.csv first].
3. run NewsDownloader.exe I scrape Weeks_or_months=7wA.

The news.csv file will now be blank. The merge will happen, but it will delete any events. I suspect the merge literally is merging 0 events. It just deletes everything from the starting week onwards. Consider implementing checks such as:

1. check if download failed. Do not merge if downloading never started or failed to complete (or configure ini whether to merge partial downloads).
2. check if news.csv is blank and/or contains less than X rows (configurable in ini). Do not merge.

This way, if the download fails, we still keep the archive intact.

Interesting note: news.csv.sav is intact with all of the events from the last successful scrape.


2) Silent mode: Merging does not occur. Activate silent mode by appending A to the Weeks_or_months ini setting. The download takes place successfully in news.csv, but for some reason the merge never takes place.

Merging does work when you run the application with GUI (removing A from Weeks_or_months=7wA in ini ).

If there is some additional parameter I'm missing, please advise. Previously, the merge was automatic after the download was completed and processed.

3) Silent mode: the news.csv events are missing "" quotes at the very beginning and end of each event (row).

old output:

new output

I didn't change anything to the ini file. But on each launch, the ini file is overwritten and you can see that the quotes are actually removed in the ini file.

Last_format_used= d[Y/M/D]","t[H:I]","c[T3]","i[T1]","n[T50]","a[T10]","f[T10]","p[T10]","r[T10]","#[Z5] .

The last update definitely broke some silent mode functions. Not sure why though; the operation should be identical with/without GUI.

-----------------------
Questions re 4.12 ini file:

1) if the Currencies or Weekdays parameter is left blank, will the newsDownloader just process all currencies/weekdays?
I see "Description" is left blank, so I assume that means any news even is processed. I wanted to confirm similar default action if blank for Currencies and Weekdays parameter.

2) Is an additional sorting input needed of the output file if the news calendar by definition is already sorted chronologically? Redundancy isn't a bad thing, but I'm trying to think of a time where the calendar was NOT sorted chronologically. Maybe if there was some missed dates and then later you want to download the missing months after you already downloaded new data.

-------
Btw
Happy new year and thanks for such a useful script.

Workaround to the bugs above is to use some sort of macro script to run it in GUI mode, and automate clicking the "ok" button. So until the next update, you can use this AutoIT script attached.

You just copy FFNewsDownloader.exe to the same folder as the News Downloader.exe 4.12 and run it. The exe version does not need any additional files to run. The .au3 source file requires AutoIT 3 to edit/run. They are identical.

Tested with Win 8.1 x64 and Win 2008 R2 SP1 x64. Make sure to disable silent mode (remove A from Weeks_or_months=7wA in NewsDownloader.ini ) so the GUI will appear.

This should help for those who schedule regular downloads.

Do anyone have the NewsDownloader.exe file? It does not seem to be attached to this forum (or within first post's Plot News v4-14.zip). Thanks.

I see the News Downloader.exe in the zip filelist.


When you unzip the contents, are you saying that you do not see it?

---------

Tested News Downloader.exe v4.14 in both non-silent and silent mode silent mode and I can confirm:

1a) download attempt with internet off results in news.csv being deleted (different but workable).
1b) News_archive.csv is untouched as merge function is skipped. Thanks.

2a) Successful download and creation of news.csv in silent mode still does not activate Merge function; so none of the news.csv makes it to the news_archive.csv. So for silent mode merge not working anymore there are two possibilities that I see:

 

1. this is correct/intentional behavior (maybe not everyone needs merging on each download request). So a switch is needed to activate merge function. Either add it to the Weeks_or_months input or have a new ini input.
2. this is incorrect behavior. The merge function needs to be re-added to the sequence after news.csv is finished being built (silent mode).

2b) The GUI run (non-silent mode) does activate Merge function ok . It's really weird that the silent mode switch has a totally different sequence from the non-silent mode.

3a) Missing leading/trailing quotes problem is still here. When running in Silent mode:

1) ini file is overwritten with missing quotes after the run

2) news.csv file contains events with missing quotes.

3b) running in GUI (non silent) mode quotes appear just fine; merging occurs.

So I think there are two separate sequences. Sequence #1 when GUI/non-silent mode is active and Sequence #2 when silent mode is active. With 4.08, these sequences were identical, so no issues regardless of which mode is active. But some subsequent update since then broke some functions so they are not identical anymore, which is why they are producing different results even with the same parameters.

Sequence #1 is perfect and functions exactly as it has before. Sequence #2 needs work.

I'll reply and take the risk of the usual trolls and haters posting something like "I thought you'd left FF".

Anyway, your NewsDownloader.ini file appears to contain the entry:
Weeks_or_months=7wA

NewsDownloader.exe is written using the Autohotkey (AHK) programming language. Alas, AHK reads any INI file entry aaa="bbb" in as aaa=bbb, i.e. the outermost quotes get dropped. In silent mode, the GUI entry gets bypassed and doesn't overwrite this, hence the quotes remain dropped, causing the problem that you describe.

So what I did in v4.14 was to add line of code that says:
If the Weeks_or_months entry contains A' (an uppercase A followed by a single quote) and the output format isn't enclosed in double quotes, then add a double quote to both the beginning and end of it, i.e. change
Last_format_used=xxx
to
Last_format_used="xxx"
which should fix all of the problems that you described.

TLDR: the short answer to your problem is that you need to change your entry in your NewsDownloader.ini file to:
Weeks_or_months=7wA'

Hope that makes sense. I tested it and it correctly reinstates the double quotes to the format, and causes data to flow to both the News file and Archive file. Try it and let me know if this fixes all of the problems that you described. The merge function should activate if the format is correct.

Thanks David,

Screw the trolls, haters job is hate...they r unable to contribute positive stuff, they r not very bright.

Ur doctors visit is always welcome.

Appreciate the fixes.

Thank you forevermaat & Hanover

I totally missed this in your update after re-reading it. But it is clear now.

Works just fine after adding the apostrophe




At least I did learn about AutoIT, which has opened the door to some more automation of other desktop apps related to trading and other things. So actually I should be thanking you indirectly for all of this. Don't apologize at all for peeking in on occasion; only good things can come of it.

[quote=hanover;11839096]{quote}I'll reply and take the risk





Hello David,

I think that more people are happy when you hear something from time to time. You have not been active for over 9 months, yet your blog is still in first place. So you seem to have done a lot right. I joined Forex Factory a year ago because your contributions have thrilled me year after year. Even though we have completely different approaches in trading, you have helped me and many thousands of traders with your volunteer work and you can be proud of that. Your dedication has brought many creative and great ideas in 13 years, many of which dealers can still benefit today. Your more than 8,000 posts can fill more than 35 books with 300 pages, I think that's an extraordinary achievement. Although there may be other opinions, I learned a lot and took a lot from your creative ideas. In addition, you have taken the trouble to change one of your indicators for me, with which I still work very successfully today.

Thank you again, certainly on behalf of many traders.

best regards