Plot News / Downloader version 4.00

Hi hanover, thanks for this interesting indicator as always.I am try to analyze the price reaction to the news , I only want those news in the att file cad-boc rate.csv which pick from your allnews.csv to be plot on chart, can I do that with your indi? and what is the DescriptionContains1,2,3 for? are they as a filter or something ? many thanks for your help .


Peter huangjin

Plot News indicator has not been designed to be used in backtesting.

The data is timestamped New York time, hence it will be GMT-4 or GMT-5, depending on whether NY is on DST or not (as you point out). Without seeing examples, I don't know what you mean by "sometimes a complete year changes" or "sometimes single events get moved or are missing". I have noticed that FF changes historical news (presumably they are fixing errors within their own database, and/or changing data as new info comes to light), hence I assume that this is the primary reason, especially if single events have been moved to different times, descriptions changed, etc. I don't use the historical data myself, hence I don't check the content of the file. It is provided on an "as is, take it or leave it" basis.

You can use the News Downloader app to build your own file, although it would take several hours to process the entire FF database (which is why I simply add a month's data to the end of the csv file, and then upload it). But you could build your own file (say) 3 months at a time, and then use a text editor to join all of the resulting output files together.

You could try running the app 'Convert news to GMT' (attached). This converts the date/time of the events in the file to a GMT based date/time, on the assumption that New York is on DST (GMT-5) from the second Sunday in March to the first Sunday in November, and on GMT-4 otherwise.

I'm not totally familiar with the XML file supplied by FF. But please feel welcome to use it, if you prefer.

____________________________________________________

The output in your first screenshot looks OK to me. What is wrong with it?

(Note that you need to change ParameterFile to NONE if you want the indicator to use all of the other settings shown in your second screenshot. Otherwise it will use the settings in the Parameter file that you specify. Read the PDF file attached to post #1 for more info)

Use the 'Plot News 4.00' indicator unless you want to have the news plot in the main chart window (obscuring everything else). The only difference between them is that 'Plot News 4.00' creates a new sub-window on your chart, while 'Plot News 4.00 main chart' doesn't.
____________________________________________________

Hi Peter,

Yes they are designed for use as a filter, for a full explanation read the PDF instruction file attached to post #1. However, the indicator is not designed to process large amounts of data. It is merely for plotting the last few weeks' data on your chart. Attempting to process several months/years worth of data could freeze or crash MT4.

You could try loading the entire file into Excel, but I don't know how well Excel would cope, in terms of sorting/filtering a large amount of data. Perhaps your best option would be to either find a text editor that's capable of sorting a large CSV file, or to load the CSV file into a database program, and sort or filter the data by the News Description. Then you could copy all of the BOC rate announcements out of the result.

If/when I get some spare time, I'll write an app that extracts data from the allnews.csv file, based on matching text within each line/row of the file. I'll include it with the next release of the news indicator.

David

Hi hanover,
this is an absolutely great idea for Manual backtesting the News.
Perfect.
However I am just interested in backtesting specific weekdays.
So for me its thursday and friday.
1.How can I choose these weekdays and disregarding the other weekdays?
2.Is there a way to show the description of the News -event for every number plotted under a Candle?
Thanks for an info.
Regards Zack

This type of filtering is not currently available. Hence you'd need to load the news.csv file into a text editor, and manually delete any rows that you don't want displayed.
The only way is to hover the mouse cursor over an individual event, and the description is shown in the help pop-up (see instruction PDF for more info). There's no way of displaying descriptions of all events simultaneously. I never intended to use the indicator in that way. There are other indicators that do this, but you invariably end up with different event descriptions overwriting each other, making them unreadable.

Hi hanover,

awesome work. What parameters do i have to use for iCustom function to use this indicator for my EA?


Best regards

Awesome tool! It's not possible yet to automatically have only news plottet which is relevant to the selected pair, right? I'm switching through different currency pairs reusing opened chart windows a lot. Would be a nice feature!

Hi Hanover,
great work on this newdownloader. I have been using it for a couple of months now and it has been very helpful in my learning! I just wanted to say thanks!
Also as of today, 3rd July 2016 it seems the Newsdownloader stopped functioning with the generic error "unable to process calendar". I have overwritten the files again from the newly downloaded zip file but still the same error.

Hi Hanover

I went to update the news downloader an it has come up with the "Unable to process calendar". I am using the latest version 4-07 modified 04/07/16 07:51, my windows is V7 professional 32 bit and also 64 bit on other machine, both are not updating today.

Any thoughts please?

Kind regards

Anyone heard from David ?

Kinda strange he has not logged in since Jun 19th.

Normally he is very responsive.

Hope all is OK with him.

Hope he just went on vacations.

He deserves the best.

I hope all is well, his work was very valuable.
In the mean time until he returns, i created a python script that grabs the news and exports it into a CSV file like his news downloader. You can find it attached for those that rely on this. However this only includes the high and medium impact news. If you want more or a different format then please let me know. I can help in his absence.
Just append the text to the news archive csv file if you want the plotter to continue plotting. Also the times are without any timeshift.

thks for offer really appreciate.

Could you do all events ?

How do I initiate the download to his CSV file ?

Thks

No problem, i have attached two csv files. news.csv just contains all the events for this week. news_archive.csv contains this weeks and last weeks.
Please note that all events that have "All Day" or "tentative" in the time field in FF have been marked at a time of 00:01 so that is can be plotted on a specific candle.

as for how to use this with his CSV files:
I can only speak for the MT4 so if you go to File -> Open Data Folder, this will bring up the data folder of the mt4 platform. Now go into the folders MQL4 -> Files. This is where all of the news downloader files are kept. Here you should see the original "news_archive.csv". I would recommend creating a backup of this file at this stage. Then there are two ways depending on what you want

Method 1:
You can just replace this origional file with the news_archive file attached to my post but be warned, you will lose all the your previous event history and you will only see the this week's and last week's events.

Method 2:
Open up the original "news_archive.csv" file and copy and paste the data from my news.csv (current week only) or news_archive.csv(last week and this week) file atached below, to the end of the file found in your news_archive file within MQL4 -> Files. This will keep all the previous weeks and months events while being able to see current events.

Then its just a matter of updating the charts: you can remove and add the Plot News indicator, or if it is already attached to the chart, just change time frames and it will automatically read the data again from the new file and plot the events on the chart.

I hope that makes sense ? If not let me know.

Thanks exidez. This is amazingly useful.

Exidez,

Thkss for the CSV files. Helpful, but then I would have to depend on you to send a new CSV daily or when FF updates the events.

The Python script tht u created which creates the CSV files would be the most ideal as tht would substitute for David's "News Downloader.exe". That would be appreciated then I would run it whenever the FF news list is updated.

Is tht possible to get the download script with instructions on how to run it ?

fxww

Hi fxww,

My apologies, I read your comment wrong previously. I see what you meant to say now.

I had also though about this, and although I will post the the csv every Sunday, if the calendar changes during the week then these will not be reflected.

I can share the script/source code but there is a little bit of playing around with python and installing the modules. At the moment the script is very basic and perhaps buggy. It is not as customizable like David's News Downloader.exe. I will create a little guide and upload the code this week for those who want it.

Ok, so I have updated my code a little bit to make it a little more customizable and compatible with different machines. I have no idea how my code will work on other systems so their may be problems as I have only tested this on Windows 7 running python 2.7 and 3.6. But first thing first.

Python:
There are plenty of guides out there so to save time I am just going to forward to this installation guide: http://www.howtogeek.com/197947/how-...on-on-windows/
This is for windows users as this is what I am currently using. So if you are using a mac or other OS, its still compatible but the installation might be a bit different.

Once python is installed and the system variables are set properly we want to open up command prompt. The sys variables will enable you to run python from command line. Go to start and search for "cmd" or "command prompt" and open it up.
Once it is open type the following:

This will install the module used in my code to parse the html calendar into a readable format for processing.
Everything should now be setup needed to run my code.

The PythonNewsDownlaoder:
To run scripts in python all you need to do is type "python <path to python file>" in command prompt. So in command prompt move to the working directory(cd <directory path>) where my python script is stored. My script takes in arguments so you will need to run my script via the cmd line as below:
The number after -w represents how many weeks you want to go back. These will be put into the archive file.
the string after -o is the name you want to output the file. The script will create(and overwrite) two files. One file is the current week only, the second file(archive file) is the file that includes the previous week(s) and also the current week. If the number after -w is 0 then both files will be the same.
It may take 10-20 seconds to process all the data. This depends how may weeks you want to go back.

Example
this will output the current weeks events in news.csv and will also output news_archive.csv which includes the previous 2 weeks and also the upcomming week.

python is open source so you can modify the code as you see fit (you will need notepad++ to change the py file). However if you find any bugs and don't know how to fix it, let me know and i will have a look. I am constantly modifying and updating the code for my own purposes as well.

Send me a PM if there is anything troubling you in the install process or running the script. I am happy to help

Attached:
This weeks calendar starting: Sunday 24 July 2016 (news.csv)
This weeks and previous weeks calendar: Weeks: 24 July 2016 + 17 July 2016 + 10 July 2016 (news_archive.csv)

Also for those that are having trouble with the above python download I have compiled everything into a executable for a simple one click use to download the calendars.
The executable can be run as above or just executing the file is the same as using "PythonNewsDownlaoder.exe -w 2 -o news". It just used default settings. I believe this only works in Windows 7 and above. PM me if there are any issues.

executable file can be found on my google drive: https://drive.google.com/file/d/0B9K...ew?usp=sharing

With method 2, I tried to change "new_archive.csv" but it didn't make any changes on my chart. Plotnew still show the past events . And I tried to change "news.csv", too but it got worse. Plotnew don't show any event on chart. Can you fix it ?

Quoting quangdk

Disliked

{quote} With method 2, I tried to change "new_archive.csv" but it didn't make any changes on my chart. Plotnew still show the past events . And I tried to change "news.csv", too but it got worse. Plotnew don't show any event on chart. Can you fix it ?

Ignored

To have any changes take immediate effect, either (1) re-attach the indicator to your chart; (2) edit the indicator (press Ctrl-I); (3) switch between timeframes

If you still cant get it to plot then follow this process:

First make sure that the news and news_archive files are placed in the correct directory.
Open up the data folder in MT4 as below:



You will still need to navigate the right folder from here. Go into the MQL4 folder and then the Files folder. You should see the original news downloader in this folder. Make sure the new news and news_archive files are placed here and overwriting any other files. You can even open them up to see if the event are current


Now you will need to refresh your Plot News Indicator by the methods used in the first paragraph. If you click on the "Experts" tab down the bottom of the MT4 you should see that the indicator has been "loaded successfully" or "initialized"


If it is still not plotting, then perhaps its the configuration of the indicator. Edit the indicator properties (CTRL+I) and have a look at the input file. Make sure that it reflects the actual name of the file. If you have any present configurations then you will have the change this in the present configuration file.


If you are still experiencing an issue then there may be an issue with excel when appending to the csv file in Method 2. Try appending it to the file with "notepad" (I recommend a program called Notepad++, but i believe notepad will also work in this case)
The events should be in the format exactly as below (excel may remove the quotes):

At least one of those steps should fix the problem or direct to solutions. Let me know if there are any further issues