Plot News / Downloader version 4.00 | Page 10 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

FFcal - Plot News indicator 692 replies

Plot News indicator (version 3.0) 402 replies

Good History Rate Downloader EA 14 replies

Plot News (version 2.0) 103 replies

Attachments: Plot News / Downloader version 4.00
Exit Attachments

Plot News / Downloader version 4.00

  • #181
  • Sep 13, 2016 11:37am Sep 13, 2016 11:37am
Hello exidez,

thank you very much for the last update I needed. (the correct year in the csv-file) For me its useful.

Now I have a new question. I remembered that the tool from hanover can download future weeks. And I could choose
quantity of weeks.

Is this possible for you to do this?
like this :
python PythonNewsDownlaoder.py -w 2 -o news

It will be great from you.
  • #182
  • Sep 16, 2016 6:02am Sep 16, 2016 6:02am
I have downloaded everything 4 times already from post #1 and the News Downloader.EXE file does not work. I keep getting the error "Unable to process calendar"
Can you please provide the MQL files and also fix the above issue please? There are numerous people on this forum getting the above error.
  • #183
  • Edited 11:28am Sep 16, 2016 11:00am | Edited 11:28am
Quoting bm0899
I have downloaded everything 4 times already from post #1 and the News Downloader.EXE file does not work. I keep getting the error "Unable to process calendar" Can you please provide the MQL files and also fix the above issue please? There are numerous people on this forum getting the above error.
I am sorry, but I'm no longer posting on trading forums on a regular basis. It is too much hard work, and I'm suffering from eyestrain and stress related issues.

If you're repeatedly getting the "Unable to process calendar" message, then the most likely reason is that the downloader app (as opposed to the indicator) can no longer recognize the (updated) format of the FF calendar. The downloader was written using autohotkey, not MQL4, hence the source code would not help you. You could try using the python-based downloader written by exidez (see his earlier posts). Beyond that, there are other news indicators around, which can be located easily enough using search engines.

Both the downloader and indicator continue to work fine on my computer, as shown in the attached screenshot. Given that we are both trying to access the same FF calendar, I don't know why it's not working for you. I'm still running Windows 7 and MT4 build 950. I can't troubleshoot problems on other people's computers.

I am sorry for any inconvenience. I will make a disclaimer note at the start of post #1.

Attached Image (click to enlarge)
Click to Enlarge Name: PPI 268.png Size: 83 KB
  • #184
  • Sep 16, 2016 11:36am Sep 16, 2016 11:36am
Quoting hanover
{quote} I am sorry, but I'm no longer posting on trading forums on a regular basis. It is too much hard work, and I'm suffering from eyestrain and stress related issues. If you're repeatedly getting the "Unable to process calendar" message, then the most likely reason is that the downloader app (as opposed to the indicator) can no longer recognize the (updated) format of the FF calendar. The downloader was written using autohotkey, not MQL4, hence the source code would not help you. You could try using the python-based downloader written by exidez...
Hey long time.
  • #185
  • Sep 16, 2016 10:05pm Sep 16, 2016 10:05pm
Quoting hanover
{quote} I am sorry, but I'm no longer posting on trading forums on a regular basis. It is too much hard work, and I'm suffering from eyestrain and stress related issues. If you're repeatedly getting the "Unable to process calendar" message, then the most likely reason is that the downloader app (as opposed to the indicator) can no longer recognize the (updated) format of the FF calendar. The downloader was written using autohotkey, not MQL4, hence the source code would not help you. You could try using the python-based downloader written by exidez...
That's really weird that the newdownloader works on some computers and not others. That would definite be hard to debug. I originally thought it was the FF calendar being different but I didn't notice anything out of the ordinary when going through the html code. Either way, there's now a solution and the indicator still works as excepted and I doubt that will be outdated. Thanks for your work hanover and I hope everything goes well.

Quoting fxdevelop
Hello exidez, thank you very much for the last update I needed. (the correct year in the csv-file) For me its useful. Now I have a new question. I remembered that the tool from hanover can download future weeks. And I could choose quantity of weeks. Is this possible for you to do this? like this : python PythonNewsDownlaoder.py -w 2 -o news It will be great from you.
I have updated the share url to version 1.3 which include the option for future weeks by the -f argument.
The new default parameters (without supplying any arguments) are two weeks before, one week ahead and the output as news.

The defaults are the same as typing the below:
Inserted Code
python PythonNewsDownlaoder.py -w 2 -f 1 -o news

or with the executable
Inserted Code
PythonNewsDownlaoder.exe -w 2 -f 1 -o news

obviously changing the numbers changes how many weeks before and after the current week the application will download and also the output file name.

both the exe and the python script have been updated.
Executable: https://drive.google.com/file/d/0B9K...ew?usp=sharing
Python Script: https://drive.google.com/file/d/0B9K...ew?usp=sharing
  • #186
  • Sep 17, 2016 12:47am Sep 17, 2016 12:47am
Quoting exidez
That's really weird that the newdownloader works on some computers and not others. ..... Either way, there's now a solution and the indicator still works as excepted and I doubt that will be outdated.
Agreed, it's strange. Many thanks for providing an alternative solution.

Quoting claudia1
{quote} Hey long time.
Good to see you're still around and hope that your trading is going well.
  • #187
  • Sep 17, 2016 1:12am Sep 17, 2016 1:12am
Hanover, firstly many thanks for building this awesome product, it seems to does the job (unfortunate I couldn't run this on my system yet ). This is exactly the kind of thing I am looking for. All I need is the updated CSV file, so that my EA can filter the high impact news when trading automatic. I don't need the indicator but just need the Newsdownloader to work to generate updated CSV file :'(
Is there a software or tool available which will only give out CSV file for an input for the EA? I couldn't find a single one via google search.
  • #188
  • Edited 7:49pm Sep 17, 2016 4:35pm | Edited 7:49pm
Quoting bm0899
firstly many thanks for building this awesome product..........Is there a software or tool available which will only give out CSV file for an input for the EA? I couldn't find a single one via google search.
I started writing news downloaders because I likewise couldn't find anything else on the net.
If you can't get the downloader to run on your computer, you could try exidez's python script (post #185).
If your EA merely needs to be aware of upcoming announcements for the current week, Forex Factory's XML page might be helpful.
  • #189
  • Sep 18, 2016 8:53am Sep 18, 2016 8:53am
Hey Folks,

Hanover.....the man...glad to see ur ok, even though ur stressed. Pls get some rest man. EXIDEZ has helped out while u were resting and wrote a great script.

I actually created a .BAT file with above python command coz I could not get the .EXE to work. My .BAT file works flawlessly after I installed the Python dev module as EXIDEZ explain in a earlier post.

Big thks to both HANOVER and EXIDEZ for keeping this absolutely brilliant tool alive.

May the pips be with both of you.
  • #190
  • Edited 7:32am Sep 19, 2016 7:06am | Edited 7:32am
Hanover - Thanks for this. You have created a wonderful tool. How does "Convert news to GMT.exe" know which time zone the initial input file is in (always in EST NYC time zone)? Also, can I please request you to add in post #1 a CSV file from 2007 to latest for High impact news only.

Exidez - thanks for this Python script with exe. Since, hanover's wonderful tool isn't working for me (& possibly others). Just wanted to ask:

(1) "python -m pip install beautifulsoup4"- isn't working for me, it comes with an error. I have downloaded python 3.5 from python.org website.

(2) Python's CSV file output, is it same as hanover's CSV format? Like "date, time, impact, currency, event description, actual value, forecast value, previous value, revised from value, FF event ID number". Please confirm.

(3) On command: "PythonNewsDownlaoder.exe -w 2 -f 12 -o news" and received error: "option -f not recognised"

(4) Is the "time zone" in GMT or time zone is in same zone as hanover's?

(5) How do you convert Python script to EXE file?

Can I please request for the following modifications:

(6) It is a lot easier to execute EXE files (able to run without Python): is it possible to have a user interface with "forward # of weeks" and "backward # of weeks" and "High Impact Only" and button for okay (like hanover)? Instead of just click EXE for output files.

(7) If no (4) above can not be done then, how do I preset "PythonNewsDownlaoder.exe -w 2 -f 12 -o news" to generate next 12 weeks of forward data?

(8) How do I preset or tell the EXE or Python to only output "H" or High Impact news only (EA news filter)?

If you can kindly post both the latest EXE and Python script (just in case future modifications can be carried out many years to come).
  • #191
  • Sep 19, 2016 10:48am Sep 19, 2016 10:48am
Hey Hanover,

I've been thinking of creating an oscillator that will aggregate the news signals.
I think many of us will be happy to have this kind of tool.
The Forecast as a reference by bars and 2 additional lines or dots for (Actual - Forecast).
One is simple (always calculated) and the second - only for condition of serious unexpected move in opposite to expectation direction:
(Forecast > Previous && Actual < Previous) or (Forecast < Previous && Actual > Previous).

The Forecast shows the anticipated trend and the Actual Events verify the direction or indicate the trend weakening and the following trend change.

It should be able to plot as a percentage change or discrete (+1 when the move is positive and -1 when negative). The Impact multiplier can be implemented as well.
The indicator can be plotted for one currency only or for the couple. That's way for GBPUSD it would be (News GBP - News USD).
I also think it could be a good idea generating additional presets csv file with all the news events, to let a user set his own Impacts and currencies for each event.
That's because some events can be relevant for currencies (for instance Crude Oil Inventories that can be relevant for USD and CAD and their influence depends on the current situation) and traders can attribute importance that differ from the FF.

Would you be interested to expand your existing indicator with this feature?
It would be great.

Exidez - Maybe you can program an MQL script running your Python exe to substitute the DownloadNews v4-00 for downloading the data?

Thank you guys. You do an excellent work!
It's a real piece of art!
Hope you will find my ideas applicable and will be more than happy to discuss these additional features, if you need any additional details.
  • #192
  • Sep 19, 2016 8:01pm Sep 19, 2016 8:01pm
Quoting bm0899
How does "Convert news to GMT.exe" know which time zone the initial input file is in (always in EST NYC time zone)? Also, can I please request you to add in post #1 a CSV file from 2007 to latest for High impact news only.
It is the base time zone of the FF calendar, which I assume is NYC EST or DST, whichever is applicable at the time.
File is attached (E&OE).

Quoting Pikey
I've been thinking of creating an oscillator that will aggregate the news signals.....
I already did something like this back in 2008 (here). However, before anybody gets too excited, the FFcal.EXE app runs only on Win XP or earlier Windows versions. Also, nobody else showed much interest hence I abandoned what I thought back then was a promising idea. Nowadays I believe that good FA is too highly nuanced to be usefully automated.

I'm trying to cut down on computer time now, and I'm focused on trading rather than programming. The latter is too time-consuming, there are 400,000 FF members and once you offer a free product everybody starts asking questions and making requests that are not necessarily helpful to my own trading approach. I respectfully suggest that these folk teach themselves programming, it is a very useful skill that has a zillion applications and you will never regret your time and effort.

This is my last reply to this thread. I really MUST get back into my analysis and trading now.
Attached File(s)
File Type: zip News events 2007-01-01 thru 2016-09-16 timestamped New York time (high impact only).zip   135 KB | 444 downloads
  • #193
  • Sep 20, 2016 4:32am Sep 20, 2016 4:32am
Hey Hanover,

Excuse my bothering you again. I promise I do it a very last time no matter if you reply or not. )

As far as you impressed your unwillingness to keep maintaining this chatting, maybe you can share the MQL source code of the Downloader and the News composite indicator with us? I program a bit on MQL, but agree it will be a waste of time to develop from scratch, moreover I'm definitely not at your level.
Hope this request not to rude, since it's about your intellectual property.
I assure you that I wouldn't request any explanation or assistance with the code.
I'd highly appreciate it.

Wish you success in the trading.
  • #194
  • Sep 20, 2016 12:48pm Sep 20, 2016 12:48pm
Quoting Pikey
maybe you can share the MQL source code of the Downloader and the News composite indicator with us?
Source code for News Composite indicator is attached. This was written back in 2008, I'm afraid that much of it will need to be re-coded if it is to run on the latest MT4 builds. Or you could try compiling using the build 509 compiler (can be downloaded from this post).

The FFcal.EXE downloader app that generated the NCI files was written using Clarion (you would need to purchase the compiler -- costs USD 2,000 ). Hence the best way forward would probably be to use exidez's python downloader (assuming that's OK with him) and then modify the code in the News Composite indicator to read his CSV format. You might also find CSVeditor useful, for editing and manipulating CSV files/formats (or just use Microsoft Excel).

Quote
I assure you that I wouldn't request any explanation or assistance with the code.
I am sorry that I can not help further. Good luck, I wish you every success.
Attached File(s)
File Type: mq4 News Composite.mq4   33 KB | 527 downloads
  • #195
  • Sep 20, 2016 1:10pm Sep 20, 2016 1:10pm
Quoting exidez
Executable: https://drive.google.com/file/d/0B9K...ew?usp=sharing Python Script: https://drive.google.com/file/d/0B9K...ew?usp=sharing

Hi exidez,
thanks for your contribution, it takes time to write SW.
Unfortunately, there is a little problem with time zones.

When downloading with ff default time zone settings (as it is done), the All day/Tentative events get wrong day(!) time stamps sometimes:
"2016/09/18","00:01","JPY","N","Bank Holiday","","","","","60346"

<event>
<title>Bank Holiday</title>
<country>JPY</country>
<date>
<=!=[=C=D=A=T=A=[ 09-19-2016 ]=]=>
</date>
<time>
<=!=[=C=D=A=T=A=[ 12:00am ]=]=>
</time>
<impact>
<=!=[=C=D=A=T=A=[ Holiday ]=]=>
</impact>
<forecast/>
<previous/>
</event>
<event>



Also, if you check the xml weekly calendar, some of the tentative events, actually DO have a schedule. In this case, the day is also wrong:

"2016/09/20","00:01","JPY","H","Monetary Policy Statement","","","","","61565"


<event>
<title>Monetary Policy Statement</title>
<country>JPY</country>
<date>
<=!=[=C=D=A=T=A=[ 09-21-2016 ]=]=>
</date>
<time>
<=!=[=C=D=A=T=A=[ 3:50am ]=]=>
</time>
<impact>
<=!=[=C=D=A=T=A=[ High ]=]=>
</impact>
<forecast/>
<previous/>
</event>

Well, this cannot be solved easily, and actually i am posting this more because i need a logical solution rather than update of the program... Partial solution would be to set the time zone to GMT... Perhaps gather the data from some other site?

And instead of data-eventid, which changes anyway so i don't see a point in having it, full time stamp (date/time/time zone) would be better...


BR, Graziani


P.S. Also, there is a small bug in the program, when you don't enter all parameters, the missing parameters stay undefined and the program crashes.

I have solved this by defining the defaults prior to try: block

limitweeks = 2
futureweeks = 1
Archive = "news_archive.csv"
CurrentWeek = "news.csv"

#argument handling for number of previous weeks to include and/or file name
try:



rather then it is now:

if not opts:
opts = [('-w', '2'), ('-f', '1'), ('-o', 'news')]
  • #196
  • Sep 20, 2016 2:17pm Sep 20, 2016 2:17pm
Also, in the first week when the year changes, date is not changed to new year:

"2014/12/31","00:01","CNY","N","Bank Holiday","","","","","50662"
"2014/12/31","00:01","JPY","N","Bank Holiday","","","","","55338"
"2014/12/31","21:00","CNY","H","Manufacturing PMI","50.1","50.0","50.3","","55338"
"2014/12/31","21:09","CNY","M","Non-Manufacturing PMI","54.1","","53.9","","53455"
"2014/01/01","00:01","CHF","N","Bank Holiday","","","","","53455"
"2014/01/01","00:01","EUR","N","French Bank Holiday","","","","","51819"
"2014/01/01","00:01","EUR","N","German Bank Holiday","","","","","51819"
"2014/01/01","00:01","EUR","N","Italian Bank Holiday","","","","","54293"
.
.
.
"2014/01/02","11:00","USD","L","ISM Manufacturing Prices","38.5","43.1","44.5","","58437"
"2014/01/03","11:00","","N","","","","","","58437"
"2015/01/04","18:30","AUD","L","AIG Manufacturing Index","46.9","","50.1","","54711"
"2015/01/04","21:35","JPY","L","Final Manufacturing PMI","52.0","52.1","52.1","","54711"
  • #197
  • Sep 20, 2016 2:22pm Sep 20, 2016 2:22pm
Quoting hanover
Good luck, I wish you every success.
All best to you too!
  • #198
  • Edited Sep 22, 2016 12:11am Sep 21, 2016 11:21pm | Edited Sep 22, 2016 12:11am
Quoting Graziani
{quote} Hi exidez, thanks for your contribution, it takes time to write SW. Unfortunately, there is a little problem with time zones. When downloading with ff default time zone settings (as it is done), the All day/Tentative events get wrong day(!) time stamps sometimes: "2016/09/18","00:01","JPY","N","Bank Holiday","","","","","60346" <event> <title>Bank Holiday</title> <country>JPY</country> <date> <=!=[=C=D=A=T=A=[ 09-19-2016 ]=]=> </date> <time> <=!=[=C=D=A=T=A=[ 12:00am ]=]=> </time> <impact> <=!=[=C=D=A=T=A=[ Holiday ]=]=> </impact> <forecast/>...
(1) "python -m pip install beautifulsoup4"- isn't working for me, it comes with an error. I have downloaded python 3.5 from python.org website.
Make sure you are running this command from the commond prompt window and not the python interpreter.
What error are you getting when you type this?
(2) Python's CSV file output, is it same as hanover's CSV format? Like "date, time, impact, currency, event description, actual value, forecast value, previous value, revised from value, FF event ID number". Please confirm.
The output is the same as hanover. However, hanover's newsdownloader had to option to include different formats and mine only outputs the default format. The format that is output can be intergrated with the newsdownloader indicator. an example of the output is below
(3) On command: "PythonNewsDownlaoder.exe -w 2 -f 12 -o news" and received error: "option -f not recognised"
Only version 1.3 and above has the -f option so please make sure you are using the correct version. 1.3 is the latest. You can downlaod from the link below:
Executable: https://drive.google.com/file/d/0B9K...ew?usp=sharing
Python Script: https://drive.google.com/file/d/0B9K...ew?usp=sharing

(4) Is the "time zone" in GMT or time zone is in same zone as hanover's?
The time zone is set to the default as forex factory and does not do any time shifting. This time should be New York Time.
(5) How do you convert Python script to EXE file?
This is a little bit involved to get this to work. I had created an exe but it is very dependant on the operating system that it is compiled on. The EXE i have included should work on windows 7 and windwos 10.
I have been using py2exe to create the EXE. There are some good guides online on how to use it but i dont want to get into it here. As fxworldwide was saying in a previous post, you can also create a .BAT file (if you are windows) and save the python command in the bat file. Then is is just a matter of executing the bat file.

Can I please request for the following modifications:
(6) It is a lot easier to execute EXE files (able to run without Python): is it possible to have a user interface with "forward # of weeks" and "backward # of weeks" and "High Impact Only" and button for okay (like hanover)? Instead of just click EXE for output files.
Ideally this is more desirable, I do agree. However I origionally wrote this code as a quick workaround and didnt expect it to be used by anyone else. As a result it will be a little timeconsuming to rewrite (or use different coding platofrms) to convert it to a GUI. I will need to read up on it and see if its worth doing. But thanks for the suggestion, i will keep it in mind.
(7) If no (4) above can not be done then, how do I preset "PythonNewsDownlaoder.exe -w 2 -f 12 -o news" to generate next 12 weeks of forward data?
"PythonNewsDownlaoder.exe -w 2 -f 12 -o news" from version 1.3
(8) How do I preset or tell the EXE or Python to only output "H" or High Impact news only (EA news filter)?
I will add this feature to the next version. For now, only hanover's indicator filteres out the low and medium impacks so i didnt include this in the origional newsdownlaoder. But this is not hard to implement.

Quoting Graziani
{quote} Hi exidez, thanks for your contribution, it takes time to write SW. Unfortunately, there is a little problem with time zones. When downloading with ff default time zone settings (as it is done), the All day/Tentative events get wrong day(!) time stamps sometimes: "2016/09/18","00:01","JPY","N","Bank Holiday","","","","","60346" <event> <title>Bank Holiday</title> <country>JPY</country> <date> <=!=[=C=D=A=T=A=[ 09-19-2016 ]=]=> </date> <time> <=!=[=C=D=A=T=A=[ 12:00am ]=]=> </time> <impact> <=!=[=C=D=A=T=A=[ Holiday ]=]=> </impact> <forecast/>...
I appreciate your help in finding these little bugs. I do exepect there to be some errors as i havent tested this extensivly. I have spent a little time trying to find this bug and it looks like its not getting the order or rows right when there are blank spaces (for example on that Sunday). I will have to spend some more time looking into this.

Edit: I spent way too much time on that then i should have. The program works are expected, What you are seeing is the difference in timezone. The XML snippet you showed me with the "correct times" has your time zone shifted into it. If you take 5 hours from that time you will get the default ForexFactory time and date (GMT-5) which is what the newsdownloader recieves from FF.
Also as for the times that DO have a schedule, this is becuase the calendar has been updated after you have already run the newsdownloader script. If you run the script again you should get an updated time. That is if the calendar on FF is also updated.

Quoting Graziani
Also, in the first week when the year changes, date is not changed to new year: "2014/12/31","00:01","CNY","N","Bank Holiday","","","","","50662" "2014/12/31","00:01","JPY","N","Bank Holiday","","","","","55338" "2014/12/31","21:00","CNY","H","Manufacturing PMI","50.1","50.0","50.3","","55338" "2014/12/31","21:09","CNY","M","Non-Manufacturing PMI","54.1","","53.9","","53455" "2014/01/01","00:01","CHF","N","Bank Holiday","","","","","53455" "2014/01/01","00:01","EUR","N","French Bank Holiday","","","","","51819" "2014/01/01","00:01","EUR","N","German...
Thanks, i woudlnt have know about this bug until it came to date. But seeing this now, I know exactly whats wrong. I will include a fix for this in the next update
  • #199
  • Sep 22, 2016 12:10am Sep 22, 2016 12:10am
BAT file to run Python after install of the Python 3.5....very simple

Rename it and remove the ".zip"

It can be edited with notepad if u want to see the inside.
Attached File(s)
File Type: zip PythonNewsRunner.bat.zip   < 1 KB | 455 downloads
  • #200
  • Edited Sep 23, 2016 12:40am Sep 22, 2016 2:10am | Edited Sep 23, 2016 12:40am
Thanks for that fxworldwide

Version 1.4 has been released to include some bug fixes.
- Fixed bug with wrong year in first week of a following year
- fixed error when all arguments are not given
- added option for high impact news only (use -I as an argument to output only high impact)


Executable: https://drive.google.com/file/d/0B9K...ew?usp=sharing
Python Script: https://drive.google.com/file/d/0B9K...ew?usp=sharing

Edit:
For those who havce already installed python, you may need to update the beautifulsoup module that is a part of this script.
update by typing the below in command prompt:
python -m pip install --upgrade beautifulsoup4
0 traders viewing now