Plot News / Downloader version 4.00 | Page 20 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

FFcal - Plot News indicator 692 replies

Plot News indicator (version 3.0) 402 replies

Good History Rate Downloader EA 14 replies

Plot News (version 2.0) 103 replies

Attachments: Plot News / Downloader version 4.00
Exit Attachments

Plot News / Downloader version 4.00

  • #381
  • Nov 19, 2020 5:18pm Nov 19, 2020 5:18pm
Quoting forevermaat
{quote} I have a python scraper in the works that is downloading the html just fine. I think for your downloader, it is simply a matter of using a different function to return html to download. (AHK or AutoIT). And seeing if you must pass along some quasi-browser headers.
please share the "quasi-browser headers" you pass with your python scraper; thanks!
  • #382
  • Nov 20, 2020 1:30pm Nov 20, 2020 1:30pm
Quoting forevermaat
I have a python scraper in the works that is downloading the html just fine.
Then (if you wanted to) all you would need to do is extract the relevant text (date, time, currency, description, impact, etc) from between the html tokens and write this data to a news.csv file, and you have created your own news downloader. The format of the FF calendar html is portrayed in a graphic here. As far as I know it hasn't changed since then.

Apart from URLDownloadToFile, AHK has ComObjCreate which allows the programmer to create com objects from various web/comms classes but I haven't done much work in this area before and I've no reason to turn something like Plot News into a lengthy R&D project.
  • #383
  • Nov 22, 2020 2:02am Nov 22, 2020 2:02am
Quoting dbarney
{quote} please share the "quasi-browser headers" you pass with your python scraper; thanks!
I think 2 different methods are employed:

Weekly json downloader was working for quite some time (daily, for over a year). For the weekly json/csv/xml scraper, the url download function was changed from urllib to requests library. Urllib didn't handle the redirects as well after DDoS attack. But requests library had no problems. Since the final file is already in machine-ready format (e.g. json), then you just use json parser + custom functions to convert into local csv in the desired time zone you want. And archiving function.

For the larger calendar scraper, cloudscraper module is used to handle the url html fetch. (Selenium could have been used, but it is slow). Then beautifulsoup4 handles most of html parsing. And then some custom manipulation functions to deal with situations like events without an exact timestamp. And archiving function.

ForexFactory's main calendar is the more cumbersome to scrape of the major fx, because of the quirks. Other calendars like fx street or investing.com you can customize the timezone and "tentative" events all have estimated times in the html. Or when downloading csv.

the weekly json/ file is machine-ready though...just missing some data columns (intentionally to make human visit main calendar?!?)
  • #384
  • Jun 1, 2021 3:12am Jun 1, 2021 3:12am
Hi everyone, How are you doing?

I am new to forex factory and I'd like to integrate forex factory with my app.
I am currently building an app for the traders and I need to get the news data from the forexfactory site.
Is there any other way to fetch the news by calling the API?

Please help me.

Thanks
Yazan
  • #385
  • Jul 1, 2021 3:14am Jul 1, 2021 3:14am
So many sites are starting to use firewalls I can't even download data into Excel from Investing.com it requires a password OMG!

We need to be able to get Current News into MT4 for each Currency pair so we don't have to keep going to different sites to read up

Also always include Actual outcome results like

! IN10TION NewsReader v09.99z lite
Attached File(s)
File Type: ex4 ! IN10TION NewsReader v09.99z lite.ex4   298 KB | 548 downloads
  • #386
  • Dec 23, 2021 8:01am Dec 23, 2021 8:01am
Hae. Can the indicator work in windows 10
  • #387
  • Jan 13, 2022 11:13am Jan 13, 2022 11:13am
great work, thanks.
so this workable by win 7 ?

Quoting hanover
Oct 29, 2020: Sorry, the Plot News EA and News Downloader no longer work, at least not on Windows 10. They are unable to penetrate the Cloudflare firewall which FF have apparently installed to prevent DDoS attacks. ____________________________________________________________________ This thread is in the Commercial area because I wanted to make it less conspicuous. Everything here is 100% FREE, no strings attached. Forex Factory have advised me that there is a virus in one of the files here. As a result, the content of this post (#1) has been removed,...
  • #388
  • Feb 21, 2023 2:12pm Feb 21, 2023 2:12pm
is anyone else able to generate historical news and holidays as a csv file?
  • #389
  • Edited Sep 29, 2023 5:22am Sep 28, 2023 8:27pm | Edited Sep 29, 2023 5:22am
I've been awol for a long time, not sure if anybody's still following this thread. Haven't looked at forex or MT4 for around 2 years now. Still haven't looked at MT5, LOL. I'm too busy working on a couple of Python projects to do any MQL4 coding.

While experimenting with Python I rewrote the news downloader. Looks like cloudflare is still blocking access to the FF calendar by bots, so I have to save the html to a file manually, and only then do I get some data for my Python code to process. This code is not in a user-friendly state, so I'm not sharing it, at least not yet.

Anyway, the purpose of this post is to share some of the outputs. As a starting point, here is a csv file of the 3,485 news events from the FF calendar for 2023 this far. This differs from all of the other news readers in that it outputs all of the actual, previous, forecast and revised values. All dates and times are UTC. The format is:
date, time, currency, description, impact, actual, previous, forecast, revised from

Hope it's useful to somebody. If it is useful, let me know, and then when I get time, I'll process 2022, then 2021, and so on. It's a somewhat tedious process because the FF calendar has to be downloaded manually one month at a time.

[EDIT] attached 2022 news events file. 4,690 news events for 2022.
Attached File(s)
File Type: csv news_output 2023.csv   266 KB | 401 downloads
File Type: csv news_output 2022.csv   357 KB | 395 downloads
  • #390
  • Sep 29, 2023 3:06am Sep 29, 2023 3:06am
Hello Hanover, thank you very much for your excellent work. Would it perhaps be possible for you to give an example of how to download the data from December 2022, for example? I can only download the current month and the month before. No other options appear for me.

Thank you
the highest point comes before the fall
  • #391
  • Sep 29, 2023 4:28am Sep 29, 2023 4:28am
Quoting Pivoter
Hello Hanover, thank you very much for your excellent work. Would it perhaps be possible for you to give an example of how to download the data from December 2022, for example? I can only download the current month and the month before. No other options appear for me. Thank you
Not sure what you mean by "download". The news events have already been downloaded/extracted from FF calendar HTML into CSV format in the file attached to the post. You can load this file into Excel or a text editor and process/manipulate the data (filter, sort, total, perform calculations, etc) however you wish. All of the news events from 1 Jan 2023 to the current date are in the file in chronological order. The first few entries in the file look like this:

"2023.01.01","20:00","NZD","Bank Holiday","N","","","",""
"2023.01.01","21:00","AUD","Bank Holiday","N","","","",""
"2023.01.02","00:00","JPY","Bank Holiday","N","","","",""
"2023.01.02","00:01","CNY","Bank Holiday","N","","","",""
"2023.01.02","06:00","CHF","Bank Holiday","N","","","",""
"2023.01.02","08:00","GBP","Bank Holiday","N","","","",""
"2023.01.02","08:15","EUR","Spanish Manufacturing PMI","L","46.4","45.7","45.9",""
"2023.01.02","08:45","EUR","Italian Manufacturing PMI","L","48.5","48.4","48.5",""
"2023.01.02","08:50","EUR","French Final Manufacturing PMI","L","49.2","48.9","48.9",""
"2023.01.02","08:55","EUR","German Final Manufacturing PMI","L","47.1","47.4","47.4",""
"2023.01.02","09:00","EUR","Final Manufacturing PMI","L","47.8","47.8","47.8",""
"2023.01.02","13:00","CAD","Bank Holiday","N","","","",""
"2023.01.02","13:00","USD","Bank Holiday","N","","","",""
"2023.01.02","20:00","NZD","Bank Holiday","N","","","",""
"2023.01.03","00:00","JPY","Bank Holiday","N","","","",""
"2023.01.03","01:45","CNY","Caixin Manufacturing PMI","L","49.0","49.4","49.2",""
"2023.01.03","05:30","AUD","Commodity Prices y/y","L","15.6%","19.1%","","19.6%"
"2023.01.03","07:29","EUR","German Prelim CPI m/m","L","-0.8%","-0.5%","-0.6%",""
"2023.01.03","08:00","EUR","Spanish Unemployment Change","L","-43.7K","-33.5K","-40.3K",""
"2023.01.03","08:30","CHF","Manufacturing PMI","L","54.1","53.9","53.0",""
"2023.01.03","08:55","EUR","German Unemployment Change","L","-13K","17K","15K","15K"
"2023.01.03","09:30","GBP","Final Manufacturing PMI","L","45.3","44.7","44.7",""
"2023.01.03","14:30","CAD","Manufacturing PMI","L","49.2","49.6","",""
"2023.01.03","14:39","NZD","GDT Price Index","L","-2.8%","-3.8%","",""
"2023.01.03","14:45","USD","Final Manufacturing PMI","L","46.2","46.2","46.2",""
"2023.01.03","15:00","USD","Construction Spending m/m","L","0.2%","-0.3%","-0.4%","-0.2%"
"2023.01.04","00:01","GBP","BRC Shop Price Index y/y","L","7.3%","7.4%","",""
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 114 KB
  • #392
  • Edited Sep 30, 2023 8:16pm Sep 29, 2023 10:51pm | Edited Sep 30, 2023 8:16pm
The files attached to post #389 had some minor issues, I have fixed them:

1. I have changed the order so that the 'impact' column/field now comes before 'description', like it does in FF calendar.
2. I have changed the order so that the 'forecast' column/field now comes before 'previous', like it does in the FF calendar.
3. The 'revised from' and 'previous' were being output in the wrong order (whenever there was a non-blank value in revised from'), I have fixed that also.

Post #389 has gone past the 24 hour editing limit, so I can't modify or delete it. So please just ignore the files attached there, and download the corrected ones below instead. Again a reminder that the news events in all of these files are timestamped UTC.
Attached File(s)
File Type: csv FF calendar news events 2023.csv   269 KB | 484 downloads

Attached File(s)
File Type: csv FF calendar news events 2022.csv   362 KB | 446 downloads


[EDIT] Uploaded 2021 news events file. (4,725 events)
Attached File(s)
File Type: csv FF calendar news events 2021.csv   365 KB | 409 downloads


[EDIT] Uploaded 2020 news events file. (4,736 events)
Attached File(s)
File Type: csv FF calendar news events 2020.csv   367 KB | 423 downloads


[EDIT] Uploaded 2019 news events file. (4,616 events)
Attached File(s)
File Type: csv FF calendar news events 2019.csv   356 KB | 389 downloads


[EDIT] Uploaded 2018 news events file. (4,591 events)
Attached File(s)
File Type: csv FF calendar news events 2018.csv   354 KB | 420 downloads


[EDIT] Uploaded 2017 news events file. (4,613 events)
Attached File(s)
File Type: csv FF calendar news events 2017.csv   356 KB | 483 downloads


Back in post #1 of this thread there's a zip file with news events from 2007-01-01 thru 2020-11-06 which means there's some significant overlap with the files above. However, events in the files attached to post #1 are timestamped New York time (UTC-5; or UTC-4 for DST), whereas events in the files above are all timestamped UTC. It shouldn't be too difficult to import these into Excel and write formulae to convert the times, if they are critical to you, and you need history earlier than 2017. New York starts DST at 2am on the second Sunday in March and reverts to normal time at 2am on the first Sunday in November. More information here.


[EDIT]
I've zipped all of the attached files and then attached the zip file to post #1 (for which there is no time limit on editing). The timestamp of the news events in all of these files is UTC.

I've also created new files that are exactly the same as the attached files, except that their news events are timestamped New York time (UTC–5 during winter and UTC–4 during summer). I've zipped all of these up and attached them to post #1 also.
  • #393
  • Oct 2, 2023 4:25pm Oct 2, 2023 4:25pm
I re-checked all of the outputs in the files I uploaded to post #392 and some of these appear incorrect in an strange way. Specifically, there seem to be cases, especially in the 2022 year, where entire months have overwritten another months' worth of news events.

I've attempted to fix this (hopefully third time lucky) and re-uploaded the zip files to post #1. I won't attempt to upload files to new posts because any errors can't be corrected after 24 hours.

Sorry for any inconvenience.
  • #394
  • Oct 6, 2023 11:16pm Oct 6, 2023 11:16pm
I've uploaded up-to-date zip files to post #1. These now have an extra 2 years of news events dating all the way back to 1/1/2015 and up to today (Oct 6, 2023). Also if Forex Factory have updated the numbers in any old news events (within this date range) then these updates will be reflected in the data.
  • #395
  • Oct 7, 2023 11:40pm Oct 7, 2023 11:40pm
I've uploaded up-to-date zip files to post #1. These now have an extra 2 years of news events dating all the way back to 1/1/2013.

Also if Forex Factory have updated the numbers in any old news events (within this date range) then these updates will be reflected in the data.

There are now a total of 132 months' worth of data in each zip file. This has given my python app a pretty exhaustive test and it appears to be working 100%. I won't be sharing it because there is still a lot of manual work involved in downloading the calendar html, and I don't want to write instructions and answer questions.
  • #396
  • Oct 22, 2023 12:26pm Oct 22, 2023 12:26pm
Quoting hanover
I've uploaded up-to-date zip files to post #1. These now have an extra 2 years of news events dating all the way back to 1/1/2013. Also if Forex Factory have updated the numbers in any old news events (within this date range) then these updates will be reflected in the data. There are now a total of 132 months' worth of data in each zip file. This has given my python app a pretty exhaustive test and it appears to be working 100%. I won't be sharing it because there is still a lot of manual work involved in downloading the calendar html, and I don't...
Many thanks for this Hanover, this is a very useful format for this information.
Everything will be okay in the end. If it's not okay, it's not the end.
  • #397
  • Oct 24, 2023 11:50pm Oct 24, 2023 11:50pm
Quoting theras2000
Many thanks for this Hanover, this is a very useful format for this information.
Thanks. Glad to hear it's helpful.

@anybody who's interested:
Files in post #1 have been updated to include news from 2011-01-01 thru 2023-10-21.
  • #398
  • Edited 10:33pm Nov 18, 2023 12:03am | Edited 10:33pm
@anybody who's interested:
Files in post #1 have been updated to include news from 2007-01-01 thru 2023-12-31.

These two zip files now cover all of the data that's available on the FF calendar. Data runs through to the end of this year (news events that haven't occurred yet have their 'Actual' value set to "").

Just a reminder that the format is:
"date","time","currency","impact","description","actual","forecast","previous","revised from","event id"

Events in the first zip file are timestamped UTC; in the second zip file New York time (UTC–5 during winter and UTC–4 during summer).

Hope this is useful to somebody.

That's it for now.

[EDIT] Sorry, but I don't know how much longer I'll be keeping these files up to date. I only rewrote the downloader as a part of an exercise in teaching myself Python. The process of obtaining the data also involves some manual work, hence there's no point in sharing the program code, and I don't have the time or the stamina for user questions and lengthy explanations. I'm not even supposed to be here, lol.
  • #399
  • Dec 4, 2023 4:57pm Dec 4, 2023 4:57pm
@anybody who's interested:
Files in post #1 have been updated again, to include latest news events from 2007-01-01 thru 2023-12-31 (specifically the "actual" and "revised from" values up until 2023-11-30, now that this date has long since passed and hence the values for the whole month are now available.

I'd like to wish you and yours a happy and restful holiday season and a prosperous 2024.
  • #400
  • Dec 5, 2023 9:36am Dec 5, 2023 9:36am
Quoting hanover
@anybody who's interested: Files in post #1 have been updated again, to include latest news events from 2007-01-01 thru 2023-12-31 (specifically the "actual" and "revised from" values up until 2023-11-30, now that this date has long since passed and hence the values for the whole month are now available. I'd like to wish you and yours a happy and restful holiday season and a prosperous 2024.
Thank you Hanover, I wish you all the best and much success in the new year. Thanks for the excellent work with the calendar events.
the highest point comes before the fall
0 traders viewing now