Plot News / Downloader version 4.00 | Page 19 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

FFcal - Plot News indicator 692 replies

Plot News indicator (version 3.0) 402 replies

Good History Rate Downloader EA 14 replies

Plot News (version 2.0) 103 replies

Attachments: Plot News / Downloader version 4.00
Exit Attachments

Plot News / Downloader version 4.00

  • #361
  • Oct 27, 2020 2:06pm Oct 27, 2020 2:06pm
BTW David....the fact u using Win 7 means I will have to call Bill and tell him........BUT not sure if it's an endorsement of Win 7 or a Rejection of Win 10.....LOL.

I already know ur position ha ha.

One Love bro.
  • #362
  • Oct 27, 2020 2:54pm Oct 27, 2020 2:54pm
I was doing a simply http screen scrape to grab the html file and parse that was working perfectly, but since 10/21, I get a "403 forbidden" error from forexfactory.com

Perhaps news.csv is not created due to the same issue?

Perhaps they no longer allow us to download the data behind the webpage automatically?

I can still do it manually though....
  • #363
  • Edited Oct 28, 2020 3:16pm Oct 27, 2020 4:19pm | Edited Oct 28, 2020 3:16pm
Many thanks for your replies.

I ran the downloader app on a Windows 10 laptop, and instead on downloading the news webpage HTML, it returns a file with the following text message:

Please enable cookies.
One more step

Please complete the security check to access www.forexfactory.com
Please turn JavaScript on and reload the page.
Please enable Cookies.
Why do I have to complete a CAPTCHA?

Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.
What can I do to prevent this in the future?

If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.

If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.
Cloudflare Ray ID: [16 digit hex number] * Your IP: [32 digit hex number separated with colons] * Performance & security by {Cloudflare}

This file normally contains the HTML of the FF calendar. It is parsed by the downloader which eventually builds a news.csv. Then it deletes the text file, and other temporary files (while it displays the 'housekeeping' message). Since the downloader can't find any news in the blue text above, it is unable to create a news.csv file.

With most web browsers it is possible to load the FF calendar, press Ctrl-U (or whatever the hotkey is) to obtain the HTML, and then manually copy/paste it into a file.

I suspect that, to combat future DDoS attacks, FF are using Cloudflare as a security measure, but it also prevents access of their website by harmless automated processes like the downloader.

Any suggestions are welcome. I don't know much about internet security. But I'm guessing that unless there is a scraper that's capable of either disabling or penetrating Cloudflare, that I could somehow call from the downloader, then I'm afraid it's the end of the road for the latter. I guess it's had a good run, and at least the Plot News indicator is still capable of getting this week's news events from the XML page.
______________________________________________________________________

Quoting forevermaat
Is there alternate methods for AutoHotKey to "visit" the page?
URLDownloadToFile is AHK's native function to scrape a webpage. However, you can alternatively call any (external) executable using Run or RunWait. Before I discovered URLDownloadToFile I used RunWait, calling wget and later curl in previous versions of Plot News. I tried both of these again just now, albeit with just the basic switches, and you can see the results in the screenshot:
Attached Image (click to enlarge)
Click to Enlarge Name: PPI 1387.png Size: 52 KB

As can be seen, wget encounters a '403 forbidden error' and curl an 'error 1020'. It seems that error 1020 is caused by Cloudflare (read this). I also attempted to run the Plot News v4-00 b600 EA, which uses MQL code to scrape the calendar and, even on Windows 7, it returns a zero size news.csv file.

I think I read somewhere in my travels that URLDownloadToFile uses Internet Explorer. But as can be seen, the AHK documentation doesn't provide this info.

I believe the only way to beat this is to find an app that's capable of disabling Cloudflare, or a scraper that is capable of penetrating its firewall, and then using AHK's RunWait command to call it. Either that or run news dowloader (i.e. URLDownloadToFile) in a Win7 environment (which somehow manages to bypass Cloudflare). The former would involve trying to find a loophole in a firewall that's been developed by a multi-million dollar company, and unless you are a hacker who's more expert in internet security than they are, the task would be futile. I am no expert, and don't have the time in any case. But if anybody does find a command or executable that disables or infiltrates Cloudflare, I'll very happily modify the the downloader to run it.
  • #364
  • Oct 28, 2020 3:46am Oct 28, 2020 3:46am
Quoting hanover
....But I'm guessing that unless there is a scraper that's capable of either disabling or penetrating Cloudflare, that I could somehow call from the downloader, then it's the end of the road for the latter. The Plot News indicator is still capable of getting this week's news events from the XML page.
Is there alternate methods for AutoHotKey to "visit" the page?

Quoting hanover
{quote} Thank you sir for the info. The downloader uses autohotkey's URLDownloadToFile command to download the FFcalendar as a HTML file....... The process seems to work fine on my old Win7 PC. I will test it on my Win10 laptop as soon as I get a chance, and report back here.
An important question is: what browser is being used by autohotkey when calling the file?

Perhaps an alternative download function may have to be used. Or a separate exe/dll that loads a more compatible headless browser will need to be used.

That's what I am thinking.

----

Regarding the xml download still working, the csv/json/xml file is designed natively to be scraped.
  • #365
  • Edited Oct 30, 2020 12:38am Oct 29, 2020 10:49pm | Edited Oct 30, 2020 12:38am
I wanted to personally thank you for all you have done up to this point.

I think that the scraper portion needs to incorporate headless browser that can more properly emulate "real" browsing so that it will properly return the page.

Or if AHK has an alternative functions to request webpage that will work.

with python, urllib function didn't work anymore. But requests.get function with redirects enabled seems to work.

Haven't tested with java yet, but will see this weekend or next weekend. But it requires having java JRE installed (just like you need python + modules installed to run py scripts)


Edit: I wanted to mention a AHK alternative: AutoIT Maybe it has download functions that AHK does not.
-----
  • #366
  • Oct 30, 2020 4:11am Oct 30, 2020 4:11am
Quoting forevermaat
I think that the scraper portion needs to incorporate headless browser that can more properly emulate "real" browsing so that it will properly return the page. Or if AHK has an alternative functions to request webpage that will work................I wanted to mention a AHK alternative: AutoIT Maybe it has download functions that AHK does not.
I wrote Plot News for my own use (it's been through four different incarnations since 2007), and during a period where I had plenty of spare time on my hands, and it didn't take much more effort to share it.

URLDownloadToFile is AutoHotKey's only native command to scrape a webpage.

I haven't used AutoIt. I understand it was a forerunner to AutoHotkey, before developers Jon Bennett and Chris Mallett wanted to take their personal design ideas down different paths. Reading the online doco it looks like AutoIt has commands like InetGet and InetRead. If AutoIt can create standalone executables then the News Downloader could potentially call one to scrape the webpage before performing its parsing. Otherwise it would be necessary to rewrite the downloader in AutoIt, a language that I have no previous experience with.

I don't know whether AHK employs a headless browser, I thought I'd read that it uses IE somewhere in my travels, but I could easily be mistaken. Its online doc doesn't mention anything.

It sounds like Python and Java can't produce standalone exceutables, i.e. they need to have some kind of runtime present.

I had a quick look at possible alternatives to wget. I don't want to download and test all 48 of them on the off chance that one happens to provide a command line option that's capable of infiltrating Cloudflare. Again, I doubt that a muti-million dollar security company would leave such a loophole, but of course you never know, you might get lucky. If somebody wants to do that, and they find a command line that achieves that, then I'll very happily include it in the News Downloader.

An alternative would be to set up a VM to run Win7 and MT4, since we know that the News Downloader's URLDownloadToFile somehow bypasses Cloudflare when run on Win7. Or you could just run Plot News and get the data from FF's weekly XML page (although it doesn't have previous week's news, or any 'actual' values). Anyway, I'm just repeating what I said in my previous post.
  • #367
  • Oct 30, 2020 9:36am Oct 30, 2020 9:36am
Mornin David,

How do I point News Downloader.exe to the XML page? This would minimum give a LIST and an ALERT on MT4 chart if I can get them into news.csv.

thks
  • #368
  • Oct 30, 2020 1:56pm Oct 30, 2020 1:56pm
Quoting fxworldwide
Mornin David, How do I point News Downloader.exe to the XML page? This would minimum give a LIST and an ALERT on MT4 chart if I can get them into news.csv. thks
Hello sir,

It is Plot News, not the Downloader exe, that points to the XML page. The only way I've been able to get it to work is as follows:

1. If you want the news 'billiard balls' (actually I think they're more like 8-ball pool balls ) to plot in a separate subwindow, download and attach the Plot News subwindow.ex4 indicator to your MT4 chart. All this does is create an empty new subwindow under the main chart.

2. Plot News is now an EA (the attached Plot News v4-00 b600.ex4 EA needs to be installed in your ...../MQL4/Experts folder). I have highlighted the settings that I have changed from the defaults. The first value in TimeOffsets needs to be 0 because the times in the XML are GMT, not FF calendar (New York) time. You can probably set the RefreshPeriod to a longer TF than M1 unless you want alerts to occur every minute. The entire DownloaderSettings section can be ignored because the data can no longer be downloaded from FF calendar due to the presence of Cloudflare. The rest of the settings should behave much the same as they did for the Plot News indicator. (Press F7 to change the parameter settings for an EA).

Please feel welcome to post any questions, and I'll do my best to answer them. Take care and stay safe.

David
Attached Image(s) (click to enlarge)
Click to Enlarge Name: PPI 1405.png Size: 27 KB
Click to Enlarge Name: PPI 1404.png Size: 85 KB
Click to Enlarge Name: PPI 1399.png Size: 31 KB
Attached File(s)
File Type: ex4 Plot News v4-00 b600.ex4   467 KB | 929 downloads
File Type: ex4 Plot News subwindow.ex4   6 KB | 933 downloads
  • #369
  • Oct 30, 2020 5:11pm Oct 30, 2020 5:11pm
BIG thks.......I seem to have got it implemented......further tests on Monday. The EA seems to create a news.csv file.

Again THANK YOU.

AM
  • #370
  • Nov 1, 2020 6:52am Nov 1, 2020 6:52am
Quoting hanover
{quote}...Reading the online doco it looks like AutoIt has commands like InetGet and InetRead. If AutoIt can create standalone executables then the News Downloader could potentially call one to scrape the webpage before performing its parsing. Otherwise it would be necessary to rewrite the downloader in AutoIt, a language that I have no previous experience with.
AutoIT CAN create stand-alone executables so the user doesn't need AutoIT installed. I think it is the Aut2Exe module. https://www.autoitscript.com/autoit3...o/compiler.htm
also the download page repeats this: https://www.autoitscript.com/site/autoit/downloads/

This might also be helpful: User functions for browsers
https://www.autoitscript.com/wiki/Us...tions#Browsers

Which got me thinking, does AHK have something similar (functions that expand what AHK can do with IE or other browsers like Chrome / Firefox / edge.) That might solve the problem if it is due to the browser not accepting certain cookies or loading certain javascript functions.


---


Quoting hanover
...I don't want to download and test all 48 of them on the off chance that one happens to provide a command line option that's capable of infiltrating Cloudflare. Again, I doubt that a muti-million dollar security company would leave such a loophole, but of course you never know, you might get lucky. If somebody wants to do that, and they find a command line that achieves that, then I'll very happily include it in the News Downloader. An alternative would be to set up a VM to run Win7 and MT4, since we know that the News Downloader's URLDownloadToFile...

I wonder why the bypassing exists? Is it connected to which browser is used natively? I think with later versions of windows, a different expectation of how to access webpages in win8+ vs win7, which is why they are letting it through. Not sure if this is browser based, or if a different protocol is used in later browsers.
  • #371
  • Edited 7:04pm Nov 1, 2020 6:21pm | Edited 7:04pm
Quoting fxworldwide
I seem to have got it implemented......further tests on Monday. The EA seems to create a news.csv file.
Sounds promising. Hope the testing goes ok.
Quoting forevermaat
{quote} AutoIT CAN create stand-alone executables.......
Thanks for the info. I haven't looked at AutoIT before so everything is news to me.

I don't understand how web browsers work 'under the bonnet'. I'm an old COBOL / C / RPG II programmer who hasn't done much work in either the web or the comms areas. I've spent most of my career developing accounting and customized database apps.

All I can do is install AutoIT, write one line of code to perform an InetRead and output the data to a file for NewsDownloader.exe to read, compile that to an exe, and then get NewsDownloader.exe to call that executable. I doubt that it will crack Cloudflare, but I suppose anything is worth a try.

Like I said before, anybody is welcome to try any of the 48 alternatives to wget, or any other scraper/crawler, and if they find one that infiltrates Cloudflare and creates a html file, then I'll add that command into NewsDownloader.exe, and the problem is solved.

Here's an example of the command to run the first of the 48 alternatives, namely aria2:
aria2c -o output.html https://www.forexfactory.com/calendar?week=oct18.2020

Unlike wget and curl, aria2c generates the correct FF calendar HTML file in Windows 7; somebody might like to download Aria2 (just click on aria2-1.35.0-win-64bit-build1.zip, and then extract the aria2c.exe from the zip) and try it on Windows 10. I'll try it when my wife returns home from work with her Win10 laptop.
  • #372
  • Nov 2, 2020 1:47am Nov 2, 2020 1:47am
I suspect cloudflare is looking at the headers and agent and blocking it based on missing information or explicit blocking of none browser agents. You can configure all of these with the right programming library where you can add the headers for the agent details etc. Have you tried doing a http trace to see if there is any other info?

in any case you need to be able to alter the headers to make it look like it's coming from a browser
  • #373
  • Nov 2, 2020 3:53am Nov 2, 2020 3:53am
Quoting EcoTrader
I suspect cloudflare is looking at the headers and agent and blocking it based on missing information or explicit blocking of none browser agents. You can configure all of these with the right programming library where you can add the headers for the agent details etc. Have you tried doing a http trace to see if there is any other info? in any case you need to be able to alter the headers to make it look like it's coming from a browser
Many thanks for your reply sir. I've enjoyed reading your profound insights into fundamental analysis. I didn't know that you were also skilled in http requests -- a man of many talents!! I expect that you know more about http requests than I do, as I'm no expert when it comes to web browsers.

[Note to self: As an alternative to URLDownloadToFile, I found some material on using GET and POST in autohotkey here. I also note that aria2 allows header info to be sent in the request, including user agent, e.g. here. I will experiment further when I get some time].

Thanks again. I wish you much continued success with your trading. Take care and stay safe.
  • #374
  • Nov 2, 2020 4:26am Nov 2, 2020 4:26am
you can get more info on what you are sending and other issues you wont normally see if you run this app while testing the code - its free

https://www.telerik.com/fiddler
  • #375
  • Nov 2, 2020 12:44pm Nov 2, 2020 12:44pm
Quoting EcoTrader
you can get more info on what you are sending and other issues you wont normally see if you run this app while testing the code - its free https://www.telerik.com/fiddler
Excellent. Looks like telerik have written several useful apps, some paid, some free.

Many thanks.
  • #376
  • Nov 6, 2020 8:33pm Nov 6, 2020 8:33pm
I'm just pleased as can be to know someone's still using Plot News + will try and set it up for myself sometime. (Don't get much chance to mess with MT4 now.)
  • #377
  • Edited Nov 8, 2020 12:16pm Nov 7, 2020 11:13pm | Edited Nov 8, 2020 12:16pm
I have an old PC running Windows 7, where News Downloader.exe continues to create a news.csv file. Hence I've updated the zip in post #1 to include all 61,889 news events on FF calendar from Jan 1, 2007 thru Nov 6, 2020. And I've attached a news.csv file below for the events for the upcoming week, i.e. Sunday Nov 8, 2020 thru Sat Nov 14, 2020.
Attached File(s)
File Type: csv news.csv   8 KB | 865 downloads
  • #378
  • Nov 12, 2020 9:51am Nov 12, 2020 9:51am
How do you normally handle events with time like "12th-18th"? How would they appear in the csv?

Attached Image (click to enlarge)
Click to Enlarge Name: forexfactory hanover handle multiple dates in single day.png Size: 91 KB

https://www.forexfactory.com/calenda...020-nov28.2020
  • #379
  • Edited 4:13pm Nov 12, 2020 3:04pm | Edited 4:13pm
Quoting forevermaat
How do you normally handle events with time like "12th-18th"? How would they appear in the csv?
Good question. I looked in the file attached to post #377 and it appears as in the image below.

Any time value on the calendar that's not in the format hh:mm[am|pm] is considered 'untimed' and has its time set to your entry in 'Set time for untimed events to' (MT4 time), effectively as a placeholder. You can also enter a 'p' in this field to have it shifted back 24 hours to the previous day. Normally you would set this to either 17:00 New York time or midnight MT4 time (both were the same time for my broker). Mine is actually set incorrectly, which explains why the date is the 11th, but I don't care as I haven't been using MT4 for a while now. The text ('12th-18th' in this case) is appended to the Description inside square brackets, so that it is not lost altogether. It can be seen how the same has occurred for other untimed ('All Day' and 'Tentative') news events also.

However, unless somebody using Windows 10 has found a way to bypass Cloudflare and scrape the FF calendar html, I guess this would be of academic interest only.
Attached Image (click to enlarge)
Click to Enlarge Name: PPI 1413.png Size: 14 KB
  • #380
  • Nov 19, 2020 3:05pm Nov 19, 2020 3:05pm
Quoting hanover
{quote} .....

However, unless somebody using Windows 10 has found a way to bypass Cloudflare and scrape the FF calendar html, I guess this would be of academic interest only. {image}
I have a python scraper in the works that is downloading the html just fine.

I think for your downloader, it is simply a matter of using a different function to return html to download. (AHK or AutoIT). And seeing if you must pass along some quasi-browser headers.
1 trader viewing now