Plot News / Downloader version 4.00 | Page 18 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

FFcal - Plot News indicator 692 replies

Plot News indicator (version 3.0) 402 replies

Good History Rate Downloader EA 14 replies

Plot News (version 2.0) 103 replies

Attachments: Plot News / Downloader version 4.00
Exit Attachments

Plot News / Downloader version 4.00

  • #341
  • Apr 12, 2020 1:57pm Apr 12, 2020 1:57pm
Quoting hanover
{quote} We discussed this a little while ago (see post #303). It appears that there would be a significant amount of work involved......
No problem. Didn't know what your schedule was like now with Covid. No pressure.

Quoting hanover
Meanwhile, it looks like Plot News downloader is creating an empty news.csv file. It started happening around last Thursday. I see that FF have changed their calendar URL from https://www.forexfactory.com/calendar.php to https://www.forexfactory.com/calendar I changed the code to accommodate this, but it still created an empty news.csv I'm still troubleshooting. It is bizarre: autohotkey's URLDownloadToFile command works perfectly when downloading other sites, but no longer for FF calendar, and I don't know why. ....
I think you should externalize the url to scrape in the inputs/ini file. So it might look like this:

Inserted Code
MonthlyURL=https://www.forexfactory.com/calendar?month={MMM.YYYY}
WeeklyURL=https://www.forexfactory.com/calendar?week={MMMD.YYYY}

also handle automatic redirects as the server. I believe the 301 redirects are being handled already by existing News Downloader.exe Must be in my case; I run task every day and I did not get an error that prevented exe from scraping and reporting to windows as task complete.

--------------------------------------------

Also, I have another minor bug to report the hotkey ^j error.

Attached Image


(it happened before, but I don't think it changed anything material in the downloading process). in other words, the download of the calendar data for news calendar.exe has continued.

but again, news calendar.exe still scrapes just fine as of a couple of hours ago.

Quoting shirotama
Can someone help me please, when i try to download forex calender i got this error {image}
has the News Calendar.exe file run successfully before?
  • #342
  • Edited 3:12pm Apr 12, 2020 2:57pm | Edited 3:12pm
Quoting shirotama
Can someone help me please, when i try to download forex calender i got this error
I think the curl files are missing from the zip, a careless error by me. Sorry. I will remedy this in the next release. See post #337 for more information.
___________________________
Quoting forevermaat
No problem...........
Good idea to externalize the URL. I will implement it in the next release.
Quoting forevermaat
Also, I have another minor bug to report the hotkey ^j error.
I've never seen this error before, and don't know how to reproduce it (in order to troubleshoot). All I can suggest is that you try leaving the 'Macro for indicator refresh' parameter blank, and refresh the Plot News indicator on your MT4 chart manually.
  • #343
  • Apr 15, 2020 2:12pm Apr 15, 2020 2:12pm
Quoting shirotama
Can someone help me please, when i try to download forex calender i got this error
I have added the curl files to the Plot News v4-16.zip file in my github repository.
  • #344
  • Apr 23, 2020 9:22am Apr 23, 2020 9:22am
Quoting pips4life
{quote} Hi Hanover, This Jan post of mine was regarding my observations of the timezone offsets for the Download News (news.csv) vs. the weekly XML file. What I had concluded was various offsets and ST/DST schedules for each file, but I was *wrong* with my conclusions. It turns out the real story is much more complex. ! Kent
Hi Kent,

I confirm fully your observations. I downloaded with the last version of the downloader the news archive (BTW now it goes back to 2000, I remember some time ago that it stopped at 2006/2007, what happened?) and what I see if I focus on the US weekly unemployment claims release (Thursday, 8:30 local time NY) is that it fluctuates between two values at the NY daylight saving switch dates (2nd March Sunday/1st November Sunday):

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot2.png Size: 169 KB


So as you mentioned one could make the hypothesis that the event timestamps are referred to NY "always on" DST, so GMT-4, without any daylight saving adjustment switch.

The issue with this hypothesis is that, having now data before 2007 where the DS switch was different (From 1987 to 2006, DST began on the first Sunday in April and ended on the last Sunday of October.) I would expect the date before 2007 to switch between 8:30 and 9:30 in April while I do not see a change in pattern in the data, they still switche on 2nd March Sunday/1st November Sunday even in the 2000/2007 range!

So a second hypothesis is that the underlying calendar is the true NY local time to which a systematic correction of an artificial DS switch (with current prescriptions: 2nd March Sunday/1st November Sunday) has been applied even before 2007.

Under this hypothesis, to obtain the correct timestamp, one could first unwind the artificial DS switch on the date and bring them back to local NY time, and then convert them to GMT using the full (period dependent, pre 2007, post 2007, rules). For the first step I tried with the "convert news to GMT" trying to use it in the opposite direction (GMT to local time) but I'm not satisfied with the results, I expected to see al the releases happening at 8:30 while I see two "wrong" values.

Does the previous reasoning makes sense? Is it true that the unemployment claims have been released always at 8:30 local time for 20 years? Is there some other reliable reference point that can be used? Do I miss something?

Thanks for any advice,
Martino

ps: of course many thanks to the Hanover for the work done and for letting us use it!
  • #345
  • Apr 25, 2020 12:55am Apr 25, 2020 12:55am
Quoting tisgno
{quote} Hi Kent, I confirm fully your observations. I downloaded with the last version of the downloader the news archive (BTW now it goes back to 2000, I remember some time ago that it stopped at 2006/2007, what happened?) and what I see if I focus on the US weekly unemployment claims release (Thursday, 8:30 local time NY) is that it fluctuates between two values at the NY daylight saving switch dates (2nd March Sunday/1st November Sunday): {image} So as you mentioned one could make the hypothesis that the event timestamps are referred to NY "always...

Hi Martino,

Thanks for bringing my awareness to the availability of data back to 2000. Yes, it was limited to 1/1/2007 as recent as April 8th, but now improved, assuming it is good data. It will hardly matter to most unless there is a source of data going back that far. Most MT4/MT5 brokers don't offer many years backward for H1 or M30 data (which would capture most news events on the hour or half-hour).

By the way, most brokers have crappy data quality! I notice plenty of bogus spikes & gaps, and the farther back you go, some have daily charts that might be 5 days/week for several years, but 6 days/week for other years! Totally bogus for channels, trendlines, pitchforks! I really would love a consistent MT4 database without all the crappy spikes and massive chunks of missing data (or even extra data, like Saturdays).

If you've built MT4 history files going back that file, it would be nice to have a file I could import, esp. back to 2000. I know there are data sources; it's just not yet my priority to work on the problem.

IMO it would be a good guess to assume that the weekly USD Unemployment Claims always appear at the same time, 8:30AM NY local, and every week, so it seems good for a calibration. HOWEVER, I just noticed that more recent years/dates are on Thursdays. A lot more older ones were on Tuesdays! I assume that the time was still 8:30AM NY local, but you're welcome to check. (The data appears consistent with "8:30AM", regardless of Tue. or Thur.)

My observations are these:
* The underlying data in FF is stored as year-round as some UTC-based time (i.e. the data doesn't observe ST/DST).
* If downloaded during winter, the entire news.csv follows the *current* NY ST schedule. (Meaning the entire file is UTC-5)
* If downloaded during USA summer, the entire news.csv follows the *current* NY DST schedule. (The entire file is UTC-4)
* The current ST/DST schedule follows the formula: 2nd March Sunday/1st November Sunday (Supposedly "2:00AM", but who knows how that was interpreted by FF).
* I'll take your word for it that ST/DST for 1987-2006 was: 1st Sunday in April / Last Sunday October.

For the dates from 2000-2006 (when the older ST/DST was in place 1987-2006), the data is incorrect. It is not truly UTC-based. There would be errors between the dates of the two different ST/DST schedules. For <=2006, the ST/DST schedule should follow, as you said, the first Sunday in April and ending on the last Sunday of October. Therefore, for true consistent "UTC" storage of the entire file, one would need to adjust with +1 hour between the different schedules for spring, and +1 hour between the two dates for the fall. Please double-check my reasoning, but here's an example:

All entire *weeks* with the incorrect Tuesday (or Thursday) entry of what I (attempted to) mark as bold need a correction. (See the b or /b in square brackets, since bold doesn't show in code blocks). Do +1hr between spring change dates, +1hr between fall change dates:
Inserted Code
"2000/01/01", ...    [b]Every year up to 2006 needs through a correction similar to that described below for 2006.[/b]
...
 
 "2006/03/07","09:30","USD","H","Unemployment Claims","340K","354K","347K","344K","45489"
 
[b]--- 2006/03/12 --- Second Sunday in March, "2:00AM NY".  All entries after this, (until DST actually begins), need +1:00 correction[/b]
"2006/03/14","[b]08:30[/b]","USD","H","Unemployment Claims","332K","348K","342K","340K","45491"
"2006/03/21","[b]08:30[/b]","USD","H","Unemployment Claims","336K","343K","334K","332K","45490"
"2006/03/28","[b]08:30[/b]","USD","H","Unemployment Claims","357K","340K","341K","336K","45492"
--- 2006/04/02 "2:00AM NY" DST truly begins First Sunday in April ---
 
"2006/04/04","08:30","USD","H","Unemployment Claims","385K","352K","357K","","45493"
"2006/04/11","08:30","USD","H","Unemployment Claims","346K","362K","388K","385K","45494"
"2006/04/18","08:30","USD","H","Unemployment Claims","352K","349K","348K","346K","45495"
"2006/04/25","08:30","USD","H","Unemployment Claims","339K","352K","355K","352K","45496"
"2006/10/03","08:30","USD","H","Unemployment Claims","308K","315K","307K","305K","45519"
"2006/10/10","08:30","USD","H","Unemployment Claims","374K","307K","308K","","45520"
"2006/10/17","08:30","USD","H","Unemployment Claims","358K","357K","373K","374K","45521"
"2006/10/24","08:30","USD","H","Unemployment Claims","350K","343K","362K","358K","45522"
 
--- 2006/10/29 "2:00AM" DST truly ends Last Sunday in October ---
"2006/10/31","[b]08:30[/b]","USD","H","Unemployment Claims","340K","341K","350K","","45523"
[b]--- 2006/11/05 "2:00AM" --- First Sunday in November.  All entries before this, back to DST ends, need +1:00 correction[/b]
 
"2006/11/07","09:30","USD","H","Unemployment Claims","336K","336K","345K","340K","45524"
"2006/11/14","09:30","USD","H","Unemployment Claims","339K","331K","341K","336K","45525"
"2006/11/21","09:30","USD","H","Unemployment Claims","323K","333K","344K","339K","45526"
"2006/11/27","09:30","USD","H","Unemployment Claims","316K","331K","326K","323K","45527"
"2007/03/06","09:30","USD","H","Unemployment Claims","323K","336K","349K","348K","49485"
--- 2007/03/11 DST begins 2nd Sunday in March ---
"2007/03/13","08:30","USD","H","Unemployment Claims","315K","334K","324K","323K","49486"
"2007/03/20","08:30","USD","H","Unemployment Claims","320K","327K","315K","","49487"
"2007/03/27","08:30","USD","H","Unemployment Claims","311K","326K","321K","320K","49488"
"2007/04/03","08:30","USD","H","Unemployment Claims","326K","319K","310K","311K","49489"
"2007/04/10","08:30","USD","H","Unemployment Claims","300K","314K","332K","326K","49490"
"2007/04/17","08:30","USD","H","Unemployment Claims","304K","316K","302K","300K","49491"
"2007/04/24","08:30","USD","H","Unemployment Claims","329K","309K","305K","304K","49492"
"2007/10/02","08:30","USD","H","Unemployment Claims","287K","299K","295K","293K","49515"
"2007/10/09","08:30","USD","H","Unemployment Claims","287K","291K","288K","287K","49516"
"2007/10/16","08:30","USD","H","Unemployment Claims","264K","286K","287K","","49517"
"2007/10/23","08:30","USD","H","Unemployment Claims","283K","269K","266K","264K","49518"
"2007/10/30","08:30","USD","H","Unemployment Claims","287K","284K","284K","283K","49519"
--- 2007/11/04 DST ends First Sunday in November ---
"2007/11/06","09:30","USD","H","Unemployment Claims","278K","285K","288K","287K","49520"
"2007/11/13","09:30","USD","H","Unemployment Claims","290K","282K","278K","","49521"
"2007/11/20","09:30","USD","H","Unemployment Claims","291K","286K","293K","290K","49522"
"2007/11/26","09:30","USD","H","Unemployment Claims","313K","287K","292K","291K","49523"
... (to present, no known problems).

The entire data file as above, AFTER making the described adjustments to specific weeks, will be a UTC-4 database. Therefore, to get true UTC, you add +4 hours to everything. The true UTC times for Unemployment Claims should be 13:30 (winter) or 12:30 (summer).

If your source file is entirely downloaded during the winter, the file will be a UTC-5 database (after the above weekly corrections). The UTC-5 times will be 8:30/7:30, so when you add +5 hours, you'll get the same true UTC times of 13:30 (winter), 12:30 (summer).


Correcting old news databases is a very low priority for me, but hopefully I'll eventually get to it. You're welcome to tackle the project and share the DB.

Kent
  • #346
  • Apr 25, 2020 1:06pm Apr 25, 2020 1:06pm
Hi Kent,

thanks for the answer and comments, very useful for me!

I started realigning the data more or less the way you suggested, so taking into account the 3 or 2 additional weeks in March and the one in October for the 2000-2006 period, but then I realized some strange fact that stopped me.

As you mention the unemployment claims release is not always on Thursday in the older data. I could not find a way to check the "true historical date", but the pattern is really strange, here is a plot of the weekday of the event. The spikes could potentially mean that there is an holiday on the correct date and the release is anticipated by one day (didnt check yet), but the systematic way it changes from Tuesday to Wednesday is really unnatural and should be an artifact...

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot1.png Size: 111 KB


Moreover, if I change event and focus on the "Crude Oil Inventories" (still weeky), I see a similar pattern for its weekday that makes me think that there is some kind of (full-day/s) misalignment:

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot2.png Size: 198 KB


Other strange things:

 

  1. If I check 2000 April 2nd, that was a Sunday, I find in the archive file several PMI releases in that day that do not happen on Sunday (usually Sunday is rather empty)
  2. I find the first "daylight shift event" for USD in 2000 on a Saturday instead of a Sunday
  3. I find a Bernanke speech in 2000 as FED chairman, but he stepped in in 2006, same for Gaithner

So there is something wrong with the data I'm using... and it's even stranger that the times of the events still look rather "reasonable" (the errors are "full quantic" days).

Maybe a reason for this is that when I downloaded the archive file I used the download.exe by setting a number of months equal to -240 and I got some issues. I remember the download was very fast but than it started working on the files for a long time (10 hours?) and at a certain point it stopped. I launched it again with a number of moths equal to -120 or less (if I remember correct) and then put the files together manually. Maybe this 2-step procedure introduced some issues in the older half of data (I do not know how it works internally). A confirmation for this comes from analyzing the file that Hanover put in its first post. It is different and the weekday for the "unemployment claims" is (except for the spikes that could be reasonable due to holidays) always on Thursday, even between 2007 and 2012 where my files shift to Tuesday and Wednesday. Here is the unemployment claim release weekday of Hanover's original archive file.

Kent, were you able to download the archive since 2000 in a single step?

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot3.png Size: 154 KB


I'll try to download the archive again since 2000, hoping it will not stop and looking at the results. If there is any suggestion on how to do it properly (or better) it is more than welcome!

Thank you
Martino
  • #347
  • Apr 26, 2020 2:08am Apr 26, 2020 2:08am
Hi Martino,

FULL STOP. The data on FF does not go back farther than 2007. When you use the News Downloader to ask for data from year 2000-2006, (or any time before Jan 2007) you get garbage.

It is clearly wrong when it has "Fed Chairman Bernanke Testifies" in Jan 2000. He was not Chair until 2006, as you already noted.

In addition, an attempt to load from Jan/2000 - Apr/2020 (244 months) will load only 160 months, starting from so-called 2000, stopping in 2013, but in reality, the events are off by 7 years, give-or-take a day or two probably due to differences in leap years. (The reason for previous variations of "Unemployment Claims" on Tue, Thur, Wed, depending on the year. There might be variation, but my guess is all are on Thursdays apart from, say, July 4th, or Thanksgiving, or... who knows.

e.g.
A badly downloaded database with "2000-2013" events has:
"2013/04/21","01:00","AUD","H","RBA Gov Lowe Speaks","","","","","16139"

This "RBA Gov Lowe Speaks" was actually on 2020/04/20.

All my previous comments about adjusting a few weeks in spring/winter during 2000-2006 are garbage-out, because the data was garbage-in.

Note: This means there are bugs with News Downloader that it even allows or attempts a request prior to 2007. I can't say whether FF served it bad data, or (more likely) it was just misinterpreted only by the Downloader, but either way, it can't retrieve data before Jan 2007.

Unfortunately, this also means that if you try using "-244 mo" (or any negative number of months from the current month), you'll have to manually calculate the try maximum yourself (because the program will run but not tell you data is wrong!).

Given that Jan 2007 is the earliest, and that it's now, for example, April, you can only ask for 13 years + 4 months = 12*13 + 4 = 160
Therefore, your entry would be "-160 mo" MAX!! If you ask for > 160, your file will be garbage and no warning!

(By the way, I just tried loading -162 mo (11/2006) vs. today, and it was wildly wrong, with no error message by the download).

My recommendation when using the News Downloader (current and presumably final version) is if you're getting close to the 1/1/2007 edge, to always use positive #months and calculate forward from 1/1/2007 or later. That way if you over-estimate the data will always be valid.

It would have been nice to have news for previous years, but as I previously stated, it's not useful for analysis without MT4 data that goes back just as far (which I would love; just haven't worked on it).

Kent
  • #348
  • Apr 29, 2020 12:07pm Apr 29, 2020 12:07pm
Hi Kent,

you are right, the 2000-2007 data are a shifted copy of the 2007-2020 ones. This is the plot of the unemployment claims curve, for instance (keeping the last month out, for scaling reasons):

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot1.png Size: 129 KB


This is bad, because for some data on the website interface you can see the historical depth is larger. For unemployment claims for example it goes back smoothly to 2000:

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot2.png Size: 102 KB


This is another example (monthly), the unemployment rate, it also goes back to 2000 on the website

Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot3.png Size: 93 KB


So its sad not to able to download these data...

For the asset data: I understand the more recent data is the most interesting... but a larger dataset is better than a small one. In relation with the issue of asset data and go back more than 10y, well it depends on the kind of date you are interested in and the frequency. I'm more interested in futures, and I can get daily candles that go back to 2000. For some kind of analysys daily data are ok. 1m or 10m data are more difficult, and if you get it its not easy usually to understand the roll rules used by the data. I used backtestmarket.com for some intraday data. More than before 2000 is less interesting because of the euro-introduction, at least for me. Intraday data in my current tests are used more for checking the news impact (with enough syncronization, not easy as seen) than other aspects (entry/exit points, technical indicators). I've also seen this website: histdata.com that reports free 10m candles for fx and other stuff, I checked eurusd and, apart from 2 outliers, does not seem bad, must check better anyway.
  • #349
  • May 6, 2020 1:29pm May 6, 2020 1:29pm
Hanover, let me start by thanking you for sharing your work with us. I just discovered this tool which sounds very promising indeed although I seem to be unable to make it work. I've copied all the files to my MQL folder but when I attach the EA to my chart I get this:
Attached Image (click to enlarge)
Click to Enlarge Name: Image1.png Size: 48 KB

Even if I add the addresses it requires to the list of allowed URLs, I still don't get anything plotted on my chart. Am I doing something wrong?
  • #350
  • Edited 4:00am May 14, 2020 1:26am | Edited 4:00am
Quoting pips4life
This means there are bugs with News Downloader that it even allows or attempts a request prior to 2007.
Sorry about that, but you get what you pay for.

I merely share some of the software that I've written for my own use. And I know that FF's news calendar only goes back as far as 2007, hence I don't need a date check.

Anybody who doubts this, go to Forex Factory's calendar and try to load any week or month prior to Jan 2007.
Try accessing the URL https://www.forexfactory.com/calendar?month=dec.2006 and see what happens (if you dare!!)
These webpages don't exist, and the EA can't possibly download non-existent data. It apparently accesses the data for the histograms from a different source.
_____________________________

Quoting mix1911
I just discovered this tool which sounds very promising indeed although I seem to be unable to make it work. I've copied all the files to my MQL folder but when I attach the EA to my chart I get this: {image} Even if I add the addresses it requires to the list of allowed URLs, I still don't get anything plotted on my chart. Am I doing something wrong?
It looks like Forex Factory has changed their calendar's URL. I have updated the files in the GitHub repository (accessible via the link in post #1).
  • #351
  • May 14, 2020 3:55am May 14, 2020 3:55am
It looks like Forex Factory has changed their calendar's URL.

Hence I have updated the PlotNews v4-16.zip file in the GitHub repository (accessible from this link) with the following changes:

 The NewsDownloader.exe app now attempts to read the calendar from the URL https://www.forexfactory.com/calendar?week=mmmdd.yyyy instead of https://www.forexfactory.com/calendar.php?week=mmmdd.yyyy

 The Plot News v4-00 b600.ex4 EA now has an additional parameter FFcalendarURL, which defaults to www.forexfactory.com/calendar. There's no need to prefix it with https:// or suffix it with ?week=mmmdd.yyyy as the EA does this automatically for you. There's also a check in the EA now that generates a popup error message if you enter a date earlier than Jan 7, 2007 (the first Sunday in the FF calendar that starts a complete week of news).

 I have also fixed another couple of minor bugs.

Hopefully everything works ok again now.
  • #352
  • Edited 7:32am Oct 26, 2020 4:29am | Edited 7:32am
Anyone else having problems with the main News Downloader.exe? Since the 21st, it is not working.


-----

Also, just friendly reminder that the end of the year is coming up soon. Will you extend the usage date another [few] years? I think you made the script good until Dec 31st, 2020. Perhaps next update can be until 2030, lol.
Edit: I misinterpreted something done to the mqh file months ago as limiting the exe file also. sorry. The main bug with exe is still active though.
  • #353
  • Oct 26, 2020 5:20am Oct 26, 2020 5:20am
Yeah, the same prob whenever FF does some change. FF had some problems and the website was down for a long time and when they came back up the downloader stopped working.

I was quietly hoping to see activity here coz I know David will check it out when he finds time.
  • #354
  • Oct 26, 2020 2:12pm Oct 26, 2020 2:12pm
Quoting fxworldwide
Yeah, the same prob whenever FF does some change. FF had some problems and the website was down for a long time and when they came back up the downloader stopped working.
Quoting forevermaat
Anyone else having problems with the main News Downloader.exe? Since the 21st, it is not working.
Also, just friendly reminder that the end of the year is coming up soon. Will you extend the usage date another [few] years? Edit: I misinterpreted something done to the mqh file months ago as limiting the exe file also.
The main bug with exe is still active
I haven't been able to reproduce the error. Downloader appears to be working fine on my PC; when 'Weeks/months to download' was set to -2, it generated the attached 'news.csv' file.

I've recompiled the app and uploaded a new zip file to github. Also took the opportunity to extend the usage date by another 12 months (btw, this has nothing to do with any mqh file, the app is written using autohotkey, not MQL).

By "not working", do you mean that the news.csv file is empty? The problem is hard to diagnose from the downloader's progress bar as shown in your video.

Anyway, please feel welcome to download the recompiled version and try again. If there are still problems, please let me know and I'll attempt to troubleshoot further. The more info I have to work with, the better.
Attached File(s)
File Type: csv news.csv   22 KB | 582 downloads
  • #355
  • Oct 26, 2020 2:48pm Oct 26, 2020 2:48pm
Ok thks David. Will redownload and test again, with more details.

Thks again.
  • #356
  • Oct 26, 2020 4:34pm Oct 26, 2020 4:34pm
Hi David,

Retested with no resolution. In summary ......when I run (double-click) News Downloader.exe it deletes the news.csv file if it's present. ( I previously inserted the one u provided today.).

The CSV file displays the info correctly in MT4, when I manually copy the one u provided. I captured the screen for each step below. U see the news.csv file I copied in manually before I run News Downloader.exe. After I press OK and it runs, the file is deleted. What u think it could be, FF format?
Attached Image (click to enlarge)
Click to Enlarge Name: Plot_News_Step03_Screenshot 2020-10-26 161847.png Size: 146 KB
  • #357
  • Oct 26, 2020 9:02pm Oct 26, 2020 9:02pm
Quoting fxworldwide
Retested with no resolution. In summary ......
Hello sir,

It won't be the FF format. But I don't know why the news.csv file would be getting deleted.

Here is what happens on my computer (don't tell Microsoft, but this computer is running Windows 7).

This screenshot was taken just before I set the downloader in motion by clicking OK. I have attempted to use the same parameters as yours:
Attached Image (click to enlarge)
Click to Enlarge Name: PPI 1383.png Size: 203 KB


Below is how the folder looks immediately after the downloader has finished running. The news.csv file is still there, at the top of the list:
Attached Image (click to enlarge)
Click to Enlarge Name: PPI 1384.png Size: 160 KB


So it appears to be running ok. It looks like we are running the same version of the downloader (mine is 422k in size, the same as yours).

I will try running it on my wife's laptop (which is running Windows 10) when she returns home, and post the result.
  • #358
  • Oct 27, 2020 12:33am Oct 27, 2020 12:33am
Quoting hanover
{quote} {quote}... The problem is hard to diagnose from the downloader's progress bar as shown in your video. Anyway, please feel welcome to download the recompiled version and try again. If there are still problems, please let me know and I'll attempt to troubleshoot further. The more info I have to work with, the better. {file}
- That's my point. The video shows that it attempts to download, then immediately skips to housekeeping just a few seconds later, then quits. Looking in the output folder, the weekly csv is missing. But the archive.csv is still there. And the csv.sav (created when the news downloader.exe is working...I'm guessing to temporarily save downloaded info) is there, with a last modified date of 2020.10.21.

So in other words, no tempnews.csv.sav file, no news.csv file, and archive.csv file is not updated. downloading seems to be completely skipped just seconds after launching.
NewsDownloader.ini settings:

Inserted Code
[Settings]
Weeks_or_months=3wA'
News_filename=C:\!newstemp\ForexFactory\forexfactory_news.csv
Archive_filename=C:\!newstemp\ForexFactory\forexfactory_news_archive.csv
Hours_shift=0
Macro_keys=
Untimed_events_time=00:00
Currencies=USD,CAD,EUR,GBP,CHF,JPY,AUD,NZD,CNY,ALL
Weekdays=Sun,Mon,Tue,Wed,Thu,Fri,Sat
Description=
Sort_output_file=1
Last_format_used="d[Y/M/D]","t[H:I]","c[T3]","i[T1]","n[T50]","a[T10]","f[T10]","p[T10]","r[T10]","#[Z5]"
Local_folder=C:\!newstemp\ForexFactory\MQL4\Files
Starting_week_month=20201027095545
Include_high_impact=1
Include_medium_impact=1
Include_low_impact=1
Include_non_economic=1
Include_unvalued=1
Include_untimed=1


This is happening on two separate computers. Win10 64 bit (thailand, tested 2 different ISPs), and Win2012 x64 bit (New York City VPS). So the bug is that the downloader.exe is not completing the download. News Downloader.exe stopped downloading almost immediately. You can even see in the 2nd video it appears to start. Then stops (first attempt).

Attached File(s)
File Type: mp4 ff_newsdownloader_hanover_download bug hidden GUI_bandicam 2020-10-27 12-02-55-378.mp4   3.4 MB | 275 downloads


Second video showing the GUI is too large:
https://www.dropbox.com/s/orl4eoih0m...8-801.mp4?dl=0

If you have a way to generate a debug log (verbose log), I could provide that for you. That would provide an exact point of failure. I'm suspecting I could be getting a 403 forbidden error. This is based on error from a different scraper that downloads the weekly csv/json/xml file. Both your scraper and this one stopped downloading around the same time, which is around when FF had DDoS attack.

---

FYI: I can visit both the weekly csv/json/xml file fine with browser and visit the calendar page also in full browser (both computers). So only the downloader cannot run properly

Please advise.
  • #359
  • Oct 27, 2020 4:03am Oct 27, 2020 4:03am
The ff xml/json file downloader was fixed. This image might help you hanover:

Attached Image (click to enlarge)
Click to Enlarge Name: ff downloader what was done to fix it.png Size: 38 KB
  • #360
  • Oct 27, 2020 1:58pm Oct 27, 2020 1:58pm
Quoting forevermaat
The ff xml/json file downloader was fixed. This image might help you
Thank you sir for the info.

The downloader uses autohotkey's URLDownloadToFile command to download the FFcalendar as a HTML file, then it parses the HTML using a series of regular expressions to extract the relevant fields (date, time, currency, event description, impact, etc):
Attached Image (click to enlarge)
Click to Enlarge Name: PPI 1385.png Size: 15 KB

The problem appears to be that the news.csv file is not being created. This is most likely because the URLDownloadToFile statement is failing, but I don't know the reason. I'll see what's involved in getting the app to trap the error and return an error code.

The process seems to work fine on my old Win7 PC. I will test it on my Win10 laptop as soon as I get a chance, and report back here.
0 traders viewing now