Attachments: Plot News / Downloader version 4.00

Plot News / Downloader version 4.00

#421
Quote
Edited 3:21pm Mar 26, 2024 2:58pm | Edited 3:21pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Quoting lmshow

Disliked

Can we get the updated news history up to end of February this year 2024? I used the plot news but it keeps downloading only for up to 2011.

Ignored

I don’t want to tie myself down to a job of providing a news file for each new month, for ever and ever. So what I did (in post 407) was provide the tools to allow anybody to download, in CSV and/or XML format, any month’s news, using a XLSX spreadsheet (and also a word doc of step-by-step instructions explaining how to go about it**).

In a nutshell, it involves using your web browser (Edge, Firefox, Chrome, etc) to obtain the HTML code of the relevant FF calendar month, and copy one particular line of this to a certain web page that converts the HTML to a column format. Then paste this column-based output into the XLSX spreadsheet (in the zip attached to post 407), which translates it into both CSV and XML formats for you to copy wherever you need it. That allows you to obtain news for any month, past or present.

For example, you could then use a text editor like Notepad to paste (append) this news onto the end a CSV file of existing news events up to but not including the given month. And then repeat this process once every month.

[** I was intending to make an instructional video but I can’t get Camtasia (which I purchased a number of years ago) to work. I’ll try downloading a different product when I get some time -- if anybody knows some good free software for recording, annotating and editing animated screen images with a voice-over from a microphone, and also has an upload to YouTube facility, please let me know.]
_____________________________________________________________

[EDIT]
I'm currently working on an app that will join any number of CSV files that you specify (these can be created individually using the process I described earlier in this post, using the XLSX spreadsheet in post 407), then filter only the news events that you request, sort them into any requested order, and output the result in either CSV or XML format. Of course you can apply any time time zone shifts to individual (or multiple) news events that you wish, in the XLSX. I've finished the app, but still writing the instructions. Not sure when I'll complete these, but when I do, I'll upload everything into my Google drive link (the same link that I've been using). That should put the finishing touch to the entire process.

Note: I've set the access to this folder on the Google drive to "anyone on the internet with the link can view". That's the most open-ended option that I could find. I hope that this allows everybody to use the link to download the files. Please let me know if you have any problems.

#422
Quote
Edited 6:20pm Mar 26, 2024 5:56pm | Edited 6:20pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've enhanced the FF calendar news XML and CSV.xlsx spreadsheet (previous version was available from the zip file in post 407) and uploaded a new version into my Google drive folder. The new version has an additional option (on the 'xml' tab) allowing you to choose between having each CDATA value output on a the same line (the way it used to work -- if you enter a 0 in the cell A20), or on a separate new line below (if you enter 1 in the cell A20). See below.

If you right click on the file in the Google drive link, hopefully there will be a download option that allows you to download the spreadsheet.

Attached Image (click to enlarge)

Click to Enlarge

Name: Image 348.png
Size: 44 KB

If you enter a 0 into cell A20 on the 'xml' tab, the output should look like this:

Inserted Code

 <event>
  <title>Final Manufacturing PMI</title>
  <country>GBP</country>
  <date><![CDATA[2024-01-02]]></date>
  <time><![CDATA[09:30am]]></time>
  <impact><![CDATA[Low]]></impact>
  <actual><![CDATA[46.2]]></actual>
  <forecast><![CDATA[46.4]]></forecast>
  <previous><![CDATA[46.4]]></previous>
  <revised_from />
  <FFevent_ID><![CDATA[134631]]></FFevent_ID>
  <group_ID><![CDATA[249]]></group_ID>
  <actual_BorW><![CDATA[0]]></actual_BorW>
  <revised_BorW><![CDATA[0]]></revised_BorW>
 </event>

If you enter a 1 into cell A20, the output should look like this:

Inserted Code

 <event>
  <title>Final Manufacturing PMI</title>
  <country>GBP</country>
  <date>
   <![CDATA[2024-01-02]]>
  </date>
  <time>
   <![CDATA[09:30am]]>
  </time>
  <impact>
   <![CDATA[Low]]>
  </impact>
  <actual>
   <![CDATA[46.2]]>
  </actual>
  <forecast>
   <![CDATA[46.4]]>
  </forecast>
  <previous>
   <![CDATA[46.4]]>
  </previous>
  <revised_from />
  <FFevent_ID>
   <![CDATA[134631]]>
  </FFevent_ID>
  <group_ID>
   <![CDATA[249]]>
  </group_ID>
  <actual_BorW>
   <![CDATA[0]]>
  </actual_BorW>
  <revised_BorW>
   <![CDATA[0]]>
  </revised_BorW>
 </event>

#423
Quote
Edited 9:45pm Apr 9, 2024 5:19pm | Edited 9:45pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Here is a zip file containing each one of the 207 months' news events between Jan 2007 (when the FF calendar first began) and (to the end of) March 2024:

Attached File(s)

news_output_ 2007-01-01 thru 2024-03-31.zip 1.9 MB | 216 downloads

The column (field) headings from left to right are:
"Date", "Time", "Currency", "Impact", "Description", "Actual", "Forecast", "Previous", "Revised from", "FF event ID", "Group ID", "Actual better/worse", "Revised from better/worse"

Here are zip files of various CSV files containing all of these 207 files joined into one file, and then sorted as described in the file name:

Attached File(s)

news_output_ 2007-01 to 2024-03_DT (CSV; sorted by date, time; timestamped UTC time).zip 1.7 MB | 202 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_CEDT (CSV; sorted by ccy, descr; timestamped UTC time).zip 1.0 MB | 179 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_DT (CSV; sorted by date, time; timestamped New York time).zip 1.7 MB | 183 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_CEDT (CSV; sorted by ccy, descr; timestamped New York time).zip 1.0 MB | 179 downloads

I've had to attach their XML counterparts to the next post, due to the forum's restriction on the number of attachments per post.

#424
Quote
Edited 9:40pm Apr 9, 2024 5:24pm | Edited 9:40pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Here are zip files of various XML files containing all of these 207 files joined into one file, and then sorted as described in the file name:

Attached File(s)

news_output_ 2007-01 to 2024-03_DT (XML; sorted by date, time; timestamped UTC time).zip 2.7 MB | 189 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_CEDT (XML; sorted by ccy, descr; timestamped UTC time).zip 1.4 MB | 168 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_DT (XML; sorted by date, time; timestamped New York time).zip 2.7 MB | 169 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_CEDT (XML; sorted by ccy, descr; timestamped New York time).zip 1.4 MB | 173 downloads

#425
Quote
Edited 9:41pm Apr 9, 2024 7:56pm | Edited 9:41pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

And just in case anybody prefers this sorted by Group ID# instead of currency/description, here are the 4 relevant files for the 207 months' consolidated values:

Attached File(s)

news_output_ 2007-01 to 2024-03_GDT (CSV; sorted by group ID#; timestamped UTC time).zip 1.0 MB | 229 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_GDT (CSV; sorted by group ID#; timestamped New York time).zip 1.0 MB | 198 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_GDT (XML; sorted by group ID#; timestamped UTC time).zip 1.4 MB | 168 downloads

Attached File(s)

news_output_ 2007-01 to 2024-03_GDT (XML; sorted by group ID#; timestamped New York time).zip 1.4 MB | 177 downloads

Group ID# is not shown on the calendar, but it a value that uniquely identifies each event type, e.g. CPI m/m, Unemployment Claims. If the description gets changed at any point, e.g. "Cash Rate" becomes "Federal Funds Rate", then if their Group ID#s remain the same, they will still remain grouped together in the file.

#426
Quote
Apr 19, 2024 4:08am Apr 19, 2024 4:08am

sabird
| Joined Oct 2017 | Status: Trader | 7 Posts

Quoting hanover

Disliked

{quote} I have made the change (assuming that I understand you correctly), and uploaded the new files CSV_to_XML_settings.json FFcalendar_CSVtoXML.exe FFcalendar_CSVtoXML.py to this link, overwriting the previous versions. There is a new setting at the end of the settings file: flag_cdata_next_line if set to 0, the <![CDATA[....]]> code block is printed on the same line as the relevant column name (which is what I think you are asking for). If set to 1, the code...

Ignored

Hi Hanover! Thanks, yes it worked, now it is printed on the same line! It was exactly what I meant. You are amazing. Thank you.

As I could test the next step only after this request, I have another question, I hope you don't mind.

<date><![CDATA[ 2024-01-03 ]]></date>
<time><![CDATA[ 03:00pm ]]></time>
<impact><![CDATA[ High ]]></impact>

1. Is it possible to remove the empty spaces before and after the values, I tried editing the .py file, but it didn't work/

2. Also, how to convert csv to xml only first 5 or 7 rows only, not all rows. Is it possible. (date till title only, or date till forecast only)

3. And, how to change the order of MM-DD-YYYY instead of YYYY-MM-DD in the py file? (if not possible its ok)

Thank you!!

#427
Quote
Edited 11:42pm Apr 20, 2024 7:23pm | Edited 11:42pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Quoting sabird

Disliked

1. Is it possible to remove the empty spaces before and after the values, I tried editing the .py file, but it didn't work/ 2. Also, how to convert csv to xml only first 5 or 7 rows only, not all rows. Is it possible. (date till title only, or date till forecast only) 3. And, how to change the order of MM-DD-YYYY instead of YYYY-MM-DD in the py file?

Ignored

Hello sir,

Why editing the .py file doesn't work: You'd need to first download python itself, and also and libraries imported in the .py code, and then either run the .py file by invoking python from the command line, or by compiling the .py to an .exe file (which is what I did) that runs standalone (directly from your desktop without the need for python or any of its libraries or utilities to be present). I only included the .py file just in case anybody was curious to see how I'd written the code, or was interested in the appearance of the python language. I was expecting comments from more experienced python coders telling me that I was a dumbass noob and that there were much easier ways of coding certain functions, LOL.

Anyway, regarding your point #2, there is a file CSV_to_XML_settings.json which you can modify using a text editor like Notepad. Its content should look something like this:

Attached Image (click to enlarge)

Click to Enlarge

Name: Image 367.png
Size: 9 KB

The settings highlighted in green determine whether the designated field (actual value, forecast value, etc) will be output to the XML file or not. If you want it to be output, leave the setting as 1. If you don't want it to be output, change the value from 1 to 0. Don't forget to save the file. The Date, Time, Currency, Impact and Event Description are always set to 1 and currently they can't be changed.

Regarding your points 1 & 3, these were done this way to make the output compatible with FF's XML file. However, I will make changes to the app that allow the user to select the format for the date, time and CDATA. (Currently these can't be changed).

I'm also planning on adding a front screen which will allow all of the settings to be easily changed, so that it will no longer be necessary to manually edit the CSV_to_XML_settings.json file, The entries you make on the screen will update the file automatically, and will therefore be retained by the app ready to pop up as the default value for the next run.

#428
Quote
Edited Apr 22, 2024 3:22am Apr 21, 2024 11:03pm | Edited Apr 22, 2024 3:22am

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Quoting sabird

Disliked

1. Is it possible to remove the empty spaces before and after the values, I tried editing the .py file, but it didn't work/ 2. Also, how to convert csv to xml only first 5 or 7 rows only, not all rows. Is it possible. (date till title only, or date till forecast only) 3. And, how to change the order of MM-DD-YYYY instead of YYYY-MM-DD in the py file?

Ignored

I’ve uploaded a new version of the FFcalendar_CSVtoXML.exe app that addresses the 3 points you made in post 426.

The necessary files can be downloaded from the following link. They can be downloaded anywhere, but the .exe and .json files MUST reside in the SAME folder.
The .csv and .xml files give you some test data to experiment with.

The new app can be run from your desktop, taskbar, or the command line. As promised, it now has a front end screen that pops up (you might have to wait a few seconds), allowing you to more easily modify all of the available parameter settings:

Attached Image (click to enlarge)

Click to Enlarge

Name: screenshot.png
Size: 22 KB

This allows you to modify what gets output to the .xml file. It should hopefully be all fairly self-explanatory except for the special symbols that can be used in the last 3 fields:

Attached Image (click to enlarge)

Click to Enlarge

Name: Image 372.png
Size: 95 KB

(if that's too hard to read, try this)

Any character(s) that don’t match any of the above symbols get output ‘as is’.

Re the xml_cdata_format setting, the token {cdata_str} gets replaced by the data, and all other characters, including spaces and punctuation, are output 'as is'. If you want to start a new line anywhere, use the special combo
\r\n
in the desired position (see the example below).

The values you enter are automatically read in from, and written out to, the CSV_to_XML_settings.json file. Hence they remain ‘sticky’ from one run to the next. You can always download the file again if you want to reset back to the original values. I have set these values to line up with what you requested, rather than achieving compatibility with FF’s own .xml file.

To achieve the latter, overwrite the content of the entire file with the following:

Inserted Code

{
    "entry_settings": {
        "root_name": "FFcalendar_news",
        "input_path": "news_output_ 2023-01-01.csv",
        "output_path": "news_output_ 2023-01-01.xml",
        "flag_actual": 1,
        "flag_forecast": 1,
        "flag_previous": 1,
        "flag_revised": 1,
        "flag_FFevent": 1,
        "flag_group": 1,
        "flag_actual_BorW": 1,
        "flag_revised_BorW": 1,
        "xml_date_format": "%Y-%m-%d",
        "xml_time_format": "%I:%M%p",
        "xml_cdata_format": "\r\n      <![CDATA[ {cdata_str} ]]>\r\n    "
    }
}

#429
Quote
May 2, 2024 12:54am May 2, 2024 12:54am

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I have performed the same operations that generated the outputs in posts 423, 424, 425, except that these now also include news events for the recently completed month of April 2024. Some of these zip files exceed FF's size limit of 5 MB hence I have uploaded them to my Google drive instead (here). This folder has been made open to the public so everybody who clicks on the link should have access.

Just a reminder that the format of the CSV files is:
"Date", "Time", "Currency", "Impact", "Description", "Actual", "Forecast", "Previous", "Revised from", "FF event ID#", "Group ID#", "Actual better/worse", "Revised from better/worse"
In the case of better/worse: 1=better, 2=worse, 0=N/A or negligible change

#430
Quote
May 23, 2024 11:42pm May 23, 2024 11:42pm

tjunjie1983
| Joined Jan 2013 | Status: Trader | 9 Posts

Thanks hanover!Nice sharing there!
I was following your instructions to get the calender into my excel sheet but for the event importance i got all 'i' instead of h for high or L for low.
Can i check with you what went wrong?

Attached Image (click to enlarge)

Click to Enlarge

Name: screenshot.png
Size: 41 KB

#431
Quote
Edited May 27, 2024 3:45pm May 26, 2024 11:48pm | Edited May 27, 2024 3:45pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Quoting tjunjie1983

Disliked

Thanks hanover!Nice sharing there! I was following your instructions to get the calender into my excel sheet but for the event importance i got all 'i' instead of h for high or L for low. Can i check with you what went wrong? {image}

Ignored

Hello sir,
FF have recently added some new columns, changing the the format of their calendar HTML.
I have amended the spreadsheet, to make it compatible.
I can't upload it here because it exceeds FF's attachment size limit of 5MB.
However, you can find it on my Google drive here -- the file is named:
FF calendar news XML and CSV.xlsx
(just right-click and select 'Download')
Good luck!

#432
Quote
May 31, 2024 11:53pm May 31, 2024 11:53pm

tjunjie1983
| Joined Jan 2013 | Status: Trader | 9 Posts

Thanks on that!
Actually i figured it out later on as well. It was the reference that is wrong..haha..
anyway thanks again!

#433
Quote
Jun 9, 2024 6:03pm Jun 9, 2024 6:03pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've updated the historical FF calendar news files to include the recently ended month of May 2024, i.e. news events from 1 Jan 2007 to 31 May 2024.
You can download these files from my Google drive here.
______________________________________

Quoting tjunjie1983

Disliked

Thanks on that! Actually i figured it out later on as well. It was the reference that is wrong..haha.. anyway thanks again!

Ignored

You're welcome sir.

#434
Quote
Jun 15, 2024 5:24pm Jun 15, 2024 5:24pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've uploaded a new version of the spreadsheet FF calendar news XML and CSV.xlsx to my Google drive here.

#435
Quote
Jul 1, 2024 8:54pm Jul 1, 2024 8:54pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've updated the historical FF calendar news files to include the recently ended month of June 2024, i.e. news events from Jan 01, 2007 (which is when the FF calendar began) to June 30, 2024.

You should be able to download all of the files listed below from my Google drive here. (I can't upload these to FF because they exceed the size limit of 5 MB).
Please post if you encounter problems.

news_output_2007-01-01 thru 2024-06-30.zip
news_output_ 2007-01-01 to 2024-06-30_DT (XML; sorted by date, time; timestamped New York time).xml
news_output_ 2007-01-01 to 2024-06-30_DT (CSV; sorted by date, time; timestamped New York time).csv
news_output_ 2007-01-01 to 2024-06-30_CEDT (XML; sorted by ccy, descr; timestamped New York time).xml
news_output_ 2007-01-01 to 2024-06-30_CEDT (CSV; sorted by ccy, descr; timestamped New York time).csv
news_output_ 2007-01-01 to 2024-06-30_DT (XML; sorted by date, time; timestamped UTC).xml
news_output_ 2007-01-01 to 2024-06-30_DT (CSV; sorted by date, time; timestamped UTC).csv
news_output_ 2007-01-01 to 2024-06-30_CEDT (XML; sorted by ccy, descr; timestamped UTC).xml
news_output_ 2007-01-01 to 2024-06-30_CEDT (CSV; sorted by ccy, descr; timestamped UTC).csv
FF calendar news XML and CSV.xlsx

#436
Quote
Jul 31, 2024 6:52pm Jul 31, 2024 6:52pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've updated the historical FF calendar news files to include the recently ended month of July 2024, i.e. news events from Jan 01, 2007 (which is when the FF calendar began) to July 31, 2024. For the latest details and download link, please see post #1.

#437
Quote
Sep 3, 2024 12:02am Sep 3, 2024 12:02am

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've updated the historical FF calendar news files to include the recently ended month of August 2024, i.e. news events from Jan 01, 2007 (which is when the FF calendar began) to August 31, 2024. For the latest details and download link, please see post #1.

#438
Quote
Oct 1, 2024 6:09pm Oct 1, 2024 6:09pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

I've updated the historical FF calendar news files to include the recently ended month of Sept 2024, i.e. news events from Jan 01, 2007 (which is when the FF calendar began) to Sept 30, 2024. For the latest details and download link, please see post #1.

#439
Quote
Edited Oct 7, 2024 3:53am Oct 6, 2024 1:37pm | Edited Oct 7, 2024 3:53am

pascal380
| Joined Jun 2024 | Status: Trader | 2 Posts

I have downloaded the folder from google drive, and the file FFcalendar_CSVtoXML.exe doesn't work. It opens as command line and then close quickly, so we don't have time to read what the error message is.

And I don't see the python source code in the google drive as you said a few messages ago.

For the file we download at https://nfs.faireconomy.media/ff_calendar_thisweek.xml in which timezone and DST is it please ?

Thank you

#440
Quote
Oct 11, 2024 5:10pm Oct 11, 2024 5:10pm

hanover
Joined Sep 2006 | Status: ... | 8,088 Posts

Quoting pascal380

Disliked

I have downloaded the folder from google drive, and the file FFcalendar_CSVtoXML.exe doesn't work. It opens as command line and then close quickly, so we don't have time to read what the error message is. And I don't see the python source code in the google drive as you said a few messages ago. For the file we download at https://nfs.faireconomy.media/ff_calendar_thisweek.xml in which timezone and DST is it please ? Thank you

Ignored

Sorry, I don't visit FF often these days, and just saw your post now.

I've re-compiled the source code (same version as supplied here) and the app seems to work fine on my PC. Please see the screenshots below. I have no way of knowing how your PC config differs from mine, hence I don't know what to suggest. For more info, please see post #410.

Just in case there was anything awry with the previous version, I've uploaded the latest versions of all of the files (including the python source code) into my Google drive into the above link. For reasons that I don't properly understand, the newly compiled EXE file is much larger than it was previously.

The data in Forex Factory's XML file is timestamped UTC, as far as I know.

Attached Image(s) (click to enlarge)

Click to Enlarge

Name: screenshot.png
Size: 10 KB

Click to Enlarge

Name: screenshot.png
Size: 18 KB

Commercial Content
/
Plot News / Downloader version 4.00
Reply to Thread
- 1 18 19 20 21 Page 22 23
- 1 21 Page 22 23

0 traders viewing now

Options

Similar Threads

Plot News / Downloader version 4.00