MQL4/5 programmers: This weekly news download code solves limit!

Quoting pips4life

Disliked

As of August 2024, ForexFactory now limits the downloading of *weekly* news file(s) to a maximum of 2 every 5 minutes (**subject to change), regardless of the file type chosen (.xml .json .ics .csv). The affected URLs are:

Ignored

//+----------------------+
//| Run data maintenance |
//+----------------------+
void DataRun(){
   if( _Period > 1 && _Period <= MaxTF ) {
      dataFilename = GetDataFilename();
            
      if(!FileIsExist(dataFilename)) {
         dataDownload(GetUrlData());
      }
      
      CheckReadData();
   }
}
//+----------------------+
//| Check & Read data    |
//+----------------------+
void CheckReadData() {    
   if(dataRead() == false) {
      dataFilename = GetDataFilename();
      dataDownload(GetUrlData());
   }
   return;  
}
//+--------------------+
//| Data File Download  |
//+--------------------+
void dataDownload(string newsUrl) {
   ResetLastError();
   dataFilename = GetDataFilename();
   string FilePath=StringConcatenate(TerminalInfoString(TERMINAL_DATA_PATH),"\\MQL4\\files\\",dataFilename);
   int FileGet=URLDownloadToFileW(NULL,newsUrl,FilePath,0,NULL);
   if(FileGet==0) {
      PrintFormat("Data download: ( %s ) successful!",dataFilename);
      GetCurrentData(dataFilename);
   }
   return;
}
//+--------------------+
//| Read the data file  |
//+--------------------+
bool dataRead() {
   ResetLastError();
   int FileHandle=FileOpen(dataFilename,FILE_READ|FILE_BIN);
   if(FileHandle!=INVALID_HANDLE) {
      //--- receive the file size
      ulong sData_size=FileSize(FileHandle);
      //--- read data from the file
      int end=0;
      while(!FileIsEnding(FileHandle)) {
         sData=FileReadString(FileHandle,(int)sData_size);
         end = StringFind(sData,"</events>", 0);
      }
      FileClose(FileHandle);
      if(end == -1) {
         cal3=1;
         if(pal3!=cal3){
            Print("Invalid data... you may need to delete "+dataFilename+" from \\MQL4\\Files\\");
            pal3=cal3;
            return(false);
         }
      }            
   } else return(false);
   /*else { //--- check for errors
         FileClose(FileHandle);
         PrintFormat("...data read: Opening file failed! ( %s ) file:  Error code = ( %d ) ",dataFilename,GetLastError());
         return(false);
   } */
   return(true);
}
string GetUrlData(){
   string myUrl = "https://nfs.faireconomy.media/ff_calendar_thisweek.xml"; //Scheduled Events              
   return(myUrl);
}

Quoting v2vboni

Disliked

{quote} Last year, I made some modifications to the news reader in my system by incorporating additional code snippets. Now, before downloading the scheduled XML file, the system verifies whether the file has already been downloaded. This week, I'm planning to introduce another feature that checks if the market is open or closed before initiating the download process. In case the market is closed, the download will be skipped, and once the market opens, the system will delete the existing file (if any) and then initiate the download. This enhancement...

Ignored

Quoting pips4life

Disliked

{quote} ...by reducing our collective overloading of the FF news server.

Ignored

//--- to download the xml
#import "urlmon.dll"
int URLDownloadToFileW(int pCaller,string szURL,string szFileName,int dwReserved,int Callback);
#import
 
//------------------------------
//  Download FF Scheduled News |
//------------------------------
//
void dataDownload(string newsURL) {
   int SetGMT = (int)GlobalVariableGet("gvGMT"); //parameter input value : Default is 2 (GMT+2  MT4 server time)
   string nygmt = "06:00";  //GMT+0 = time equivalent is 01:00 NY time
   nygmt = (string)(SetGMT<0 || SetGMT>0 ? ( (string)((6+SetGMT)>24 ? "0"+(string)((6+SetGMT)-24) : (string)(6+SetGMT)) ) : "06" ) + ":00"; // GMT+0
  
   int nygmtHour = getHour(nygmt); //06:00 GMT+0 or 01:00 New York Time
  
   if (nygmtHour > -1){
      nygmtHour = nygmtHour; // output must be 08:00 in GMT +2 (if paramter input value is 2)
      
      if (nygmtHour > 23) nygmtHour = nygmtHour - 24;
      if (nygmtHour < 0)  nygmtHour = nygmtHour + 24;
   }
  
   int mfHour = MathAbs(nygmtHour-1); //05:00 GMT+0  or 00:00 New York Time
    
   datetime serverTime = TimeCurrent(); // Get the current server time
   int dayOfWeek = TimeDayOfWeek(serverTime);  //0 = Sunday;
   MqlDateTime strl;
   TimeToStruct(serverTime,strl); // gets result as "hh:mi"
   int downloadCond = 0; //download condition
   // Check if it's Sunday and download once between 08:00 to 23:00 GMT+2
   if(!FileIsExist(dataFilename)) downloadCond = 3; // news xml file does not exist
   else {
      if ( (strl.day_of_week== 0 && strl.hour >= nygmtHour && strl.hour <= 23)){
          downloadCond = 1;
      }
      // Check if it's Monday to Friday and download once between 07:00 GMT+2 to 19:00 GMT+2
      else if( (strl.day_of_week >= 1 && strl.day_of_week <= 5 && ( strl.hour >= mfHour && strl.hour <= 12)) ){
          downloadCond = 2;
      }
   }
  
   if (downloadCond > 0) {  
      ResetLastError();
      dataFilename = GetDataFilename();
      string FilePath=StringConcatenate(TerminalInfoString(TERMINAL_DATA_PATH),"\\MQL4\\files\\",dataFilename);
      
      // Check if lastDownloaded.txt exists and open it for reading if it does
      int fileHandle = FileOpen("lastDownloaded.txt", FILE_READ|FILE_TXT);
      string lastDownloadedDate;
      
      if(fileHandle > 0){
         // Read the content of the file as a string
         lastDownloadedDate = FileReadString(fileHandle);
         FileClose(fileHandle);
      }
      else {
         // If lastDownloaded.txt doesn't exist, create it and set the initial last downloaded date to a distant past
         fileHandle = FileOpen("lastDownloaded.txt", FILE_WRITE|FILE_TXT);
         if(fileHandle > 0){
         //if (fileHandle != INVALID_HANDLE) {
             FileWriteString(fileHandle, "1970.01.01"); // Set initial last downloaded date to January 1, 1970
             FileClose(fileHandle);
         }
         lastDownloadedDate = "1970.01.01";
      }
      
      // Convert the string to datetime
      datetime lastDownloaded = StrToTime(lastDownloadedDate);
      
      // Check the download condition and update the data file accordingly
      if (downloadCond == 3 || (downloadCond == 1 && lastDownloaded < serverTime - 60 * 60 * 1000) || (downloadCond == 2 && lastDownloaded < serverTime - 12 * 60 * 60 * 1000)) {
         int FileGet = URLDownloadToFileW(NULL, newsURL, FilePath, 0, NULL);
      
         if (FileGet == 0) {
            PrintFormat("Data download: ( %s ) successful!", dataFilename);
            GetCurrentData(dataFilename);
         }
      
         // Update the last downloaded date in lastDownloaded.txt
         fileHandle = FileOpen("lastDownloaded.txt", FILE_WRITE|FILE_TXT);
         if (fileHandle > 0) {
            // Update the last downloaded date
            string newDownloadedDate = TimeToStr(serverTime, TIME_DATE);
            FileWriteString(fileHandle, newDownloadedDate);
            FileClose(fileHandle);
        }
      }
   }
  
   return;
}
 
 
 
//+------------------------------------------------+
//| Subroutine: getting the name of the .xml file  |
//+------------------------------------------------+
//deVries: one file for all charts!
string GetDataFilename() {
   int adjustDays = 0;
   switch(TimeDayOfWeek(TimeLocal())) {
      case 0: adjustDays = 0; break;
      case 1: adjustDays = 1; break;
      case 2: adjustDays = 2; break;
      case 3: adjustDays = 3; break;
      case 4: adjustDays = 4; break;
      case 5: adjustDays = 5; break;
      case 6: adjustDays = 6; break;
   }
  
   string fileName = "";
   string sName = "";
  
   if( workNews == event_scheduled) { sName = "_Scheduled.xml"; calendardate =  TimeLocal() - (adjustDays  * 86400); }
   else calendardate =  TimeLocal();
  
   fileName =  (StringConcatenate(TimeYear(calendardate),"-",
          PadString(DoubleToStr(TimeMonth(calendardate),0),"0",2),"-",
          PadString(DoubleToStr(TimeDay(calendardate),0),"0",2),sName));
   return (fileName); //Always a Sunday date
}
 
//---
int getHour (string timeString) {
   string result[];
   string needle = ":";
   char u_needle = (char)StringGetCharacter(needle,0);
   int k = StringSplit(timeString, u_needle, result);
   if (k > 0) return (char)result[0];
   return -1;
}
 
//+----------------------+
//| Data Run                |
//+----------------------+
void DataRun(){
   dataFilename = GetDataFilename();
   dataDownload(GetUrlData());
   CheckReadData();
   return;
}
 
//+----------------------+
//| Check & Read data |
//+----------------------+
void CheckReadData() {
    if(dataRead() == false) {
        dataFilename = GetDataFilename();
        dataDownload(GetUrlData());
     }
     return;
}
 
 
//+--------------------+
//| Read the data file  |
//+--------------------+
bool dataRead() {
   ResetLastError();
   int FileHandle=FileOpen(dataFilename,FILE_READ|FILE_BIN);
   if(FileHandle>0) {
      //--- receive the file size
      ulong sData_size=FileSize(FileHandle);
      //--- read data from the file
      int end=0;
      while(!FileIsEnding(FileHandle)) {
         sData=FileReadString(FileHandle,(int)sData_size);
         end = StringFind(sData,"</events>", 0);
      }
      FileClose(FileHandle);
      if(end == -1) {
         cal3=1;
         if(pal3!=cal3){
            //very rarely happen
            Comment("Invalid data... you may need to delete "+dataFilename+" from \\MQL4\\Files\\");
            pal3=cal3;
            return(false);
         }
      }            
   } else return(false);
   /*else { //--- check for errors
         FileClose(FileHandle);
         PrintFormat("...data read: Opening file failed! ( %s ) file:  Error code = ( %d ) ",dataFilename,GetLastError());
         return(false);
   } */
   return(true);
}
 
//---
string GetUrlData(){
   string myUrl = "https://nfs.faireconomy.media/ff_calendar_thisweek.xml"; //Scheduled Events              
   return(myUrl);
}
 
//+------------------------------------------------+
//| Subroutine: getting the name of the .xml file  |
//+------------------------------------------------+
//one file for all charts!
string GetDataFilename() {
   int adjustDays = 0;
   switch(TimeDayOfWeek(TimeLocal())) {
      case 0: adjustDays = 0; break;
      case 1: adjustDays = 1; break;
      case 2: adjustDays = 2; break;
      case 3: adjustDays = 3; break;
      case 4: adjustDays = 4; break;
      case 5: adjustDays = 5; break;
      case 6: adjustDays = 6; break;
   }
  
   string fileName = "";
   string sName = "";
  
   if( workNews == event_scheduled) { sName = "_Scheduled.xml"; calendardate =  TimeLocal() - (adjustDays  * 86400); }
   else calendardate =  TimeLocal();
  
   fileName =  (StringConcatenate(TimeYear(calendardate),"-",
          PadString(DoubleToStr(TimeMonth(calendardate),0),"0",2),"-",
          PadString(DoubleToStr(TimeDay(calendardate),0),"0",2),sName));
   return (fileName); //Always a Sunday date
}

Quoting v2vboni

Disliked

{quote} I have implemented an input parameter that accepts the GMT offset value for the MT4 server (the time of your broker's platform server). You can enter this value according to your server's time zone directly from your news reader (with my system, I use another tool to accept this input via a global variable)...

Ignored

   if (downloadCond > 0) {
       ResetLastError();
       dataFilename = GetDataFilename();
  
       string FilePath = StringConcatenate(TerminalInfoString(TERMINAL_DATA_PATH), "\\MQL4\\files\\", dataFilename);
  
       // Check if lastDownloaded.txt exists and open it for reading if it does
       int fileHandle = FileOpen("lastDownloaded.txt", FILE_READ | FILE_TXT);
       datetime lastDownloaded = 0;
       if (fileHandle > 0) {
           // Read the content of the file as a string
           string lastDownloadedDate = FileReadString(fileHandle);
           FileClose(fileHandle);
  
           // Convert the string to datetime
           lastDownloaded = StrToTime(lastDownloadedDate);
       }
       else {
           // If lastDownloaded.txt doesn't exist, create it and set the initial last downloaded date to a distant past
           fileHandle = FileOpen("lastDownloaded.txt", FILE_WRITE | FILE_TXT);
           string initialDownloadedDate = "1970.01.01 00:00:00";
           if (fileHandle > 0) {
               // Create the initial last downloaded date
               initialDownloadedDate = "1970.01.01 00:00:00";
               FileWriteString(fileHandle, initialDownloadedDate);
               FileClose(fileHandle);
           }
           lastDownloaded = StrToTime(initialDownloadedDate);
       }
  
       // Get the current server time
       datetime currentTime = TimeLocal();
  
       // Check the download condition and update the data file accordingly
       if (downloadCond == 3 || (downloadCond == 1 && currentTime - lastDownloaded >= 60 * 60) || (downloadCond == 2 && currentTime - lastDownloaded >= 12 * 60 * 60)) {
           int FileGet = URLDownloadToFileW(NULL, newsURL, FilePath, 0, NULL);
  
           if (FileGet == 0) {
               PrintFormat("Data download: ( %s ) successful!", dataFilename);
               GetCurrentData(dataFilename);
           }
  
           // Update the last downloaded date in lastDownloaded.txt
           fileHandle = FileOpen("lastDownloaded.txt", FILE_WRITE | FILE_TXT);
           if (fileHandle > 0) {
               // Update the last downloaded date
               string newDownloadedDate = TimeToStr(currentTime, TIME_DATE | TIME_MINUTES | TIME_SECONDS);
               Print("newDownloadDate = ",newDownloadedDate);
               FileWriteString(fileHandle, newDownloadedDate);
               FileClose(fileHandle);
           }
       }
      //Print("lastDownload = ",lastDownloaded);
      //Print("currentTime = ",currentTime);
      //Print("downloadCond 1: ",currentTime - lastDownloaded >= 60 * 60);
      //Print("downloadCond 2: ",currentTime - lastDownloaded >= 12 * 60 * 60);
   }
   return;
}

Quoting pips4life

Disliked

...The affected URLs are:
https://nfs.faireconomy.media/ff_calendar_thisweek.xml
https://nfs.faireconomy.media/ff_calendar_thisweek.json
https://nfs.faireconomy.media/ff_calendar_thisweek.ics
https://nfs.faireconomy.media/ff_calendar_thisweek.csv
...

Ignored

Quoting v2vboni

Disliked

{quote} The download feature can be utilized by any news reader available or for any data file that can be downloaded through the MT4 platform's custom tools and indicators. It has been updated to include hours, minutes, and seconds, ensuring it operates as intended. This feature allows users to manage the download conditions for news data based on their desired time interval. It accomplishes this by comparing the current time with the last downloaded time stored in a file. If the user wishes to update the news data every hour (Sunday), every 12...

Ignored

Quoting emmzett

Disliked

{quote} It's worth noting that as of now the request limit implementation by ForexFactory is flawed and erroneous. The request limit is applied on all four urls together which is wrong. The correct implementation would be to implement a separate request limit for each single url. Example: An MQL program requests the news calendar in XML format, and if it ignores the sent cache headers and unconditionally re-requests too often the limit kicks in. So far so good. But if a second separate MQL program requests the calendar in a different format (e.g....

Ignored

Quoting pips4life

Disliked

...There is one hypothetical solution that I *don't* know how to implement, but it would eliminate the need for any of the various News indicators/EA's/programs to be updated by their authors...

Ignored

Quoting emmzett

Disliked

{quote} Technically that's possible. It requires two things: 1) Interceptiong the download: For this the url has to be an input parameter. In many cases this is not given. 2) Replacing the FF server to serve the intercepted HTTP request: that's a proxy server which delivers files cached elsewhere. Not possible for an average user. In summary, if the url is an input and the user runs a proxy server for serving calendar export requests, than nothing needs to be done by the original MQL author. Look at your own code: the url is hard-coded. You see,...

Ignored

Quote

Disliked

The affected URLs are:
https://nfs.faireconomy.media/ff_calendar_thisweek.xml
https://nfs.faireconomy.media/ff_calendar_thisweek.json
https://nfs.faireconomy.media/ff_calendar_thisweek.ics
https://nfs.faireconomy.media/ff_calendar_thisweek.csv

Quoting pips4life

Disliked

...and that proxy server's purpose is only to intercept these 4 URL's...

Ignored

Quoting pips4life

Disliked

...I would caution that this is not proven AFAIK. Maybe someone could comment if this is the situation or not. It might be...

Ignored

Quoting emmzett

Disliked

{quote} That's not possible with standard programming techniques. You cannot intercept those calls with standard APIs, especially if those urls use HTTPS. What you think about is a "man-in-the-middle" attack. You can definitely implement a proof-of-concept for this but there is no way to publish an application to the average FF user which essentially hacks into the user's operating system and installs a "man-in-the-middle" layer to intercept HTTPS. There is absolutely no way to succeed with such an approach, even if it's technically possible. So...

Ignored

2025-Mar-02 v1.01 by pips4life
   Big changes to 'P4L_News.mqh', and this program, so that it definitely tries to update with 1st tick(s) after UTC 06:00 (= 01:00 EST, or 02:00 EDT)
      at the start of a new week, since the FF server updates the news files on Sunday 00:00:00 (midnight) NY.  If it fails, it keeps
      trying every 5 minutes until it succeeds, after which it resumes the normal interval 'TimeBetweenNewsUpdates_HptMM' which can be
      every 8 hrs, 60 minutes, or any interval >= 5 minutes (Short values are not recommended because it will block out other
      uncoordinated news because they have no chance to download the news file(s) within the 2X-per-5minutes limit).
      With these changes, this version is much better for longer update intervals like 8hrs, Daily, even Weekly.
   New var 'TimeBetweenNewsUpdates_HptMM'(8.0) to specify the time interval between news updates.
      (Note: The news files rarely change, and at most are updated by FF once-per-hour; they do NOT contain any
       'Actual' news values, only the event titles, times, 'Forecast' and 'Previous' values, ...)
   The original 'get_FF_news(...)' function changed; it has a new argument!  The old argument definition can be uncommented
      in the 'P4L_News.mqh' file if needed. (The old (overloaded) function will call the new definition, but it's not recommended
      if downloading multiple news types).
   This program demonstrates the downloading of up to all 4 news file types, and never downloads more than 1 type per 5minutes, no matter
      the interval.  Note: Incoming ticks are necessary to trigger the download(s), either when market is open, or starting MT4,
      changing the TF, or do a chart 'Refresh'
   New vars: downloadNews_typeXml(true), downloadNews_typeJson(false), downloadNews_typeIcs(false), downloadNews_typeCsv(false).  
      ONLY ONE format is downloaded each 5 minutes, after which the regular interval between downloads resumes.
      This staggering of downloads gives a chance for other news programs to use the 2nd download-per-5minutes.
      NOTE!  This intentional staggering every 5 minutes, plus only using 1 or the 2X-per-5minutes allowed by this
      program means that if downloading all 4 formats, it will take incoming ticks for 15 minutes before all 4 files
      are updated with new weekly files!  (And longer still, if other uncoordinated news programs use up the 2X limit;
      not to worry, though, because this indicator will keep trying every 5 minutes until the new weekly file(s) have
      been updated at least once for the week, regardless of the update interval).
   As always, this code and the include file were compiled and tested on both MT4 and MT5.

2025-Mar-05 v1.02 by pips4life
   Good news!  ForexFactory responded to a request to now provide independent limits of
      2X-per-5minutes downloads for EACH of the four news file types: .xml  .json  .ics  .csv
      Therefore, I removed the 'lastAttemptAny' variable from the previous version, which
      served the purpose to download only 1 format per 5 minutes (which therefore would take 15+
      minutes to download all 4).  Now the downloads of multiple types are attempted without delay.  
      (If any 1 or more fail, those specific types will re-try every 5 minutes until success).
   New var 'runNoMoreOftenThanOncePer__seconds'(5) which saves CPU wasted by every tick (though small by most standards).  
      The user could increase this to much larger if desired.  The only advantage to a smaller
      number is that if you manually delete one or more of the downloaded news files (in the
      Common /Files/ folder), it will attempt to download within the specified '5' seconds.
   Minor updates to the logic that checks the file timestamp so that if the file is deleted by
      the user, it immediately attempts an update rather than waiting a 5 minutes interval.
   Updated the commonly used 'getModTimeOfSavedNews' function (P4L_News.mqh) to use only
      FILE_MODIFY_DATE, instead of both FILE_EXISTS and FILE_MODIFY_DATE.
      One is sufficient, and saves about 30% of wasted CPU.   Even so, each tick only uses
      about 200 microseconds unless it downloads an updated news file(s).  Each file type
      takes around 125ms, but rarely occurs, per the time interval you specify.

Quoting v2vboni

Disliked

... I'm struggling to do this because it seems the MT4 server time sometimes returns GMT -1 temporarily on my end... and then changes back to GMT +2 after a few seconds or even just a second or two ; )— It's frustrating for me. ...

Ignored

Quoting pips4life

Disliked

{quote} First off, your multiple posts got me thinking about a problem with my code, namely that the first download of the week should be repeated every 5 minutes *until* it succeeds at least once, and then it can check *much* less often (8hrs? Daily? Weekly?) without there being a risk of having the previous weekly data in the file(s). I fixed that problem (v1.02)! AFAIK, everything should be good now. Thanks for raising the issue! I am using the MQL functions 'TimeGMT()' and 'TimeGMTOffset()' and 'TimeLocal()'. With these functions, I've never...

Ignored

Quoting v2vboni

Disliked

... With regards to, "StringConcatenate(TerminalInfoString(TERMINAL_COMMONDATA_PATH),"\\File",, dataFilename)" ... I cannot use this one. If I have multiple MT4 instances (even with the same broker credentials) accessing the same file simultaneously, I may encounter issues due to file-locking conflicts or race conditions. 1. File Access Conflicts Problem: If two MT4 instances try to write to the same file at the same time: The file may become corrupted. One instance might overwrite the other’s changes. The TERMINAL_COMMONDATA_PATH directory is shared...

Ignored