Plot News indicator (version 3.0) | Page 12 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

FFcal - Plot News indicator 692 replies

Plot News (version 2.0) 103 replies

How to plot on indicator over another 3 replies

Plot on graph - Indicator help 2 replies

Indicator to plot lines for multiples of 10 within last days range 2 replies

Attachments: Plot News indicator (version 3.0)
Exit Attachments

Plot News indicator (version 3.0)

  • #221
  • Dec 25, 2011 4:39am Dec 25, 2011 4:39am
Quoting AltosTrader
It is the weekend - there is no news until market opens. You are showing history but no headlines in your screen shot.
Just to clarify:

(1) The little numbered circles show both past and future events. Your VertSpacingPips entry controls how far above or below the price candles these are plotted (or the spacing between them, if they're being plotted in a sub-window - see (b) below). You can switch them off altogether by setting VertSpacingPips to 0.

(2) The rolling window of upcoming events shows only future events. The number of events is controlled by your ShowUpcomingXEvents entry. If this is set to 0, then the rolling window is not displayed.


The following applies to both (1) and (2):

a) Only news events that have been downloaded from the FF calendar by running the Download News script (which calls the FFcal.exe app), and that meet all of the filter criteria (FromDate, ThruDate, PlotTheseImpacts, PlotTheseCurrencies, etc) that you set, will be considered for processing.

b) If you set MainChart to TRUE, both the numbered circles and rolling window of upcoming events will be displayed in the main chart window, and hence the sub-window will be left empty. If you set MainChart to FALSE, they will be plotted in the sub-window (as shown in the screenshot).

All of the above applies irrespective of whether the market is open or closed. Conversely, whether the market is open or closed has no bearing on what is displayed.
Attached Image (click to enlarge)
Click to Enlarge Name: 2011-12-25_2337.jpg Size: 175 KB
  • #222
  • Dec 26, 2011 2:39am Dec 26, 2011 2:39am
Hah ! I was not aware FFCal was available so far ahead of market opening. Sorry to confuse. Being in AUD I just load up News on my Monday morning. Thanks for pointing that out.
Cheers,
Altos.
  • #223
  • Dec 26, 2011 5:40am Dec 26, 2011 5:40am
Quoting AltosTrader
Hah ! I was not aware FFCal was available so far ahead of market opening. Sorry to confuse. Being in AUD I just load up News on my Monday morning. Thanks for pointing that out.
Cheers,
Altos.
Altos,

You can run FFcal for any week - past, present or future - that you wish, without restriction. The question is whether the news events are available in FF calendar (which you can always check - here), at the time it is run. FFcal simply downloads whatever it finds, exactly as it is.

David
  • #224
  • Dec 26, 2011 7:18am Dec 26, 2011 7:18am
Quoting hanover
Altos,

You can run FFcal for any week - past, present or future - that you wish, without restriction. The question is whether the news events are available in FF calendar (which you can always check - here), at the time it is run. FFcal simply downloads whatever it finds, exactly as it is.

David
it's good for spitting out a csv of calendar data. problem is there's not much point updating past events if using your indi with the intentions of backtesting with it. unfortunately you chose to base your 'time' from the current clock value which completely screws things up for backtesting (updating future object y positions, rolling news feed for upcoming events), and for some reason you chose to have it reload the file every real n minutes which is somewhat pointless and only harasses the hdd, but worse it causes mini mt4 freezes while it loads the file if you have a lot of data in the csv. this only gets worse with each chart you have the indi loaded onto as they all choose to reload the file at the same minute if they are all set to the same reload period.

people might not realize any of these things if they intend to use your indi for backtesting. basically you can't use it for backtesting, so don't intentionally export lots of past data else you will have mt4 periodically hang every time the indi reloads the data from file. this is obviously not good when live trading.

awhile ago i gave myself a coding day and wrote my own indi, so i'm sorted.
Congratulations ScalaFX for coming out of the closet!
  • #225
  • Dec 26, 2011 1:29pm Dec 26, 2011 1:29pm
Quoting nubcake
awhile ago i gave myself a coding day and wrote my own indi, so i'm sorted.
Sure, your points are valid, but the indy was never intended to be used in the manner that you're describing.

As a programmer, surely you're aware that it's impossible to provide a solution that caters for every conceivable requirement of every individual. Hence the "take it or leave it" disclaimer in post #1.

The logical solution would have been for me to do as you have apparently done, i.e. simply not share my work publicly. I would have saved myself hundreds of hours of 'unnecessary' work had I done so.
  • #226
  • Dec 26, 2011 1:57pm Dec 26, 2011 1:57pm
Quoting hanover
Sure, your points are valid, but the indy was never intended to be used in the manner that you're describing.

As a programmer, surely you're aware that it's impossible to provide a solution that caters for every conceivable requirement of every individual. Hence the "take it or leave it" disclaimer in post #1.

The logical solution would have been for me to do as you have apparently done, i.e. simply not share my work publicly. I would have saved myself hundreds of hours of 'unnecessary' work had I done so.
it goes without saying.

if you don't want to backtest and happen to leave mt4 always running then you are ready to go right out of the box with minimal setup. but, for anyone who wants to backtest (like myself, and i know others also) then you will have to roll your own indi or open the csv and manually read through it as you match to a chart. i call this the 'old school' way of doing it, like when traders would print charts and lay them out for study on their desks and whatnot. i shudder at the thought. frustration is surely a source of ingenuity.

point remains that users should not export a mass of historical data as it will, and does, make mt4 freeze when it is time for the file to be reloaded. people need to keep that in mind so that their live trading isn't interfered with and to keep it all running nicely.
Congratulations ScalaFX for coming out of the closet!
  • #227
  • Dec 26, 2011 2:53pm Dec 26, 2011 2:53pm
Quoting nubcake
it goes without saying.

if you don't want to backtest and happen to leave mt4 always running then you are ready to go right out of the box with minimal setup. but, for anyone who wants to backtest (like myself, and i know others also) then you will have to roll your own indi or open the csv and manually read through it as you match to a chart. i call this the 'old school' way of doing it, like when traders would print charts and lay them out for study on their desks and whatnot. i shudder at the thought. frustration is surely a source of ingenuity.

point...
Fair enough, I can see how supplying a CSV of thousands of news events could create the impression that I'd intended the indy to be used for backtesting.

I did use the original version of the indy to perform some very rough visual backtesting of specific news events, e.g. non-farm payroll, interest rate changes. In this version, all of the filtering was performed by the FFcal.EXE downloader, hence the CSV file contained only the selected news events (and thus avoiding the processing bottlenecks that you describe). And of course there was less history available at that time. When I rewrote the indy as v3, some kindly folk had suggested that it would be more user-friendly to use indy parameters instead, but by that point I was more focused on delivering a 'warning-of-upcoming-events' indy than a backtester.

Meanwhile, the FFcal downloader remains unchanged, hence it's still possible to for more computer-savvy folk to have the filtering performed at that point in the process, should they choose.

If you'd pointed out the shortcomings prior to writing your own backtester, I would have attempted to find solutions for you. However, like you say, frustration is a great motivator for ingenuity. And it's always better to have your own code, over which you have total control.
  • #228
  • Dec 26, 2011 3:16pm Dec 26, 2011 3:16pm
Quoting hanover
Fair enough, I can see how supplying a CSV of thousands of news events could create the impression that I'd intended the indy to be used for backtesting.

I did use the original version of the indy to perform some very rough visual backtesting of specific news events, e.g. non-farm payroll, interest rate changes. In this version, all of the filtering was performed by the FFcal.EXE downloader, hence the CSV file contained only the selected news events (and thus avoiding the processing...
i wondered why you seemed to have overlapping concepts in the exe and the indi, along with redundant data in the csv's (11111000). in fact it pissed me right off trying to understand the different output formats and their contents which you totally changed from file to file. mystery solved.

the data loaded in a backtest is fine at the start, but quickly goes downhill from there as the objects don't update in sync with new bars printing. anyway, same old story... too much data + scheduled file reloads = poopoo for live trading. the current version is just not designed with backtesting in mind anymore, so trying to use it in this way creates conflicting issues for those who don't realize.

Quote
If you'd pointed out the shortcomings prior to writing your own backtester, I would have attempted to find solutions for you.
i would bet a whole 10 dollars that i did, awhile ago. around the time you added the subwindow / main window output choice. unless i imagined it. anyways, did it myself instead.

Quote
However, like you say, frustration is a great motivator for ingenuity. And it's always better to have your own code, over which you have total control.
damn straight!
Congratulations ScalaFX for coming out of the closet!
  • #229
  • Dec 26, 2011 7:01pm Dec 26, 2011 7:01pm
Quoting nubcake
i wondered why you seemed to have overlapping concepts in the exe and the indi, along with redundant data in the csv's (11111000). in fact it pissed me right off trying to understand the different output formats and their contents which you totally changed from file to file. mystery solved.

the data loaded in a backtest is fine at the start, but quickly goes downhill from there as the objects don't update in sync with new bars printing. anyway, same old story... too much data + scheduled file reloads = poopoo for live trading. the current version...
Nubs,

I think the file format got changed somewhere in v2 also. Can't remember the exact reason why, it might have had something to do with the fact that v2 used a different calendar (was it fxstreet's?) to source the news.

Speaking of my memory lapses (wait 'til you reach 50, LOL), you did in fact discuss the backtesting back in post #172; judging by my reply I might have been influenced by your "a sub window would be a hack that would get around this issue entirely anyway" comment, but again, I'm not sure. Anyway, my apologies.

Cheers - and all the best for 2012,
David
  • #230
  • Dec 27, 2011 1:28am Dec 27, 2011 1:28am
Quoting hanover
Nubs,

I think the file format got changed somewhere in v2 also. Can't remember the exact reason why, it might have had something to do with the fact that v2 used a different calendar (was it fxstreet's?) to source the news.

Speaking of my memory lapses (wait 'til you reach 50, LOL), you did in fact discuss the backtesting back in post #172; judging by my reply I might have been influenced by your "a sub window would be a hack that would get around this issue entirely anyway" comment, but again, I'm not sure. Anyway, my apologies.

Cheers...
no i think i fucked that up.

i'm confident i was under the impression that the file reloading was tied into the count of how many bars had elapsed (and not current pc time), and this was tied into updating object positions and loading future objects. i've since come to realize that file reloading is tied to the pc time.

i think mt4 fails when trying to add objects too far into the future, so when you get the initial data loading and object plotting a range of future objects are perfectly set (in the sub-window) until you reach the point where mt4 had stopped bothering to plot future objects.... and so until you get a new file reload you are stuck with no further upcoming future objects as you plow away in the backtest.

let's say mt4 plots objects 200 bars into the future and no further, and you go through a day of m1 bars in a minute or so. you go right over those initial future objects quite fast and are left in a void of no data until you mess with the indi properties to force it to re-init and reload new data.

i'm sure i've come across the same future objects issue in my own mql ages ago. i can't remember if the objects get created but with bogus values, or if they simply aren't created at all. i can't be bothered testing, so i assume they don't get created at all since mt4 surely has a cut-off point where the array that stores future dates has a set size that can't be exceeded. my guess is that when in mql we refer to bar zero mt4 actually resolves this internal array index into something like the internal arraysize minus 200 (or whatever the cut off point is), so that the last 200 positions (or whatever) in the array are the near future bars and then everything before that are the current and past history bars. incrementally increasing the internal array with each new bar and referencing data like this with an offset where our mql index numbers are actually negated from the end of the array plus some extra buffer makes it so that no array data shuffling needs to be done to maintain the history data in their correct index positions, and having the extra buffer of data at the end of the array permits the plotting of near future objects such as we do in our news indi's and whatnot.

tl;dr
Congratulations ScalaFX for coming out of the closet!
  • #231
  • Jan 25, 2012 4:08pm Jan 25, 2012 4:08pm
Hi David,

I think I've found something weird in the csv file you attached in your first post. Specifically I looked at NFP days ("Non-Farm Employment Change"), and found the time can be either 8:30 or 7:30 or even 9:30. However according to US BLS website it's always 8:30 EST/EDT:

http://www.bls.gov/bls/archived_sched.htm

I checked all monthly dates from 2007 to 2011 and they are always 8:30.

I also browsed your thread, found that you posted previous csv's:

 

  1. #76: http://www.forexfactory.com/showthre...85#post4162385

  1. #92: http://www.forexfactory.com/showthre...78#post4280978


They both have similar problem. In the one from #92 it was either 11:30, or 13:30 (looks like GMT) until the last month (12/3/2010) when it became 8:30.

BTW I was not able to run your indicator since I use Windows 7. However I found the csv file quite useful for me for my own backtesting purpose.

I understand this is FF's data problem, not yours. My question is do you have a clean copy of the correct time and would you mind posting it? Or if you know where I can get similar data for free?

Thanks and best regards,

screen

  • #232
  • Jan 25, 2012 5:01pm Jan 25, 2012 5:01pm
Quoting screenname
Hi David,

I think I've found something weird in the csv file you attached in your first post. Specifically I looked at NFP days ("Non-Farm Employment Change"), and found the time can be either 8:30 or 7:30 or even 9:30. However according to US BLS website it's always 8:30 EST/EDT:

http://www.bls.gov/bls/archived_sched.htm

I checked all monthly dates from 2007 to 2011 and they are always 8:30.

I also browsed your thread, found that you posted previous csv's:

 

  1. #76: http://www.forexfactory.com/showthre...85#post4162385

[list][*]#92:...

Hi,

I've assumed that, if you turn personalized settings off, that FF calendar was always set to New York time (i.e. GMT-5 or GMT-4, depending on whether or not NY is on daylight saving).

The Plot News download process simply takes the data supplied "as is", by saving the FF calendar as an HTML file, and I've likewise assumed that this would accurately reflect the raw (unpersonalized) calendar times.

Any 1 hour adjustment that I've made for daylight saving was sourced from here. This seems to suggest that, for every year since 2007, daylight saving in NY started on the second Sunday in March, and ended on the first Sunday in November.

Apparently there's something that's not quite right in one of these three areas. My apologies; the data was presented in good faith.

I'm sorry, I don't know where you can get this type of data elsewhere. You could perhaps try the news calendars on other forex sites (easily found using Google).

It's possible to run the indy on Vista or Win7, if you follow the instructions here. However, the indy may not be suitable for backtesting - see nubcake's recent posts (#224 onward) for the reasons why.

David
  • #233
  • Jan 25, 2012 6:25pm Jan 25, 2012 6:25pm
Quoting hanover
Hi,

I've assumed that, if you turn personalized settings off, that FF calendar was always set to New York time (i.e. GMT-5 or GMT-4, depending on whether or not NY is on daylight saving).

The Plot News download process simply takes the data supplied "as is", by saving the FF calendar as an HTML file, and I've likewise assumed that this would accurately reflect the raw (unpersonalized) calendar times.

Any 1 hour adjustment that I've made for daylight saving was sourced from here....
Thank you for your response. I know where you got the .xml file. I noticed that some of the data is now correct in FF, e.g.,

http://www.forexfactory.com/calendar...y=2011-3-4&c=2 (It was 9:30 in your latest CSV but it's 8:30 now).

But some others are still incorrect, e.g.,

http://www.forexfactory.com/calendar...y=2009-7-2&c=2 (still 7:30)

Forgot to mention that I noticed some other discrepancies and reported to Twee in December. At that time I used your csv (which gives the correct time for that particular day) and some other website as evidence. I thought it was just DST problem but looks like it's deeper than that. The Factory cannot boast having the best calendar if it cannot get the historical data correctly.

Thanks for the tips on Win 7!! I'll try your indy.

BTW I'm not running backtests on CrapT4 (as Steve Hopwood calls it), but I will make use of the csv in my own infrastructure for analysis.

screen
  • #234
  • Jan 26, 2012 3:07pm Jan 26, 2012 3:07pm
Quoting screenname
Thank you for your response. I know where you got the .xml file. .......
Screen,

Thanks for your reply.

Just for the record, I don't use the XML (perhaps you're confusing me with Derk Wehler's indy?) primarily because it doesn't provide the 'actual' value from the news announcement. I wanted my indy to be able to provide me with historical details, in addition to warning of upcoming news.

Instead, I use wget to download the FF calendar for the user-specified week, which creates an HTML file. Then I parse the HTML (this was too slow using MQL4, hence I wrote an external Clarion app). The HTML may have different timestamp values from the XML; I have no way of knowing how FF's IT folk create the latter.

David
  • #235
  • Feb 23, 2012 10:21am Feb 23, 2012 10:21am
Quoting hanover
Screen,
Instead, I use wget to download the FF calendar for the user-specified week, which creates an HTML file. Then I parse the HTML (this was too slow using MQL4, hence I wrote an external Clarion app). The HTML may have different timestamp values from the XML; I have no way of knowing how FF's IT folk create the latter.

David
Hi David,

Sorry to bother you but I cannot user your great program because I don't use windows. Instead I want to integrate it in a LUA program. I am looking at the same approach as you did but I was interested if you could give me a head start by sharing your logic (programing code). That way I could read your code and use it to strip the html in lua.

Thnx in advance for your reply.

Regards,

Mark
  • #236
  • Feb 24, 2012 4:46am Feb 24, 2012 4:46am
Quoting deraaij
Hi David,

Sorry to bother you but I cannot user your great program because I don't use windows. Instead I want to integrate it in a LUA program. I am looking at the same approach as you did but I was interested if you could give me a head start by sharing your logic (programing code). That way I could read your code and use it to strip the html in lua.

Thnx in advance for your reply.

Regards,

Mark
Hi Mark,

I am very sorry, but I am unwilling to share the Clarion code that processes the HTML.

Having said that, there is no real magic to it, just a lot of sweat and trial-and-error that eventually resulted in around 2,200 lines of code.

The screens and file structures were created using the Clarion Enterprise Edition development suite, which retails for around USD 2,000. Then you would need to learn the nuances of Clarion, and translate the code from Clarion to LUA. Hence it would probably be simpler (and cheaper) for you to start from scratch with LUA, in any case.

David
  • #237
  • Feb 24, 2012 5:46am Feb 24, 2012 5:46am
Quoting hanover
Hi Mark,

I am very sorry, but I am unwilling to share the Clarion code that processes the HTML.

Having said that, there is no real magic to it, just a lot of sweat and trial-and-error that eventually resulted in around 2,200 lines of code.

The screens and file structures were created using the Clarion Enterprise Edition development suite, which retails for around USD 2,000. Then you would need to learn the nuances of Clarion, and translate the code from Clarion to LUA. Hence it would...
i bet half of that code is replicated in the indicator anyway and now serves no function.

deraiij : go play around with the calendar page on this forum. click on the little calendar up in the top left and figure out how the different types of calendars pages are generated based on the particulars of the URL passed to FF. once you work out how to manually form a url that will actually load a valid FF calendar page you are ready to go. once you have done that all you need to do is programmatically read whatever webpages you need from the FF calendar pages (i don't know LUA and how you would go about this, but in windows i'd be digging around wininet or winsock to read a webpage). once THAT is done you just need to figure out the actual html structure, ignore all of the garbage html data that surrounds the calendar data, and then just mine the details out between the html tags. this is all just basic string parsing. once you have mined the data dump it into an array that holds all of the current news data, sort the data and also ensure you haven't doubled-up on data in the database, then just spit it all out into a csv file or however you want to store it.

then it's just a case of deciding how you want to display this info back to yourself on your trading platform or in a separate application.

i guarantee you that in windows, all of this.... the reading a webpage, parsing the html and mining out the relevant data, generating a valid database, and then nicely displaying this data back to you in mt4.... can all be done in under 2000 lines of code. not just the database stuff, but the whole process.

unless it somehow takes around 8 or 9 hundred lines of code just to download a webpage then it's just not.... that.... difficult.

my personal news indi is 701 lines of code and has the capability of displaying more info than just the calendar data. i also have a webpage mining and database creating set of code that digs through hundreds of html files on my hdd to get its data, and it's only 411 lines of code. that's 1,112 lines of code so far..... the only thing i don't bother to do at this time is automatically download webpages off the internet, but i seriously can't imagine THAT requiring another 1000 lines of code.

keep it simple, functional, and to the point. you'll have it done and debugged within a few days.

the most annoying part i found was dealing with characters in webpages that you aren't expecting which appear online as perfectly normal basic text but are in fact using a unicode value other than what the basic ascii character value is for the visually same char... things like the ' and " and non-breaking space characters are little bitches that i came across which need to be dealt with for example when they aren't just the normal ascii ', ", and space character. none of this is at all difficult, but it's a case of looking through the text in your database to find text that doesn't look correct, finding where this was read from, and then going back and reading character by character the particular line of text that generated this poorly formed text string and figuring out which characters are in the string and which ones exactly are the problem and are not within the expected ascii range for that particular char. once you know which bullshit unicode chars you have to substitute when you are reading them in then you are good to go with this part and just have it automatically replace the bullshit chars with similar normal ascii chars. off the top of my head the char 160 in ascii is a capital A with a circle or circumflex above it, but in html this represents a non-breaking space and is functionally the same as declaring &nbsp. shit like that can ruin your day when you think you are finished, but discover you aren't and have another random char appearing in your text that needs to be dealt with that you did not know about.

then you also need to deal with when they explicitly state a unicode character via the &#nnnn; method. again, this is just a case of finding these unicode declaration strings and replacing them with some reasonable ascii substitute. just more basic string handling.

a lot of this shit is just tedious, but none of this should be particularly difficult. 90% of it is just dealing with text and shuffling it around to substitute text and then mine out what you need, and the other 10% is sorting and file reading / writing.

if you are a new coder then just don't even bother. just don't. if you are experienced then this should be fairly standard stuff to crank out. read webpage, mine data out of string, write db, done... then read db, display however you want it, done.
Congratulations ScalaFX for coming out of the closet!
  • #238
  • Feb 24, 2012 12:54pm Feb 24, 2012 12:54pm
Thanks David & nubcake for your replies!

I will give it a crack myself. I already had the reading html and writing to the database part. I already started with the search and substitution, I only hoped I could have some code to make it easier

Regards,

Mark
  • #239
  • Feb 24, 2012 6:19pm Feb 24, 2012 6:19pm
It works!

After all it was much easier than expected

Regards,

Mark
  • #240
  • Feb 24, 2012 10:14pm Feb 24, 2012 10:14pm
@deraaij: well done.

@nubcake: the 2200 lines is the total size of the module, not merely the section that converts the HTML to a CSV file.
0 traders viewing now