Plot News indicator (version 3.0)

Agree, there are a whole slew of issues because I don't understand how to make Plot News work on Win7, and I'm too lazy to go to the bother of learning how, especially as there are other good free news indys available.

Nubs, your explanation of the UAC considerations was excellent; I really appreciated it. I now suspect that my antiquated version of Clarion (hasn't been updated since 1997, and I'm unwilling to shell out the US$2,000 needed to do so) creates 32-bit EXEs that are somehow incompatible with 64-bit Win7. But I can't be sure, and I can't be bothered trying to troubleshoot the problem any further, especially as there's no monetary incentive for me to do so. As my uncle used to say (something like) "Goodwill is more important than money, but it doesn't keep your creditors off your back". LOL.

Interestingly, FF commissioned Derk Wehler to write a news indy (here) and also provided an XML version of the calendar, as part of the project. But that was a long time ago.

@Brucewhain: I explained all of the other reasons why I'm unwilling to commercialize Plot News in my earlier post. I'm glad you still find the product useful, and that it continues to run OK for you.

David

Well it may be there is an issue with software liability, but it's pretty unlikely a single .mql file could be used to collect anything in court - in the download new.csv a FF scenario. Interesting FF actually paid money at one point with an events indicator in mind. I'll leave it alone now but but seriously don't understand the issue with windows 7. Ver. 1.27 works fine. exactly the same so long as platform is in Program Files. Of course the download csv from FF scenario would obviate FFcal for the general user.

3 things :

1. you don't have to do anything to 'make it work on win 7'. just about everything works on win 7 except dos programs... and even those can be dos-boxed if desperate. people who can't make shit work on win 7 just don't understand their own OS and that's their issue. it really is just a case of MOVING MT4 TO SOMEWHERE NOT CONTROLLED BY WINDOWS (and as i keep saying, my mt4 lives in a folder on my desktop).

nobody who does this ONE THING... this ONE TINY THING... ever 'has problems with mt4 on win 7'. it's just that simple. anyone using win 7 and mt4 who can't do this one tiny thing gets what they get. *shrug*

2. i used to use your news indi, then i wrote my own. but i still used the ffcal program to do the downloading since that aspect was already done and not a wheel i felt the need to re-invent. i'm on ... wait for it.... windows 7 64-bit!

so i really don't know what all the problem is that someone claims they had. sounds to me like their pc or OS install is just shit. period. with enough effort and talent it is still possible to break win 7. most of us just aren't that 'gifted' if you get my drift. for all anyone knows they could have bumped the tower with their knee like i've done a million times.

3. there isn't even a need for ffcal. you can do it all within mql. when i get my lazy ass around to it i'll eventually roll my own news downloader in mql as a part of my news indi, plus a few other news sources i'll make it download. i actually do a lot more with mine as i have a lot more data that i feed it, but getting the data was still an external process through another program and then another script i wrote specifically for text parsing and dumping the data into a csv, in the same vein as ffcal but not as elegant or efficient. something like over 70 meg of html was parsed into 2 csv data files (since i hit the openoffice row count limit and was unable to visually check that the data looked correct).

...

bottom line is that anyone who can't get their shit to work only has themselves to blame. the lazy route is to just maintain compatibility with the ever-changing ff format. the 'it has to be compatible because it is a damn ex4 file' route is to just do the download and parsing in an mql script and ditch ffcal... but again, not even necessary. for any reason. at all.

the laziest route is to have people update the csv file by hand each week themselves and tell them to piss off.

...

i'm not actually saying to do anything. i'm just replying because people still feed each other misinformation and it rubs me the wrong way, and you've come to the wrong conclusion that win 7 is magic and hates 32 bit programs as well as mt4 and certain ex4 files. nope. all totally false, and refer to point 1. it really is just that simple.

oh, and i don't even know if the virtualstore is considered a part of UAC or not. UAC hassles me when i go to run certain permission-requiring programs (like wampserver, or install a new version of directx, etc), but i think the virtualstore is a separate issue, and in this case is THE issue that people have... not UAC. it's all just confusions about the virtual store. move mt4 to somewhere that it can call home and the virtualstore doesn't become a factor at all and windows won't use this feature as it has no reason to. as per previous ramblings, virtual store is just to stop shitty programs spamming files all over the program files directories and wherever else that windows monitors. it just redirects the file input/output to an area safely tucked-away elsewhere without the program needing to know any different. stick a shitty program somewhere that windows doesn't care about and it can read/write wherever it likes, and people looking for those files will find them exactly where they expect them to be.

what's the point of the virtual store? as i said, to keep shitty programs from spamming windows locations. delete a user account on the pc and there goes their virtual store files from spammy programs and whatever other personal files associated with their account.... and all the while the program files directory is as clean as a whistle with no files spammed anywhere, so if / when you fully uninstall a program the windows program files directory WILL be able to clear the program out of that area properly.

i should also add that the 'slew of issues' i was referring to was actually point 1 above. when people can't even figure-out how to get mt4 running properly on their pc then trying to get them to figure-out how to use indi's and scripts etc is just a nightmare that FF surely wouldn't care to dip their toe into, and would rather let silly shits like us deal with it while they still get traffic across the site.

Yes virtual store (not a web shopping experience) was a big problem for me in the beginning - and computer science is definitely not my strong suit - but think that not so recent windows updates have allowed initial download of MT4 into program files (this based on first hand experience) and made it so its being locted there doesn't get it torn up and thrown in the trash of that virtual store thing. Still Windows 8 no doubt offers new challenges.

i'm not entirely sure i understand what you just said. i think you are saying that it is possible to install mt4 in the program files directly. yes. this is the case, and always has been. you can install whatever you want into the program files directory. what windows won't allow is for programs in the program files directory to read / write into anywhere within program files. it redirects the I/O to the virtual store directory which is tucked-away in the depths of the currently running windows user account along with desktop shortcuts / files, desktop image info, etc etc etc.

what you probably don't understand is that if you install mt4 into prog files is that it will run perfectly fine.... up until you start dicking around adding indi's and indi-specific files (such as ffcal and the csv files etc). when you do that you set yourself up for failure because YOU expect those files to be what changes when you run ffcal, but what actually happens is windows does all of the file I/O into the virtual store.... so you end-up with whatever files YOU added to the mt4 directories, as well as the virtual store updated and current versions that were created when you ran ffcal to update the csv file. whatever you dumped into the mt4 directories is like being in stasis and just never changes.

when you go plonking around in your mt4 directory and see that the csv files haven't been updated your mind goes blank. "oh win 7 is shit and hate mt4". no. win 7 just updated those files in the virtual store, instead of messing around within program files.

so rip mt4 out of there and stick it somewhere else. then just use like normal. windows won't need to protect itself from file spam and so the directory structure YOU expect when you poke your nose around the mt4 directly WILL be exactly as you expect, and no dark windows majik will have occurred to ruin your day and expectations of where certain files are.

think of the virtual store as the naughty corner for file I/O that should have otherwise been directed to the user accounts appdata or whatever directory. it's like how some games now all dump their save files and whatnot into the user account game directory... keeps things tidier than letting programs add all kinds of crap into where the program resides in the program files directory. you have your programs, and then you have the added files living elsewhere that can be wiped nice and cleanly. programs that don't adhere to this newish regime have their file i/o automagically directed to the virtual store, since they are old programs that have no reason to understand the new 'stick your files here' concept of the users account directory, or for programs that simply suck and choose to not follow this design and want to write files next to their exe's living in the program files directory.

i should note that this might be technically incorrect, but it's a close-enough model that i manage to get by more than fine, unlike some others from the sounds of things.

i'll shut up now because i'll just keep going if i don't. no one hates on my win 7. my precious!

Ugh!

When I first got Windows 7 it not only took it's choice among all files downloaded into P-FilesMT4 and carted them off to the virtualstore but - apparently because MT4 was not considered compatable with the new OS then - took the entire MT4 program and divided it up between virtualstore and P-Files. Result: (as i remember) experts folder end up in vs so suddenly there are no indicators. I overcame this by various means, the most effective being that Take Ownership feature that everyone was downloading for a while. Believe it has now been defanged.

What you don't seem to realize is that there is no way to use Plot_News with FFcal unless the platform is located in Program Files. The path for updating the csv file cannot be customized. No doubt hanover was trying to cope with this mysterious-for quite-a-while to me virtualstore thing (designed I beleve to make your surfing habits available at all times without permission for anyone - law enforecement, market research or just hackers - who wanted it) and would have people tell him "but there IS NO EXPERTS FILE!" and stuff like that.

I believe Microsoft has seen the folly, and resentment, that their new policy caused, and instuted fixes. As far as I can tell the virtualstore has stopped updating completely whereas even after I "Took Ownership" of Program Files it was expanding at such a rate that I was afraid it would use up my hard drive. One time I just f#8g deleted it. But it doesn't do that anymore. Also believe updates have relaxed Windows' restructions on what you can do in Program files because I do everything I need now. The Platform and Plot_News have functioned perfectly for at least a year and a hlf in Program Files.

lol, or you could just move the folder and be done with it. and i have ffcal in my desktop copy of mt4, so i don't know what you were talking about when you said ffcal needs to be in program files, unless this is something that has since changed. afaik ffcal only cares to output wherever it happens to be.

admittedly i haven't used it in a long time, so unless it has been otherwise updated and actually broken since the copy i have, then.... nothing. i think you might still be a bit confused about the whole thing, or i just don't understand what you were saying.

Another illuminating post; many thanks. IMO my biggest problem is that I don't have a Win7 PC, hence I can't troubleshoot and experiment myself. Thus any 'conclusions' I reach are largely formed from the observations of users who are probably even less computer literate than I am. Anyway, I will likely purchase a Win7 PC soon, so hopefully that will rectify things to some extent. When I do, I'll re-read all of your posts that I've bookmarked, and they should make even more sense as I get some 'hands on' experience.

I attempted to write a news downloader in MQL4, but I found the string manipulation (extracting the relevant tokens out of a large HTML 'blob', by processing it character by character) was way too slow. It was taking anything up to 7-8 minutes to parse 170kb, if I recall correctly. Perhaps there was a more efficient way of coding it, but in the end I gave up, and rewrote my downloader in Clarion. So I'd be very interested to know if you encounter the same speed problems, and (if you're willing to share) how you overcome them.

Bruce,

You should be able to type a full path/filename into the field highlighted in the screenshot. Pretty sure it works OK with WinXP — it was certainly intended to take a full path like c:\Program Files\My MT4\experts\files\news.csv

Obviously you need to get the spelling, punctuation and spacing exactly correct, for Windows to recognize the path/filename.

If you simply specify news.csv then it expects to find that file in the same folder as FFcal.EXE

David

O mah god! You know why I thought that? I just push the Setup button every time and never look at the main interface. Well that's good news, from a versatility stanpoint...

Win 7 x64 here. Installed with your installer in folder programs x86 and works fine. Just my 2 cents. Been using this since 2008 and love it! in fact,the only issues i ever had with it was with xp back in the earlier days.

Thank you again for providing us with your time, help and this and many other Ea's and indicators you have made. It is realy appreciated. Alot of work goes into these. Alot being a real understatement here.

when you get win 7 you'll realize that i've written many many words to describe something that is actually rather simple to understand when you observe it and undeserving of the amount of text i've written on it... my win 7. my precious!!!

as for speed concerns parsing text, it sounds like you may be doing it a harder way than necessary. i don't know how ff formats its data, but surely there is some 'format' to it which lends itself to jumping over known-useless parts of the html and then just searching for specific markers that you can anchor to and start pulling data from that point onwards. especially things like css identifiers in stuff like p or table html tokens can make it super simple to 'find your way' around a html file of data, and then use those as jumping-off points to start reading in chunks of relevant data. i mean, this stuff is output by pc's automatically so it has a structure to it that you can use to your advantage and know where to expect certain pieces of data to be.

because i don't really want to go back to using delphi / pascal i tend to write code in whatever i'm currently using, so that's mql or like right now i'm coding some non-website stuff in php and running it on my pc using wampserver. it's easier to physically type in c-like languages versus pascal with it's million " := " assignment tokens, so back then mql was the way to go as far as i was concerned, so that's what i used to parse the text despite a proper exe certainly being superior and an option.

what year is it? 2012? hmm it was probably 14 months ago when i parsed the 71 meg of html, and then a different news source with just over 50 meg of html... so grain of salt with how i remember things. but yes, the way i went about it took a long time to get through the text. if i remember correctly i took somewhere around an hour for the 71 meg?

edit :
on an average dual core 2.8
/edit

so what's that... roughly a meg or less a minute? i might be being generous, and perhaps it took 2 hours. i really don't remember, but i don't think it was anywhere longer than somewhere between 1 and 2 hours. it'd be either around a meg a minute or half a meg a minute. not exactly lightning fast. there are ways and means of being efficient with the string handling which i did not do AT ALL! i'm reading lines of text at a time from the files and passing copies around by value between functions to sanitize the strings etc. if i cared for speed i would have just read in one entire file at a time and then inspected chunks from the string and passed index values and length sizes of whichever part i'm currently looking at to the other functions etc, instead of passing back and forth string copies. i imagine that would make for a significant boost... i mean, it could only get faster compared to how i was doing it. one fast file read instead of many smaller bursts, plus no string copying and passing around and instead just one main string that you reference parts of by index and length.

i'll say this though, one part that i wasn't looking forward too was sorting the data when combining it all together. usually selection sort is my go to sorting method since it's so basic and is usually more than adequate, but once you start dealing with decent amounts of data the time it takes exponentially increases until it basically locks your pc up.

it turns-out that with a simple 'comb sort' it takes merely a moment, as in 1 second or so. i was super-surprised at how quickly comb sort works. selection sort takes forever. i'm not even sure if i ever managed to wait long enough for it to finish selection sorting before i dumped that in favor of comb sort. altogether i have 1 news source that is spread over 2 csv's due to it being 17 meg and too many rows to inspect with openoffice if you keep all of the data in 1 big file, plus another 1.5 meg file of csv data from a different news source. each of those files is comb sorted individually.

each time the news indi runs it gets the data from the ffcal csv and also the csv files i've generated from the other sources and then does a comb sort of the combined data (after dumping useless data that is before the date range i care to deal with) and combines them all together. so that's the ffcal csv, plus a 9 meg csv, another 8 meg csv, and finally a 1.5 meg file of data i generated from another news source, then large parts of it all is dumped for being not relevant, and then whatever remains all gets comb sorted by the news indi when it initializes, and it takes all of a second or 2 where mt4 is locked-up while it does this and then it's finished and mt4 is running normally again (this is in backtesting i should add... the whole point of why i made my own news indi and wanted to get all of this other data).

comb sorting is just sexy, especially when you are using it to combine previously sorted data into one larger heap of combined and sorted data. i think the more pre-sorted the data is the faster a comb sort is, but i'd have to revisit the comb sort code to check this and i might be wrong. even if all of the data was unsorted i'm sure it would still only take a few seconds to sort it all together anyway.

edit 2 : the final consideration with all of this is that the data being parsed shouldn't really amount to a whole lot in the end anyway and should really only take a few moments. read-in the page for a weeks worth of news, parse, add to database, load the previous week, rinse and repeat until overlapping with existing data that is already sitting at the very end of the currently known data.... overwrite the old file with the newly updated data. keeping in mind things like data in the current week that hasn't occurred yet and so still needs to be overwritten on the next update to include the figures that finally were associated with the news event, but that's a minor quibble to overcome.

shut up nubcake.

I'll have to look at comb sort. I've used shellsort in the past, it seems to work well enough. If I have to sort a list of identically formatted strings, I prepend a sort string to the font of them, while I'm loading into an array. Then I shellsort the array, and finally extract the strings again. Not that efficient, especially if there are a lot of strings in the original list, but it can be done in a few lines of code.

Re the slowness of parsing the HTML: I attempted to use MQL4's StringFind() to search for certain tags, but it not much faster than a character by character parse, if I recall correctly.

The approach I currently use (Clarion) is to first parse the HTML and build an output file, splitting it into lines (rows) that start with <tr> and end with </tr>, and outputting to a text file. Then I parse each row of this file, and simply pull the relevant field data (date, time, currency, description, etc) from between each <td> and </td>. The impact and revised from (if you want them) are a little more tricky, but they always follow a given signature string. Not too difficult.

shut up hanover.

redundant text file. you could achieve the same thing by building an array of strings if you were dead-set on maintaining a line-by-line structure, but then you are just creating complexity where none is needed as you could still just read each table row from one main string that holds the entire html text, rip that section of text to a separate string... deal with it... and then move on to the next table row section, similar to how you seem to do it with the text file. it does sound like you've made it harder on yourself, but i can appreciate why. sometimes it's "easier" to make it a bit more complex or redundant but to maintain a certain schema as a means to an end. but this can create traps.

shellsort sounds to me like comb sort but applied a bit differently. i'm not that familiar with different sorting algo's since i've use selection sort for 99% of the stuff i've needed to sort, then a type of bucket sort combined with selection sort for one particular application i wrote, and then i came across comb sort which rocked my world.

Just one more little question (shut up bruce) pleeeze...? lest my ignorance show again (And my german landlady said once that I was the only foreign speaker that ever lived in her establishment who's german was actually worse after a year than when he moved in - so you know how I would do at computer code, so please don't get angry at any misunderstanding I might still have of the possible situations that a professional coder might find himself in including relations with perspective and revenue customers and legal issues) but...

What if - and I'm not saying this is actually an if, but a purely hypothetical if - what if it was somehow agreed that FF would be contacted regarding Plot_News (by someone or a group who are actually a going concern of course, which would leave me out except as an eager prospective user) and they subsequently agreed to use a version of PN in which the csv file was downloaded by users from - perhaps - the Calendar Page of the FF website - including the requsite reduced and laterally emphasized depiction of a black-background M5 chart with "0"-size billiard balls, thus enabling a compulsory once-a-week visit by users (but i go every day at least just to read news about the news, and many would probably want "Actual" outcomes on their charts, enabling still further of compulsory same) thereby leaving the parsing and the provision of a useable downloadable file (it having already passed through the throes of creation) and the questions about how to download it, to FF, who after all are the ones who originally contemplated the idea and paid something to have it realized in code...

Coding of the original mechanism to be used by FF to isolate the necessary information in the csv file totally aside - and the current version seems to work fine - Would using such a version, with the csv downloded from an FF webpage, not result in a minimum of work for the creator(s) in providing FF with an indicator usable in all current versions of Windows plus MT4 and (another big coding excursion probably) MT5?

It seems to me after hearing you both that this (strictly hypothetical) option might be more feasible (if at all) than what i had in mind before.

Because otherwise, one scenario that pops up - and please forgive me for bringing up this epic-scale bad-case scenario which actually is after all none of my business - is that of some future FF-savey coder coming along and coding an exactly similar indicator, compatible with both MT7 and Windows Infinity, thereby cashing in on hanover's intellectual property with no knowledge or notification to his heirs or assigns and no legal recourse. Because - and don't know where my authority comes from to say the following except that it's obvious: It is the one obvious and correct way to do the thing. (And because I know there are coders who would with complete knowledge of what they were doing think nothing of it whatsoever, having through my recent FF request for advice re. an indicator had the displeasure of dealing with at least one such "funny" coder.)

(last question on the subject I promise as this die hard has foremost to consider the self-preserving necessity of being permitted to remain a die hard. ((afraid now I am indeed "over thinking this".)) B.

so you are saying that ff should make a file downloadable to use with an indicator... when we already DON'T NEED THAT FILE ANYWAY? why would we go from ripping the data automagically to making it so that someone needs to download a file and stuff around with that? your proposition is worse than the current situation, and has the minimum of benefit (if any at all) for FF and the maximum lack of benefit for ff news indi coders and indi users!

why don't we just do like i said in a previous post... make all users just hand-add the data themselves? because that would suck, right? right.

as for 'other coders' coming along and 'cashing in' on this idea, well, i've had that very thought myself. i have my own news indi, and it is visually similar to hanover's. does hanover own the sole rights to all news indi's? and if you think so, then what part of that ownership is mine as i know hanover has taken advice of mine and made alterations to his indi from my suggestions? to put it humbly (lol) my own news indi is superior for a variety of reasons. it's why i bothered spending my time making my own, because what i wanted, and needed, i could do... so i did. hanover's indi was not fulfilling the requirements i had, so i had no choice but to roll my own version from scratch. what you wrote seems to imply that i have no opportunity to do anything with what is 100% my own work. if / when i find the need for something i tend to make an attempt at it myself because in general it will be far more suited for what i want than other options out there. because it has similarities of one sort or another surely does not restrict my rights as sole coder of what i've written.

you are still over thinking this... a lot.

Quickest way to get the job done, IMHO, given the limitations of Clarion.

IP is a gray area IMO, exactly where to draw the line. For example, some ideas are sufficiently generic that I doubt they could be patented. Anyway, it's irrelevant if neither of us are intending to go commercial with our indys. I'm certainly not, for the reasons I gave previously.

I likewise wrote my indy to meet my own requirements.

I don't see where you would get that idea.

Since the indicator is so similar to the Bloomberg one suppose it's a cloudy issue, though noone's made an issue out of it and don't see why the would care. If there continues to be a free forex platform long into the future soneone will probably make a pretty much identical indicator or revive the current one.

[quote=hanover;5509984]
hi Hanover i'm new to this side and i'm residing in singapore can you let me know the download links of your news indicator and what should i do about it as i really in need of a good news indicator.my gmt is +8..pls knidly help I'm really in dire need of this news indicator..thank you so much..

Hi,

I have withdrawn Plot News from the public domain after receiving a complaint from a user that it was crashing his computer, and threatening to hold me responsible.

Hence Plot News is no longer available. I'm sorry, but I don't want to risk any further problems.

Anyway, euclid has developed an excellent news indicator — here.

David