Plot News / Downloader version 4.00 | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

FFcal - Plot News indicator 692 replies

Plot News indicator (version 3.0) 402 replies

Good History Rate Downloader EA 14 replies

Plot News (version 2.0) 103 replies

Attachments: Plot News / Downloader version 4.00
Exit Attachments

Plot News / Downloader version 4.00

  • #41
  • Nov 5, 2015 6:12pm Nov 5, 2015 6:12pm
I was hoping you'd do that but it felt too cheeky to ask for source code, fantastic, thank you!

I also stumbled across this https://import.io/ , which could potentially create alternative news sources such as http://uk.investing.com/economic-calendar/, Reuters, etc pretty simply and deposit the data as a CSV into the files folder using a custom API which can be called using the wininet.dll functions from MT4. Pretty neat solution I thought.

M
  • #42
  • Nov 5, 2015 10:45pm Nov 5, 2015 10:45pm
Quoting mlewis
I also stumbled across this https://import.io/ ........
ML,

Hey thanks, that's pretty cool. I ran it using the FF calendar page as a test, and it produced a very tidy CSV; the only problem with FF is that the 'impact' and 'revised from' values are embedded inside (rather than between) the HTML tokens, and hence were omitted from the CSV. There may be a way around this, I will continue to experiment.

Nonetheless, I still like parsing the HTML manually, as I tend to prefer to work with my own code, rather than within the constraints imposed by somebody else's 'black box'. The News Downloader.EXE app is written using Autohotkey; apart from all of its macro/hotkey facilities, it's a surprisingly versatile programming language that can be run interpretively with debugging facilities, and also compiles/compresses to standalone EXEs that offer some protection against reverse engineering/piracy. I spent a few days expanding the language by writing my own functions for string/number/array/object processing/formatting/conversion etc, just as I did with MQL4.

My knowledge of API programming is not great. There's probably a much better approach than the ShellExecuteA routine that I copied some years back, into the downloader script. It seems to require the full path of the executable that is to be run, hence the need to store the folder name in the INI file. Messy, but it does the job.

Re the News Downloader.EXE app, I ssume that you're aware that you can include an uppercase 'A' in the 'Weeks/months to download' which automatically bypasses the settings dialog window in all future runs; it simply gets the settings from the NewsDownloader.ini file, and continues to do so until the 'A' is removed by manually editing the file. Then you can have it bypass the "Now go to MT4, select a chart with Plot News, and then press Ctrl-J" message by leaving the 'Macro for indicator refresh' setting blank. That way, the app executes without any operator intervention (assuming you want it to run that way, when called from your EA).

David
  • #43
  • Nov 6, 2015 4:15am Nov 6, 2015 4:15am
Well I'm on build 902 for my sins, so ShellExecuteA no longer works after the build 600+ switch, so ShellExecuteW would be the fix I think for everyone else crazy enough to regularly update their MT4:

http://forum.mql4.com/67340

Also you can do this for the data path which is simpler:

string terminalDataPath=TerminalInfoString(TERMINAL_DATA_PATH);

I wasn't aware of that but was wondering how best to do it! great!

Mark
  • #44
  • Nov 6, 2015 6:00am Nov 6, 2015 6:00am
Quoting hanover
{quote} The News Downloader.EXE app is written using Autohotkey; apart from all of its macro/hotkey facilities, it's a surprisingly versatile programming language that can be run interpretively with debugging facilities, and also compiles/compresses to standalone EXEs that offer some protection against reverse engineering/piracy. I spent a few days expanding the language by writing my own functions for string/number/array/object processing/formatting/conversion etc, just as I did with MQL4.
Hanover,
That's great to know how you actually wrote the software. It was really clever of you. I was wondering if you could share the Autohotkey code with use, so we can try to adapt the app in order to retrieve other kinds of databases from the web... it would be extremely useful for many applications. Let's say, for example, to retrieve stock market news from Yahoo... to be constantly scanning the market for certain keyword. This keywords could later be used in a neural network app that can try to predict the stock fluctuations from its news release. It is just an idea. But there are many other applications, really.
  • #45
  • Nov 6, 2015 9:39am Nov 6, 2015 9:39am
Quoting mlewis
Well I'm on build 902 for my sins, so ShellExecuteA no longer works after the build 600+ switch, so ShellExecuteW would be the fix I think for everyone else crazy enough to regularly update their MT4: http://forum.mql4.com/67340 Also you can do this for the data path which is simpler: string terminalDataPath=TerminalInfoString(TERMINAL_DATA_PATH); I wasn't aware of that but was wondering how best to do it! great! Mark
Many thanks for the info.

Quoting pedrosimao
I was wondering if you could share the Autohotkey code with use
Sorry, but I'm not willing to do this. Two main reasons:

1. Read this post. Perhaps I'm paranoid, but I've put a lot of hard work into this software, and I don't want to run the risk of having anybody else make money from selling it. The button in the downloader that points back to this thread is there for good reason: if anybody attempts to sell the app, the buyer can click the button and immediately see that he could have obtained it free here, and has therefore been defrauded. The previous version of Plot News was downloaded more than 50,000 times, hence I intentionally posted version 4 in the commercial section, so that less folk would see it.

2. It really wouldn't be that helpful, especially for the uses that you're suggesting. The HTML code in every website is completely different, hence most of the code in the downloader would need to be rewritten. AHK is pretty easy to learn, I picked it up in just a few days. You can download AHK here, and a good IDE here. And of course there's no reason why you couldn't use any other programming language of your choice.
  • #46
  • Nov 6, 2015 10:46am Nov 6, 2015 10:46am
You seem to go to great pains to obfuscate your code and engineer libs to handle shit that shouldn't be so hard. Then you just give the software away anyway.

I think you are making things a lot harder on yourself than is necessary. Never parse html by hand. Since you like to use Clarion it seems there is support for dealing with XML... which should get you all you need for parsing HTML.

http://www.softvelocity.com/clarion/...xmlsupport.pdf

Then you never need to fuck about with walking through strings. Just walk the DOM objects and get their data as needed.

I have a PHP website that crawls about 8 other websites every 15 minutes and aggregates their data. Doing that by walking through strings would suck major ass and would require probably 20 times more code per website than I currently have and would be far more susceptible to failure. Realistically I scrape data from a site with around 20 to 50 lines of code, because dealing with a DOMDocument object and letting it handle the nitty-gritty of splitting the cURL'ed GET response string saves a tonne of work.

Even AutoHotKey seems to have an API for dealing with an HTML DOM.

http://autohotkey.com/board/topic/11...an-html-table/

Scraping websites should be the easy part of all of this. Don't make it 80%. Make it the 20%. There are a lot of problems that are already solved and ready to go. Not all black boxes are opaque.
Congratulations ScalaFX for coming out of the closet!
  • #47
  • Nov 6, 2015 3:33pm Nov 6, 2015 3:33pm
Quoting nubcake
You seem to go to great pains to obfuscate your code and engineer libs to handle shit that shouldn't be so hard. Then you just give the software away anyway......
Many thanks for the info.

I'm not really trying to obfuscate anything, just taking some simple and obvious steps to prevent others from selling stuff that I've spent some time on. I suppose it would be simpler not sharing anything at all. Needless to say there are a number of indicators and EAs that I've written that will never get posted on forums like FF.

The version of Clarion that I have was purchased in 1995 and is 16-bit. As you know I ran WinXP for a long time, but I bought a new PC with 64-bit Win7 almost 2 years ago, and Clarion 16-bit won't run (and it would cost US$1,000+ to upgrade it). I had a friend attempt to re-partition my disk so that I could optionally run a 32-bit version of Windows, but he couldn't get it to work. But I didn't pursue this further, because now that I've discovered AHK, I don't use Clarion any more. I've rewritten a few of my old Clarion apps, the Plot News downloader is one of them.

No doubt I've made things harder on myself by reinventing the wheel, but I don't mind parsing the HTML, it gives me some practice using regexs, which I quite enjoy.

Your programming knowledge, especially OO, API, and web stuff, is streets ahead of mine. I'm an old dog 1980s COBOL programmer, and it's fair to say that most of the time I'm too lazy to learn new tricks. My health is not so good these days which doesn't help. This year I've spent more time trading than programming, tinkering with AHK was a rare exception.
  • #48
  • Nov 6, 2015 5:52pm Nov 6, 2015 5:52pm
Quoting hanover
{quote}Clarion 16-bit won't run
What? Not even in compatibility mode or anything? That's shite.

Maybe this... https://groups.google.com/forum/#!to...on/mTXJt3NTCyU

Quote
(and it would cost US$1,000+ to upgrade it).

Screw that. Just learn pascal or python or php, or something. Shit, if you HAD to you could try using Node.js. There are free options out there which I would have to imagine are more robust than AHK. It seems AHK can do more than I previously knew.

Quote
but I don't mind parsing the HTML, it gives me some practice using regexs

Do you hate life or something? Should we call somebody for you to do a welfare check? Fuck... that.

Quote
especially OO, API, and web stuff, is streets ahead of mine.

Not really. It's more... just around the corner. None of that stuff is overly complicated once you use it. It's always, always, going to be foreign noise looking at it from the outside. But once you start using it things become clear, and quickly. You've been doing this for long enough that it'll no doubt be easy. It's not like you are starting from scratch trying to learn to program as well as learn a tonne of current-day tooling and verbiage.

Quote
I'm too lazy to learn new tricks

These aren't that new, and are more just a twist on old tricks. It's doing a back-flip that you've done a thousand times, but a slightly different way that helps prevent you landing on your head so much and uses less effort as well.

Quote
My health is not so good these days which doesn't help.

Well that sucks.

Quote
This year I've spent more time trading than programming, tinkering with AHK was a rare exception.

Probably not a bad thing.
Congratulations ScalaFX for coming out of the closet!
  • #49
  • Nov 6, 2015 10:56pm Nov 6, 2015 10:56pm
Quoting nubcake
What? .......
Re Clarion, thanks for the link. Confirms pretty much what I thought. My Clarion days are behind me now.

I've found AHK to be surprisingly functional and stable, for a product of its type (screenshot FWIW --- and I see Fortran is still in use --- another name from the distant past ). Was planning on teaching myself Python (and might still do so) before I discovered AHK. I've no doubt that the more widely used products offer greater functionality, more libraries/objects, better IDEs and interfaces to more products, databases, etc, simply because more programmers use them, and hence (especially if they're open source) they get developed to a more comprehensive level. However, for better or worse, I like writing my own reusable code libraries for use as building blocks in bigger projects. If that means that I need to see a psychiatrist, well so be it.

I wrote the News Downloader because I wanted to plot news events on my charts, nothing more. I have no reason to showcase my outdated programming skills, and I'm not looking to sell anything. I retired from the corporate software world in 2005 and have no great desire to work office hours again, regardless of whatever might be on offer. I have a couple of other business interests that require some focus, and given both my current situation and the challenges imposed by forex, a few profitable years of trading would pretty much complete my 'career satisfaction' bucket list.
Attached Image (click to enlarge)
Click to Enlarge Name: PPI 066.png Size: 105 KB
  • #50
  • Nov 7, 2015 3:04am Nov 7, 2015 3:04am
Hi Hanover,

Thank you for posting this for free and for all the hard work you have put in.

Thank you again.

Dale
  • #51
  • Nov 8, 2015 9:27am Nov 8, 2015 9:27am
David
was there any day in past... when you never shared something useful or helped someone ?
I bet not a single one.

Regards
100 Fold Challenge->Interested? ->https://www.forexfactory.com/thread/32152
  • #52
  • Nov 12, 2015 10:36pm Nov 12, 2015 10:36pm
Thanks hanover for another great MT4 app and it's good to see you back. I used to use FFcal with added speech from CodersGuru worked very well by hearing the news being read out..

Thanks again
Rod
  • #53
  • Nov 15, 2015 8:43am Nov 15, 2015 8:43am
Thanks Hanover (smarty pants). I don't know why you offered this to all of us, but it showed generosity that surprises me. One question; do you have the code at hand to call this indicator within expert advisor.
  • #54
  • Nov 17, 2015 6:08am Nov 17, 2015 6:08am
Quoting Rhomeroo
do you have the code at hand to call this indicator within expert advisor.
See posts 39 thru 45.
_________________________

@Dale, forexsaint, Roddo: many thanks for your kind words.
  • #55
  • Nov 17, 2015 3:03pm Nov 17, 2015 3:03pm
Once again Thank you.​
  • #56
  • Nov 25, 2015 1:58pm Nov 25, 2015 1:58pm
Hello thank you for make this great program hanover just one question though, why when i try to plotting (step 4. run the download news v4-00 and step 5. Attach the plot news v4-00 indicator) when i do both step my chart still same, i don't see anything that similar with your ss
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot2.png Size: 213 KB


after that i try to compile with my metaeditor "download news v4-00 script" and found error there
Attached Image (click to enlarge)
Click to Enlarge Name: Screenshot3.png Size: 209 KB
then i try to update hanover --- extensible functions (np) and hanover --- function header (np) here (build 600+ compatible (mine is build 910)

can you please tell me any ways to solve this problem? or maybe some insight?

or anyone who already solve this problem please help me T_T

Thank you so much for anyone who try to help me, and sorry for my bad english
  • #57
  • Nov 25, 2015 4:29pm Nov 25, 2015 4:29pm
Quoting shirotama
can you please tell me any ways to solve this problem?
The script will compile correctly ONLY using build 509.

Do NOT attempt to compile the mq4 files. (I will delete the mq4 files from post #1 to avoid any further confusion).

Please try the following:

1. Delete ALL mq4 files (related to Plot News or News Downloader).

2. Re-run the installation process (as explained in the QUICK START GUIDE in post #1), and everything should work correctly.
As explained, you must run the NewsDownloader.exe once manually first (see step 3). This should load news events into the news.csv file (you can check to see what's in this file by using a text editor like Notepad). Then the Plot News v4-00 indicator will read this file plot the events on your chart.
  • #58
  • Nov 26, 2015 12:57pm Nov 26, 2015 12:57pm
Hi hanover,

Maybe I m late with this idea, or not. I try to find how can I plot the future move of the price according with the forecast numbers from the next events (news).
I mean, if next event with forecast will be in 60 min, a pair will price in the forecast more or less and close to that event. I think you get the idea.

I will not go further with, only if you are interested.

Ty and have fun.
  • #59
  • Nov 26, 2015 1:29pm Nov 26, 2015 1:29pm
Quoting shirotama
can you please tell me any ways to solve this problem? or maybe some insight?
In addition to my previous comments, I've just uploaded a new version that fixed a couple of minor bugs. I don't know if it will solve your problem, but you might like to try it. Simply go through the process in the QUICK START GUIDE in post #1. When downloading the news files, make sure that they overwrite the previous versions of the files.

Quoting barbones
Hi hanover, Maybe I m late with this idea, or not. I try to find how can I plot the future move of the price according with the forecast numbers from the next events (news). I mean, if next event with forecast will be in 60 min, a pair will price in the forecast more or less and close to that event. I think you get the idea. I will not go further with, only if you are interested. Ty and have fun.
I'm possibly stating what might already be obvious but......... the only way to find out if an idea is profitable is to test it yourself. There are a lot of different categories of news events (inflation, employment, growth, production, etc), and a lot depends on how their outcomes can potentially impact future central bank policy, and how the market as a whole perceives what that impact will be.

I haven't conducted any research as to what extent price might anticipate a certain outcome, and hence the effect gets partially 'priced in' before the announcement. It might be worth looking in to.
  • #60
  • Nov 26, 2015 2:09pm Nov 26, 2015 2:09pm
The idea it s not simple and i ' m not a coder but I have some experience how to implement a concept. Also my english it s not so great.

I think that all that will came it s the thing that will impress the market or if others think in the same way and they have resources, why not to try to go where market will be at the next event. Priced in movements that we see most of the time.

Now, let's imagine: what if from now till then, we will have on the chart the price from the future like a line. Chart line, Simple trendline from now to a point in the future, whatever. You get the idea.

We know that the main driver of today economies are Central B, we will call this A.
We know that CB look, in this period of time at inflation. B.
We know employment is another part lets call it C.

Let s assume that RED are the stronger then Orange and Yellow, but in the end last 2 types will cont in some form.
In this moment we have 3 types line. In the start are independent. Only for now. Red will drive the market but the influence of last 2 types will change the power from the begining.

In this moment we combine this 3 line in 1 that have the main power form RED and, in time will become stronger or weaker because of the new smaller events.

Percentage between this 3 types of events will define the Main Line - the power - strong UP/down for his future plot.

Now, about the horisont in the future. A starting point have to be defined to a have a future direction? Yes and No.
Let s assume we are in the middle of a established trend/direction. Like today on Euro USD. 80% chances to have Rate UP at USD and new QE for Euro.
This predictions (forecasts) support daily changes. Every day, according with new developments and other things that can have an effect on this OLD forecasts, will have to readjust the "Old" forecast. The Main Line.

This is only the big picture about a thing I know he exist somewhere in some servers.

On small steps, this "future" price line can be made from :
1. Initial manual input for 3 4 categories.
2. Input from the next events that will generate the power in combination with point 1.
3. Major levels from the past. (To be analysed)
0 traders viewing now