Jack's Automated System Development Journal

OH! another important thing to keep in mind that the free patch only works with builds up to 409..

I had an old installer handy for myself but after googling around you can find a few options to download it:

http://www.mediafire.com/?sxyes74vvoib3u9

*I take no responsibility if this screws up your computer, I did not package this file, you use it at your own risk*

It is an IBFX-aus build. You will need to add in the srv entries for your desired broker by hand so you can get your broker's spreads.. or just use the spread tool (another script that can be found online by googling around) to set them by hand to whatever you desire.

Note, you don't need to use build 409 for converting the CSV files, just for backtesting with tick data. Probably best to do as I do and create multiple MT4 installs for various purposes or stages in your process. (I install them all to a C:/MT folder to keep tabs on them all.)

Lol. Many thanks Jack. I was about to ask where you got your 409 but you were faster anticipating my question!
One question though. Are both your 409 and the one you use for convertiong the CSV files from IBFX? I tried with a 406 build but it didnt accept the files I used from another broker.

See above

You gotta play with the server files (.srv files, you'll see them, just add in the ones from your other installs) in the config directory to point to your desired broker's server.

So if you generated the fxt files on your regular MT4 install (like the one you use to trade with your broker) and you wish to retain the same conditions and spreads for testing on 406, you should copy over the server entries within the file so you can access a demo account on the 406 build.

Of course, once that's all setup, you don't need to be logged in to do testing. When I launch my testing MT4 instance, I remove all charts (so it doesn't have a reason to pull a new history file) and let it log in for just a second or two to grab recent spreads and trading conditions... then I delete the account from the navigator window which effectively logs out, and test away without any issue.

If the demo account you deleted before was the only registered account at the time, then the next time you start MT4, the demo account will still be there (despite the fact that you deleted it, MT4 is wonderful like that) so you can do the login, acquire spreads and trading conditions, delete account to log off trick again.


Wow... you know, when I think about it.. MT4 is a real pile of BS with all the work we gotta do just to shoehorn it into doing what we want. lol

Thanks for the info. I really appreciate it. I'll try it out right away... it was really bothering me that I didnt get it to work with the patch script and 10$ a month plus 100$ for the tick data suite seemed a bit much just to get metatrader to work.

I guess thats what we get for wanting free stuff, for the purpose of meaningful research and not only screwing around with some indi's and charts... Its possible to get it to work but often its a real handful of work to do so. Guess thats the tradeoff...

But its amazing what we can get nowadays for free. High quality tick data, software that at least works with some tweaking... some years ago that wasnt possible.

Anyways thanks again!

Ok, noticed last week that Sunday's open was a bit shaky and didn't treat the EA well.. did some research and testing over the weekend and found most Sundays between 5pm and 9pm EST/EDT just box around the strategy and don't really contribute to overall returns.

So I'll be flicking the switch back on around 9pm tonight instead of now.

It also struck me that this is a method of filtering, and I didn't filter much during my testing.. heck, the only filter I put in was a spread filter to prevent it from trading should the spread raise above a point (which I figured would help the EA avoid taking trades during wacky news releases and such.)

So... a simple 'discrete' filter for Sundays pre-9pm should do the trick, I'll start adding it in now.

EDIT: here's an article from MetaQuotes that covers some basic filters:
http://articles.mql4.com/914

I stayed away from them at first since people get in the trap of overusing them and killing any edge a system might have... or trying to use them to filer out enough bad trades that an edge is formed... usually both actions are fruitless. But in this case, I can sparingly use them and not impact the strategy too much.



EDIT2:

Also, with hurricane Sandy blowing through NYC tomorrow, the mayor of NYC has put Zone A on evac notice. That means most of the big firms on Wallstreet will be running with just a hand full of people instead of their full capacity. I'm really curious about how this will affect the NewYork session and US Equities tomorrow.

*thinks*

If anyone is interested, I'm currently taking an online (free) university level course through Coursera.org on computational investing.

It's by the Georgia Institute of Technology.

I may find out I'm already past the level of knowledge the course conveys, but in reading the course overview it appears to be a good fundamental base for people who want to analyse trading performance and portfolio theory. It's not directly related to forex or EA programming, being mostly equities based, but they will be using Python to build out market simulations and to test modeling and . They'll also be covering some analysis and stats used to compare performance, so theory and tool wise it's still applicable.

It's already in week 2, but I think registration might be still open:

https://class.coursera.org/compinvesting1-2012-001

Again, it's free and by a decent US university... no pressure

If you haven't already heard, US Equities are closed today since NYC is battling Hurricane Sandy.

My better judgement tells me to shut down my algo.. in case the market gets out of whack without equities driving sentiment.. at least for the NYC session.

Mm...

Other part of me wants to let it run and see how she handles the extra chop.

So far it's up for the day. Guess time will tell.

-----

Oh, and MyFXBook told me they will correct the way the tracking page is accounting for the balance corrections from before. So I'll be able to post growth (%) charts. Woo! Just waiting on them to make the change.

Jack, enjoying your posts so far, look forward to being able to contribute.

Made it through the fist day of US equities being shut down due to hurricane Sandy. Today is day two, and I hope it will be more of the same.

http://i.imgur.com/OOiRq.png

That's better... without unexpected news announcements and bad ticks the algo chugs along smoothly.

---

On another note. With all the talk about tick data lately I went back to my backtesting MT4 install and started playing around. I ended up corrupting the FXT files somehow (now only showing 70% modeling quality with errors) so I think I tried to run a test without running a required script first... sigh..

I started to do the conversion and prep of tick data again and am having problems with the old version of metatrader not wanting to pull a broker's trading conditions or complete the login process.

So I grabbed a trial of the tick data suite to hold me over as I try to fix the free method of tick data testing. I might just end up buying the suite to save myself the headache if this is going to happen often.

I can tell you how I managed to get it to work:

I used your mt409, so I also downloaded the newest build from the SAME broker...
I made an account with the newer version.

then I converted all tickdata using the newer version, let it copy to their respective folders.
I closed the new metatrader, opened it again. Closed it.
afterwards I copied the fxt and hst files over to the old installation.
I also copied over the config folder(overwrited everything).
I ran the old version, started the script, and it worked.
However, when doing backtests it only seems to work with the most accurate test method.

Since I copied over the whole config folder, the 409 and newer version probably do not need to be from the same broker though. Didn't test it since its not important for my purposes.

Hope that helps.

Edit: copying the config folder from the new version may also get all current settings... which was, what you were looking for?
Meaning... start new mt4, let it pull the info. Close it. Copy config folder over to old mt4. Start old mt4. Check if it worked.
Didn't test it though.

Thanks, I'll play around some more.. looks like I'll be putting out a full guide even if I didn't intend to just because I'll be documenting the process this time around in case I mess up the data files again.

For now, the trial of the tick data suite is working rather well. I might renew the trail (use a different email to register) to hold me over while I sort stuff out.

Ok, time for another performance update. I think I'll keep these weekly.

The last update covered performance from the initial start date to Oct 26th.

This update will cover the period from Oct 29th to Nov 2nd, this past trading week.

Notable changes:

 

1. I added a time filter to cut out the first few hours of Sunday night, and to stop taking on new positions in the last few hours of Friday evening. The code and logic checks out, same with backtests, but I gotta wait til Sunday to make sure it follows through live.

Notable Stats:

 

1. This last week saw a 1.37 lots of volume turned on the account, bringing the total notional value trade on this algo up to $370,000. This is a slight reduction of average volume per day, but it also coincided with a less volatile time in the target pair.
2. The algo is currently pushing a profit factor of 1.10, this isn't bad, but I'd like it to be higher.
3. Total profit for the week was $11.47, or up ~3.82% from the previous update balance of $300.51
4. Profit since inception is $11.98
5. Account high was ~$314, which represents a really shallow loss from top as it stands (equity high, not balance high since there was a realization of one big gain that had an offsetting position from an opposite signal bringing the equity down net.)

Quick balance graph:

http://i.imgur.com/AyUUf.png

No cash adjustments, corrections, or show news events causing outliers in the return results (positive or negative) to note this period.

So far so good. My time is going into adding additional strategies from this point, so more code blocks and findings to come.

I am happy to report that most of the week the algo ran unchecked as I was traveling on a mini-getaway to greener places. Been hiking the last two days. Fun stuff. Will get back to focus on this and other bits come Monday.

Cheers all!

Hi Jack,

Cool thread!

A few months back I wanted an EA to trade SD zones that I mark on the chart, so I paid a programmer to make this. This fine gentleman (PM me for his name) made 8 versions of the algo as we tweaked what it could do.

Surprisingly through these tweaks, I've found a mind-numbingly simple automated entry and semi-discretionary exit method.

The reason I wanted to post here is because one of the more profitable concepts I've been using (and it was a mistake to discover this) is closing overnight trades about 15m ahead of NYO.

Throughout the day I loosely manage the trades according to what the current sentiment is and let the EA take automated trades and I'll target either nearby stops/barriers/ADR.

This has led to my best all time performance!

Over about 150 trades over the last 6 weeks I've noticed my average trade duration is 7 hours. My hit rate has been slowly climbing and is currently around 44%. My average win is about 2.5 my average loss but this has been dropping as my hit rate increases.

Not to go off on tangents, I just wanted to share the idea of automated entry and then semi-discretionary exits that are handled differently depending on the time of day and sentiment.

Thanks!

Well, speaking of semi-discretionary exits.. there's news coming up soon that might drastically affect the algo. I'd normally let it trade through the news release, but this is one that has people expecting a rate cut while there's still a good chance no cut will take place. So no matter what the outcome, there's a high likelihood of volatile movements.

Something in my gut is telling me to hit the 'pause' button while this plays out, since the strategy isn't really created to trade news.

I was going to wait for another half hour or so, but I had a pretty clean moment to pause it now so I took it (nothing in play, just closed out a few winners.. no need to get caught up in a trade that might last more than an hour and into news times.)

So yeah, I'm holding off on all my manual trading for related pairs as well til post news release. I bet I'll just feel silly if the impact is minimal, but gotta follow what feels right.


Week started strong otherwise (I know it's only Monday, this is just an update preview):

http://i.imgur.com/aVmnz.png

(yes, that's a growth by % chart instead of a balance chart, MyFXBook finally fixed the account balance update correction I asked for..)

PS. Will be linking directly to the myfxbook page soon and making it public..

I tend not to focus on hit rate alone, and rather look at values like profit factor or expectancy.

Your hit rate when taking the average win vs loss into account seems really good BTW.

Right now, my strategy has a hit rate of 78%, but it wins less than what it loses and only has a profit factor of (as of this writing) 1.16.. I expect these numbers to change a bit as more trades are added to the sample and won't consider them anything to trust and depend on til I get a few thousand trades accounted for... right now the live account only has 237 trades (it dos trade often) with an average trade length of 4 hours and 56 minutes.

Welp, my gut was right. Most AUD pairs jumped ~40-60 pips within a second. During the rate announcement (and they held rates instead of going with the analyst forecast of lowering.)

While that wouldn't have been a disaster, it would have potentially taken a loss or two had my algo been online.

So it was a good call.

Taking the Algo offline for major news has been a lifesaver. Doing it again today. Even though price might move for me if the algo is in a directional trade, the risk isn't worth it, and in reviewing the fist week's performance where a news release took out a few stops when compared to this last week of avoiding news.. the difference is clearly visible in the account's equity curve.

Account hit a milestone today that I'll detail at the week's end in the next update.

Til then:

--

I stumbled upon this wikipedia article that I found interesting:

http://en.wikipedia.org/wiki/Taleb_distribution

I've always liked Taleb's work but never realized they used his namesake to describe fundamental flaws in trading strategy expectancy.

--

I'm now looking into a way to flatten things before news releases automatically. . . so far I've been just turning off EAs from MT4 then manually closing out trades when it appears to be a good spot (in profit or not.)

I imagine this will involve using a library that pulls a news feed from a reliable source, but "reliable" is the key word. Anyone have experience in doing this within an EA?

I'm also thinking I might need to include some logic that tells the EA to start up again when price movements and trading conditions settle out again.

The trouble I foresee here though is not being able to backtest test it properly.. so likely this will be something I run on demo for a few weeks to work out any bugs and ensure it's doing the job right while I manually manage the live account.

--

Progress toward additional strategies to add to the account has been going slower than I'd like. My workstation just finished running an optimization set that took 55 hours to complete (1280 total combinations in this test run) and it turned out pretty fruitless. I need a faster computer.. lol.

I've come up with a few ideas that work well on this year's history data, but fail hard on previous years, so they don't meet my requirement of working for at least the past 3 years.

I know I could always apply the first strategy to as many other pairs as possible and see if it would work with adjustments made to the system, but I was hoping to diversify first into a completely different method/strategy before porting what I have in production.

--

I've been looking at some commercial systems and seeing what makes them tick... I don't want to copy them and I never want to buy one.. but they've been very interesting in terms of brainstorming new ideas.



That's it for now, just a few thoughts rattling around in my mind.

Friday Update Time!

Notable Changes:

 

1. Nothing to the main live account as of yet. Still manually turning the EA off during news releases. Haven't gotten a chance to look into automating this past the thoughts I had on the subject from a post ago.

Notable Stats:

 

1. 313 trades to date.
2. $578,000 in total notional trade volume to date. Part of the pickup here is the account scaling up automatically as it got a little profit to work with. Nothing huge, all programmed into the risk and money management components.
3. -$2.13 paid in swap interest to date. I foresee this as a potential drain on the account, ~0.6% of my account value over nearly a month soaked up by interest since the strategy often holds past NYC close. I'm going to spend some time on this later, maybe there's something I can do to lower this.
4. The milestone news was passing the 10% return mark earlier in the week. Current net profits on the account stand at $45.44, or 15.15%. Woo!
5. Profit factor improved to 1.24 during this last week as well, trading conditions seem to have favoured my strategy. I can't count on this to continue forever and I expect to have weeks where the market isn't so agreeable with the EA. Just nice when it works well like this.

http://i.imgur.com/nWLbk.png
You can see what I mentioned earlier, how things have worked out well since I actively started turning things off during news.

Since it's nearly been a month, I wanted to check out some other analytics on the account, like how the long trades vs the short trades are matching up and such, so I pulled up the directional analysis on the pair my strategy is running on:

http://i.imgur.com/LfbCu.png

Green line is profit on longs, red is for shorts.

The nice part about this is how when the strategy suffers on one side, it does ok on the other to smooth things out (not perfectly, but nice.) And when both directions work well, things progress really nicely.

I know it might seem silly to be pleased with such a small dollar amount of returns, but again I'm new to programming and this is just an experiment in writing my own, so I'm pleased with the % return so far and how things have been going. I'll be adding more capital to this account should things keep up, maybe a few grand transferred over from one of my manual accounts..or heck, if I keep getting this rate of return (unlikely, but one can dream ) I may just leave it be and let it compound on itself since a low deposit to high balance always makes for a better story.

That's it for now. Friday night and I gotta hit the town with friends.

Oh, one last quick thing, I pulled up a drawdown chart by days from MyFXBook, and thought it was rather funny:

http://i.imgur.com/HIaKb.png
(I believe this includes floating losses as well as realized.)

You can see the improvement in the last two weeks from flattening things out and turning the EA off during news.. earlier such news events were stopping one order management method right out hard and causing the other to hold a loser longer than it should. Recovery from these drawdowns were quick, but all I can think about is how much higher the return would have been to date if I had been turning it off during news from the start of going live. Not making excuses or anything, just realizing that my expectancy in my backtests had me thinking it was ok to leave it on, when reality was the news events should have been filtered and I would have seen how much smoother things ran without them affecting trades.

Also, kinda looks like the market is giving me the finger on this chart, no? lol..