Jack's Automated System Development Journal

Has it been a week already? Friday UPDATE time!

(Little earlier than usual, but with an average trade duration of 4+ hours and the algo being flat by 2pm, it made no sense to let it take new trades with the weekend coming shortly.)

Notable Changes:

1. I've started two experimental accounts on the side to test a few theories out. One is live, one is demo.
   

2. The new live account takes nearly the same strategy implemented on my production live account but jacks up the size and takes a slightly different approach to managing drawdowns. It also does away with the "Dasher" method mentioned earlier and sticks to the "Runner" method. It's only $50, and I won't report on it here, since the main reason I've put it in play is for experimenting. Any profit/balance will just go back to my manual trading account, and some of the changes might make it into production.
3. The demo account is running a modified version of the production strategy on a new pair. I spent a little time this last week doing some porting to other pairs and testing of variables and think I found a good setup, but it's gotta run for a week or two on demo to make sure there's no bugs.

Notable Stats:

 

1. 394 trades to date. I thought we'd breach 400+ by this time but a few days this week were slower than usual.
2. This week saw some very abnormal price movements from what the strategy is used to trading with on the main pair. It seemed to last for about two days following a news event. I shut the algo off during the news release (as normal) but it really took a beating after I turned it back on. Market seemed to grind slowly in every direction that could harm the algo during this time, kinda frustrating really, but I sat it out.
3. Given the rough trading conditions, drawdown touched a new high of 9.48% this week. This wouldn't show in the drawdown chart since the day it occurred saw some recovery putting the final number a lot lower, but I thought I'd mention it since it was notable. The strategy recovered quickly from this, and given my backtest results I can expect periods of worse drawdown, but still not a pleasant sight. (EDIT: I thought I'd mention again that the algo has shifted up to a higher size during the last week and a bit, so the chart below will look more jagged but the pip return is still well within what's expected.)
   
4. The EA finished the week net positive, and now sits at a total return of 16.41%, or $49.22 in profit on the $300 account. This is only a little over 1% since last friday, but I'll take that over a net loss any week.
5. $782,000 in notional volume traded since inception... getting close to my next milestone of a million pushed.
6. Given the rough trading encountered, profit factor decreased to 1.18 during this period.

http://i.imgur.com/uUqjE.png

I'm going to start rolling the chart window forward in the next few updates so it focuses more on recent changes. When I start doing that, I'll post the link to the MyFXBook page so anyone can review the full chart. (I haven't posted that link yet since there's still a tab that shows which pairs are being traded even though the trade history is private, and I'm asking them if it's possible to hide that as well. Or I might just break down and say which pair it is, since I've already given away enough for people to guess by this point. lol.)


Anyway, I'm hoping that next week my algo won't take it in the pants as much as it did this week.

I've come up with a series of milestones I feel are notable given my Algo's strategy and methods:

 

1. First 10% returned.
2. First 50% returned.
3. First 100% returned.
4. $500k Notional Value Traded.
5. $1M Notional Value Traded.
6. $5M Notional Value Traded.
7. Survived the first month. (Without significant loss.)
8. Survived half a year.
9. Survived 1 year.
10. 100 Trades Executed.
11. 500 Trades Executed.
12. 1000 Trades Executed.
13. 5000 Trades Executed.

Some have already been achieved and I've noted them in the update posts. I just thought I'd drop this down as an example of the less important trading stats I've been keeping tabs on (the important ones are drawdown and profit factor.) Basically, if I can make some of the higher notional value, time survival, and trades executed milestones it will tell me that my strategy is clearly working in the live environment the way I expected it to work.

It's that time again! Friday update!

(I promise I'll post more useful content than just updates of my account soon.. just been busy as heck lately.)

Notable Changes:

1. Took down the demo experimental account I mentioned during the last update, as I found a flaw in the strategy of the new pair I've been testing and I need to work around it. The other experimental account running the initial strategy with adjusted (more aggressive) risk management on the initial pair is still going and performing well. (Note: if you haven't been keeping tack, these are just comments on separate tests I've been doing outside of my production live account. The live account has had no major changes since last update.)

Notable Stats:

1. 478 trades to date.
2. Price movements on the target pair was within what the strategy was designed to handle, with one really good day and a few days that were a bit boring.
3. Drawdown has been kept under check, not making any new highs there and staying well below 3% (at any given time) during the week. I like that.
4. Week started strong and preserved capital after a nice gain on Monday, finishing the week at 27.48% total return to date. That's $82.43 profit atop of the $300 account starting balance.
5. I really thought I'd breach $1MM in notional trading volume this week but the account stopped short of it by just a few thousand. To date: $996,000 in notional value has been turned on this strategy. This high volume still shocks me given the account balance size..
6. Lastly, on this week's good results, the profit factor has increased to 1.26, which I'm pleased about as well.

http://i.imgur.com/NWKbK.png

(I wanted to post a return by % chart instead of a $ balance chart but MyFXBook is generating that chart to look all squished and I can't figure out how to display it better.)

In going over some of the analytics of the trade history, I noticed a rather interesting outcome when sorting winners and losers by the hour of the day:

http://i.imgur.com/VudwX.png

After 400+ trades, I'm starting to think this isn't just a coincidence and will probably spend some time this weekend figuring out what happens at 7am that fakes out my strategy and forces it to take a loss. :|


Also, in the interest of transparency, I'm going to open the MyFXBook page to the public! (Crowd goes "Ooooh", "awweee")

http://www.myfxbook.com/members/FXGe...journal/408319
(I will put this link on the first post as well.)

I was hoping they'd change a setting to allow me to cover up the pair I'm trading, but it doesn't look that that feature will be implemented anytime soon. So with great hesitation, I've made the MyFXBook page public, because I think transparency trumps my paranoia of people thieving my strategy. (Of course, the trade history is still hidden, but everything else is verified and open.)

Cheers everyone!

Very interesting Mr.Jack !
Subscribed.....

Thoughts on exiting a trade that has yet to reach a take profit target:

One common thing I see with most systems is the fixed take profit and stop orders regardless of how the trade plays out. I've always felt that a strategy has to be simple, but still as dynamic as possible when it comes to trade management.

In manual trading, I can create a decision tree for possible outcomes of my trade, and extend that to include handling different market conditions. For instance, if price instantly rockets toward my target, clearly there's momentum there and maybe I should hold out for a little more profit.. at least, leave something on the table to see how far it rides out. Of course, you'll have to justify the use of any trade management technique with hard stats on what would have happened in the past had you traded in such a way (and if it truly is an improvement over your initial fixed stop and TP.)

So when I set out to create the first strategy I've been using in my algo, I wrote down a few key ideas I wanted to test out:

 

1. Will measuring short term volatility be useful in setting stop and TP levels?
2. Is there special conditions (filtered from the main strategy) where going for more pips in my TP is statistically preferable?
3. How much of my edge is lost as time passes from my entry? And can I measure this or create a time curve where I know the point where no edge continues to exist in the trade?
4. Then the standard stuff like, what stop level is optimal over the long run? What take profit has the best mix between expectancy and hit rate?

What surprised me was the 'edge lost over time' question and the relationship I found. When I trailed a stop as time progressed (not as unrealized profit gained, but just time's passing,) performance suffered, but when I lowered my TP as time went on, even if that means into the red, the equity curve and relative drawdown improved quite a bit. (This won't be true for every strategy, but try it yourself on your own strategy to see if it could be an improvement.)

It was tricky at first to get the right mixture of controls for lowering the TP. I ended up settling on two main variables, the time after the entry is made to start lowering the TP, and how steep the TP is lowered as each bar passes. I picked these two since there was some code available that did most of this already in my library of example EAs (so I got lazy here,) but I'd like to extend it later to make the steepness curve with time (maybe less intense at first, then dropping off faster as the trade runs really late.

Now, don't get me wrong, there's always a max stop level set that's within the strategy's acceptable risk tolerance. This addition was more about trying to lose less than the full stop out, but only if it became clear the signal hasn't yielded a decent trade.

Finding the right mixture took a bit of time, and a few nights in a row with my desktop running optimization tests. Staggering out start times from a few minutes after the trade was entered, to many hours, and cross testing that against the steepness of reduction for each time period.

I'll admit though, it's hard to watch the pair bounce about and see that your algo is about to exit a position that probably will be rather profitable if I hold on longer, but I only allow myself to adjust positions by flattening out before news or flattening out at the end of the week... so I just sit there and remind myself that no matter how much I think something is going to happen, the stats of my tests tell me that letting the TP fall is the most profitable in the long run.

This can somewhat be reflected in my distribution of duration vs profit:

http://i.imgur.com/Ur0Zf.png

You can see the hard line where losses get no worse (which is ~2% risk,) but each trade between 0 and 2% in the red were taken out early by the TP being lowered into the red. This is why profitable trades drop off as time goes past a point (with the exception of a few outliers way off to the right, that's just the "Runner" method of trading lighter size but averaging in over time.)

This is why I like strategies that trade frequently, it's been about a month and a half now since I flicked the switch on live for this strategy and I have a decent sample size to analyze like this.. clear patterns emerge that aren't just noise.

Now, this data brings up two new questions:

1. For the cluster of losses that have a duration of around a day and a half, with no chance of becoming profitable, should they be taken out more aggressively than ones that have only been on for less than a day? (I think yes, but need to code the steepness of reduction into a curve as I mentioned above, right now it's a fix amount per bar.)
2. And, is there a chance I could be making more on the trades that reach their TP within the first hour? (That thick green cluster on the left side.)

Anyone ever play with time decay TP/SL orders like this before?

The weeks are just flying by :S

Friday update time again!

Notable Changes:

1. None this week, still testing new variables out in my experimental account on the new pair I wish to add to the production live account.

Notable Stats:

1. 542 trades to date. Which breaches the 500 trades milestone I set out earlier.
2. The week was kinda boring overall, no huge gaining days, no steep loss days.. just price movements that gave a little profit, then took it back, rinse and repeat a few times. Looked like I was going to end the week up ~3% overall but today saw one jerk of a move that had me caught in the wrong direction and taking a stop loss on without enough time before close for the algo to recover (it doesn't increase risk to recover or anything, just the blend between Dash and Runner (the two order handling methods from the first page) that allow a small directional bias to form.)
3. Week ended nearly unchanged: 28.05% return to date, or $84.15 in total profit. That's just $1.72 up overall. Better than a loss I suppose.
4. Another milestone was reached at the start of the week though: Over $1M in notional volume has been churned! At weeks end, the total notional value traded sits at: $1,157,000
5. Profit factor slipped down to 1.23 to date given the this week's time with the algo spinning it's wheels. I'm pleased with anything over 1.20, and we'll see that become harder to keep above over thousands of trades and possible bad weeks ahead.

http://i.imgur.com/NCODi.png

(as per the release last week, you can see the stats for yourself on MyFXBook here, fully verified)

I'm hoping next week will bring better results, but volatility might die down as we approach the holidays in December. I'm also going to have to pick a time to turn the algo off entirely til the start of January.

Still interested in this. See full post for details.

Yay, major news coming up in ~2 hours, plus more 3 hours after that.

Turning the algo off for the evening I guess, just managing the last position to flatten out fully now.

I'm indirectly proving why liquidity drys up so much during news releases.. lol

Was out of town and didn't get a chance to update on Friday as I usually do. So to catch up:

Belated Friday Update Time!

Notable Changes:

1. I'm experimenting with code that takes in a news calender and marks times when it needs to flatten risk (pre-news) automatically. This has jumped high up on my priority list and you'll see why in a bit.
2. Back to the drawing board with the strategy being ported to a new pair, despite surviving backtests, running the strategy on the new pair has all sorts of risky issues I just noticed during forward testing. Going to reconfigure and try again soon. I'm aiming to take on GBP/USD and some yen crosses but I'm finding that most USD quoted pairs tend to be harder to find statistical edges that last more than a few months... probably because so much activity is already on them from quant firms. Sigh.

Notable Stats:

 

1. 664 Trades to date.
2. I made a huge mistake this week and forgot about a major news release (rate decision) that affected my pair and didn't turn it off in time. I set my phone alarm to ring 20 min before the decision, but I was on the subway when the alarm went off and by the time I got home it was already too late. I'm not making excuses here, it was entirely my fault for not planning my day out better and I'm not pleased about it. Sure enough, the EA was heavy in a directional trade when the news release took the pair hard against it by quite a few pips. Luckily, the stop losses kicked in and took me out of the trades before more damage was done, but in the process I gave back quite a bit of profit (~13% of the account value from peek (~31% mid-week) to trough (~18 on Thursday.)) When I trade manually (my normal trading that is, remember that EAs are still entirely new to me in comparison) this kinda loss would be considered a complete disaster (and be much more costly in a $ sense,) so I gotta be more careful with an EA I'm running when not around a computer.
3. All that being said, the damage was limited by the order and risk management I put in place. Nothing beats hard stops, and without them, the loss would have been considerably worse given how far the pair moved post news, possibly even putting the account into the red, so I'm glad I put the effort into risk control earl on in this EA's development.
4. Last week's performance ended negative, finishing at 22.77% in profit to date down from the 28.05% mark at the end of the week before. The EA recovered a bit form the spill, and this week so far has seen additional recovery putting me back in the profit range of the pre-news screw up. (note that current returns are back up near 29%, but this update is just for last week's activity.)
5. $1,427,000 in notional value traded to date.
6. PF slipped to 1.19.

You can see the dip, caused by not minding the news, and recovery, up on the MyFXPage I posted.

In a positive light, if this is how things are when times are bad, it's pretty rosy overall.

i am a fan of bots, i have 2.

there is something that is very good with your bot, it has this upward sloping equity curve, so MM is right and it's not over leveraging. it's not in gambling, instead it is trading.

all the best on your endeavor!

Thanks

That's the idea, to take my manual experience and understanding of risk and apply it to automated trading. I can't say it's been a perfect start, as I've found the process to be a little frustrating as I go (mostly because translating what I want to do into code can take a bit longer for me as I'm not a programmer by trade, and I'm not used to having something trade all the time since my day usually ends when I leave my trading desk,) but it's sure been a fun learning experience so far.

This reminded me, I need to update this thread.

I think I'll switch to monthly updates from weekly, so as not to bore everyone when the EA is just spinning it's wheels waiting for the market to give it favourable conditions again.

Cheers!

Notable changes:

1. Being the holidays and all, I've taken some time off work and greatly cut down on my manual trading at home. The idea was to pick a time to turn off the EA entirely, and I had previously thought the 20th of Dec was a good target, but soon did I find myself planning out ways to keep it running just to see what happens.
2. So I started by turning off the 'Dash' (link is to the post where I describe Dash and Runner methods oftrade management method entirely, since it is far more sensitive to price not acting as expected, and kept 'Runner' going on its own. I decided not to move the buying power from Dash to Runner, and instead just let Runner keep up at the same settings it was before. Runner, if you don't remember, was my nick name for the method of taking a smaller nibble on my trade signal at first, then if the trade is still on later and the signal is still valid it would add in some more size. It also has a much wider stop loss which I felt would come in handy should the markets act..well..odd.
3. And odd they did. During the week or so that past from the 20th to now all the EA did was spin it's wheels and risk more than was necessary for pitiful gains. I should have just turned it off entirely as I first planned. I will detail the cruddy trading environment the holidays bring later in this update.
4. This month is not only quarter end, but year end, and it will mark the change to monthly updates from weekly on the EA stats directly. Instead, I'll focus more on useful content and try to write an entry weekly for that instead of boring performance stats.

Notable Stats:

1. 789 trades to date. (Which is also the final number of trades for 2012 and Q4.)
2. The last little while (about two weeks) ended up positive by a few percentage points, but it carried risk the entire way and really wasn't worth the little gain made. (as mentioned above. Period ended with a total gain to date of 34.26%, ending 4.85% up in December.
3. $1,780,000 total notional value traded (in AUD to be exact, even though the account is in USD, the pair is AUD based.)
4. 532.1 pips won, a decrease from the last major period I reported. This is mainly due to flattening out before a few news releases in the last month, as it wasn't an opportune time to close out the open positions entirely I just flatted out the exposure instead.. since both long and short signals are running independently, the temporary hedge is often small compared to the positions put on by the EA (just enough to balance it out) and that makes the 'pips' earned kinda out of whack when compared to profit during the period. (Clearly this isn't the best way of handling news, but I'm not sure how to code in a state that lets me flatten the EA then pick back up with the position and directional biases it had pre-news once the market has settled down..)
5. Profit factor slipped again to 1.17, mostly because the algo turned volume without gaining much (thus my complains about risking for little gain.) Not pleased about this either. (I'm not entirely cynical, I'm still happy there wasn't a loss to be posted. )

2012's final equity curve looks like this:

http://i.imgur.com/Y4ddK.png

You'll notice the lower volume in the last 6 days, that's from dropping down to just Runner and not letting Runner have Dash's buying power. Good thing too, it would have been profitable but the drawdowns would have been painful.


Let's pull up some of the more interesting stats:

Long signals got their asses handed to them the last month.
http://i.imgur.com/Mespj.png
Most of the profit the short side of things made was gobbled up by long losses. In fact, the total pips and $ made from long trades has gone negative.

During the EA's downtime this next few weeks, I'll be investigating what went wrong and trying to figure out if there is something I can do to help filter out these losing trades.

You can also see how quickly the EA gains profit when both long and short trades are doing well, in late Oct and early Nov the EA was flying. I'll be reviewing that time too to see if there's anything that can be identified to mark the 'in tune' period of the market and maybe have the EA up the risk slightly to benefit from it in the future.


Lastly, let's look at what the holidays bring us:

http://i.imgur.com/wz1Jr.png

(Now, I never thought I'd pull up a chart and give even a hint about the strategy itself, but consider this a candid holiday treat/tease.)

Here we see a 30m chart of the target pair (AUD/NZD) with period separators turned on so you can see each day sectioned out.

The first few days it's acting 'normally', at least, somewhat behaving how I'd expect and how my EA is designed to work within. Gentile coasting and giving the ability for the EA to get a bit of a directional bias as the trend looks solid...

Then the last few days it's a bloody mess. No consistent trend, and fast acting jerk moves before the EA can figure it's in a range and remove any directional bias. And the best part is, there's no major news during the last few days, as from what I can tell this was just the result of liquidity drying up in NZD and some punter having their way with the pair batting it around (just looking over NZD/USD and some crosses makes it seem that way.)

Anyhoo... the year is over for this EA. Looking to start it back up around the 7th of Jan in the new year, we shall see.

Look forward to more in this thread in the mean time.

Guys,

Quick hint / note about MT4's strategy tester.

If you're like me, and make your EA very verbose in the comments it makes to the log, and also make an abusive use of the 'optimize' feature to generate all sorts of statistical data points to measure when setting variables for the EA... then you might be surprised to find the log files get rather large rather quickly.

Take a look at the logs that holds logs on systems testing within MT4's install directory. Not the broker connection and trading log files, but the ones found here:

MetaTrader Directory > tester > logs

My folder was 63Gigs in size by the time I noticed. And that's just a month's worth of testing gone by of me running backtests on tick data, or optimization tests overnight.

Crazy.

:S

Even now, about a week after I last cleared the logs out, I'm sitting on around 3Gigs of text files. I'm going to go through them and see if it's a problem with the way I write EAs.. like if it's writing something to the logs every tick or something... :S

Anyone else notice this?

Ok, I figured this out. If you botch a sendOrder or modifyOrder call within the EA, and your EA is set to keep retrying a failed order til it goes through, then the combination has some of the EA's I've put together or tested trying to resend an order nearly every single tick... considering I test with tick data, that could be a hundred times each minute worth of history is backtested... so that explains the huge log files alright. LOL

Think I'll add a sleep period after a failed order to my list of things to work on..

4kb. you dun goofed.

...consequences will never be the same. :S

lol. Really though, it was some bad code trying to send an order every single tick update. Consider this a 'learned the hard way' warning to other n00b EA writers out there.

wow, the balance and equity remains very tight together, and still sloping upwards.
shows a lot of discipline!

i do see some opportunity. the ea doesn't maximize profit taking. but it may just be the key to maintaining the slope that it has today. great job still and keep up the good work!

im just an end user so i have nothing to contribute on the EA development but the discipline is what keeps me coming back.

http://i.imgur.com/hKga6.jpg

Thanks!

The general idea was to take my manual trading experience and translate that into my EA's method of managing orders, exposure, and trade signals. When I manually trade, I consider all floating equity as the account's balance, be the gains/losses realized or still in open orders. And with that, holding onto a loser without reason (as if I was hoping it comes back, or just avoiding a loss) is the exact behaviour I spent a long time working to crush out of my trading habits... so writing an EA that does it would be against my goals.

There will be floating losses at times, which shouldn't be anything huge, since the EA does is grab an average price while the my trade is still valid (that's the key right there, the trade still hast to be valid, statistically proven by my testing to still be within acceptable risk and price boundaries from the signal.) So any basket of orders is to just get an average entry price, not average a loser.

If a trade goes hard against it, the EA starts cutting losses before serious damage is done. You can see and example of this in early December, as the EA had to deal with a snap movement against a position it was already in thanks to a news release which I failed to manage correctly (I usually flatten out exposure before major news since the EA isn't designed to handle news just yet on its own, but this instance I just didn't keep track of it properly.)

There will be realized losses in the future... My backtests have proved that periods of 2-3 months in a row can be negative, so I'm expecting not-so-great months to take place in the future. For now though, things are going generally smooth.

After all, this is just a fun experiment for me on the side (seriously, the stuff I do for fun outside of work is often trading or market research related, I'm gonna burn out one day if I keep my interests so focused, lol.) But if things keep going this well for the EA, and my ability to code keeps improving as I practice and adjust what I've got so far, I might start doing this more seriously and put a decent chunk of change on the line.

Anyhoo.. thanks again for the kind words.

no liquidation of SL... volume trading... (puke. nose bleeding. having fever right now). i hear mfoste1 resounding in you. but he goes further by adding to winning trades!

mfoste1 did say.. there will be a hard stop..

nevertheless, both of you are treating fx not just trading but as an investment. like a reasonable person should

by the looks of it, you already understood how to "get an average entry price, not average a loser." i PM'd you the link of a great trader here in ff and how he adds to winning trades and when to cut it lose. there is a hyperlink of the logs of his trades to better understand the behaviour. i followed your experience in downloading historical data from page 1 (thank you for extensively documenting it) to do just that.

you never asked for any help (and cant offer you technical help on your ea development). but perhaps, allow you to allow yourself better profit taking targets. obviously you already have the right mindset to be able to execute it. g/l and em watching you from a microscope! lol.. no, em not suggesting you change anything. but maybe, just maybe, you can better the tp's.

as f_xcm studies says, more than 75% of the time, retailers are on the right position if the SLs were not too tight and winners allowed to run as far as they could. your system has ample SL, the tp can evolve, imho.

overall you are doing great!

Well, to properly respond to you I'd have to talk more about the strategy and reveal too many details about it, but there is stops in place and they get hit quite often. Look back on the 2nd page (I think) where I talk about the two separate ways of managing signals once a position is taken (Runner and Dash were my nick names.)

Trust me, if I explained why it's doing that it's doing you'd understand it isn't as simple as straight low return to high risk with high hit rates. I want to detail it, but then my strategy would be greatly exposed, and I'm not about to do that anytime soon.

About letting winners run, recently I've been playing with the idea of closing 80% of a trade should the trade entry fall within a specific zone of major support or resistance and letting that last bit ride as long as possible. This, however, wouldn't be related to the main strategy so much as S/R trading.. plus I have no idea how I can code an EA to pass off an order ticket to another sub-process running a different strategy, while the main process resumes looking for new signals in the main strategy. So this idea might take a while to code out.

That being said, I saw the link you sent me and I've seen a few styles like that before (anti-grid or snowball..) I haven't used anything like it in my own work, but I was thinking of experimenting with longer term trend trading on major pairs with legging into the trade as it moves in my direction (only, instead of on a set grid--hate grids and stay away from them in general--it would be legging in on pullbacks.) I have yet to code up anything for this but it's on my list of strategies to explore since I have a lot of manual trading experience doing the same.