Currency Scan

Because I was going to use the index itself as an asset I evenly weighted the index across the pairs incorporated, called it a Balanced Dollar Index. I treat the others the same way, even weighting, so I can build ML models against it and evenly distributed trades across it. Which to be honest is a waste of time because my allocation to FX now could be swallowed up by post apocalyptic bottle caps... It does give me additional insight though to broader trends and cycles in the market.

The small amount of lift we saw in the EUR/USD was enough to pull the bullish signal in the cyclical indicator off the table. Now we have something of a mess on our hands as both the EUR Index and the USD Index have bearish signals in the cyclical indicator. Just short of getting a good entry on a B/O the EU looks like no-mans land right now (at least to me). Outside of a little bit of sell-off in the NZD things are quiet.

Sorting, Syncing and Trimming Data

The next step in our little project is getting our data ready. We're going to drop our next subroutine right below the last one. In this instance - just to refresh - I'm actually putting it into the sheet object where it will execute as opposed to a module. When the data comes out of MT4 it's pretty clean for the most part. You will occasionally find something that printed way off the grid or you might have a couple of missing data points. For this type of modeling I don't sweat either of those things to much. You'll know your short data just by getting to the bottom of the range where and inspecting the final row. If you want to fill those in, you can, but when you're dealing with 100,000 rows, one or two missing points is not going to have any impact on the modeling quality.

Data points that appear to be misprints will be manually corrected if I can find accurate data. Usually I'll manually review a line chart of the data and look for anything that looks odd. It's rare to find anything now, although back in the early 2k's you would see crazy stuff all the time. It's not a bad practice though.

Our sync15m routine includes the following routines

1. sorts the data from ascending to descending
2. trims the data to the sample size I want (the example below is just an example)
3. resorts back to ascending
4. removes unused cells to reduce the size of the worksheet (this is more beneficial when doing a large trim)

EURGBP is something I'm going to inspect a little more. Not ready to pull the trigger on it just yet as we're in a pretty solid bullish trend in the monthly. I'd like to see that stall and I may get in here for a reversal, try to ride out a cyclical move back down for a couple of weeks.

Calling PowerShell from VBA

Today's single line of code was brought to you by PowerShell, where you damn well better not have spaces in your directory if you want to call it from VBA.

So my current process goes like this. Get a text from the Currency Scan app that let's me know one of my triggers has occurred. The text will also include an image of the CS... it works pretty well. I have had some issues with the image not showing up though. Especially when markets are active and it fires off a bunch of times in the course of a day. I think maybe too many of those images may be exceeding the amount of "unlimited data" I can download or something. So I thought as a backup to that, I'd also push it out to a web directory so I can view it there as well. It will also eliminate me having to run it locally as much when I just want to check things. With that I'm calling it a day.

This works...

This not so much... (Even though you can call that file directly from the PS ISE and run it.) Who knew.

While I'm in the processes of integrating the machine learning models into the CS I'm also going to spend some time reviewing and testing newer/alternative research around the processes. Right now I'm looking time-series normalization for supervised machine learning. If any of my fellow quants' know of something worth reviewing let me know. Thanks.

Okay... I've been kind of keeping track of what is resonating here and I'm going to make a few adjustments. When I post a scan up it seems to generate pretty decent follow through, page views go up and subscribers increase. If I start posting the blah, bla, code this, blah... not so much. Page views flatten and I lose people. I want to keep this thread as useful and interesting as possible for the majority of the audience of FF. So no more blah, blah, blah...

If you want to follow along on the deeper stuff, the math, the coding, exchange rate determination, complexity etc. I'm going to launch a blog just for that (nothing there yet). It will be listed in my profile.

It's important to me , even if you don't care to understand the underlying research, you have access to the basis for my work. Maybe it will even spur you on to new directions in your own work.

So going forward I'll keep this pretty much centered on posting scans that don't fit well into other threads and I'll stop boring you to tears with all the underlying stuff. - JP

Hi FXA, congrats on your new thread. personally I find your analyses very interesting ( from other thread, its little different then mine) but if we can shier our thoughts at the beginning of the week to find the big trend ( weekly trend) would be nice( little homework every weekend). I always scan the pairs at the end of the week so if you can do the same thing your way it would be awesome. any thoughts?

I think that's a great idea. I'll do my best to get in here each Sunday night and drop a scan and comment. Pretty soon the other components will be integrated and I'll have a lot more time. Things are moving forward, I'm just taking my time and enjoying the process and trying to learn any new thing I can.

The data-processing template has been completely reworked and is much more efficient than it was previously. It's still too slow overall so I'm going to tweak it further but I'm making some good headway. I can now grab 1.6 million rows of data (24 total currencies), clean it, trim it, process and normalize ten inputs (for 16.3m data-points), break it down into three set's of data (training, validation and out-of-sample) for each pair in about one and half minutes.

Because I don't need to do all of that at the same time - I'd rarely be running the ML stuff on more than two or three pairs at a time - I'm going to break it down into chunks of four or six, which will reduce the bloat and improve processing. The other, more important goal, with this new template was a need to improve my ability to change, add and modify the inputs. This makes extending and updating those much easier. That will make the modeling a lot more interesting and a lot less cumbersome.

I should be building models next week off the new excel-app and then it's a process of of integrating the c++/xll directly into Excel and the Currency Scan.

The initial test run with the new input structure generated a handful of models that produced 54% accuracy across both the long and short-side on all three data-sets (+/-%). That may seem paltry, but 54% accuracy can make you a rock-star over the long haul especially if it's part of a larger process.

54% with 2:1 or more RR and avg pip gain of over 50p will give you over a million dollars of profit over 100 trades starting from 10k, if properly aggressively position sized. So yeah, if you're willing to post that kind of data, I'm willing to take a serious look at it and figure out how to trade it.

Pip I'll have to go back and look at what the average gains are when I get a chance, not sure if they would hit that 50 pip target or not. Accuracy and return are two different paths of course. The model was based off of 15 minute data with a 24 hour outlook, I'm not certain how posting it here on the forum would work out as there is no way for me to automate delivery to VBulletin as far as I know and FF doesn't seem interested in making a place for it (I've asked... got no reply). To put it somewhere else generates all kind of additional server cost for me and the minute I charge I get booted down into the commercial abyss, where I'd rather not be... Not sure what the answer is on that one.

If I get something figured out I'll post up. On a different note I've gotten a little side tracked with a couple of data science competitions, one with a hedge fund and the other with a bank. Given the depth of the field I doubt I'll do very well, the top 1000 slots are typically separated by a 0.00x percentage of improvement, but it's a good opportunity to sharpen my skill set and challenge myself a little. Right now I'm brushing up on bias & variance and trying to bring a little balance to my predictions, which are leaning to deep into one side of the data-set.

I'm also working with Tableau a little, thinking about turning the scan into a more active/dynamic piece of data where one could click on the pair and additional fundamental data would show up... something to think about. First things first, need to get my submission finalized for the comps and get back to finishing the CS models, then get the c++ knocked out so they can be called from vba. Either that or I'm just going to break the formulas out by hand and hard code them into a cell, you can also call c++ directly from a cell so there is that as well... what ever is fastest. I'm all about the speed!

Trading for me has been nill. Which I kind of expected but I do keep things running, if something pops it will let me know. Still waiting for asymmetry. - JP

good luck on the competitions, sounds exciting and fun.

I'm sure you know, 3:1 RR starting at 10p/30p run over 200 trades, at 40% win rate (actually 35%), with 25% risk (very aggressive) would net over a mil on a 10k starting cap or less (carrying up to 80% dd), most stomachs wouldn't handle it all the way and pull out before hand (yours truly included), however, toning down the risk per trade to real world levels and having the numbers you described equals triple digit yearly returns, definitely "turtle territory". 15m data over 24hours is good to catch start of important swings. If it is fairly simple to do, it might be interesting to compare 24h/36h/48h/60h and 72h outlooks as well, using the same 15m data, since major moves are often delayed by a day, two or three.

Lol... welcome to the sickness. You can model this stuff so many different ways, you'll make yourself crazy. What you're suggesting though, isn't unusual and you'll often see specifications for data to be trained to a time frame and then +100 periods and then +200 periods etc. I do think there is a danger in that though. Over the years I've been doing this, I've seen a lot of shifts in the phasing of the cycles currencies go through. If you start looking for the most profitable period to hold a trade based on a specific model you might be data snooping and could find in a few weeks that the phase has changed and your model is no longer any good. When I was managing money I would hold the trades for 24 hours, run the model, if it stayed the same I stayed in the trade, if it changed I reversed the position. That way I could minimize the transaction cost but didn't really have to think too much about trying to optimize myself into the poor house.

To be really honest though, as profitable as it was, it was a hard model to trade. There was a lot of volatility in the returns. This is how some of the analysis you see in the scan came into existence. I wanted to filter those short term signals through something else and it worked. Nothing is perfect though, some of that filtering cost me missed trade opportunities but it came with a lot less drama (and phone calls from clients) so it was worth it to me. I did find that, for me, trading these raw algo models at anything over 4:1 was almost too much for my personal risk profile and in monte-carlo testing they would almost always blow an account up at anything over 10:1. If you start stringing some losses together it can get ugly quick and make recovery in any reasonable time period impossible. Not something you ever want to explain to your clients. Slow and low was always the key to surviving and profiting with these types of models.

Have to agree with you on that one. One needs to show some profitability or at least return to even as reasonably soon as possible after a string of losses, for one's own sanity, or if trading for clients (unless one has incredibly iron patience). There are those who can trade and hold with 5:1 or more and do well but those are rare traders (read: holders). It's a rare person who can go with 10 straight losses and still have the guts to stick with the main plan and load up just as usual for the 11th trade after 10 hits to the gut and losses to equity.

There have been many "tests" done on this, but obviously anything less than 1:1 is negative expectancy and not good unless (you're trading as an exchange member with near zero transaction costs and/or) you are super quick to get out of any negative trade and not carry a negative trade for more than two or three pips.

For most other method traders, swing traders or position traders, 1:2, 1:3 maybe 1:3.5 max, any anything in between those ratios, is reasonable and can be very profitable, given that 50% win rate you talked about that started this subtopic.

Almost every test I've done over the years, show that 1:3.3 or a hair over 1:3 is the ideal zone to be in. It allows profitability at a 33% win rate, which gives you room to find a better edge on the method. A slight increase to 38% or 40% and it becomes super duper profitable. The only problem with this is that if the win rate drops to 25% for a sustained length of time, the method and account is toast. Of course no method is fail safe and/or stays profitable forever. To me, finding a small edge that will work for a few months straight during the year, and then maximizing the gains on that edge is one of the best shots for triple digit returns. With that, a few super profitable months on any method(s), can/will make a good year. Of course I'm talking about trading a personal account here.

Hedge funds run into thinking about liquidity and execution after a certain time, and would be ecstatic with a 40% to 50% annual gain, however as you know, there are hedge funds that will bank 400% plus or more in any given year, and for the most part they are not trading with 80% win rate. I can say this with confidence, years ago one client of a company I worked for was one of the original "turtles", and the previous year his currency fund went from over 100M to over 400M. He was a rock star that year in the industry, and was in the top 10 (top 5?) published returns for the year of all registered funds. I can't say his name, but it's similar to a famous Oscar winning actor. Old timers will be able to figure out the name. Most of the young guns here, don't even know what the "turtles" are other than "Ninja Turtles". Ha. I talked to him a few times and his overall method (at least parts of what he shared) was super simple, and most of it had to do with holding good trades (not scalping).

There used to be a genius here named Ronald Raygun that now runs 10 to 20 types of EAs at once on various servers he put together and put in different parts of the country. He's doesn't write much here anymore, but he does come online and check in and read sometimes. His different EAs are designed to trade in different types of overall market conditions. If he ever checks in, it would be good to get in touch with him, because he is also a data freak (I used that word in the most complementary and highly regarded way).

Anyway, sorry to imply a request for data runs over different TF sets. I don't think that was my intent.

15m data over a 24 hour outlook with 50% win rate over 200 to 300 trades is as I mentioned enough to fund your retirement account indefinitely.

Not even going to lie, I'm pretty excited by this. It means a lot to be able to validate your work amongst the best of the best. Out of 225 data-scientists I'm currently ranked 34th (was as high as 32 this morning). The competition is incredibly tight. The difference between my log loss score and first place is 0.006. Splitting hairs at this point.

There is some pretty good company in the competition as well...


https://medium.com/@Numerai/encrypte...ba8#.ix74majjk
https://numer.ai/home

I doubt as this competition continues to grow I'll be able to continue to rank this high. I think we're going to see more and more scores eventually converge very close to the most efficient models. For me the real measure of success will continue to be how close I can get to the best score. I really feel like if I can keep the difference in the realm of something less than 0.01 I'll be modeling at a very high level.

congrats. hope it continues. fascinating work.

Thank you. It's already having an impact on my model work for myself and the scan. That Log Loss function wasn't something I was aware of but its robust and a good way to measure the error rate. The downside with this stuff is it's time consuming and I've honestly spent too much time on it this week. Need to strike a balance, kind of addicting though when you start chasing scores, but that's not getting any real work done. I'll probably leave comp models to the weekend.

I saw an article yesterday about how the dollar is back in the driver seat. It's definitely up but it's not into a range that has any of my longer term warnings triggering. I'd really hoped to have had another stab at the USDCAD again. I may be glad I got in those earlier trades when I did, things may just keep sliding sideways as we get closer to the summer, I might be on my thumbs until this fall/winter. We'll see how it plays out.

Barely hanging on... I've dropped a bunch of spots but still remain - if only slightly so - above the capital allocation level at the hedge fund. Competition is so tight! The models I'm making as a group are getting progressively better on average but they still trail this single model by a few points. Would love to get my score pushed a little higher, I think there is a potential bias in the dataset that could move things around a bit and I'm guessing I'll get kicked through the uprights (or into the net if you will) when they run the algo's on the balance of the data.

Well I continued to drop in the overall rankings which I figured I would as there were a lot of contestants who were camping on better scores. They have since started dropping those in. This morning I was ranked 72 out of 314 other Data Scientists but was still above the capital allocation line. I made a small change this morning (ensembled 2 of my best models) and improved my logloss score from 0.69047 to 0.69036. That moved me up several spots and will hopefully keep me above the allocation level when the balance of the data-drops.

Today is the last day for submission. Monday I imagine we'll be working on new data.