The Big Kahuna: Full FX dashboard (MT5 only)

>>>> UPDATE - sorry this one's off to the recycle bin. It's become way too valuable to dole out here. But I'll leave the posts so you can get the idea and code similar >>>

This is badass.

It's a fully-featured panel for the fastest read across the currency universe I could imagine and robust predictive volatility modelling. Based on this thread here (https://www.forexfactory.com/thread/post/15461331#post15461331) and an original panel by MoodyBot.

Change log:

03/03: unpublished changes

2_9_4: Regime detection a/c to curves
This was the first run of wiring everything together, and it's good but needs work. The regime detection was very basic just to prove principle, but it surfaced the stats of the curves and revealed hidden structures that we can improve on. The good news is curves work very well across majors. Across crosses - the basic curves work but event based ones are useless. Not degrading quality, but just useless. It's enough to work on though. So now, we freeze the curves as is and move on.

2_9_3: Consumption curves 1/3
Curve families created for news categories. Daily templates for each pair and historical created with construction curves shown on stats page, hourlies on majors. Uses kernel on event time (macro) or session (CB events) normalised to produce hourly weightings. Written to CSVs for later use.

2_9_2_2: Events 1/3 d/l, categorised and uplifts created per pair - unpublished
Uplifts created per pair, wired into stats. Couple of % performance increase but large jump in robustness due to dispensing with global uplift variables. Stats surfaced on global page, majors and its own news stats page.

2_9_1: UI changes 21/2 currently using dummy data - unpublished
Global page:
- click to chart across all pairs loading default template (BK_HAR.tpl). If not installed will use default template.
- YZ/HAR colums replaced with one column showing values in use, now defaulting to HAR not Auto
- Addition of uplift column showing future impact of events on HAR
Majors page:
- Uplift figure and click to chart added to all panels with heatmap colours.
- New mode toggle added cycles between News - the current default, Cats (categories) and Hourlies. Cats disuplays the category and label for today/tomorrow and construction method for consumption curves. Hourlies displays hour by hour forecast of vol
Stats page:
- Two new modes cycles between HAR stats, News stats and Curves
- News stats shows summary of all downloads, detail on today and previous 20 days categorisation of day and construction method.
- Curves shows curve templates of all pairs across session and categories of events
Fixes:
Click FX in Majors was doing nothing, not sure when this happened. Fixed.
Click to collapse removed from Stats.

2_8 19/2 Timezone shift (unpublished)
Shifted start time from London midnight (the easiest and most popular) to London fix at 4pm. Option implied vols update at 4pm, regimes literally reclibrate at this time. Negligible performance increases (around 1.5%), but 40% drop in RMSE (root mean squared error - ie the big oversized days are not as bad) The downside - statistically less robust DM test against YZ & ATR. However, end result is telling - on balance now the HAR is about 30% better on majors than YZ and ATR, 20% on crosses, 22% overall. This is pretty much bang on what I'd hoped for.

2_7 11/2 Base improvements (unpublished)
- Refactored unified cache
- 3-step init screen (data/calcs/continue) now with default two year history, retry and sensible single broker requests for data not already cached in user history. Let it sit on first run, else you'll swamp broker with unnecessary data requests. If any fail, retry. Once done, click calculate, then skip or continue. Skip skips failures.
- Detailed stats screen, with detailed stats per pair, full graphs (click pair/forecast) of last 20 days historical performance and DM test to show superiority against YZ and ATR models
- Refactored HAR uisng seasonality and fitting to convert RV to range now calculating against every pair individually as opposed to deriving from crosses (historically). Improves baseline performance and crosses.

2_6 cancelled - refactor of the HAR went in disastrously wrong direction

12/11: Update: 2_5
- Massive data restructure and better predictive performance. Also merged thread so appreciate feedback but it's gone

FEATURES LIST - 2_4
Features (initialisation):
- Detailed pre-initialisation screen with data-fetching, symbol validation and pre-calculations of vol models and jump risk
Features (global view):
- Volatility prediction using HAR-RV model with 6 months lookback @5m resolution, walk-forward optimisation with 3 months out of sample. Crosses derived from majors for HAR. Not for YZ in case there's a problem with one of majors history.
- Incorporated back history of HAR-RV, Yang Zhang and ATR accuracy rates (past 20 valid trading days)
- Auto mode that switches between most accurate based on rolling MAE comparison over 20 days.
- Dynamic efficiency ratios of cost of trade made using spread:forecast
- Strength meters using normalised Bloomberg DXY ratings across commodities, equities, bonds and currencies with session lookback. In layman's terms, these are accurate. Most are not.
- Jump risk stats of 10 biggest moves (high-low) over 1 minute over past 30 days across majors with chart click-through for each
- DST aware visual sessions and world clock
- Pip calculator with risk/SL updated dynamically
- News now and next. Flags for news exactly +/- 1440 minutes with user config buffer. Additional optional FX events of cut/roll/fix.
Features (majors view):
- Majors view with visual representation of range prediction vs realised using Auto, HAR or YZ
- News aggregated by pair with now/next and optional fix/roll/cut and timer to next high impact
- Volatility switch between Auto and HAR/YZ

ROADMAP updated 3/3
2_91 UI changes. As dash is used as a debug surface too, I'll prioritise UI changes for events, curves and hourlies using dummy data
2_92 Event download, processing, categorisation and display. Creation, application and display of HAR uplifts. Check performance stats.
2_93 Consumption curves, template creations, mapping to events, creation of curves today and 20 day historical, and display
2_94 Curve stats and regime detection module to validate both before using in production.
2_95 Regime detection 2.0. Improve granularity of regimes. Surface stats. Live modify forecasts based on regime change.
2_96 Baseline dash changes: switch on caching, test, write outputs to CSVs, test indicator can pick up. Final UI changes likely to majors page to make live stats more accessible
3_0 Full release
+
Indicator 1_0 Creation of indicator to display hourlies, speedometer, regime detection, plus jump to tail risk events

GEEK NOTES 23/2
HAR-RV outperforms Garch in FX, and if well calibrated, even outperforming ATM IVs too (except in high vol events). All majors and crosses are now calculated directly off the data with no derived calculations (from version 2_7 onwards). HAR contains jump parameters, seasonality, walk-forward optimisation with 6 months out of sample.
Auto mode in the panel to switch between modes was there during construction as HAR was behaving erratically. Now it's stable, advise to keep it to HAR at all times.
Model from 2_7 onwards works London fix - London fix. This is the optimal time for vol models given it aligns with real money movements (discovered, not widely used by academics).
- Currency strength use BBDXY %s, normalised across 7 currencies. They'll need to be updated annually ofc, but better than DXY weights.

USE:
1) You'll need a live account with a broker that provides data for the equities, bonds, and commods data to work. It'll work with a demo account but some symbols won't be available. However even MT5 server demo accounts have the majors, crosses, gold and silver and delayed data on equities.
2) You need to subscribe to all the symbols below. If you've a broker where they've renamed them all you'll have to enter a gazillion pair names. No way around this. The symbols are changeable in the settings if your broker has different.
3) You can adjust the local time via settings but I've never tried it outside of UK. Hopefully works
4) Timezones and DST. Spent an eternity on this; it should keep up with clock changes at all times, all locations.
5) Set your chart to black, save as template.
6) The init screen will tell you the state of your MT5 data. Choose to reload or skip if you're in a hurry. Big calcs cache so it'll be fast after the first run.
SYMBOLS:
All majors + crosses
Commodities:
Gold: XAUUSD
Silver: XAGUSD
Brent: BNO
WTI : USO
Nat gas: UNG
Equities
S&P: SP 500m
FTSE: FTSE100
Euro: EUSTX50
Nikkei: JPN225
Hang Seng: HK50.
Bonds and credit
1-3yr: SHY
7-10yr: IEF
20+yr: TLT
US H/yld: HYG
US invest: LQD

NB Stats page screenshot from unpublished version

V2_5 notes
OK, so getting the robust model working properly as a baseline is *the* most important thing, else it's a pig with lipstick. Everything else spins off from here, which is why I pulled the first version. Not having any old shite floating around, especially given there's NOT ONE single HAR-RV model out there in retail in FX. Not one! Shows the massive gap between retail and industry who all use these models (usually but not always, blended with ATM IVs).

So, there were a number of problems; the calibration was failing silently and falling back on a hard-coded magic figure. Bad enough. Even if it had worked, the walk forward sample was in sample. And the crosses weren't being derived correctly due to triangulation errors (violation of Cauchy-Schwarz inequality if you're interested). Caching issues. Turns out errors cluster like volatility. Robust debugging and logging now available if you need to get under the hood.

Layman's terms, a shit show.

Good news is its fixed, and it's performing bloody well. And exposes the obvious that you need to see proper stats, up front to be sure of the reliability.

Version 2_5 attached to main. If you've 2_4, burn it please and never mention it again.

Stats (20 day median error) is 21.3% for HAR vs 27% for an ATR across majors. Crosses, fairly useless 25% vs 25.9%. Short version - it's performing 25% better across majors than a dumb ATR. I should bloody well hope so.

Mate, I don’t even know why you’re on FF.. greater than 99% are sitting thinking…wtf??

I’m currently working on my own dashboard using Claude and Chat GPT…. It’s like asking the wife to go grab a monkey wrench from the garage… you’ll get something but it’s a lottery as to whether she knows what it is, where it is, has she chucked it out, changed it or refuses to get it and says too much data this week.... she’s walking over…

It's wizardry! I know that both your missus and mine may be distantly related, and that this scale of dev looks ridiculously complex. But the simple difference is Kiro. $20 and an hour, you'll see what I mean. I was lucky - I met a guy working for AWS who worked on it who gave me the heads up, so got in early on beta. Now fully launched.

It's like the difference between your nan using Alexa and Bill Gates programming an operating system.

Thanks for the heads up, I’ll check Kiro out

Update:
I've been working on this for a while now, and not published intermediate versions because it takes a long time to get to production quality (switching on caching, checking it works and doesn't pollute etc).
I'm updating the versions above albeit unpublished to anyone who's interested can see what's going on. The end point is taking forever; partly because until Opus 4.6 I hit a roadblock of daft errors, but also because the next stages are insanely difficult conceptually (to me at least, probably akin to a kids crossword at Citadel).

The end goal is hourly forecasts but the road to get there is torturous. Once you've got daily volatility down pat for a vanilla model, you need to add event awareness in, so that it accounts for NFPs, CPIs etc. So events need to be categorised and uplifts applied.

Then for hourlies, you need templates of consumption which varies across events, times, pairs. An NFP on EURCHF is different from an interest rate decision on GBPUSD. All of these consumption templates need to be calibrated, smoothed and anchored across multiple timezones to form hourly weightings, which is what gives you forecasts.

Then you need to write this data to CSVs (old school I know), for a separate indicator to pick up and display on a chart dynamically, as RV <> forecasts.

final note, there'll be no source code for the final version - it's worth a few quid and while I'm not selling it, I'm not inclined to gift it to those that might.

Anyways... month or two out for version 3.0 based on current progress

Progress reports:
2_9_2
So I've got events in and calibrated to give uplifts. Nothing too complex here - all events in the past 2 years classified as CB, Macro London, Macro NY, pre-event day, or none. Impact assessed to produce uplifts by each pair. Pre-event is the day before macro or CB. I mentioned before this is a proper edge; now just simply quantifying it properly.

The stats are done and genuinely surprising stats - like CB events double the range on GBPUSD on average. What is useful is immediately, is you can now see which pair is likely the most volatile. I follow volatility directly so this is a gift.

The uplifts for crosses on USD news days too are really surprising, and I'd have never have guessed the relative impacts. The stats are surfaced on the global page as % uplifts (see below).

And the HAR performance? Couple of % improvement but also removal of a dirty secret. In order to calibrate the last version I added a global uplift to paper over the HAR undershooting. You'd get fired for that in a work setting but I was in a hurry. Anyway, this global variable (now in user settings) is set to 1, ie no change.
Stats on the news have their own page too for summary of what's computed.

2_9_3 Consumption curves
So divided the currency universe into families of curves, and applying the same labels as news, curves for each family. Then on the day of the event the curve is constructed using a kernel either set to time (macro) or session (CB), normalised, and decomposed into hourly weightings. Stats page shows the construction curves used for each day. Useful as a chocolate teapot to trade but used for debugging.

This is now shown in hourlies on the majors page too - see attached. Bear in mind the left is 4pm London (now the start of the HAR trading day).

And here is where I thought I'd be tweaking a bit more and diving into an indicator to show hourlies. But oh no. Not remotely.

Here's the next learning point. Where I'm getting to is a predictive band around price on each hour - 80% of the time it'll return here type thing. And combined with regime detection, bobs your uncle (trend/reversion trades at a deep or shallow discount). But in order that this indicator doesn't become the world's most colourful, labour-intensive sack of shite, not only do the consumption curves need fully quantifying in their effectiveness in ways that are above my pay grade, but so does the regime so that it's not just a colourful band that runs along the chart. Fair enough. And then - allegedly - you use the regime to map back to adjust your consumption curves.

Christ. All I wanted was a band around price to give me a rough idea. But I read enough to convince me - basically the trading world is full of bullshit stats; there's one doing the rounds on TV at the moment using infamous dilbert ICT 'techniques'. No need to add to the nonsense.

So 2_9_4: curve stats and regime detection module. Full of stats I don't yet understand. Pinball loss anyone? Currently burning through Opus credits as we speak

2_9_4 Now we're getting somewhere.
So now, we have curves done, forecast of hourlies and actuals. In addition we've a regime filter that works in shadow mode - doesn't impact anything at present. The attached shows what we currently have - hourly forecasts and hourly actuals (in gold). As luck would have it - honestly - we're in a suddenly Iranian-war induced high vol scenario so they don't look exactly impressive at present. It means that HAR forecast is way under; it's no knowledge of the future, and we're not currently adapting to the most recent information, other than oh, it was volatile yesterday.

The stats though - showing probability bands encapsulating price are broadly OK. So we're on the right track, just needs improving.

And this brings us to what the next phase needs to be. And thankfully I listened to GPT whispering in my ear.

So the HAR forecasts are as good as they can be - regularly returning around 40% improvement over ATR. This is brilliant. Locked.
Event uplifts work well. Improve core forecasts, reduce RMSE. They're not granular by design - CPI is treated like an NFP because we adjust this later. Locked.
Curves. Base curves I didn't monitor because they're so well known to be obviously sensible (Asia is quiet who knew). Event/family curves work well on majors (up to 40% improvement), do nothing for crosses. Not entirely surprising. So they add some value, just not tons. Locked.

So these three elements are untouchable. Now we need to move from the noddy regime I first implemented through to a more sophisticated one v 2_9_5 (in progress).
So regime detection is an artform - the basics of it are very simple, but bearing in mind it requires all of the above to function well, I can see why you never see proper regime detection at retail level.
We measure three things. The cadence is hourly but measured off 5m returns.

1) Trend - simply the distance travelled from London fix in our case. within a certain band of the fix = range. Simple. We add a persistence - must be 2 continuous hours so it doesn't flip back and forth. So state is trend or range

2) Speed - we measure cold,neutral,hot - slow,at expected speed, fast. If we have a forecast of X pips over one hour, if it gets there in 20 minutes, it's running hot, if 90 minutes, running cold for eg. Neutral - on time. Within tolerances

3) Highvol/normal - This is the height of the range per hour. If it's higher than normal or just normal. Why not smaller than normal? Because the stats in 2_9_4 revealed we only need this binary state, and that adjust for low hours is where you get ridiculous values. It's asymmetric I know.

So combining these three measures we end up with say "trend hot highvol". And what if the next hour it suddenly reverts or changes? It becomes unstable - these are times when strategies need to step aside. In reality we blend last hour and last 6 to get this figure, but you get the idea.

What we're doing is capitalising on trends in volatility and stepping out the way when it's all over the show. Just in the same way we capitalise on price trends, we're doing identically for volatility surfacing data that's invisible to the naked eye - a market running cold or hot on a price chart without the concrete statistical base behind it is virtually invisible to the naked eye.

Anyway - so we get those labels and then what?

Take today for example. it would say hot trend highvol on EURUSD. For sure. Since around 5am I'd imagine. Then we need empirically adjusted calibration- not to our core forecast, event uplifts or curves. But to the remaining hours of our day (which finishes at 4pm). So that when, like today, our forecasts are way below what's actually happening the system is responsive to it rather than stubbornly telling us well that's not what's supposed to happen.

These calibrations are different in length and type according to the regime and the day type (Non,pre,macro Ny or Lon, CB). And in order to do it properly rather than guess, we need to surface the stats produced by each regime to add meaningful calibration

In addition, regime detection is a useful measure for virtually EVERY kind of strategy if done right. A simple MA crossover works (to a degree) with good regime detection. Breakouts strategies work far better. Reversion. All it is is a filter, just a bloody complex one.

What it does mean is the indicator cannot stand alone - it'll need to read a CSV hourly minimum from the main dashboard. Cant see that's much of an issue tbh, but worth knowing.

And onto V 2_9_5 : improvements in the regime stats including a horror show of debugging, to answer a few questions.
Firstly, are the regime bands sensible. Does it flip around or is it relatively stable. Good news is now, across our 3 dimensions we've fairly standard timings- normal/high vol medians around 5 hrs, Cold/neutral/Hot around 2/3 hrs (this is a noisy signal), and Trend 4-6h. This is in line with most inst. models which is where I'm aiming. Now we know regime labels are solid and useful (they surface notably different stats according to regime) they've migrated to the main panel.

So that's out the way. Then it's on to calibration. Can we improve the hourlies with these regimes? Yes, but get it right.... there's some defaults already in which give us something approaching sensible curves (see screenshot) but they need tweaking so that an 80% confidence band is right +/- only a few %. This is called regime conditioning; taking advantage of persistence in trends of price and volatility to adjust your intraday forecasts. Work to be done.

There is, of course there is, another form of conditioning - intra-day conditioning. Basically, adjust on the basis of what's just happened. Harder to backtest. I'll come to that eventually.

And while I've burned through nearly a months credits in a few days, I'll park this for a while, and think ahead to the indicator spec. We can't just have a dumb box around price - it need to be anchored to both the trend, and visually show amount of dispersion left at any given moment. It's got to 100% be simple and intuitive and free of clutter too. Needs some thinking time.

OK - freezing this for the month. 2_9_5 is done, surfacing coarse regime stats (I started with granular - wrong way around). So there's two entirely separate versions of regime stats on one page. Simple and complex. From here next month to do list:
1- Confirm curves constructed correctly. Add event times to curves, mark vertically. Allow scrollback through 20d history

2- Calibration process: Increase lookback days to 60 for regimes. Factor (global) and then meanmult/dispmult by family and regime. This impacts the size of the bands (meanmult) and how wide/dispersed over time they are. Target 80/95% coverage across all regimes and families. May need a recalc stats on regime page

3- Targeted calibration: Any pairs that consistently over or undershoot need more surgical calibration. Check secondary stats are functioning correctly, not entirely convinced by figures I'm seeing compared to course. Wire to live. CSV Output

4- Intraday conditioning module (new):
A) - inputs of hourly forecast, specific shape curve, day forecast, event flags
- derived hourly actuals
- discrepancy signal - tbc - maybe log error. Reliability weight too - proximity to event/overall time of day/regime
B) Update rules: level only scaling vs shape aware redistribution or residual correction. Veering towards level only
C) Update day-level forecast - maybe. I'm reluctant on this only because I can see the potential for multiple versions to confuse. Needs careful data structurign. Cap floors and ceilings.
D) Testing. I've no idea how to test this efficiently - stuff updating hourly requires careful planning so you don't introduce look-ahead bias. Needs at the least to see on/off to visually check it's not madness
E) Ensure CSVs are updated by this module

5 - Main dashboard changes. Small UI changes to do to Majors panel on forecasts display. Add hours of continuous regime to label (move PipV column along). Global stats needs some reordering. Maybe remove YZ entirely too. Ensure flush button is deleting CSVs.

6- Indicator
Ensure DST aware and CSV parsing first (runs at 30 secs past hour). Contains
a) control panel
- with on/off toggles
- Local time inc DST
- CSV last update (hours since)
- jump risk navigation (from main dashboard, useful for stress testing)
- maybe pre conditioning and post conditioning toggle (for backtesting)
- auto/no updates - update the centre to be day's data. No updates because in live you don't want to accidentally pull yesterday into the mix.
- speedometer and trend stats

- data panel
a) regime label and hours since change
b) early regime change detection (see below)
c) daily cum. and hourly cum.

- main chart
d) verticals for fix and events (inc. previous days)
e) single line for trend trajectory
f) Daily forecast above/below fix on both left and right of fix. Maybe right is conditioned forecast?
g) From price open - box showing hourly forecast with 80/95% coverage lines
h) From current price - box showing remaining in hour and another one for day.

This indie will calculate both remaiing (based on variance) and early regime detection, neither of which will appear the main dashboard. The idea is to keep entirely separate forecast logic and display metrics only for dash

The early regime detection - used everywhere - something like 4hr vs 15/30m RV cross. When over certain threshhold for X minutes, will flag that regime is likely to break.

****

Whether or not that'll be doable next month we'll see, but that's the plan.