Diary of a Pro Quant

Good points yoriz! I did some more research and it sounds like Random Forest and XGBoost are different enough to where I should keep both. I'm going to bring in some new models too, at least temporarily. In python, introducing new models doesn't involve much extra code. If I have time today, my plan is to bring in about 10 models for ISM and see which perform the best looking at metrics like mean squared error.

I revised my ISM script and basically added every Scikit Learn linear regression model to see how they all compare. Here were the results, with each respective model and its test Mean Absolute Error.

1. LinearRegression - 1.179
2. Lars - 1.179
3. Ridge - 1.179
4. HuberRegressor - 1.186
5. ARDRegression - 1.191
6. BayesianRidge - 1.195
7. TweedieRegressor - 1.209
8. ElasticNet - 1.238
9. GradientBoostingRegressor - 1.291
10. XGBRegressor - 1.294
11. RandomForestRegressor - 1.334
12. TheilSenRegressor - 1.335
13. OrthogonalMatchingPursuit - 1.439
14. KernelRidge - 1.485
15. SVR - 2.134
16. KNeighborsRegressor - 2.203
17. LassoLars - 5.137
18. SGDRegressor - 9125281268857.1 (Something clearly went wrong.)

There are almost 20 models that accomplish the same task so I'm working to differentiate which models do a better job and which perform poorly. Once I get some sense of that I can prioritize studying the better ones first. I'm also considering whether I may get better results by taking the mean/median of multiple models (ie top 3 or top 5)... My initial observation is some of the simpler models actually had the best out-of-sample results.

This is just a starting point, I'll review other metrics later (possibly today).

Reran it with a couple of new metrics, now sorted by median absolute error:

GradientBoostingRegressor()

Mean Absolute Error: 1.277

Median Absolute Error: 0.775

R2 Score: 0.877

XGBRegressor()

Mean Absolute Error: 1.294

Median Absolute Error: 0.93

R2 Score: 0.855

HuberRegressor()

Mean Absolute Error: 1.186

Median Absolute Error: 0.983

R2 Score: 0.92

ElasticNet()

Mean Absolute Error: 1.238

Median Absolute Error: 0.991

R2 Score: 0.902

LinearRegression()

Mean Absolute Error: 1.179

Median Absolute Error: 1.071

R2 Score: 0.916

Lars()

Mean Absolute Error: 1.179

Median Absolute Error: 1.071

R2 Score: 0.916

Ridge()

Mean Absolute Error: 1.179

Median Absolute Error: 1.073

R2 Score: 0.916

RandomForestRegressor()

Mean Absolute Error: 1.478

Median Absolute Error: 1.085

R2 Score: 0.82

ARDRegression()

Mean Absolute Error: 1.191

Median Absolute Error: 1.087

R2 Score: 0.916

TweedieRegressor()

Mean Absolute Error: 1.209

Median Absolute Error: 1.104

R2 Score: 0.908

BayesianRidge()

Mean Absolute Error: 1.195

Median Absolute Error: 1.126

R2 Score: 0.911

TheilSenRegressor()

Mean Absolute Error: 1.351

Median Absolute Error: 1.202

R2 Score: 0.92

KernelRidge()

Mean Absolute Error: 1.485

Median Absolute Error: 1.273

R2 Score: 0.914

OrthogonalMatchingPursuit()

Mean Absolute Error: 1.439

Median Absolute Error: 1.286

R2 Score: 0.852

KNeighborsRegressor()

Mean Absolute Error: 2.203

Median Absolute Error: 1.657

R2 Score: 0.561

SVR()

Mean Absolute Error: 2.134

Median Absolute Error: 1.791

R2 Score: 0.591

LassoLars()

Mean Absolute Error: 5.137

Median Absolute Error: 5.674

R2 Score: -1.5641893023281903e+29

SGDRegressor()

Mean Absolute Error: 3867747843212.178

Median Absolute Error: 4027267144524.002

R2 Score: -23.456

When sorting by Median Absolute Error, the results are more in line with what I would have expected.

Thoughts:

1. GradientBoosting is interesting that it did significantly better than everything else and it's R2 is decent. It must have had a very bad prediction or two for the mean absolute error to be so much higher than the median. I should add in the max_error for the next iteration of this... I'm slightly concerned about low median and high mean. My understanding is this model, in theory, should get beaten by XGB so I'm slightly less interested in it despite commendable performance here.
2. XGB did well, as expected. I really thought this would be the best model, but it looks like it's got competition. Not a fan of this R2. The mean absolute error isn't great, but we already know it can drop into the 1.25 range after parameter tuning.
3. Huber did good too, and it has a higher R2 than the above two models. This one may have potential, I'll have to research it further.
4. ElasticNet does pretty good too, and from what I've read it sounds like it should be a strong model.

At this point, I'm confident in XGB. I'm going to study Huber and ElasticNet next to see what potential they've got. I'll look more into GradientBoosting after that.

Added some more key metrics, sorted by Mean Squared Error this time:

ARDRegression()

Mean Squared Error: 2.421

Root Mean Squared Error: 1.556

Mean Absolute Error: 1.191

Median Absolute Error: 1.087

Max Error: 3.492

R2 Score: 0.916

HuberRegressor()

Mean Squared Error: 2.424

Root Mean Squared Error: 1.557

Mean Absolute Error: 1.186

Median Absolute Error: 0.983

Max Error: 3.541

R2 Score: 0.92

Lars()

Mean Squared Error: 2.456

Root Mean Squared Error: 1.567

Mean Absolute Error: 1.179

Median Absolute Error: 1.071

Max Error: 3.544

R2 Score: 0.916

LinearRegression()

Mean Squared Error: 2.456

Root Mean Squared Error: 1.567

Mean Absolute Error: 1.179

Median Absolute Error: 1.071

Max Error: 3.544

R2 Score: 0.916

Ridge()

Mean Squared Error: 2.458

Root Mean Squared Error: 1.568

Mean Absolute Error: 1.179

Median Absolute Error: 1.073

Max Error: 3.545

R2 Score: 0.916

BayesianRidge()

Mean Squared Error: 2.571

Root Mean Squared Error: 1.603

Mean Absolute Error: 1.195

Median Absolute Error: 1.126

Max Error: 3.625

R2 Score: 0.911

TweedieRegressor()

Mean Squared Error: 2.631

Root Mean Squared Error: 1.622

Mean Absolute Error: 1.209

Median Absolute Error: 1.104

Max Error: 3.665

R2 Score: 0.908

ElasticNet()

Mean Squared Error: 2.712

Root Mean Squared Error: 1.647

Mean Absolute Error: 1.238

Median Absolute Error: 0.991

Max Error: 3.795

R2 Score: 0.902

RandomForestRegressor()

Mean Squared Error: 2.807

Root Mean Squared Error: 1.675

Mean Absolute Error: 1.294

Median Absolute Error: 1.046

Max Error: 5.34

R2 Score: 0.868

TheilSenRegressor()

Mean Squared Error: 2.84

Root Mean Squared Error: 1.685

Mean Absolute Error: 1.364

Median Absolute Error: 1.141

Max Error: 3.546

R2 Score: 0.918

OrthogonalMatchingPursuit()

Mean Squared Error: 3.032

Root Mean Squared Error: 1.741

Mean Absolute Error: 1.439

Median Absolute Error: 1.286

Max Error: 4.882

R2 Score: 0.852

GradientBoostingRegressor()

Mean Squared Error: 3.095

Root Mean Squared Error: 1.759

Mean Absolute Error: 1.285

Median Absolute Error: 0.717

Max Error: 4.752

R2 Score: 0.878

XGBRegressor()

Mean Squared Error: 3.511

Root Mean Squared Error: 1.874

Mean Absolute Error: 1.294

Median Absolute Error: 0.93

Max Error: 4.77

R2 Score: 0.855

KernelRidge()

Mean Squared Error: 3.57

Root Mean Squared Error: 1.889

Mean Absolute Error: 1.485

Median Absolute Error: 1.273

Max Error: 5.647

R2 Score: 0.914

SVR()

Mean Squared Error: 6.242

Root Mean Squared Error: 2.498

Mean Absolute Error: 2.134

Median Absolute Error: 1.791

Max Error: 5.743

R2 Score: 0.591

KNeighborsRegressor()

Mean Squared Error: 7.032

Root Mean Squared Error: 2.652

Mean Absolute Error: 2.203

Median Absolute Error: 1.657

Max Error: 5.571

R2 Score: 0.561

LassoLars()

Mean Squared Error: 31.589

Root Mean Squared Error: 5.62

Mean Absolute Error: 5.137

Median Absolute Error: 5.674

Max Error: 10.474

R2 Score: -1.5641893023281903e+29

I pulled out SGDRegressor(). It had the lowest MSE, but crazy high numbers for everything else. It's clearly wrong and just adds noise.

A few closing thoughts for tonight:

1. My favorite model right now is still XGBoost. It didn't have the best performance in the ISM example, but extreme gradient boosting should lead to better/quicker results.
2. I read CatBoost (category, not the animal) is still better but it isn't a part of SciKit-learn. I don't think it's worth investigating, yet.
3. For my purposes, speed doesn't seem to matter. I read in one example where SVM took something like 90 minutes to train a model while XGBoost took just 48s. My data sets are small and it took about 2s to train all 20 models. So where there's a tradeoff between accuracy and speed, I'll take accuracy for these models.
4. I'll continue to run this template script on other economic data to see how the various models do, but adding/removing models won't be my immediate focus. Improving features, better preprocessing, and implementing cross-validation (which I still haven't done yet) are all higher priority. I'm itching to get started with MLOps too... When/if I decide to switch up the models, most of the code is already completed so it would be a fairly easy transition.
5. My favorite evaluation metric is median absolute error. I think median is better than mean in this case as the mean gets pulled up by the occasional wild estimate. In practice, I'd be able to identify these wild estimates so I'm not too concerned with them. For that reason, median makes more sense to use.
6. Predicting economic data has been a way to practice linear regression ML. For price though, my understanding is classification is better suited. Instead of trying to predict what the close of EUR/USD will be tomorrow, predicting whether it will be higher or lower is the better route. That's an additional reason I'm not going to focus overly much on the exact model to use since I'll need to use classification models anyway.
7. Feature engineering is going to be the most important aspect. Great features + good preprocessing + a good model should be a winning formula. None of the ML is a secret, the edge (if there is one) is in the features.

By the way, if you're reading this and everything sounds foreign to you, rest assured machine learning is probably a lot easier than you think. Its A LOT to wrap your head around, and each aspect is important, but no one part is very difficult.

Thanks for the updates. I am following your experiments with great interest. Nice to see you consider many different ML techniques and not just focus on one. I am too lazy to try them all, so I am hitchhiking on your efforts :-)

I don't have experience with predicting economic data. However, when predicting price action the accuracy of the models is not a good indicator for the live performance. If the model successfully predicts the direction of large bars, and fails at smaller insignificant moves, then you can still make a good profit even though the model accuracy is bad.

Consider adding a backtester to your code and use Sharpe or Monte Carlo to rate the performance of your model, rather than the raw accuracy of your model.

Thanks yoriz, I always appreciate your feedback! I'll definitely add performance metrics into an strategy I produce. ML classification actually lends itself very well to vectorized backtesting.

So I'm about to start on a new script... I'm debating between building another regression model, this time for Retail Sales, or building a classification model for price (predicting 1, 5, or maybe 21 days out).

I build a rough classification model to predict whether tomorrow will close higher or lower for USD/JPY. The results were exceptionally poor with an accuracy rate of 54%. The CAGR was -.50 and the Sharpe was just .20... Only up from here!

I moved the strategy from a python file to a Jupyter notebook and the results improved.



Of course, the results didn't really change as everything runs exactly the same. However, something could have been wrong with the dataframe I originally used. (It's easy to accidentally remove some of your data by rerunning the same code if you're not careful.)

In any event, the results don't matter too much at this point. What's important is I've got a template to work off of now.

I added a new feature and the results improved significantly. Here are the metrics version 2 produced on the test set:



That appears to be a great system, but I'll admit I'm skeptical. As they say, if it seems to be too good to be true it usually is!

Um... Same code in plain Python vs embedded in Jupyter notebook gives better results?! That's unexpected. Perhaps your system had outdated ML libraries that are now updated by installing Jupyter?

Nah, I'm using VS Code which can use .py or .ipynb files. I think it was just a dataframe issue where I reprocessed it too many times.

I think I know what's going on here. My price data for USD/JPY comes from Yahoo Finance and my new feature's data comes from Stooq. I think they could have different close times so my new feature may include the first part of the t+1 session for USD/JPY. I'll need to confirm, but that makes sense more than really coming up with a >2 Sharpe strategy on my second try.

Ouch... yes, that sort of thing can happen. With H1 data you can often see the timezone shift and correct for that, but with D1 data there is not much you can do other than look for a different data source.

I normalize all my Forex data to UTC+2 or UTC+3 based on New York Daylight Savings. In other words: I don't have Saturday or Sunday bars. First tick comes in Monday morning at 0:00, last tick on Friday evening at 23:59. I reconstruct the D1 bars from those ticks.

In addition I added a sanity check for most symbols based on characteristic peaks/valleys. The most obvious landmark for CHF for instance, was at 11:30 (UTC+2) on 15 Jan 2015 when the world crashed. Use that to double check your time shift.

Yep, it looks like I'll need to find a new data source. Yahoo! Finance has been convenient to use, but I see they've got some big gaps between their open and close which shouldn't happen during the week:


Moreover, I can't figure out when their open and close prices correspond to. Reconciling against ForexFactory data shows it doesn't look like 5pm EST, but it's not clear it really aligns with any other logical time. I'll just move on, I'm overdue to get a new data source anyway.

I'll see about OANDA or Stooq as a replacement.

Stooq didn't appear to offer historical forex data, despite having it on their website and offering a ton of other historical data... I winded up using AlphaVantage, which daily data roughly matches ForexFactory's.

They do offer daily Forex data, but not intraday. For example: https://stooq.com/q/d/l/?s=eurusd&i=d

It looks like it on their site, but it won't download. I think its something specific to forex because stock tickers work...In any event, AlphaVantage worked so I've got the data I need.

ISM PMI is due tomorrow and the FF forecast is for 46.0. Here's what my ML models are forecasting. This time I added in the FF forecast as a feature.



And here are the old models without the FF forecast. Some actually have better error scores, XGBoost being one of them.



Either way, it looks like we may be looking at slightly better than expected data.

Cool thread. My trading also evolved into statistical based strategies about 5 years ago. But with a focus on equities.

I am looking to possibly diversify into FX with a main focus on news events. Looking forward to sharing ideas. At one point about 7 years ago I was physically writing down how many pips of volatility there was for each news event and for which pairs.

Welcome aboard Fx3423454, I'm glad to have you join us!

You said you're using statistical based strategies, what are you using to do you analysis? (ie Python, Excel, R, C#) There's nothing wrong with equities, plenty of money to be made there too!

News trading would be great, if I can get a semi-accurate estimate. So far it's mostly just been an exercise to practice machine learning, but I think there's real potential for calling ISM correctly. On the other hand, there are some data's I'm not too optimistic. Non-farm payrolls is one. It's fun to forecast, but realistically there's not a clear right answer in the end. If you call it right, but it's revised the next month were you really right? If you're wrong, but it's revised later to where you would have been correct then were you really wrong? Hard to say. Also, data's like Retail Sales I'm not optimistic about. We might be able to put together a good model, but it seems like we can't beat traders at places like JP Morgan that might have access to Chase's credit card transaction data.