Diary of a Pro Quant | Page 2 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Is random really random? 231 replies

Pro Traders in a Random Market 60 replies

Walking the walk 2 replies

Alpari Direct Pro vs Alpari MT4 Pro 4 replies

Trading Diary/Journal 5 replies

Attachments: Diary of a Pro Quant
Exit Attachments

Diary of a Pro Quant

  • #21
  • Jun 7, 2023 1:38pm Jun 7, 2023 1:38pm
Quoting yoriz
{quote} No, I used Java (and Apple Numbers for the plots). For these simple experiments probably any language will do, although admittedly Python is very convenient for this kind of scripting. However, I do most of my ML using proprietary algorithms that are not available in the Python libraries. Implementing them in Python would be too slow and implementing them in C as a loadable Python library is too much hassle for my taste. Hence I do mostly work with Java which nowadays is as fast as C++ (without the complexities of writing robust C++ code)....
Very interesting! I prefer Java to Python, but finance has (mostly) adopted Python so I have too.

Quoting yoriz
{quote} Ah, I see. That is different from both methods I tried. Since we are looking at seasonal effects on a daily basis, it is very important the matching is exact. However, here it matters what seasonal event we are trying to find. Certain holidays are always at the same calendar date (and therefore not on the same day of the week). Other events are always on the same day of the week but on different calendar dates (e.g. returning to office on Monday after holiday, anticipating an upcoming holiday on Friday, etc.). So it is hard to tell which...
You're right, there's a few different ways of doing this and it's not clear which approach best. This beta version of cyclical charting is low confidence anyway so I'll readdress it at some point. After another iteration or two of improvements I may integrate it into an ML model. It's probably never going to be good enough to use for trading decisions by itself, but may add value as a component of a larger model.
  • #22
  • Jun 7, 2023 1:55pm Jun 7, 2023 1:55pm
Quoting ForexStats
I prefer Java to Python, but finance has (mostly) adopted Python so I have too.
Python is easy to learn and has several excellent ML libraries, so that is an excellent choice for most purposes!
  • #23
  • Jun 13, 2023 9:43pm Jun 13, 2023 9:43pm
Sorry for the brief hiatus here, I'm working on an ML model at the moment. I'm actually newer to time series forecasting with machine learning, but based on what I've read XGBoost looks like the way to go. I'm trying to get something basic up and running, then make a real model once I have the fundamentals down. Scikit learn and Keras also look good for their particular use cases.
  • #24
  • Jun 28, 2023 12:39pm Jun 28, 2023 12:39pm
I created some very simple ML models using logistic regression and extreme gradient boosting to predict the 1 day future return of the S&P 500. I didn't try to make anything useful quite yet, it's more just getting it to work. The logistic regression model was 76% accurate and extreme gradient boosting came in at 86%. Nothing impressive, but at least the models work.

I thought the feature importance aspect was really interesting. It basically says how much each feature matters towards the model's decisions. Here's a chart of my practice model:

Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 15 KB



My next steps will be:
- Correct encoding of features
- Implement cross-validation
- Implement pipelines
- Select better features

Selecting better features is my last step, and its where the rubber meets the road. It's tempting to do it right away, but without a robust infrastructure the features won't really be graded correctly.

I'll adapt this to FX later. I'm using the S&P 500 first because there's more history and more indicators (features) to use. All the code is reusable so it's just a matter of changing out the data/features.
  • #25
  • Jun 30, 2023 3:27pm Jun 30, 2023 3:27pm
Quoting ForexStats
My next steps will be: - Correct encoding of features
It turns out that encoding didn't make much difference. I tried a few things out and the accuracy ranged from 85% to 87%, barely any different from my starting point of 86%.

On to cross-validation.
  • #26
  • Jul 2, 2023 10:18am Jul 2, 2023 10:18am
Using historical data on regional Manufacturing PMI data, I created 5 basic machine learning models to predict ISM Manufacturing PMI.

Per the ForexFactory.com Calendar, the market expects the data to come in at 47.2. The mean forecast my models produced is 49.1.

Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 15 KB


Interestingly, all the models have a higher forecast than the ForexFactory.com estimate so we might be looking at better than expected data tomorrow.
  • #27
  • Jul 3, 2023 12:47pm Jul 3, 2023 12:47pm
Quoting ForexStats
Using historical data on regional Manufacturing PMI data, I created 5 basic machine learning models to predict ISM Manufacturing PMI. Per the ForexFactory.com Calendar, the market expects the data to come in at 47.2. The mean forecast my models produced is 49.1. {image} Interestingly, all the models have a higher forecast than the ForexFactory.com estimate so we might be looking at better than expected data tomorrow.
The data came in at 46.0.

This was way below expectations and its fair to say my models did poorly. Thiel-Sen came the closest, but still wasn't very close.

I'll tune them and try again next month!
  • #28
  • Jul 6, 2023 9:40pm Jul 6, 2023 9:40pm
Quoting ForexStats
{quote} The data came in at 46.0. This was way below expectations and its fair to say my models did poorly. Thiel-Sen came the closest, but still wasn't very close. I'll tune them and try again next month!
I added a new feature, the prior ISM PMI. That seemed to help the models quite a bit:

Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 16 KB


Thiel-Sen was only off by .5 this time, still with no hyperparameter tuning.
  • #29
  • Jul 6, 2023 10:09pm Jul 6, 2023 10:09pm
I adapted my models for NFP using ADP and the prior NFP as features. The models came out pretty bad to be honest, with a high error on each. Still, here's the shot in the dark prediction:

Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 20 KB


I'll see what else I can add as features next month and try to get the error down.
  • #30
  • Jul 7, 2023 10:14am Jul 7, 2023 10:14am
Quoting ForexStats
I adapted my models for NFP using ADP and the prior NFP as features. The models came out pretty bad to be honest, with a high error on each. Still, here's the shot in the dark prediction: {image} I'll see what else I can add as features next month and try to get the error down.
It came in at 209k, which was pretty far off the best model at 255k. That was expected, the question is how to improve it...

My initial thoughts for new features:

  1. Sum of previous 4 weeks of Initial Jobless Claims
  2. Challenger Job Cuts
  3. PMI data, particularly employment components

I may be able to leverage some alternative data, like Google Trends, but it won't be on the next iteration of this.

  • #31
  • Jul 7, 2023 1:00pm Jul 7, 2023 1:00pm
Quoting ForexStats
My initial thoughts for new features:
I like how you add additional information sources to your model! Often people try to predict XYZ using only the historical data of XYZ (aka reading charts). Adding more information to the mix certainly helps.

As for "trying next month": I hope you do not mean that literally? Try to obtain historical data and test your models using a Walk Forward approach. That helps with overfitting, and tests the real-life performance of your model.
  • #32
  • Jul 7, 2023 2:28pm Jul 7, 2023 2:28pm
Quoting yoriz
{quote} I like how you add additional information sources to your model! Often people try to predict XYZ using only the historical data of XYZ (aka reading charts). Adding more information to the mix certainly helps.
Thanks for chiming in yoriz! You're 100%, historical data of XYZ alone won't have much predictive value apart from a possible momentum or mean reversion factor. That's a part of it sure, but insufficient really so I always try to bring in any other relevant data I can.

Quoting yoriz
As for "trying next month": I hope you do not mean that literally? Try to obtain historical data and test your models using a Walk Forward approach. That helps with overfitting, and tests the real-life performance of your model.
No, definitely not. When I say "try next month", I just mean we only really get one shot to make an NFP prediction so I'll have to wait to next month for my call on July's employment. In the meantime though, I'll train the models on all the historical data.

I forgot to mention this previously, but my training set was 2010-2021 and test was 2022-Present. I should have retrained the models on the test set too, but didn't take the extra step knowing model quality would be low either way without better features. The ISM models I made aren't that bad really and include 5 independent data series as factors so I'm optimistic they'll do better for the next data release. I'm less confident on the notoriously difficult NFP, but we'll see just for fun.
  • #33
  • Jul 9, 2023 12:59pm Jul 9, 2023 12:59pm
Happy Sunday! What a great time to do some hyperparameter tuning. I was researching the tradeoffs of grid search vs random search and it sounds like random search is probably the better option. Apparently grid search is better in a sense that it will find the optimal hyperparameters, but its very computationally expensive. Alternatively, random search just guesses randomly which sounds terrible but I read that within 60 guesses it should come within 5% of the optimal solution. It's basically the option that's pareto principle friendly.

I think what I'll do is use random search on experimental models, and grid search for ones going into production. I'm open to other options too, anyone else into ML is welcome to chime in on their preferred approach!
  • #34
  • Jul 10, 2023 9:53am Jul 10, 2023 9:53am
Quoting ForexStats
Happy Sunday! What a great time to do some hyperparameter tuning. I was researching the tradeoffs of grid search vs random search and it sounds like random search is probably the better option. Apparently grid search is better in a sense that it will find the optimal hyperparameters, but its very computationally expensive. Alternatively, random search just guesses randomly which sounds terrible but I read that within 60 guesses it should come within 5% of the optimal solution. It's basically the option that's pareto principle friendly. I think what...
So, from the 5 models I've used XGBoost really should be the best. I did some basic hyperparameter tuning and the model didn't actually improve much. For ISM, the MSE went from 1.29 to 1.25. Better, but I had hoped for more. I'm not sure if it's just this one model, or if the default settings are normally pretty close to optimal.
  • #35
  • Edited 2:10pm Jul 10, 2023 1:59pm | Edited 2:10pm
Quoting ForexStats
I did some basic hyperparameter tuning and the model didn't actually improve much.
In my experience the inputs you feed to the model (and how you pre-process them) are much more important than hyper-parameter tuning. As a basic example (which you probably already know):

  1. open[i] --> using absolute prices as an input causes the model to learn behavior around specific prices and perform worse once future prices drift away
  2. open[i]-open[i-1] --> using the change in price during the last bar is better as it also works when future prices are higher or lower
  3. (open[i]-open[i-1])/open[i-1] --> using the relative change in price, also named "the return", works better once prices are significantly changed from the prices you trained the model on since it normalizes the price
  4. log(1 + ((open[i]-open[i-1])/open[i-1])) --> using the log-return allegedly works even better, although personally I never noticed any improvement over the relative change.

Normalizing and pre-processing your inputs can improve your models much more than hyper-parameter tuning ever will.

  • #36
  • Jul 11, 2023 11:22pm Jul 11, 2023 11:22pm
Quoting yoriz
{quote} In my experience the inputs you feed to the model (and how you pre-process them) are much more important than hyper-parameter tuning. As a basic example (which you probably already know): open --> using absolute prices as an input causes the model to learn behavior around specific prices and perform worse once future prices drift away open-open[i-1] --> using the change in price during the last bar is better as it also works when future prices are higher or lower (open-open[i-1])/open[i-1] --> using the relative change in price,...
Thank you! I preprocess much of my data and sometimes normalize it. It hasn't made as much difference as I initially expected, but I'm still newer to ML. For hyperparameter tuning, I actually just ran a for loop over a few critical parameters and optimized that way. I'm sure grid search and random search will come in handy later, but a simple for loop did pretty good by itself.

You mentioned log returns... That's an area I've had a hard time wrapping my head around. The industry does prefer log and there are some good reasons for that, but your results can come out significantly different than standard percentage change. And it's strange because one or the other isn't really right or wrong to use, but render meaningfully different results. Admittedly, I don't use log returns often but that's probably the better approach.
  • #37
  • Jul 11, 2023 11:45pm Jul 11, 2023 11:45pm
I put together some ML models tonight for CPI. They use the prior CPI, average hourly earnings, import prices, WTI, and gasoline as features.

The FF calendar has 0.3% and 3.1% for the MM and YY forecasts.

M/M, my models came up with an average estimate of 0.2%:
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 15 KB


And Y/Y, we're at 3.4%.
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 15 KB



The models seem to be in conflict, calling both lower than expected M/M inflation and higher than expected Y/Y inflation. This may be because the inputs are slightly different. I'll be interested to see which do the best and refine the models.

An alternate approach would have been to call the index level, then derive the M/M and Y/Y change from there. That presents a few problems though since it doesn't align with the M/M and Y/Y data I'm already using. While I'm not a fan of that approach at the moment, I may go that direction if my current approach leads to poor results.

I plan to add Core CPI next month.
  • #38
  • Jul 11, 2023 11:49pm Jul 11, 2023 11:49pm
By the way, this is all for fun and practice at the moment. I don't expect any of these ML models to perform well yet. I've got all the scripts so it's very little work to add/remove features, rescale the data, tune parameters, etc. It may not be a lot of work from here to make something genuinely worthwhile.
  • #39
  • Edited 9:12am Jul 12, 2023 8:43am | Edited 9:12am
Quoting ForexStats
I put together some ML models tonight for CPI. They use the prior CPI, average hourly earnings, import prices, WTI, and gasoline as features. The FF calendar has 0.3% and 3.1% for the MM and YY forecasts. M/M, my models came up with an average estimate of 0.2%: {image} And Y/Y, we're at 3.4%. {image} The models seem to be in conflict, calling both lower than expected M/M inflation and higher than expected Y/Y inflation. This may be because the inputs are slightly different. I'll be interested to see which do the best and refine the models. An alternate...
The data came in at 0.2% and 3.0%.

My M/M models all nailed it, with the exception of XGB. XGB is still probably the best model though so I'm not reading much into this.

The Y/Y models all missed this one, but XGB came close along with Random Forest. The average estimate wasn't terrible either at 3.4%. A few thoughts on this:

  1. The Y/Y change from last month was a full 1.0% drop. That's a big drop, I wonder if the models struggled to deal with it.
  2. I may add month of the year as a feature next time to help with seasonality.
  3. One possible problem here is the data around Covid, since we're dealing with a lot of data that was very exceptional. It's outlier data, but legitimate also so it's hard to know if I should simply remove it or include it. I'm leaning towards removing it since it definitely didn't represent normal times. It may be as simple as setting the training dates to go until February 2020 and test dates June 2020-Present. That won't work with I implement cross validation though.
  4. Interestingly the Theil-Sen Regressor model, which was built to handle outliers well, did not do well on CPI Y/Y. It's done very well in other forecast though so I'm not sure what to make of this yet.


I'm thinking of removing the Linear Regression and Random Forest models. They're like less sophisticated versions of Theil-Sen and XGB so they shouldn't do as well in theory, and thus far they've come up a little short of their more sophisticated counterparts. I've known this all along, but wanted to see how they'd do anyway.

  • #40
  • Jul 12, 2023 1:21pm Jul 12, 2023 1:21pm
Quoting ForexStats
I'm thinking of removing the Linear Regression and Random Forest models.
I prefer Random Forests because:

- very fast to train!
- RFs simply ignore weak or useless inputs, while NNs can go crazy if you feed it weird inputs.

This means I can freely feed it any indicator I can find (thanks, mladen ;-)) and later read back the "relevance" of that input to determine if it is worthwhile to keep it.
0 traders viewing now