Machine Learning with algoTraderJo

Quoting algoTraderJo

Disliked

2- This never happens for me. I predict a return MLE and MSE so they are not price values that can be higher/lower in an illogical manner. My predictions are, for example, an MLE of 0.2% and an MSE of 0.1%, there is no way in which the MLE can be lower than the MSE in an inconsistent way. Structuring the outputs in this way is crucial for reproducing my results.

Ignored

Quoting PipMeUp

Disliked

{quote} It happens twice with my dataset from 2007 to 2015. 29/10/2008 MLE=-9.003751139075197E-4 MSE=3.8783981800940843E-4 09/05/2010 MLE=0.002561915803222095 MSE=-0.004921393412012904 Note: It is possible that the MLE is negative (when the D future prices are all lower than the current price) but in this case MSE must be more negative (and vice-versa).

Ignored

Quoting algoTraderJo

Disliked

{quote} For me it never happens because the MSE and MLE are calculated from the current open. If all values within the next 12 bars are above the current open then the MSE is 0, in no case can it be negative.

Ignored

Quoting PipMeUp

Disliked

{quote} Ok so you cap the values to 0 in the training target. But nothing prevents the forecast to be negative, and it often is. Do you cap the forecast as well?

Ignored

Quoting PipMeUp

Disliked

Is anybody able to reproduce Jo's curve? None of mine remotely look like his. Using A=1, B=80, C=6 and D=12 on USD/JPY from 2007 to today, I get breakeven with a 15% max DD. 2.5 years to recover. With E/J A=0, B=130, C=9, D=12, I got a profit factor of 1.096. Using 16 bar horizon, E/J loses money and U/J improves a bit (PF=1.094). Note: I use a fixed size MM. With 1% MM it turns to a disaster. This is what the trades look like for E/J {image}

Ignored

Quoting PipMeUp

Disliked

Is anybody able to reproduce Jo's curve? None of mine remotely look like his. Using A=1, B=80, C=6 and D=12 on USD/JPY from 2007 to today, I get breakeven with a 15% max DD. 2.5 years to recover. With E/J A=0, B=130, C=9, D=12, I got a profit factor of 1.096. Using 16 bar horizon, E/J loses money and U/J improves a bit (PF=1.094). Note: I use a fixed size MM. With 1% MM it turns to a disaster. This is what the trades look like for E/J {image}

Ignored

Quoting MathTrader7

Disliked

{quote} I do not use F4 framework, I use my own simulator. I gave a try to the latest Jo's method but I couldn't reproduce Jo's results. It may be because I don't have access to the same history of ticks as he does, or I may estimate the parameters of the model incorrectly... Or there is some bug in my simulator

Ignored

Quoting PipMeUp

Disliked

Is anybody able to reproduce Jo's curve? None of mine remotely look like his. Using A=1, B=80, C=6 and D=12 on USD/JPY from 2007 to today, I get breakeven with a 15% max DD. 2.5 years to recover. With E/J A=0, B=130, C=9, D=12, I got a profit factor of 1.096. Using 16 bar horizon, E/J loses money and U/J improves a bit (PF=1.094). Note: I use a fixed size MM. With 1% MM it turns to a disaster. This is what the trades look like for E/J {image}

Ignored

Quoting algoTraderJo

Disliked

{quote} This is always problematic, one of the reasons why I chose F4 was that there is a community behind it (so bugs get found/fixed) plus the whole source is available. It is very difficult to compare things when private frameworks are being used, especially if there is different data, different ML algorithms, etc. If you have any questions, just ask, I am interested in giving you as much info as you need to reproduce using your own kits.

Ignored

Quoting MathTrader7

Disliked

{quote} I have to admit that you're absolutely right! I'm going to give up further development, test and debug of my own simulator (the current state of my simulator took a lot of time, and it still needs a lot of effort for further development).

Ignored

    public void process(Candlestick candle) // hourly candle
    {
        // window is an array of C+D+1 candles. Shift.left moves the element one place on the left.
        Shift.left(window, candle);
        if (window[0] == null) return;
        // tz is Europe/Paris = GMT+1/+2
        long timestamp = Time.convert(window[C + 1].getTimestamp(), tz);
        // A is midnight Paris time (0:00 GMT+1/+2)
        if (Time.getHour(timestamp) == A) {
            double[] returns = new double[C + 1]; // +1 is for the intercept
            double[] excursions = new double[2];
            returns[0] = 1; // this is for the intercept coefficients
            for (int i = 1; i <= C; i++) {
                returns[i] = (window[i].getClose() - window[i - 1].getClose()) / window[i - 1].getClose();
            }
            // MinMax is a helper to keep the min and max values
            MinMax minMax = new MinMax();
            for (int i = C + 1; i < window.length; i++) {
                minMax.update(window[i].getHigh());
                minMax.update(window[i].getLow());
            }
            // Compute the MLE/MSE
            double close = window[C].getClose();
            excursions[0] = (minMax.max - close) / close;
            excursions[1] = (close - minMax.min) / close;
            // samples and targets are double[][] with B rows (The contents of the S and T matrices)
            Shift.left(samples, returns);
            Shift.left(targets, excursions);
        }
        else if (samples[0] != null) {
            // Time.now() is the time of the last tick received
            timestamp = Time.convert(Time.now(), tz);
            if (Time.getHour(timestamp) == A) {
                double[] query = new double[1 + C];
                query[0] = 1; // don't forget the intercept
                // The last element in window is the candle we just received
                // pos is the index C bars before ()
                int pos = window.length - C;
                for (int i = 1; i <= C; i++, pos++) {
                    query[i] = (window[pos].getClose() - window[pos - 1].getClose()) / window[pos - 1].getClose();
                }
                // Turn the arrays into matrices
                Matrix q = new Matrix(new double[][] { query });
                Matrix S = new Matrix(samples);
                Matrix T = new Matrix(targets);
                // Normal equation
                Matrix St = S.transpose();
                Matrix StS = St.times(S);
                Matrix coeff = StS.inverse().times(St).times(T);
                Matrix forecast = q.times(coeff);
                // Forecast
                double mle = forecast.get(0, 0);
                double mse = forecast.get(0, 1);
                // Trading rules
                // Note position is set to null when the strategy is notified the SL was hit.
                if (mle > mse) {
                    double stopPrice = instrument.getBid() - 0.5 * atr.getLast();
                    if (position == null) {
                        position = openLong(stopPrice);
                    }
                    else if (position.isLong()) {
                        position.trailStop(stopPrice);
                    }
                    else {
                        position.close();
                        position = openLong(stopPrice);
                    }
                }
                else if (mle < mse) {
                    double stopPrice = instrument.getAsk() + 0.5 * atr.getLast();
                    if (position == null) {
                        position = openShort(stopPrice);
                    }
                    else if (position.isShort()) {
                        position.trailStop(stopPrice);
                    }
                    else {
                        position.close();
                        position = openShort(stopPrice);
                    }
                }
            }
        }
    }

Quoting PipMeUp

Disliked

I simulated using JPY as the account currency to prevent the rate fluctuations to distort the result. I completely rewrote the strategy to make sure it is correct. I still get negative results. MLE and MSE ARE capped to zero as seen in post 356: the initial value before the loop is set to 0 (lines 13-14). Here is an example where MLE is negative: the highest high is below the previous candle close. {image} I trust my simulator. I thoroughly tested it. Also this simulation has a single position open at a time with just a SL. That's a very easy use...

Ignored

Quoting levkly

Disliked

Hi algoTraderJo, Thanks for posting the code. 1. Do you perform cross validation before trading live or you have other methodologies in order to avoid over optimization? 2. How you choose the predictors (inputs ) ? Do you use other parameters except past N days returns like fundamentals data , sentiment ...? Thanks Lev

Ignored

Quoting algoTraderJo

Disliked

{quote} There is a very important difference. I don't use the close prices for the calculation of the MSE/MLE I use the open prices. This is why you can get a negative MLE/MSE while I never get this result. I always make a comparison from the open of the trigger hour to the subsequent highest/lowest values. You will never get a negative if you do things this way, I do not artificially cap the values I do never get any negative ones due to this fact. I also calculate returns based on candle opening values (open[t]-open[t-1]/open[t-1]), I never use...

Ignored