Machine Learning with algoTraderJo | Page 23 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Algorithmic Quant Trading (Machine Learning + Stat-Arb) 25 replies

Forex with Machine Learning Software Project 1 reply

Machine Learning + Retail Forex = Profitable? (Quant) 1 reply

Potential new machine learning style software. 79 replies

My most recent advancements into machine learning 16 replies

Attachments: Machine Learning with algoTraderJo
Exit Attachments

Machine Learning with algoTraderJo

  • #441
  • May 19, 2015 3:25pm May 19, 2015 3:25pm


I don't even know what else to ask?! I even downloaded and re-installed the F4 4.4.14 from scratch, took your exact files for the EURUSD from post 409 and cannot reproduce the past results. Not even close.

I am using Strategy Tester 0.39 The only difference I could possibly think of is that I am compiling 32bit Win - but I cannot imagine this making any difference.
Trying the post 408 (original file settings - settings of A = 10 (GMT +1/+2), B = 130, C = 7 and D = 20. Stop at 40% of the daily ATR(20), ) I get this result:

Total trades: 3640
Longs: 1798 Shorts: 1842
Final balance: 385209.65$
Max DD: 45.25$
Max DD Length: 3315
PF: 1.05
R2: 0.98
Ulcer Index: 22.97
Time elapsed: 0:03:10.511000

Attached Image (click to enlarge)
Click to Enlarge Name: results.png Size: 74 KB


It is similar with the other settings - e.g. post 428 (A=13 GMT +1/+2, B=150, C=3, D=8 with a 30% of the ATR(20) as a stoploss)
Total trades: 2400
Longs: 1413 Shorts: 987
Final balance: 668401.09$
Max DD: 48.09$
Max DD Length: 2256
PF: 1.07
R2: 0.97
Ulcer Index: 22.64
Time elapsed: 0:02:32.010000

Attached Image (click to enlarge)
Click to Enlarge Name: results_1.png Size: 70 KB
  • #442
  • May 19, 2015 3:40pm May 19, 2015 3:40pm
Quoting KaBo
I don't even know what else to ask?! I even downloaded and re-installed the F4 4.4.14 from scratch, took your exact files for the EURUSD from post 409 and cannot reproduce the past results. Not even close. I am using Strategy Tester 0.39 The only difference I could possibly think of is that I am compiling 32bit Win - but I cannot imagine this making any difference. Trying the post 408 (original file settings - settings of A = 10 (GMT +1/+2), B = 130, C = 7 and D = 20. Stop at 40% of the daily ATR(20), ) I get this result: Total trades:...
Don't despair my friend I'll reproduce on a windows machine and will send you the results along with a link to get the exact library source I use. You definitely have a setup problem somewhere.
  • #443
  • May 19, 2015 4:31pm May 19, 2015 4:31pm
Here are the steps from an F4.4.14 unstable release on Windows using Visual Studio 2010:

  1. Make sure you can compile the Release candidate first without any modifications (make sure the DLL is created inside the bin folder)
  2. Close Visual Studio 2010.
  3. Download and unzip the latest Strategy tester version.
  4. Download the latest DATA_NST zip file from Asirikuy and replace the files inside the Strategy tester/history folder with them
  5. Make sure you can run all diagnostic tests in the Strategy tester without problems
  6. Download the zip file attached to this post
  7. Replace the munay.c file in your F4 Source/dev/TradingStrategies/src folder with the one inside the zip (easier to replace an already existing strategy to ensure everything is exactly the same)
  8. Open up Visual Studio 2010 again.
  9. Recompile F4
  10. Copy the newly created DLL file (inside the F4 Source/bin/.../) to your StrategyTester root folder (replace the AsirikuyFramework.dll that comes with the tester). It is vital to replace the DLL! (otherwise no changes take place)
  11. Replace the ast.config in your StrategyTester/config folder with the one inside the zip
  12. Copy the munay_eurusd.set file inside your StrategyTester/set folder.
  13. Open a command prompt and go to your Strategy tester folder, then run python ast.py (this is what I do, I prefer to use the command line tester, much cleaner)

You should have these results after the test is done (I attach the exact tester output):

Attached Image (click to enlarge)
Click to Enlarge Name: results.png Size: 66 KB


Daniel and another Asirikuy member have confirmed that they are able to reproduce following the above steps.
Attached File(s)
File Type: zip full_setup.zip   2 KB | 490 downloads
  • #444
  • Edited 4:44am May 20, 2015 4:00am | Edited 4:44am
algoTraderJo, have you ever tried to apply the same ML or similar to US stocks?
I would be glad to reproduce your results with FX first but don't have access to the same data as you have. If you could share the data just for one FX pair you experimented with I could compare the results of applying ML to both FX and US stocks.
  • #445
  • May 20, 2015 6:03am May 20, 2015 6:03am
Quoting birdyalex
algoTraderJo, have you ever tried to apply the same ML or similar to US stocks? I would be glad to reproduce your results with FX first but don't have access to the same data as you have. If you could share the data just for one FX pair you experimented with I could compare the results of applying ML to both FX and US stocks.
Sorry the data I have is proprietary (got it from Asirikuy) but you can get data from Dukascopy or from FXCM for free (Pipmeup posted some data). To answer your question I also do ML for stocks successfully, using similar techniques. In stocks you must be extra careful with mining bias because it's very easy to get things that work just by chance. Also when doing machine learning it is incredibly important to always retrain models on every trading decision using a moving window and avoid the classic academic mistake of doing a "training in sample" with a "testing out of sample" that stuff never works when you move to live trading. It is also important to always measure actual trading simulations, measuring success by "accuracy of predictions" is another classic pitfall academics tend to fall into.
  • #446
  • May 20, 2015 6:17am May 20, 2015 6:17am
Got it, algoTraderJo, thanks!
I will post the results here when it is done. Won't you mind if I will post in the thread some results for the stocks as well?
  • #447
  • May 20, 2015 6:55am May 20, 2015 6:55am
Quoting birdyalex
Got it, algoTraderJo, thanks! I will post the results here when it is done. Won't you mind if I will post in the thread some results for the stocks as well?
I would prefer it to keep this thread only about Forex. If you want open up another thread so that we can discuss your stock machine learning results.
  • #448
  • May 20, 2015 9:12am May 20, 2015 9:12am
Quoting algoTraderJo
Sorry the data I have is proprietary (got it from Asirikuy) but you can get data from Dukascopy or from FXCM for free (Pipmeup posted some data).
One more question for you if I can - did you convert somehow timestamps of Pipmeup's posted data or used it as is when you did a comparison in #436?
  • #449
  • May 20, 2015 9:25am May 20, 2015 9:25am
Quoting birdyalex
{quote} One more question for you if I can - did you convert somehow timestamps of Pipmeup's posted data or used it as is when you did a comparison in #436?
I always trade using GMT +1 non-DST/+2 DST timestamps so the data posted by Pipmeup was automatically corrected within the framework I use to match this setup (GMT +1/+2).
  • #450
  • Edited May 21, 2015 9:02am May 20, 2015 5:03pm | Edited May 21, 2015 9:02am
A very interesting thread, which I have just spent the last few hours reading. I particularly like the move to hourly bars to avoid the complications of being limited by the constraints of daily bars. Also, the idea of using excursions from the open very much mirrors what I have been working on over the last few months - however I've rendered the idea as an indicator; see my blog post at http://dekalogblog.blogspot.se/2014/...indicator.html for more details.

I've now subscribed to this thread and shall follow it with interest.
  • #451
  • May 21, 2015 3:54am May 21, 2015 3:54am
Datamining biais. I think it is time to talk about it.

I can now reproduce the experiment. KaBo will certainly do as well soon. But is this system a real one or a lucky realization? AlgoTrader said that he found A, B, C and D using a genetic algorithm. If each of these parameters are limited to a range of only 20 possible values there are 20^4 possible systems. Out of 160000 random systems it is almost certain to find a few ones generating fantastic returns(*). How can we check whether this system works or is a fluke?

If the expectation of a system is E any big enough subsample of the trades should have approximately the same expectation. In the simulation the long and the short trades have the same return (both participe for half of the result). I modified the simulation to open a position in only 25% of the cases. I ran 50 times the simulation. Note that the SL is trailed. In this modified version some trades will be made that the original strategy won't take because it is already in a trade. Out of the 50 simulations 85% were in profit. The 25% remaining were in loss smaller than 40%. But 40% is an normal DD for the system. Can I conclude this system is real?

(*) If 160000 people toss a coin 15 times approximately 5 of them will get only heads.
No greed. No fear. Just maths.
  • #452
  • May 21, 2015 9:38am May 21, 2015 9:38am
Quoting PipMeUp
Datamining biais. I think it is time to talk about it. I can now reproduce the experiment. KaBo will certainly do as well soon. But is this system a real one or a lucky realization? AlgoTrader said that he found A, B, C and D using a genetic algorithm. If each of these parameters are limited to a range of only 20 possible values there are 20^4 possible systems. Out of 160000 random systems it is almost certain to find a few ones generating fantastic returns(*). How can we check whether this system works or is a fluke? If the expectation of a system...
Conduct a White's Reality Check test and a Monte Carlo Permutation test, both of which are explained in this book - http://eu.wiley.com/WileyCDA/WileyTi...470008741.html and companion website here - http://www.evidencebasedta.com/

Basically what you would do is detrend and randomly re-order your time series data set and then take the "position vector" of your system ( in this case the positions indicated by the ML algo output ) and apply it to the re-ordered time series. Do this N ( large number ) times to build a null hypothesis distribution and compare your algo results on the real data to this distribution and apply your statistical test of choice e.g. p-value.

A more computer intensive version of the above would be to retrain the algo on each randomly re-ordered series to create a third null distribution - the distribution of spurious returns that the algo can give. We know that this is spurious because the random re-ordering deliberately breaks any coherence the underlying real data may have. This can be taken one step further by not even using the real data itself, but statistical measures of it. Take the mean, standard deviation etc. and create a completely hypothetical time series with the same attributes as the real data. Train the algo on N random realisations of this hypothetical distribution and compare the real results to this "obviously" spurious distribution. At a bare minimum the algo results on real data must be significantly better than those obtained on this distribution - if not, how can you possibly tell random results from real ones?
  • #453
  • May 21, 2015 10:08am May 21, 2015 10:08am
The following is what I was able to reproduce

  1. with 2pips one-way costs
  2. without reinvestment
  3. using PipMeUp data for E/U as is, i.e. without any timestamps conversions
  4. forecasting MLE/MSE using vanilla regression with intercept and open/open returns
  5. ignoring trading signals when one of forecasting values were below zero
  6. stoploss was set to 50% of 20day ATR
  7. risk per trade was set to 1%
  8. when the signal generated was the same as a current trade direction then stoploss / risk per trade has been reseted
  9. when the signal generated was in the opposite direction to current trade direction the position has been flipped
  10. maybe something else I forgot to mention...

Attached Image (click to enlarge)
Click to Enlarge Name: Screen Shot 2015-05-21 at 17.55.19.png Size: 103 KB


The best one in terms of Sharpe ratio that I was able to find using 3 generations of 10 size population for [A B C D] parameters from [ 0 5 3 3] to [23 25 15 20] using integer search and where B parameter was multiplied by 10 is the following one

Attached Image (click to enlarge)
Click to Enlarge Name: Screen Shot 2015-05-21 at 17.59.02.png Size: 79 KB


For the time being I not sure whether there is a bug in my code or maybe it all about GMT conversion or...
algoTraderJo what do you think about the results?
  • #454
  • May 21, 2015 11:11am May 21, 2015 11:11am
Quoting babelproofre
{quote} Conduct a White's Reality Check test and a Monte Carlo Permutation test, both of which are explained in this book - http://eu.wiley.com/WileyCDA/WileyTi...470008741.html and companion website here - http://www.evidencebasedta.com/ Basically what you would do is detrend and randomly re-order your time series data set and then take the "position vector" of your system ( in this case the positions indicated by the ML algo output ) and apply it to the re-ordered time series. Do this N ( large number ) times...
To be honest I don't understand the test you propose. If I detrend the dataset (EUR/USD returns you mean?) and shuffle them I'll simply get a random walk of some sort. If I apply the trades at the same points of time for the same duration and with the same lot size, I'll simply get a random outcome with expectation 0 and a variance proportional to the volatility of the market and the average duration of the trades. The performance of the backtest is so good that it will pass the test with a p-value way above 3. This test won't reject the result. Again, flipping a coin 15 times and getting 15 heads is an extremely unlikely event. But if 160000 people do it one of them will generate the event with a good probability. This system was selected from a set of systems of such an order of magnitude.
No greed. No fear. Just maths.
  • #455
  • May 21, 2015 11:15am May 21, 2015 11:15am
Quoting birdyalex
The following is what I was able to reproduce with 2pips one-way costs without reinvestment using PipMeUp data for E/U as is, i.e. without any timestamps conversions forecasting MLE/MSE using vanilla regression with intercept and open/open returns ignoring trading signals when one of forecasting values were below zero stoploss was set to 50% of 20day ATR risk per trade was set to 1% when the signal generated was the same as a current trade direction then stoploss / risk per trade has been reseted when the signal generated was in the opposite direction...
FYI when A=0 the spread is more around 7.
No greed. No fear. Just maths.
  • #456
  • May 21, 2015 1:10pm May 21, 2015 1:10pm
Quoting PipMeUp
{quote} FYI when A=0 the spread is more around 7.
thanks for your two cents. in that case the system turns to completely unprofitable system.
will try to find what could be wrong
many thanks PipMeUp!
  • #457
  • Edited 3:11pm May 21, 2015 2:45pm | Edited 3:11pm
Quoting PipMeUp
{quote} To be honest I don't understand the test you propose. If I detrend the dataset (EUR/USD returns you mean?) and shuffle them I'll simply get a random walk of some sort. If I apply the trades at the same points of time for the same duration and with the same lot size, I'll simply get a random outcome with expectation 0 and a variance proportional to the volatility of the market and the average duration of the trades. The performance of the backtest is so good that it will pass the test with a p-value way above 3. This test won't reject the...
Rather than go on a lengthy exposition about the tests, I would refer you to my earlier links and also add White's original paper, http://www.ssc.wisc.edu/~bhansen/718/White2000.pdf , and the R ttr-tests package http://cran.r-project.org/src/contrib/Archive/ttrTests/ which implements it as well as some other tests. I also have some zipped files with various papers that relate to these types of tests and also a zipped file that provides MATLAB/Octave code for them. I would be happy to let you have them but, for some reason, I cannot attach them to this post.

Note! In my earlier post the description I gave was more about the permutation test. White's test is sampling with replacement rather than re-ordering.
  • #458
  • May 21, 2015 3:08pm May 21, 2015 3:08pm
Quoting birdyalex
{quote} thanks for your two cents. in that case the system turns to completely unprofitable system. will try to find what could be wrong many thanks PipMeUp!
I think this is the reason
FXCM avg spread 2.4 pips <-- commission is included
Dukascopy avg spread 0.85 pips <-- commission is not included
No greed. No fear. Just maths.
  • #459
  • May 21, 2015 4:01pm May 21, 2015 4:01pm
Quoting babelproofre
{quote} Rather than go on a lengthy exposition about the tests, I would refer you to my earlier links and also add White's original paper, http://www.ssc.wisc.edu/~bhansen/718/White2000.pdf , and the R ttr-tests package http://cran.r-project.org/src/contrib/Archive/ttrTests/ which implements it as well as some other tests. I also have some zipped files with various papers that relate to these types of tests and also a zipped file that provides MATLAB/Octave code for them. I would be happy to let you have them but, for some...
This paper is incomprehensible to me. I thought West's notations were awful this guy beats him by a mile! Is there an unciphered version somewhere?

From the few I could get he proposes to test the hypothesis that the strategy performs no better than some benchmark like buy and hold or random. The problem I see is that if the strategy overfits the dataset or is just extremely lucky, the results will always beat 99.99% of the random benchmarks. How would be solved the problem of the 160000 people flipping 15 coins and one of them gets 15 heads? This is better than 99.997% of the other participants. How can I say that his coin is fair or weighted? His "strategy" has no free parameters. Perhaps he is able to do it again and again at will but you only have one successful sample at hand. Same question if I just tell you that someone flipped a coin 15 times head without telling you how many other people tried?

What about a genuine strategy that performs no better than half of the possible random entries?
Attached Image (click to enlarge)
Click to Enlarge Name: RandomWalkTrials.png Size: 174 KB


EDIT: in the picture I meant 35%
No greed. No fear. Just maths.
  • #460
  • May 21, 2015 4:40pm May 21, 2015 4:40pm
Quoting birdyalex
The following is what I was able to reproduce with 2pips one-way costs without reinvestment using PipMeUp data for E/U as is, i.e. without any timestamps conversions forecasting MLE/MSE using vanilla regression with intercept and open/open returns ignoring trading signals when one of forecasting values were below zero stoploss was set to 50% of 20day ATR risk per trade was set to 1% when the signal generated was the same as a current trade direction then stoploss / risk per trade has been reseted when the signal generated was in the opposite direction...
The GMT offsets and the DST are extremely important because market opening times and daily volatility cycles also change with DST. You need to correct your data to the proper timezone. Also use a higher spread as pipmeup said.
1 trader viewing now