Backtesting discussion | Page 3 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Backtesting programs 20 replies

Best backtesting Period 3 replies

good backtesting software 5 replies

Custom backtesting code 9 replies

Backtesting vs. Thought Process 10 replies

Attachments: Backtesting discussion
Exit Attachments

Backtesting discussion

  • #41
  • Feb 28, 2016 6:06am Feb 28, 2016 6:06am
Quoting NaughtyPip
Guys I have a question regarding back testing. (Sorry to butt in on the thread) I am working on a mechanical trading method that has produced fair results over 300 samples. I want to have this coded and then run more extensive back tests to take out the leg work.. and then of course forward test it. The question is would it be possible that your broker or meta quotes notice if you were using a profitable ea and subsequently, retrieve the ea from your computer for themselves? Or should I take off my tin foil hat?
When it all boils down to it, protecting your code is a pretty difficult exercise. For example exe decompilers and brokers can access your system in MT4. Consider using external offline systems using API interfaces with the broker if you want more solid protection but just your trade history alone could be reverse engineered by those that really want to find out how your system ticks.

The question is, will your system amongst the thousands of others warrant particular attention? If you just stay under the radar, there is probably a good probability that your system will remain confidential to you. There would be EA developers on this forum and those with direct experience in the brokerage houses on this forum that would be able to give you solid advice. Ask around and hopefully someone in the know will pop up.
  • #42
  • Feb 28, 2016 8:03am Feb 28, 2016 8:03am
Quoting Ffl
Do you consider that the system is working or just lucky month??? )
First, I think it is better test the system in a timeframe of 2 years. And the system should have an edge and a positive expectancy over long time.

Secondly, back testing involves no money, therefore no pressure when you have some losses, and you won't fear for profits and hope for book loss to be gone.

I think using a live account with some small money like $200 to test and practice should be better.
  • #43
  • Feb 28, 2016 5:46pm Feb 28, 2016 5:46pm
Quoting NaughtyPip
Guys I have a question regarding back testing. (Sorry to butt in on the thread) I am working on a mechanical trading method that has produced fair results over 300 samples. I want to have this coded and then run more extensive back tests to take out the leg work.. and then of course forward test it. The question is would it be possible that your broker or meta quotes notice if you were using a profitable ea and subsequently, retrieve the ea from your computer for themselves? Or should I take off my tin foil hat?
How naughty of you to suggest that a broker might do something like this! :-)

But really, is it such a big deal if someone copies what you have made; unless of course you intend to sell it as a commercial product.

As already answered, your EA can be copied. To me the more important thing is whether a broker can make a successful EA perform badly. This seems to have happened when Million Dollar Pips (MDP) became so successful in 2011. I made $60K with this EA in just 3 months running it on 3 different brokers. As I am sure did many others. It works by taking limit orders on a price spike and trading the price back just like a rubber band does after being stretched. By the end of 2011 this EA was no longer profitable. The elasticity seems to have gone out of the spikes on MT4. I wonder why (not much) and how that was done.
  • #44
  • Feb 28, 2016 9:07pm Feb 28, 2016 9:07pm
There is another element to backtesting that hasn't yet been discussed, and that it the platform used to carry out the testing.

Many here speak of MT4 Strategy Tester and good quality Tick data, but if you study the way Strategy Tester works you will be disappointed to find that those ticks you searched for, for so long, aren't actually used during backtesting.

MT4 only uses the OHLC data of any TimeFrame used, and then simulates the tick sequence when painting a bar. If you run your backtest on M1 all the ticks are simulated. The reason being that MT4 doesn't keep tick data in its history, only the OHLC.

Moving to M5 does the same, but also uses the M1 data to at least align price every minute. Moving to higher timeframes does provide addition actual prices for each OHLC bar contained therein, but the randomness of ticks prevails between these prices.

So if you are trying to scalp using an EA, the results you see in backtest are not much better than random.

Backtesting on short TFs will aid in coding by checking your logic and ensuring the proper sequence of steps is taken, but don't expect to see realistic profit results.

If on the other hand you code your EA to only look at prices at the open or close of a bar then you stand a chance of realising your equity curve when you go live, as the BackTesting is using real data (the open or the close) rather than the simulated ticks.

But watch out for a few other traps:
1. if you place pending orders, Strategy Tester will fill you at your nominated price, but in reality price could jump past.
2. similarly Stops will be realised as set and not take account of tick moves and jumps in price, or not be seen at all as the simulated ticks dont reach your stop level, when in reality they might have.

So if you want to use MT4, I suggest you write your code to only make logical decisions on the open of every M1 (or higher TF) bar. Don't use pending or stop loss orders, but open or close trades based on the open price only.

By all means have hard stops in place but have the EA close any orders at the open.

If all else fails then try testing using a different platform - no recommendations here, but would be interested in what others are using.

If you truly must test off tick data then create your own test platform using a spreadsheet and do your code iterations as each new tick arrives.
" check out The Traders Outpost "
  • #45
  • Feb 28, 2016 9:19pm Feb 28, 2016 9:19pm
Quoting RedLineFred
There is another element to backtesting that hasn't yet been discussed, and that it the platform used to carry out the testing. Many here speak of MT4 Strategy Tester and good quality Tick data, but if you study the way Strategy Tester works you will be disappointed to find that those ticks you searched for, for so long, aren't actually used during backtesting. MT4 only uses the OHLC data of any TimeFrame used, and then simulates the tick sequence when painting a bar. If you run your backtest on M1 all the ticks are simulated. The reason being that...
Great info Fred.

Cheers
  • #46
  • Feb 28, 2016 9:27pm Feb 28, 2016 9:27pm
Quoting Copernicus
{quote} Great info Fred. Cheers
Very interesting. Could you explain why we should use the Open rather than the Close of the bar?
  • #47
  • Edited Feb 29, 2016 8:24pm Feb 28, 2016 9:45pm | Edited Feb 29, 2016 8:24pm
Quoting sydap2003
{quote} Very interesting. Could you explain why we should use the Open rather than the Close of the bar?
Best for Fred to explain however IMO it fast-tracks backtest processing without compromising accuracy by using the MT4 strategy tester option of 'open prices only' provided you seperate the signal bar from the trade execution bar. Having a signal bar and a trade execution undertaken on the same bar can run you into trouble given the random in-fill between OHLC. I remember a few times I would get miracle equity curves when signals and execution points were placed within the intervals between OHLC.

It's also particularly embarrassing when using 'control points' you produce a stellar equity curve and then find that the reason for the result was that the spread spanned the range of the bar incorporating signals and trade executions. You might find a few of these equity curves displayed on FF . For example below is an image of the equity curves from a bogus attempt using control points that lie within the spread and the chart to the right is the reality.

Attached Image


PS If you still want to use MT4 as your back-test solution for short range strategies such as scalping strategies and insist on tick data as a solution, then it's worthwhile having a look at Birts Tick Data Suite to improve back-test reliability but it limits you to those brokers that provide tick data such as Dukascopy....but it is a lot of effort when there are more robust dedicated backtest solutions around.
  • #48
  • Feb 29, 2016 1:24am Feb 29, 2016 1:24am
Quoting sydap2003
{quote} Very interesting. Could you explain why we should use the Open rather than the Close of the bar?
You could use either but it made more sense to me to use the open as it gives me one bar to execute the trade logic of the EA before a new bar comes along.
Using the close leaves you hanging, if for example, you want to trade the close of the Friday daily bar as there is no time left to trade.
" check out The Traders Outpost "
  • #49
  • Feb 29, 2016 2:48am Feb 29, 2016 2:48am
Quoting RedLineFred
would be interested in what others are using.
Hi Fred,

Been using Forex Tester for a while now and like it a lot. Simple things like having full control over auto-scrolling speed and pause/start/stop for instance has made all the difference to my style of backtesting, which is to generate signals off of candle closes and always open trades at market.
  • #50
  • Feb 29, 2016 11:42am Feb 29, 2016 11:42am
Quoting sydap2003
{quote} How naughty of you to suggest that a broker might do something like this! :-) But really, is it such a big deal if someone copies what you have made; unless of course you intend to sell it as a commercial product. As already answered, your EA can be copied. To me the more important thing is whether a broker can make a successful EA perform badly. This seems to have happened when Million Dollar Pips (MDP) became so successful in 2011. I made $60K with this EA in just 3 months running it on 3 different brokers. As I am sure did many others....
Hey syday2003, are you asking why it stopped working?
  • #51
  • Feb 29, 2016 4:37pm Feb 29, 2016 4:37pm
Quoting NaughtyPip
{quote} Hey syday2003, are you asking why it stopped working?
Yes I would love to have a definitive answer from someone in the know.
  • #52
  • Mar 1, 2016 8:18am Mar 1, 2016 8:18am
To echo what's already said, 1500+ out-of-sample trades get's you to statistical significance. OOS can be fiddled with though which will ruin your results. This makes walk-forward more meaningful but less practical. Who has time to walk forward 1500 sample in real time? There is where Monte-Carlo Analysis can add some insight.
When you hit 50, coffee becomes part of a proper modeling pipeline.
  • #53
  • Edited Mar 2, 2016 8:14am Mar 1, 2016 8:48am | Edited Mar 2, 2016 8:14am
Quoting TheFxAnalyst
1500+ out-of-sample trades get's you to statistical significance.
There is no statistically signicant number of trades.

Certainly, all things equal, a higher number of trades in a test is preferable. However, when considered irrespective of the length of the test period, more trades do not lend more significance to results.

100 trades made over a ten year test period is most certainly more meaningful than 2000 trades made over a 1 year test period.
  • #54
  • Mar 1, 2016 9:52am Mar 1, 2016 9:52am
Hi everyone!

Happy to see that this thread got so much reactions and help
I got one little question about my own backtesting,
if i'm going live, How long does a normal latancy take in time? because i never experienced latency am i a bit curious if it's a big problem for my live experience!

Have a great day everyone!
Frank
  • #55
  • Mar 1, 2016 12:40pm Mar 1, 2016 12:40pm
Quoting Ffl
Hi everyone! I wanted to start my first thread about the question that i'm facing right now, The question after how long do you think a ''system'' is worth calling a system. How long backtesting do you consider necessarily before you decide if the system WORKS To place this to my own perspective, i've been working for months on my own way of trading and i finally seem to get somewhere, i've been backtesting for one month straight and picked the days that seemed the best for the testing (all in this month!!!) with a in my opinion good result for...
I am not an experience traders, but I will be confident if a strategy give me at least 2 wons and 1 loss in a 3 trading position, and that strategy is work for a month or more.
At least, a month with demo account for me, then I go to cent account.
  • #56
  • Mar 1, 2016 4:40pm Mar 1, 2016 4:40pm
Quoting sydap2003
{quote} Yes I would love to have a definitive answer from someone in the know.
Well the logical explanation would be that once enough money is pumped into the market to take advantage of a particular loophole, the loophole disappears. Which is why you want as little people as possible knowing about your loophole.
  • #57
  • Mar 1, 2016 8:43pm Mar 1, 2016 8:43pm
Usually 1-2 months is enough.

However it's hard to say, I have seen backtests for 10 years with good tick data, and while profitable in 9 years, the results were bad in one ENTIRE year long, only to be profitable again after that.
  • #58
  • Mar 2, 2016 8:13am Mar 2, 2016 8:13am
Quoting AcerPhoon
Usually 1-2 months is enough. However it's hard to say, I have seen backtests for 10 years with good tick data, and while profitable in 9 years, the results were bad in one ENTIRE year long, only to be profitable again after that.
Depending on your research goal, any test period can be adequate.

If you want to observe how a strategy fares across varied market conditions then the longer the better. If you wish to jump straight into trading with reckless abandon then a small test period is enough to placate your better judgment.
  • #59
  • Mar 6, 2016 7:01pm Mar 6, 2016 7:01pm
This has worked out to be a very useful thread on back testing. I have some data related to some of the issues/questions raised.

By way of background I prepared 9 SET files using FX Autotrader Elite with 2015 tick data. Each has over 200% return for 2015. I have been running these on demo since 1 January 2016 and since then have also looked at their 5-year back test results. Results for some of these are shown in the attached. They deal with two issues.

1. Over what period should we optimise a new strategy?
The data shows that if we optimise over one year (2015 in my case), then some strategies pass the five year back test; others do not. We can re-optimise the strategy using tick data from a "down" year and then re-run and we can generally achieve better results when using these re-optimised strategies over the 5-year back test period. That gives me more confidence that these strategies will perform profitably going forward.

2. Does the back test give the same result as a forward test over the same period?
In the example shown on the attached, yes it does in the sense that equity curves for the two tests are comparable over 8 weeks, so in this case we pass that test. The forward test on demo actually was a little more profitable, and I believe that that was due to using a conservative (largish) spread for back testing. Better to be pleasantly surprised that rudely awakened. In all, the three that were positive in back test were also profitable in forward demo test over same period; and the six that were negative (in DD) in back test also lost on forward demo.

I have only just started trading the currently profitable SET files (three out of 9) on a live account, so insufficient data as yet to see how live compares to demo.
Attached File(s)
File Type: docx Compare Back Test Periods.docx   1.2 MB | 267 downloads
  • #60
  • Mar 6, 2016 9:03pm Mar 6, 2016 9:03pm
Quoting sydap2003
This has worked out to be a very useful thread on back testing. I have some data related to some of the issues/questions raised. By way of background I prepared 9 SET files using FX Autotrader Elite with 2015 tick data. Each has over 200% return for 2015. I have been running these on demo since 1 January 2016 and since then have also looked at their 5-year back test results. Results for some of these are shown in the attached. They deal with two issues. 1. Over what period should we optimise a new strategy? The data shows that if we optimise over...
Nice stuff. Great to see some robustness testing. :-)
0 traders viewing now