Questionable Strategy Tester Results. | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Strategy Tester Results 4 replies

Why does strategy tester discard results? 3 replies

2009.03.11 15:41:15Tester memory handler: tester stopped because not enough memory 19 replies

Strategy Tester produces inconsistent results 0 replies

MT4 EA - strategy tester - backtest results inaccuracies 15 replies

Attachments: Questionable Strategy Tester Results.
Exit Attachments

Questionable Strategy Tester Results.

  • #1
  • First Post: Sep 21, 2006 12:10pm Sep 21, 2006 12:10pm
Attached are several backtest reports with astonishing reults over time ranges from 6 weeks to 5 years. The most astonishing is the 5 years gbpusd backtest.The EA uses money management to set lot size. On first long term backtest, it stopped in the middle due to excessive lot size. I contacted IBFX and was advised that they limit lot sizes to 50 for Mini and 100 for Standard accounts. I set max lot size to 90 and retested. On $1,000 deposit, end profit is over $17,000,000.

I tested on IBFX, FXDD and NF with similar results. I also test "every Tick" and "Control Point" models and again similar results.

Many distrust backtest results and this may be why. On 2006-09-19, I started EA on $5,000 demo on the 4 major pairs. So far, the account is down $453.

Any comments would be appreciated.

Wackena
Attached File(s)
File Type: zip StrategyTesterResults.zip   731 KB | 673 downloads
  • #2
  • Sep 21, 2006 12:35pm Sep 21, 2006 12:35pm
Quoting wackena
Attached are several backtest reports with astonishing reults over time ranges from 6 weeks to 5 years. The most astonishing is the 5 years gbpusd backtest.The EA uses money management to set lot size. On first long term backtest, it stopped in the middle due to excessive lot size. I contacted IBFX and was advised that they limit lot sizes to 50 for Mini and 100 for Standard accounts. I set max lot size to 90 and retested. On $1,000 deposit, end profit is over $17,000,000.

I tested on IBFX, FXDD and NF with similar results. I also test "every Tick" and "Control Point" models and again similar results.

Many distrust backtest results and this may be why. On 2006-09-19, I started EA on $5,000 demo on the 4 major pairs. So far, the account is down $453.

Any comments would be appreciated.

Wackena
Wackena,

A couple thoughts come to mind:

Did you include commissions, spreads, and slippage into your testing? In my experience that can account for up to 30% of your equity.

You can not run a 5 year test and then declare a system dead after 2 days. You simply don't have enough data to tell one way or the other.

While I didn't study every curve in detail, it looks to me like your strategy did better a couple years back than it does now. It may have already been fading a bit.

Your strategy was trading 100 lots in the end. I am assuming that you are trading less than that. I would suggest that you eliminate the position sizing algorithm and test a single contract. This will give you an idea of where the system is working and where it is not. Position sizing can do a wonderful job of helping you through poor performance but it's not useful in doing a raw evaluation of a strategy.

When done properly backtesting can tell you a lot about your strategy and the possibility of future performance.

Best wishes,
Phil
  • #3
  • Sep 21, 2006 12:53pm Sep 21, 2006 12:53pm
If you want to check the accuracy, go through the trades the EA made by hand on the chart and make sure everything is correct. In the end I would handtest the strategy, and like Phil said spreads can make a system that turns 1,000 into 1,000,000 almost worthless to trade. Im not saying you're not onto something, it might be a very good system, but nothing can replace handtested results!
  • #4
  • Sep 21, 2006 1:03pm Sep 21, 2006 1:03pm
Quoting philmcgrew
Wackena,

A couple thoughts come to mind:

Did you include commissions, spreads, and slippage into your testing? In my experience that can account for up to 30% of your equity.

You can not run a 5 year test and then declare a system dead after 2 days. You simply don't have enough data to tell one way or the other.

While I didn't study every curve in detail, it looks to me like your strategy did better a couple years back than it does now. It may have already been fading a bit.

Your strategy was trading 100 lots in the end. I am assuming that you are trading less than that. I would suggest that you eliminate the position sizing algorithm and test a single contract. This will give you an idea of where the system is working and where it is not. Position sizing can do a wonderful job of helping you through poor performance but it's not useful in doing a raw evaluation of a strategy.

When done properly backtesting can tell you a lot about your strategy and the possibility of future performance.

Best wishes,
Phil
Phil,
Thanks for the comments. The slippage is set to 3 in the EA. I didn't know how to set commissions and spreads in the Strategy Tester module. Please advise.

I agree on snap judgements on demo performance. Attached is Excel file that shows that over time there are good periods and losing periods. Even last month of Sept. has loss. click the Chart tab in the excel file.

I have tested fixed lost size and win/loss ratio is the same.

Again thanks,

Wackena
Attached File(s)
File Type: zip ST-excel charts..zip   391 KB | 554 downloads
  • #5
  • Sep 21, 2006 1:22pm Sep 21, 2006 1:22pm
Quoting wackena
Phil,
Thanks for the comments. The slippage is set to 3 in the EA. I didn't know how to set commissions and spreads in the Strategy Tester module. Please advise.

I agree on snap judgements on demo performance. Attached is Excel file that shows that over time there are good periods and losing periods. Even last month of Sept. has loss. click the Chart tab in the excel file.

I have tested fixed lost size and win/loss ratio is the same.

Again thanks,

Wackena
Wackena,

I am a TradeStation guy and have no idea how MT4 works.

Testing with fixed lots should not change the win/loss, it just reveals the true equity curve without the benefit of position sizing. Adding position sizing is the last thing you do before finalizing your testing and going live. A fixed lot equity curve would be more likely to reveal periods of poor performance and give you a more realistic expectaion of system performance.

Overall, I think it's great that you are going through this exercise and that you are asking these kinds of questions.
  • #6
  • Sep 21, 2006 1:54pm Sep 21, 2006 1:54pm
another thing to note, is that until the modeling quality says 90% your test is not accurate. the longer the timeline of you test the better its going to look because there is less and less data. this is especially true on interday trades.

you might look to importing accurate data into mt4
  • #7
  • Sep 21, 2006 2:14pm Sep 21, 2006 2:14pm
Quoting witchazel
another thing to note, is that until the modeling quality says 90% your test is not accurate. the longer the timeline of you test the better its going to look because there is less and less data. this is especially true on interday trades.

you might look to importing accurate data into mt4
I had the same thoughts. On FXDD demo, I downloaded new .hst files from http://www.alpari-idc.com/en/dc/databank.php and used Period Converter script to reset each timeframe. Same results and model quality. When using "Every Tick" model, what else can I do to improve Model Quality? Is this only based on .hst data quality? Is there some other site to download data to test on MT4 platform?

Thanks,
Wackena
  • #8
  • Jul 25, 2007 8:48am Jul 25, 2007 8:48am
I have better result about 20Mln for the last 2 months, check this post:
http://www.forexfactory.com/showthread.php?p=1507134
It open trades outside of candels !!!!

Quoting wackena
Attached are several backtest reports with astonishing reults over time ranges from 6 weeks to 5 years. The most astonishing is the 5 years gbpusd backtest.The EA uses money management to set lot size. On first long term backtest, it stopped in the middle due to excessive lot size. I contacted IBFX and was advised that they limit lot sizes to 50 for Mini and 100 for Standard accounts. I set max lot size to 90 and retested. On $1,000 deposit, end profit is over $17,000,000.

I tested on IBFX, FXDD and NF with similar results. I also test "every Tick" and "Control Point" models and again similar results.

Many distrust backtest results and this may be why. On 2006-09-19, I started EA on $5,000 demo on the 4 major pairs. So far, the account is down $453.

Any comments would be appreciated.

Wackena
  • #9
  • Last Post: Jul 25, 2007 9:19am Jul 25, 2007 9:19am
Wackena,

this kind of problem could happen for many reasons:
- quality of data
- if you scalp for a few pips; as the tick movement is only estimated in a M1 candle even at 90% modelling quality (but it seems it's not the case of this EA)
- related to the point above, usage of data inside M1 candle: I would limit the EA to use only the open or the close of the M1 candle.
- usage of wrong indicators: some indicators are wrongly coded and "know the future" so of course they are successful !
...

It is difficult to say just seeing your statement as it needs further investigation.
0 traders viewing now