What information is important in backtesting analysis? | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Backtesting programs 20 replies

Is backtesting necessary before Walk Forward Analysis? 7 replies

MT Information (Programming, Backtesting, Tips ...) 9 replies

Cannot get the spread information in backtesting 3 replies

Important Information for Germany 0 replies

Attachments: What information is important in backtesting analysis?
Exit Attachments

What information is important in backtesting analysis?

  • #1
  • First Post: Apr 15, 2014 5:58pm Apr 15, 2014 5:58pm
What information do you guys think is important when backtesting strategies?

I think most people have a tendency to look first at return, but I can see importance in managing drawdown risk and standard deviation. Is there anything else that you would look for in optimal backtesting of strategies?

I am sure what you look for differs in automated strategies vs manual strategies.
THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER.
  • #2
  • Apr 15, 2014 6:02pm Apr 15, 2014 6:02pm
In my opinion, the backtester should be used only to confirm that the strategy is trading according to rules (if you were talking about EA). The only valid way to test profitability of a strategy is forward on a Live environement.
MT4/MT5 EAs/Indicators/Alerts coder
  • #3
  • Apr 15, 2014 6:17pm Apr 15, 2014 6:17pm
Thanks for the response Ferru.

You don't believe in predictive financial modeling at all? I believe that backtesting strategies is simply a tool to analyze the future predictability of your strategy and not the answer. I still think it is a powerful tool. Obviously forward testing live is the only true test for system viability but all future data is to become past data so why discount historical data?

I do think backtesting should be used to verify that the system is working correctly but I also believe that it is a great tool to predict the future viability of a strategy.
THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER.
  • #4
  • Apr 15, 2014 7:07pm Apr 15, 2014 7:07pm
I look mostly at:

Total net profit / Maximal drawdown

The higher the value the better equity line looks.
"To hold, you must first open your hand. Let go." - Lao Tzu
  • #5
  • Apr 15, 2014 7:25pm Apr 15, 2014 7:25pm
Quoting emanontrader
Thanks for the response Ferru. You don't believe in predictive financial modeling at all? I believe that backtesting strategies is simply a tool to analyze the future predictability of your strategy and not the answer. I still think it is a powerful tool. Obviously forward testing live is the only true test for system viability but all future data is to become past data so why discount historical data? I do think backtesting should be used to verify that the system is working correctly but I also believe that it is a great tool to predict the future...
You can't predict future datas with past datas ... So your analysis of past datas would only tell you what your strategy would have given you in the past.

That's what I believe ...
MT4/MT5 EAs/Indicators/Alerts coder
  • #6
  • Apr 15, 2014 8:30pm Apr 15, 2014 8:30pm
I look at consistency of results.

By the way vox dei, maximizing total net profit / max drawdown will tend to give the best in sample results but the worst out of sample results (least generalizable) of any metrics I have tested. That good looking equity line on training data has a high probability of becoming a jagged, ugly equity line in live trading. The more that you optimize, the greater the chance of overfitting.
  • #7
  • Apr 15, 2014 8:38pm Apr 15, 2014 8:38pm
Quoting FerruFx
In my opinion, the backtester should be used only to confirm that the strategy is trading according to rules (if you were talking about EA). The only valid way to test profitability of a strategy is forward on a Live environement.

This is the only thing that the strategy tester should be used for.

Everything else is 100% worthless.

Test an EA for a Month and Backtest this same time frame.

See how worthless these actually are.

It's not that history doesn't predict the future, it is that the way these run, as pure of history and data as you can find will still not match.

If these actually worked, would anybody still be trading manually?
Money Can't Buy Happiness. Poverty Can't Buy SHIT! You Choose!
  • #8
  • Apr 15, 2014 8:43pm Apr 15, 2014 8:43pm
Quoting Magix
If these actually worked, would anybody still be trading manually?
Exactly. There are so many holy grail backtests around there ... Why develop new strategies !
MT4/MT5 EAs/Indicators/Alerts coder
  • #9
  • Apr 15, 2014 8:48pm Apr 15, 2014 8:48pm
Quoting FerruFx
{quote} Exactly. There are so many holy grail backtests around there ... Why develop new strategies !
Money Can't Buy Happiness. Poverty Can't Buy SHIT! You Choose!
  • #10
  • Apr 15, 2014 9:50pm Apr 15, 2014 9:50pm
Live vs Backtest.
Same EA, same Set file trading on the EXACT account that it is being tested...
Attached Image(s) (click to enlarge)
Click to Enlarge Name: a1.png Size: 43 KB
Click to Enlarge Name: a2.png Size: 35 KB
Money Can't Buy Happiness. Poverty Can't Buy SHIT! You Choose!
  • #11
  • Apr 15, 2014 11:07pm Apr 15, 2014 11:07pm
Quoting FXEZ
I look at consistency of results. By the way vox dei, maximizing total net profit / max drawdown will tend to give the best in sample results but the worst out of sample results (least generalizable) of any metrics I have tested. That good looking equity line on training data has a high probability of becoming a jagged, ugly equity line in live trading. The more that you optimize, the greater the chance of overfitting.
Thanks for your valuable output, FXEZ. It's good to know.

Btw, could you develop a bit on your process? TIA
"To hold, you must first open your hand. Let go." - Lao Tzu
  • #12
  • Apr 15, 2014 11:31pm Apr 15, 2014 11:31pm
Agree with FerruFx,

Also in forward testing spread and market conditions affect substantially on the behaviors of indicators used.

Would add, if back testing on MTF then it is not worth wasting time, in my opinion unless someone have a prove method to back test MTF

For example: If i am trading on 5M by analyzing 15m.

In back testing
It always shows "predictive future" which is actually "stored past" , as in back testing history of the bars for higher time are already stored from past.

In Forward Testing
If I receive a signal on 5M and and analyze 15M, I would be 1 to 3 Bars Lagging in my analysis. Thus, my forward test results would not be the same at all.
  • #13
  • Apr 15, 2014 11:47pm Apr 15, 2014 11:47pm
"In my opinion, the backtester should be used only to confirm that the strategy is trading according to rules (if you were talking about EA). The only valid way to test profitability of a strategy is forward on a Live environement."

You know I use to think that too but my research has lead me to believe that backtesting can be a very valuable tool and I mean literally. Most people underestimate the sheer amount of money that it takes to move liquid currencies like EURUSD even one pip, and the amount of money that exchanges hands every single day. Historical Data has all that information stored but it's almost impossible to get the information that is needed from it. When people trade "setups" they are actually trading imbalances in the market that they can identify, but a past bar has a lot more information than most people realize.

A past bar is a snapshot in time of the sheer volume of transactions and the outcome of those transactions, say for example the 4 hour bar that happen today, if the same bar happens again in the future and have the same OHLC and the same weighted ratio, with at least 70 percent chance the same outcome would almost certainly happen, just by the sheer volumes involve.

The hardest thing to do is to get the proper weighted ratio inside the candle, and by that I mean how much buying of AUD and selling of CHF and Buying of JPY and so on, even though it might be a EURUSD candle other currencies almost certainly played a role in either weakening the USD or strengthing the EUR, what happens in Vegas don't stay in Vegas in Forex. Although a candle might look the same they almost always have a different weighted ratio inside the candle and that is why a lot of times a perfect setup goes wrong because even though it looks the same it gives more or less pips. It's an incredible tool to have if you know how to get the information.
Being a trader is lonely, but being a great trader is lonelier still
  • #14
  • Apr 18, 2014 6:29am Apr 18, 2014 6:29am
Quoting vox dei
{quote} Thanks for your valuable output, FXEZ. It's good to know. Btw, could you develop a bit on your process? TIA
I consider consistency to be like waiting for a bus. If the bus is on time day after day, that is a consistent bus. If it is alternately early or late, then the bus is not very consistent. In like manner when profits from a given equity curve are consistent, profits arrive at regular intervals.

If I compare two equity curves and the first makes more $$$ but the 2nd makes a profit on more periods, then I would consider the 2nd system to be more consistent. When optimizing, often the higher profit systems "win" with statistics such as net profit, profit/drawdown, Sharpe ratio, profit factor, etc. I'm not so interested in the magnitude of profits when looking for consistency as I am in the number of periods that are profitable. If you divide up an equity curve into 10 or 20 "periods" you can then objectively identify over how many periods a given equity curve is profitable. If Equity curve A is profitable 13 of 20 periods and equity curve B is profitable 15 of 20 periods, then equity curve B is ahead in terms of consistency.

A given system will have "average" performance. During some periods, a given system may outperform its average performance by quite a lot due to market variance. When we maximize certain statistics (such as profit/dd) we tend to find those systems that just outperformed with nearly flawless equity curves. However, going forward they are very likely to revert to their average performance. This means they are likely to draw down to "catch up" with their lower true average performance line. The saying "trees don't grow to the sky" explains why these systems that previously outperformed tend to come back down to their true average performance.
  • #15
  • Apr 18, 2014 7:20am Apr 18, 2014 7:20am
Quoting FXEZ
{quote} I consider consistency to be like waiting for a bus. If the bus is on time day after day, that is a consistent bus. If it is alternately early or late, then the bus is not very consistent. In like manner when profits from a given equity curve are consistent, profits arrive at regular intervals. If I compare two equity curves and the first makes more $$$ but the 2nd makes a profit on more periods, then I would consider the 2nd system to be more consistent. When optimizing, often the higher profit systems "win" with statistics such as net...
Thank you for the thorough walk through your process
"To hold, you must first open your hand. Let go." - Lao Tzu
  • #16
  • Apr 19, 2014 2:18am Apr 19, 2014 2:18am
Another important factor to take into account when back testing is when there were major new announcements, this often seems to be forgotten, and there were some major events with regards to the EURO, all this is not seen on the charts.
  • #17
  • Last Post: Apr 19, 2014 6:11am Apr 19, 2014 6:11am
Quoting emanontrader
What information do you guys think is important when backtesting strategies? I think most people have a tendency to look first at return, but I can see importance in managing drawdown risk and standard deviation. Is there anything else that you would look for in optimal backtesting of strategies? I am sure what you look for differs in automated strategies vs manual strategies.
http://www.amazon.com/Trading-System...N2VHMNKB5FR0Q7
0 traders viewing now