Optimization results question | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Fast optimization by indicators ranges. 1K results for 1 sec. 0 replies

Getting empty optimization results from EA backtest 4 replies

Getting more detailed MT4 optimization results 1 reply

1 passes done during optimization results discarded insignificant. 10 replies

Metatrader Optimization produce random results 0 replies

Optimization results question

  • #1
  • First Post: Sep 11, 2008 9:43am Sep 11, 2008 9:43am
In optimizing an EA, which result would you consider the most important? Profit, profit factor, drawdown, etc.?

and why?
  • #2
  • Sep 12, 2008 12:10pm Sep 12, 2008 12:10pm
for me, i just look at profit $. the other measures just come out proportional to profit $, and drawdown is just inverse generally speaking.
  • #3
  • Sep 12, 2008 7:44pm Sep 12, 2008 7:44pm
thanks.

Quoting rael
for me, i just look at profit $. the other measures just come out proportional to profit $, and drawdown is just inverse generally speaking.
  • #4
  • Sep 12, 2008 8:16pm Sep 12, 2008 8:16pm
Return %, drawdown, the distribution of the profits over time, "smoothness" of the equity curve, number of trades, estimated transaction costs (average spread * number of trades), averaged profit per trade, averaged profit per successful trade, number of successful trades versus number of unsuccessful trades, stuff along those lines.
  • #5
  • Sep 12, 2008 8:38pm Sep 12, 2008 8:38pm
Ummmm, ok. I take it you take all of those into consideration when evaluating results.

Quoting Rabid
Return %, drawdown, the distribution of the profits over time, "smoothness" of the equity curve, number of trades, estimated transaction costs (average spread * number of trades), averaged profit per trade, averaged profit per successful trade, number of successful trades versus number of unsuccessful trades, stuff along those lines.
  • #6
  • Sep 12, 2008 8:52pm Sep 12, 2008 8:52pm
the #1 most important factor: the modeling quality! if trades are being scalped for TPs and SLs of less than 15 pips or so (depending on the pair) a 99% MQ is critical! otherwise, M1 (90%) is usally sufficient....

the #2 most important factor: THE NUMBER OF TRADES! also, the more years in the backtest, the better...... allowing for assessment of the performance in several types of market conditions....

even this doesn't guarantee anything.... see my "Statistical analysis fallacy" thread.
  • #7
  • Sep 12, 2008 8:58pm Sep 12, 2008 8:58pm
Is that thread here on FF?

Quoting tdion
the #1 most important factor: the modeling quality! if trades are being scalped for TPs and SLs of less than 15 pips or so (depending on the pair) a 99% MQ is critical! otherwise, M1 (90%) is usally sufficient....

the #2 most important factor: THE NUMBER OF TRADES! also, the more years in the backtest, the better...... allowing for assessment of the performance in several types of market conditions....

even this doesn't guarantee anything.... see my "Statistical analysis fallacy" thread.
  • #8
  • Sep 12, 2008 9:30pm Sep 12, 2008 9:30pm
Quote
Ummmm, ok. I take it you take all of those into consideration when evaluating results.

Yes, and more.

Return % is obvious.
Drawdown is obvious.

Google "Sharpe Ratio" when you have time.

Distribution of profits over time is less obvious, but more important. Are all of your profits happening because of 1 or 2 trades? Or is it a lot of little profits? This matters because if too much of your profit is dependant on a small set of trades you might just be getting lucky, and different conditions might end up making the system unprofitable.

"Smoothness" of the equity curve is a way of saying % of a drawdown compared to return. If you have large bursts of profit then large drawdowns you're going to see the above problems, this is just another way of recognizing the above.

Estimated transaction costs are very important. I have a system that's profitable 90% of the time, but it only generates 5 pips per trade and relies on hundreds of trades. So try that on the GBP/USD and you'll be lucky to see a single pip profit per trade. Point being, I don't use the system because it's merely working for the broker. Goes to the same point as "number of trades" ... the system may be over-trading.

Average profit per trade, successful and combined, goes to all of the above. Number of successful trades versus unsuccessful gives you a way to determine expectancy. A higher expectancy system usually ends up with a lower return ratio, and a higher return ratio usually lowers your expectancy. You should have a firm idea of how your system performs on both.

Test different situations here and see how it impacts your drawdown, trade distribution and returns.

I have a list of approx 100 items I check during backtests. You'll probably want to start researching your own list.

Quote
the #1 most important factor: the modeling quality! if trades are being scalped for TPs and SLs of less than 15 pips or so (depending on the pair) a 99% MQ is critical! otherwise, M1 (90%) is usally sufficient....

Nod. If you're trading for small time frames this is going to be brutally important. If you're trading only on daily data, and only on-close of the day, it's a lot less.

Quote
Is that thread here on FF?

If you plan to have a reliable trading system you'll want to start doing your homework. =)
  • #9
  • Sep 12, 2008 9:38pm Sep 12, 2008 9:38pm
yes

Quoting nondisclosure00
Is that thread here on FF?
  • #10
  • Sep 12, 2008 9:49pm Sep 12, 2008 9:49pm
Drawdown % is very important. Many systems perform profitably but have so much drawdown that they are untradeable at the margin level you expect / desire. And I don't think MT4 computes drawdown correctly - or margin used and things like margin calls.

Profit factor is also important - and a high # of trades and average profit per trade. I don't do system testing in MT4 so I don't run into some of its limitations that were mentioned by others in this thread.

I will just say that optimizing a system to obtain better report statistics is generally the wrong thing to do. It typically means the system isn't that robust and so you're looking for a set of elusive parameters that performs "best". I've never had success with the systems I've produced using that basic development strategy.

However, if after you are done developing your system, you decide to optimize your money management strategy (and by that I mean position sizing not entry/exit) to best fit your risk profile, then I'd say that's a good use for optimization.

But running mindless optimization on system parameters and picking the system that performs best according to the stats of your choice, isn't the way to go IMO.
  • #11
  • Sep 12, 2008 10:26pm Sep 12, 2008 10:26pm
drawdown % is a matter of philosophy. personally, i accept a complete wipeout of my account as a risk in order to achieve unparalleled returns....

that doesn't mean i put all my investment capital into my strategy....

it means that forex is what you make of it.
  • #12
  • Sep 14, 2008 5:34pm Sep 14, 2008 5:34pm
Quoting tdion
drawdown % is a matter of philosophy. personally, i accept a complete wipeout of my account as a risk in order to achieve unparalleled returns....

that doesn't mean i put all my investment capital into my strategy....

it means that forex is what you make of it.
I agree! It is what you make of it! And I'm also convinced that no matter how well a system performs in the past that the chance of total wipeout as you so aptly put it, is still a possibility. And so I'm developing systems on both fronts or with both philosophies - steady growth and go go growth with higher risk of implosion. An understanding of drawdown is important for both strategies IMO.

For the traditional approach, drawdown is used to measure possible future risk so that positions may be sized to encapsulate the desired risk. And a way to know when the system stops working is when historical drawdown levels are breached.

For the go go growth approach, drawdown still needs to be within a certain limit *most of the time*. The understanding is that eventually the account will implode if it is pushed far enough. But hopefully you will have taken out so much $$$ due to the hyper aggressive nature of the strategy and / or money management (position sizing / progression) being used that it won't matter. Like starting up 5 $500 accounts and hoping 1-2 of them go up 10x before all of them have blown. When the desired profit benchmark is reached, you pull out the funds (according to your plan) and start over rather than let it ride, because it will blow up eventually. In this case blowout is just part of the process and not a total defeat as in the traditional approach.

My own philosophy is that yes - eventually just about any system will implode or cause a margin call on an account, if that system is traded aggressively. And if you're going to make real $$ trading you need to make your $$ when the system is performing well - meaning you push it when it is working. So the way to approach this is to expect eventual blowout and withdraw funds at certain benchmarks. That way you're always ahead in the game as long as you don't blow out right at first. And you have a strategy of starting small and reloading to mitigate that. When the account does blow, it's of small import because the startup capital is small, unlike the traditional approach with smaller % returns and larger capital requirements. So I think we're somewhat on the same page if I'm interpreting your philosophy correctly!
  • #13
  • Last Post: Aug 7, 2009 6:45am Aug 7, 2009 6:45am
Quoting tdion
the #1 most important factor: the modeling quality! if trades are being scalped for TPs and SLs of less than 15 pips or so (depending on the pair) a 99% MQ is critical! otherwise, M1 (90%) is usally sufficient....

the #2 most important factor: THE NUMBER OF TRADES! also, the more years in the backtest, the better...... allowing for assessment of the performance in several types of market conditions....

even this doesn't guarantee anything.... see my "Statistical analysis fallacy" thread.
How to get 99% modeling quality ? And is 90% MQ in M1 even possible ?
0 traders viewing now