Backtesting Discussions - I had a 99.72% success on a strategy. | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Strategy Backtesting Platforms 24 replies

Anybody had success with Renko and running an EA? 0 replies

Strategy backtesting - platform needed 0 replies

Cash & Carry (The "Perfect" Hedge) Open Thread Discussions 64 replies

Backtesting Strategy with S&R Levels 0 replies

Attachments: Backtesting Discussions - I had a 99.72% success on a strategy.
Exit Attachments

Backtesting Discussions - I had a 99.72% success on a strategy.

  • #1
  • First Post: Oct 19, 2006 8:26am Oct 19, 2006 8:26am
Hello Everyone,

I imagine you all are a bit familiar with the EA I'm testing, Flyer!. I've been continuing to make improvements and do further tests on those improvements. My discussion here though I want to be regarding the accuracy of Strategy Tester in MT4. The reason being is that after making an improvement I considered to be worthwhile on my EA, I achieved a 99.72% win ratio out of 362 trades from 2/15/06 to 10/13/06 on the GBP/USD.

It seems totally incredulous that any program could achieve this kind of accuracy. It took 1 loss out of that whole period for a drawdown of 0.91%.

It is run on the M1 timeframe which is supposed to be the most accurate setting available (even though the formula for the modelling quality on this setting gets capped at 25%).

So how reliable is a 99.7% success rate on Strategy Tester? Who else has experience with this kind of situation only to see the EA not perform as well on live trading? And furthermore, can you explain why in live trading the results were so much different... what factors did not occur in Strategy Tester that happened on live trading.

I am attaching the report from Strategy Tester so people can look at these results.

Eventually, I may perhaps release this version of my EA to the forum, but before I do this time I don't want to get overly excited about these results before I do more testing.

Mainly, I'm just looking for some thoughts and considerations to the strategy tester. I'm also a very skilled computer programmer and have considered even writing my own program to back test strategies so I can make sure the accuracy of what I'm getting is true. However, it is a time consuming process to do such a thing and I'm also spending a lot my time actually trading the markets.

Ok, there you have it. Thoughts and ideas are much appreciated.

GirlFlyer
Attached File(s)
File Type: zip GBPUSDTEST.zip   27 KB | 1,235 downloads
  • #2
  • Oct 19, 2006 8:42am Oct 19, 2006 8:42am
Quoting GirlFlyer
Hello Everyone,
It seems totally incredulous that any program could achieve this kind of accuracy. It took 1 loss out of that whole period for a drawdown of 0.91%.

It is run on the M1 timeframe which is supposed to be the most accurate setting available (even though the formula for the modelling quality on this setting gets capped at 25%).
You called out the results yourself, which is good. Forward testing always works better because you're letting the EA do what it's supposed to do, as it's supposed to happen. Certain things like requotes and slippages don't happen in back-testing. Also, backtesting removes all the variables such as what would happen to your margin levels or equity levels if you had the same EA running on several pairs like you want to? Backtesting can't represent that one time when all four trades collided together in the negative and almost crippled the account making your total results only about 40% of what the backtester told you it was, for only one pair. I know some of these seem like unlikely scenarios, but they still happen in forward time, and can't be properly backtested.

Backtesting tries to present the optimum circumstances when you really need to test for the worst circumstances.
  • #3
  • Oct 19, 2006 9:22am Oct 19, 2006 9:22am
I'd be willing to place a large bet that it is the modeling quality that's giving you the skewed results. I had an EA that I backtested on my IBFX Live account using 1M data (the EA trades on 30M timeframes). The backtester consistantly yielded ~98.7% w/l ratios, with the average win greater than the average loss. Needless to say it could turn $250 into millions and millions in a matter of months with only 28% drawdown. I even forward tested it for about two weeks. It produced about 6-7 trades. All of them were winners, but the amount won was less than the backtesting indicated it should have been.

The modeling quality was only around 50% though. I then tested it using Alpari data (making the modeling quality 90%), and the exact same EA with the exact same settings, over the exact same time period turned into a EA that was barely profitable. Instead of $250 -> millions and millions, it was more like $250 -> $287 in 9 months.
  • #4
  • Oct 19, 2006 9:56am Oct 19, 2006 9:56am
Quoting Loki
I'd be willing to place a large bet that it is the modeling quality that's giving you the skewed results. I had an EA that I backtested on my IBFX Live account using 1M data (the EA trades on 30M timeframes). The backtester consistantly yielded ~98.7% w/l ratios, with the average win greater than the average loss. Needless to say it could turn $250 into millions and millions in a matter of months with only 28% drawdown. I even forward tested it for about two weeks. It produced about 6-7 trades. All of them were winners, but the amount won was less than the backtesting indicated it should have been.

The modeling quality was only around 50% though. I then tested it using Alpari data (making the modeling quality 90%), and the exact same EA with the exact same settings, over the exact same time period turned into a EA that was barely profitable. Instead of $250 -> millions and millions, it was more like $250 -> $287 in 9 months.
Well, that's the strange thing and I understand you.... but according to MetaTrader, when you run a test on the M1 setting, you cannot get better than 25% modelling quality. Take a look at their webpage...

http://articles.mql4.com/93

I quote a portion of it here, but the rest of the article, explains it better.


**** Quoted from above reference


Data modelled on one-minute timeframes are rated very high: 90% of quality.

Reasoning from the same formula, the one-minute data modelled cannot be of quality exceeding 25 per cent since data of a smaller period are not used for modelling of these data. For client terminal, there are simply no data of a smaller period, but there are tick data. However, tick data are not stored in the client terminal by standard means.

In themselves, one-minute data are the most detailed data about the price movement. And, using them, one can model the price movement inside of bars of larger periods. At that, discrepancy between the real price movement and the ticks modelled will not be large due to presence of one-minute reference points: Open, High, Low, Close.

****** End Quote

So, by their own reference, data on the M1 is really 90% quality even though the formula only generates 25% by its very nature.

Also, I'm already using the Alpari Data, so again, I'm up to standard by everybody else.

But, you're right. Forward testing really is the only way to do it.

GirlFlyer
  • #5
  • Oct 19, 2006 10:23am Oct 19, 2006 10:23am
GirlFlyer, I guarantee you that it's not accurate.

Trust me, I know a few highly successful traders, all of them I consider pro, and they would be the first to tell you that if you can create a strategy that's 75%, heh, you need to do whatever it takes to find money to trade it...

I know a guy who has a strategy that has a higher % than 75%, and he's going to start a hedge fund and blackbox the strategy... so what does that tell you...
Mr. Trend
  • #6
  • Oct 19, 2006 5:00pm Oct 19, 2006 5:00pm
Hello Girlflyer,

I think the MT4 has some bugs in their internal calculation. (I am currently trading live in MT3 that gives all the correct I want. However my demo on MT4 from the same rule always gives slightly different trades. The only MT4 EA I made is just for the championship.) I haven't tested your EA so I won't say anything about your results. However I would give you some problems of my testing result on my own EA.

1. The hard-coded time-frame experiences: I coded the indicator on the Hourly (MODE_xxx as you know). However when I run backtest on different time frames, I got different results. As you stated correctly M1 timeframe give you 25% quality and other time frame give you 90% quality. I am very surprised. Then I "forward" tested the EA and found holy shit - the EA did trade differently on different timeframe I am using and then I backtested it again on those timeframes and found out that the trades weren't taken in the backtest for some reason I couldn't figure out. The ONLY trade that is accurately traded according to the rule are the one on the hourly chart and the backtest on the hourly timeframe showed the correct trades. (Therefore I think you should actually test on the timeframe that it is "supposed" to be trading in your codes).

2. longer term of trading: I noticed that you only got data from Feb 2006 but not earlier. The problem with that is that your strategy may not be able to survive with adjusted market condition. For my own example, one of my EA I developed can make 10x increase in equility within 3 months in 2004. However it basically make a standstill onwards. Therefore a change in market condition may actually make one's EA fail. That's why when I find a good EA, I put it in the trading LIVE ASAP. That's the only way to test it.

Conclusion: If you got good results in testing in M1 timeframe and that gives you the correct trade on your demo. My advise is to put it in the live testing in the M1 timeframe and make your million dollars!

Good luck
  • #7
  • Oct 19, 2006 5:04pm Oct 19, 2006 5:04pm
Hi GirlFlyer,

i think it's too soon to tell that your EA is working and successful,

We should go forward testing, give it 2 months, and collect the results, work out the 'black' area. And start to get rich.

2-months forward test results ? has anyone got it?
  • #8
  • Oct 19, 2006 5:12pm Oct 19, 2006 5:12pm
I'm not trying to be negative at all, if you guys can get EAs to trade effectively, go for it. I'll get in on it if it looks promising.

However, no computer reads the Wall Street Journal, no computer knows what people are thinking, and, believe it or not, this market cannot be figured out mathematically. There are too many variables that are not predictable. That's why human judgment will always be better than electronic evaluation.

Please, if you want to see your account increase, please, start reading the news, use longer time frames, watch other markets, like futures, equities, and commodities, because these are all inter-related.

I am not presumptuous, I just see a lot of people letting computers do what they ought to be doing themselves, and not doing it very well.
"The time to act is when others show signs of tire." --W.D. Gann
  • #9
  • Oct 19, 2006 5:46pm Oct 19, 2006 5:46pm
i have not had much luck with mt4 for backtesting, from what i can tell it is worthless, tradestation works alot better, even excel works better. i guess you do get what you pay for.

after i created the indicator for flyer i went ahead and tested it for awhile (foward) and i found it profitable with a expectancy ratio of about 0.109 (38% winrate , 1.81 payrate). While it is profitable, that ratio only lets me bet about 0.7% a trade with out huge risks of ruin and only provides a 8.2% chance of double your money after 1000 trades (and a 22.3% chance of loss). It does a great job of predicting the direction of the market, but it tends to retrace and eat up the profits.
  • #10
  • Edited 6:04pm Oct 19, 2006 6:01pm | Edited 6:04pm
Witchazel,

My point exactly. No matter how you try to automate this, please tell me who doubled their account with an EA? The perfect computer program would consolidate all the world's money in one account in a matter of days.

This market is based on two things, greed and fear. Until a computer can feel greed and fear, a computer can't trade effectively. Until you feel greed and fear, you can't control them. Until you control them, you can't capitalize on other people feeling them...
"The time to act is when others show signs of tire." --W.D. Gann
  • #11
  • Oct 19, 2006 6:39pm Oct 19, 2006 6:39pm
there are many EA's that you can double your money in 1000 trades. but 1000 trades is a very long time
  • #12
  • Oct 19, 2006 7:04pm Oct 19, 2006 7:04pm
GirlFlyer,

my V105A doesnt has ' GapSize=0.0014, FlatMarketSize=0.0015' parameters, why?

Chad
  • #13
  • Oct 19, 2006 7:18pm Oct 19, 2006 7:18pm
Quoting chad
GirlFlyer,

my V105A doesnt has ' GapSize=0.0014, FlatMarketSize=0.0015' parameters, why?

Chad
This is my own private test version that I'm running, that's why you don't have it yet.
  • #14
  • Oct 19, 2006 7:27pm Oct 19, 2006 7:27pm
Quoting dukeofdents
I'm not trying to be negative at all, if you guys can get EAs to trade effectively, go for it. I'll get in on it if it looks promising.

However, no computer reads the Wall Street Journal, no computer knows what people are thinking, and, believe it or not, this market cannot be figured out mathematically. There are too many variables that are not predictable. That's why human judgment will always be better than electronic evaluation.

Please, if you want to see your account increase, please, start reading the news, use longer time frames, watch other markets, like futures, equities, and commodities, because these are all inter-related.

I am not presumptuous, I just see a lot of people letting computers do what they ought to be doing themselves, and not doing it very well.
Hi Duke,

I agree with you on most parts. Even though I'm working on this EA and studying everything, this has just been a project I started to help my understanding of the market better and test my own strategies. EA's are what they need to be "Advisors"... They can help you make decisions, but are not the ultimate answer, at least not in the stage of the development process we have them in today.

On the point of mathematics though, everything can be defined mathematically. The universe is founded in mathematics and so is the market. The problem is finding the equation that fits the situation and in this case the equation of the market is one that is ever changing and ever evolving, but it has one. It was said once before that a computer could never beat a human grand master at chess, well... guess what, it was finally done. Eventually someday, we may find that a computer algorithm may approximate the performance of good old human intuition and understanding.

Until then, let's all keep doing our research and work, and we'll make it there!

Happy Trading
  • #15
  • Oct 19, 2006 8:01pm Oct 19, 2006 8:01pm
No one can predict which the direction the market is going to go in any given moment and no one can predict where the market will go on a longer timeframe. All we have are edges.

When you have an edge that has a greater than 50% probablility of success you trade it with proper money management EVERY time it throws a signal. The reason we don't trade when we should and trade when we shouldn't and sabotage our chances of success is because we THINK we can predict the outcome.

For example - "The last three trades worked this way exactly, so this one will too." So we risk 10% instead of 1% and get burned. Then the next time we see the setup we remember the painful loss when we were wrong so we hesitate and miss the opportunity.

EA's are great tools! Written correctly -- an EA is the equivalent of a slot machine in Vegas. It works all the time for us with no emotion. It doesn't have to make huge money in a short period of time. All it has to do is use an edge to make small consistent profits.

A little encouragement for those who believe that EA's can be a valuable trading tool.

Cheers,
Jeff
  • #16
  • Edited Oct 20, 2006 12:24am Oct 19, 2006 10:19pm | Edited Oct 20, 2006 12:24am
There are a lot of questions about the effectiveness of EA's and what they can do. My feelings are these:

1. I look at the current automated championship going on and the whole first page is complete with success stories of EA's. This is live, forward trading with posted results. http://championship.mql4.com/2006/pa...ndex/1/-equity Just think of all the people not in the championship with effective EA's.

2. Yes, fear and gread are the components of a market. News is also. But these factors are built into the price: how fast a market moves, where it moves, how fast it moves back, etc. That little tick and where it has been, and how fast it has moved, has all these factors built in, and thus can be measured quantitatively. Thats all the market is: a quantitative expression of emotion. (Unless of course the government intercedes-which is still emotion-just at a more powerful level)

My opinion is the benefit of an EA taking out the emotions of trading far outweigh its limitations. They have to be constantly tweaked, yes. But there are certain laws within the maket that can be predicted.

It seems like every book I have read about trading say the same thing: Have a system, trade it to a fault, and leave your emotions at the door. Sounds like an EA to me. I think it can be done, I think it is being done.

I also agree the Universe was founded by mathematics. There is no "Holy Grail" mathmatical formula for the market because the maket is made of people- everyone of which are imperfect and behave irrational at times. This is how we are created. So the maket will behave in an imperfect, unpredictable manner at times. But we are creatures of habit, also, which can be predicted much of the time.
"Its not where your at, but where your coming from"
  • #17
  • Sep 7, 2008 1:53pm Sep 7, 2008 1:53pm
You must have gone in the attic of FF to find this thread, its two years old.
  • #18
  • Sep 7, 2008 5:02pm Sep 7, 2008 5:02pm
wish what u r saying is true caz i think that would never happen
{Promotion Removed}
  • #19
  • Last Post: Sep 9, 2008 6:12am Sep 9, 2008 6:12am
where is yr system i was reading the whole post but nothing here viewing yr system posts here sre for sharing systems not just taking about imaginary systems.by the way nice site i like the google add sense which in i i hope the readers clicks from here bring u extra more money ,huh is this all what u want ff
{Promotion Removed}
0 traders viewing now