Statistical analysis fallacy | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Technical Analysis Fallacy 63,568 replies

The Ultimate Fallacy in Technical Analysis 99 replies

"Technical Analysis Fallacy" thread people, I need your help 54 replies

Technical Analysis Fallacy Redux 23 replies

Attachments: Statistical analysis fallacy
Exit Attachments

Statistical analysis fallacy

  • #1
  • First Post: Edited Sep 8, 2008 1:09am Sep 7, 2008 8:41pm | Edited Sep 8, 2008 1:09am
Below is a graph of trades for the EURUSD over the past 8 years. One trade per day, based on the direction of an MA cross. (Remember, all the "experts" say trading dailys is "easier" )

What is the point?

The first half of the graph shows modest profit over the first 900+ trades.... the remaining 2/3 you would have lost your A$$.

HAHA all you forex statisticians out there.... explain WHY the algorithm failed even though the huge population of the first few years leads you to conclude (statistically) that this is a profitable/breakeven strategy.


WHAT IS THE BIG PICTURE OF THIS THREAD?????

It doesn't matter how many trades you try to base your kelly criteria for your setups on.... the market WILL CHANGE so you can't really conclude anything.
Attached Image (click to enlarge)
Click to Enlarge Name: TesterGraph.gif Size: 11 KB
  • #2
  • Sep 7, 2008 10:22pm Sep 7, 2008 10:22pm
Quoting tdion
Below is a graph of trades for the EURUSD over the past 8 years. One trade per day, based on the direction of an MA cross. (Remember, all the "experts" say trading dailys is "easier" )

What is the point?

The first half of the graph shows modest profit over the first 900+ trades.... the remaining 2/3 you would have lost your A$$.

HAHA all you idiot forex statisticians out there.... explain WHY the algorithm failed even though the huge population of the first few years leads you to conclude (statistically) that this is a breakeven strategy.


WHAT IS THE BIG PICTURE OF THIS THREAD?????

It doesn't matter how many trades you try to base your kelly criteria for your setups on.... the market WILL CHANGE so you can't really conclude anything.
Were all doomed?

Leon
Living the adventure in my head.
  • #3
  • Sep 7, 2008 10:39pm Sep 7, 2008 10:39pm
no.

it will behoove you to concede that luck plays a part of your success or failure, if you can't explain WHY THE TRADES WORK.

This is a very simple example.... that there are millions of TA systems that will look just like this graph.... if you played them long enough.

I would be interested to see how James16, DIBS, etc would play out in an 8 year challenge..... if specific rules were presented.... for any timeframe.... I have 1 minute data for EURUSD for 8 years (90% modeling quality)... and can get my hands on tic data if necessary (99% modeling quality)...

I would code it myself.... and verify with my eyeballs that the trades are opening and closing where they should.

Any takers?
  • #4
  • Sep 7, 2008 11:24pm Sep 7, 2008 11:24pm
The proposition is a straw man, the market changes, big deal.
This does not invalidate any statistics, just your system.
The breaking of a wave cannot explain the whole sea.
  • #5
  • Sep 7, 2008 11:42pm Sep 7, 2008 11:42pm
craig,

maybe you missed the point. in this example, after 900 trades, some statical type will say "i have a huge population size." he'll throw the trades into matlab and say "i have my bellcurve distribution showing 95% confidence levels that i can expect 100 pips a month with my system" or something like that.

you get the idea.

and then WHAM! throw it out the window. my point is, forget statistics in forex..... unless you can tie them to some real world fundamental data. otherwise, you are curve-fit and possibly walking into the jaws of the beast.

Quoting Craig
This does not invalidate any statistics, just your system.
  • #6
  • Edited 2:44am Sep 8, 2008 2:20am | Edited 2:44am
Here is another graph of a system I made (1989 - 2008) trading daily charts.... there are over 1000 single lot trades here (I verified the entries were realistic by checking the charts after the tester ran.) Only one position open at a time.

The first post kinda makes you think twice about the system's future expectancy, huh? Never-the-less, probably worth rolling the dice on if you are playing with money you can afford to lose.
Attached Image (click to enlarge)
Click to Enlarge Name: TesterGraph.gif Size: 18 KB
  • #7
  • Sep 8, 2008 4:33am Sep 8, 2008 4:33am
Quote
HAHA all you forex statisticians out there.... explain WHY the algorithm failed even though the huge population of the first few years leads you to conclude (statistically) that this is a profitable/breakeven strategy.
Because past performance is not always indicative of future results?

Ie: Things change. Especially trending behaviors.

Actually tho, I wouldn't conclude this is a viable strategy on that timeframe. You don't really have enough data from just 8 years (let alone a few at the beginning). You have to consider the distribution of returns, not just the rate of returns. Still, after some 50 bad trades in a row it should be obvious that the approach has seriously changed, and one would be wise to study why. Anyone using a stat-based approach should know when to re-evaluate.

Also keep in mind that the euro changes a lot with the countries that join or leave the ECB. The concept didn't start till 1992, but the big changeover deadline wasn't until 2002. So a huge shift in the way the currency worked happened roughly 6 years ago, you should therefore expect a major shift in the way it trends. Without enough post-switchover data we can't really determine these things, and it's just not an old enough currency for that. As an aside... I would also deeply question the validity of data that old, it's probably a composite.

Long story short... I don't think there's anyone that expects things to keep working for years and years without some changes being required. Imagine what would happen if the UK joined the ECB. The cable would disappear and the euro would undergo a radical shift in behavior, I doubt any system designer w/ a brain would allow themselves to be blindsided by such a change.
  • #8
  • Sep 8, 2008 9:52am Sep 8, 2008 9:52am
Quoting tenantro
Craig makes a valid point, the screenshot you attached only shows that the particular EA you tested does not work very well.It does not prove that all the rest of us who try to apply statistics to trading are delusional incompetents as you would like to believe (probably so that you can feel better about yourself).
hi

what kind of analysis was necessary to do after the frist 900 trades to know that it was a failed system?

thanks in advance.
  • #9
  • Sep 8, 2008 10:54am Sep 8, 2008 10:54am
Quoting tdion
no.

it will behoove you to concede that luck plays a part of your success or failure, if you can't explain WHY THE TRADES WORK.

This is a very simple example.... that there are millions of TA systems that will look just like this graph.... if you played them long enough.

I would be interested to see how James16, DIBS, etc would play out in an 8 year challenge..... if specific rules were presented.... for any timeframe.... I have 1 minute data for EURUSD for 8 years (90% modeling quality)... and can get my hands on tic data if necessary (99% modeling quality)...

I would code it myself.... and verify with my eyeballs that the trades are opening and closing where they should.

Any takers?
yes! i'll pm you once my EA is complete. Will be good to be proven wrong with solid high quality data
  • #10
  • Jan 5, 2009 7:00pm Jan 5, 2009 7:00pm
Quoting tdion
Here is another graph of a system I made (1989 - 2008) trading daily charts.... there are over 1000 single lot trades here (I verified the entries were realistic by checking the charts after the tester ran.) Only one position open at a time.

The first post kinda makes you think twice about the system's future expectancy, huh? Never-the-less, probably worth rolling the dice on if you are playing with money you can afford to lose.

check your data... euro did not exist in 89.. your off by more than 10 years..
Keep it simple stoopid....
  • #11
  • Jan 5, 2009 7:44pm Jan 5, 2009 7:44pm
Quoting Kurka Fund
check your data... euro did not exist in 89.. your off by more than 10 years..
The data is there.

Ive always assumed that it is a combination of all the currencies exchange rates before the one currency was formed.
Living the adventure in my head.
  • #12
  • Jan 5, 2009 8:27pm Jan 5, 2009 8:27pm
Basically you are right, LeonLorenzo.

This topic crops up in these forums every once in a while: Before the Euro existed, whe had the ECU (European Currency Unit). It was conceived -as a basket of european currencies- as far back as in 1979, although of course it didn't have physical existence. In fact, when we made the switch to the Euro (which at the time was exactly the same thing, just with a different name) we still had to wait for a couple of years before it had a physical existence. Details here:

http://en.wikipedia.org/wiki/European_Currency_Unit

The composition of the basket was locked in the early nineties, I believe. Any data before that is probably quite worthless. I actually wonder whether any data before the real introduction of the currency has value in terms of analysis.

As a side note, I still remember New Year day in 2001 (or was it 2002?) when it was introduced: many of us were coming back from parties, etc... and there were lines in nearly all the ATMs, since we were all so curious to see Euro notes for the first time. A funny sight those lines, given the state (of drunkenness) in which many of us were. I still wonder how many cards ended up trapped in the ATMs that morning...
  • #13
  • Edited 11:57pm Jan 5, 2009 11:42pm | Edited 11:57pm
"forget statistics in forex"

I see your point, however, when you say statistics, I presume you are referring to trading systems?

These "statistics" are what most traders dwell on. They go from system to system, course to course, and never find a profitable system, or a system they can stick with. They venture from emas, to MACD's, to Elliot Waves and even fractals in an attempt to 'get it right'.

What is missing is......Statistics.... because statistics can also mean expectancy, and that is one statistic that cannot be ignored.



Can you make a system that wins 30% of the time (forget about profitability)?

If you can, then you need to win 2.3 times more than what you lose, so never take a profit less than that. You are already at breakeven!

Now, what if you set your stop to breakeven when you have reached $X profit, and you get stopped out at breakeven 30% of the time? Then your average win only needs to be 1.67 times more than what you lose. That's manageable!

No matter what system you are trading, you need to make sure that the statistic of expectancy is positive.

e.g. If your average loss is $100 and you win 30% of the time, then you need to be making at least $234 per winning trade, there’s no point in making $200. These statistics can't be ignored. As for the system, use whatever you like, it really doesn't matter as long as you created a positive expectancy for yourself.


BTW tidon I design a lot of systems and that chart is typical of curve fitting or error of some sort.




Here's a statistic for you. I have placed a buy order on the EURUSD at 1.3677, and a stop at 1.3526 and a profit target of > 1.4700, no less.

That's a potential profit of 1023 pips, risking 151pips, or 6.7 times my risk!!! This type of trade (if triggered) has been profitable for me around 50% of the time. Would you risk $1 to make $6.7 with a 50% chance of winning? It's not a bad statistic regardless of the system.

DISCLAIMER: THIS IS NOT AN INVITATION TO TRADE AT ALL!!!!
  • #14
  • Jan 20, 2009 5:38pm Jan 20, 2009 5:38pm
Its not the statistics that failed, it was the so-called 'analysis using statistics' that failed.

Just because you have large population does not guarantee anything about the real population because of a confidence interval or a test. Sometimes quite the opposite. Such a large test population could have guaranteed false results just because of the size.

Plus there is no analysis here, are those trades independent of each other? What about lurking variables?

Stats is backed up with mathematical fact and it works very, very well.

And of course the "market changes" all the time. It changes for a reason. Probably thousands of them. Could these be quantifiable into variables? I don't know.

This is why so many private sector jobs require Ph.D's for their statisticians.

I for one am not one, so think for yourselves.
  • #15
  • Jan 20, 2009 9:09pm Jan 20, 2009 9:09pm
I'm not following your post loa1452, but it appears that you are talking about trading systems?

I trade using the same systems as other people, yet I have been profitable nearly every month for 14 years.

Here's my results for this week.

http://img242.imageshack.us/img242/7255/chart10nr9.jpg

What system did I use? It doesn't matter.
  • #16
  • Jan 20, 2009 9:39pm Jan 20, 2009 9:39pm
I am talking about people trying to use statistics to prove something that they don't know enough about.

I am not saying anything about trading here. I am simply stating how statistics will fail if certain assumptions are not met, and some are too quick to state that stats simply don't matter. I am in no way saying a system is not profitable, how else would it be possible to make money?

Sure, stats probably don't matter too much when trading just because of the shear amount of variables and connections to other markets and psychology etc, which is all involved in each and every trade.

To be clear, I am talking about statistics as a mathematical science, i.e, doing actual hypothesis testing, using confidence intervals and what have you.
  • #17
  • Jan 21, 2009 12:08am Jan 21, 2009 12:08am
On top of the faulty application of the statistics is the equally possible faulty interpretation of the results.

For example, a test was done and it proved that taller people are more intelligent than shorter people. The flaw was that they tested little children too.


Here is the essence of my point regarding expectancy and it’s an important point for anyone wishing to make this a business.


You have 2 poker machines in front of you.

MACHINE A costs $1 for the possibility of a $2 prize.

MACHINE B Costs $2 for a $1 prize.

They both have similar win rates, and both are quite unpredictable as to when they will pay out.

There are two approaches.

You can try to decipher payout prize patterns, and get an understanding of the mechanics of the machine, build models and even do a degree in the machines construction.

OR

You can just use machine A, and get through losing streaks, use smart money management, and take calculated bets.

Believe it or not, this is the choice of every trader.
  • #18
  • Jan 21, 2009 6:35pm Jan 21, 2009 6:35pm
Yep.

I couldn't imagine anyone carrying out a statistical test without knowing how to interpret the results to themselves.

Just getting the assumptions met to do a test is a hard enough task. The test itself is easy.

I'm not sure what you are saying with you machine A and machine B statement. I don't know anyone that would attempt to apply statistics to their trades unless they had an extensive knowledge in math. If it was easily doable, everyone would be a millionaire.

Why would anyone pick machine B if machine A costs less, has a higher payout, and has similar win rates? Furthermore, you would never win with machine B. You pay more to play than the prize itself...

There is a book called "how to lie with statistics". It explains the scenario when people would choose to interpret results differently.
  • #19
  • Jan 21, 2009 6:41pm Jan 21, 2009 6:41pm
Quoting tdion
craig,

maybe you missed the point....
The whole point is.. tdion, has absolutely no idea what statistics is or how it is used. And apparently his 'statistical type friends' are just as simplified.
  • #20
  • Jan 21, 2009 7:24pm Jan 21, 2009 7:24pm
"I don't know anyone that would attempt to apply statistics to their trades unless they had an extensive knowledge in math"

It back to that statistic of expectancy, and it's simpler than most people think.

Many people will jump into a trade and set a stop of, for example, 40 pips. When they reach an open profit of 20 pips, they worry about losing it and close the trade happy that they made a profit and their analysis was right.

Without even realising it, they are unneccessarily gambling on MACHINE B.
1 trader viewing now