Determining randomness in a trading system | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Successful Trading & Determining the Probability of a Trading System 10 replies

Fooled by randomness 15 replies

Luck, Randomness, and Chance in Trading 0 replies

Thoughts on randomness and systems 40 replies

Poll: Are trends rooted in randomness? 16 replies

Attachments: Determining randomness in a trading system
Exit Attachments

Determining randomness in a trading system

  • #1
  • First Post: May 18, 2012 8:25am May 18, 2012 8:25am
Here is a backtest for a bit over 2 years ( 2010-04-05-2012-05-17) for a purely mechanical system. Attached is the profit curve.

# Trades: 967
Profit factor: 1.29
% Wins: 49.12
Av. Win: $35.37
Av. Loss: $26.40

I want to determine the probability that this is a result of randomness. What is the best way to do that?

Thanks in advance.
Attached Image (click to enlarge)
Click to Enlarge Name: StrategyTester.gif Size: 7 KB
Swedish steel!
  • #2
  • May 18, 2012 9:54am May 18, 2012 9:54am
Quoting grovsnus
Here is a backtest for a bit over 2 years ( 2010-04-05-2012-05-17) for a purely mechanical system. Attached is the profit curve.

# Trades: 967
Profit factor: 1.29
% Wins: 49.12
Av. Win: $35.37
Av. Loss: $26.40

I want to determine the probability that this is a result of randomness. What is the best way to do that?

Thanks in advance.
easy, change the parameter inputs, change the underlying instrument, change the backtesting period, maybe change the time-frame. If it still works, then it is likely not the result of randomness.
The less universal a trading system is, the more likely it is to stop working in the long-term.
But don't forget to include trading costs. Most systems turn negative once they are included.
  • #3
  • May 18, 2012 11:47am May 18, 2012 11:47am
Quoting Custos
easy, change the parameter inputs, change the underlying instrument, change the backtesting period, maybe change the time-frame. If it still works, then it is likely not the result of randomness.
The less universal a trading system is, the more likely it is to stop working in the long-term.
But don't forget to include trading costs. Most systems turn negative once they are included.
Cheers. The numbers look similar to this when testing a 10 year period. About 5000 trades, however, this can still be randomness imo. I want to determine the likelyhod that it is random.
Swedish steel!
  • #4
  • May 18, 2012 11:52am May 18, 2012 11:52am
Quoting grovsnus
Cheers. The numbers look similar to this when testing a 10 year period. About 5000 trades, however, this can still be randomness imo. I want to determine the likelyhod that it is random.
well, then I would suggest you to buy a statistics book and see how statistical significance is calculated and how degrees of freedom are related to that. By the way, you are correct that 5.000 trades is not enough, the lowest amount of data points pointed out in statistics books is +10.000 data points (i.e. trades). The more data you have of course, the better.
  • #5
  • May 19, 2012 1:32pm May 19, 2012 1:32pm
just glanced at your results and I can observe that your system has an expectation of 0.13 which is a break even or marginally profitable system.

I think you are cutting winners too soon.those results from the backtest seem rather too optimistic.

I believe backtesting is overhyped, you'd need a degree or intensive studies on statistics to be able to correctly interpret the data.

but fortunately the research is out there for some systems like simple moving average crossover which has a win rate of between 35-45% win/loss of 3:1 measured from the 1870s so its unlikely to change. most of the other solid trend following systems range between 30-50%. so if you have a trend following system you can estimate the performance using the lower %s
wo-yoy! wo-yoy! wo-yoy! wo-yoi! wo-yoy-yoy-yoy!
  • #6
  • May 19, 2012 2:02pm May 19, 2012 2:02pm
Quoting Custos
By the way, you are correct that 5.000 trades is not enough, the lowest amount of data points pointed out in statistics books is +10.000 data points (i.e. trades).
This is not correct. Any statistical test on trading that has a lower confidence band above 0 suggest a non random outcome with the assigned confidence level and a positive expectancy. The more the better but 10 000 is not a relevant number. It's more important to include all market states in the calculation.

From the stats looks like its random entry (~50%) that tries to position it self to catch a fat tail movement in the market.
  • #7
  • May 19, 2012 2:28pm May 19, 2012 2:28pm
Quoting Rikers
This is not correct. Any statistical test on trading that has a lower confidence band above 0 suggest a non random outcome with the assigned confidence level and a positive expectancy. The more the better but 10 000 is not a relevant number. It's more important to include all market states in the calculation.

From the stats looks like its random entry (~50%) that tries to position it self to catch a fat tail movement in the market.
Thanks for the input!
  • #8
  • May 19, 2012 2:30pm May 19, 2012 2:30pm
Quoting Custos
Thanks for the input!
Is this irony?
I was expecting from you some high math rape that wold make me look silly...
  • #9
  • May 19, 2012 3:03pm May 19, 2012 3:03pm
Quoting Rikers
Is this irony?
I was expecting from you some high math rape that wold make me look silly...
I know this Forum is so full of sarcastic statements that it's almost hard to believe that somebody writes a normal thank you statement. I am not all knowing and always glad to learn from others. So, again thanks.
  • #10
  • May 20, 2012 4:38am May 20, 2012 4:38am
Quoting Rikers

From the stats looks like its random entry (~50%) that tries to position it self to catch a fat tail movement in the market.
That's exactly what it is.

Now, I studied some statistics back in uni, 10 years ago, most is gone due to vodka though.

What is best statistical test to check for nonrandomness?
Swedish steel!
  • #11
  • May 20, 2012 7:54pm May 20, 2012 7:54pm
you can do a test on a lower confidence band as:

X-Φ-1(p)*(std.dev)*sqrt(n) where:

X= last value
Φ-1 = inverse standard normal cumulative distribution
std.dev = standard deviation
n = number of periods
p = P(Z<=z)=1-alpha/2
alpha = 1-confidance
confidence = desired confidence like 0.95 or 0.99

if it is above the first value you have a positive expectancy within that system for the desired confidence

if this is to much post the standard deviation and I'll calculate for you...
  • #12
  • May 21, 2012 2:51am May 21, 2012 2:51am
Quoting Rikers
you can do a test on a lower confidence band as:

X-Φ-1(p)*(std.dev)*sqrt(n) where:

X= last value
Φ-1 = inverse standard normal cumulative distribution
std.dev = standard deviation
n = number of periods
p = P(Z<=z)=1-alpha/2
alpha = 1-confidance
confidence = desired confidence like 0.95 or 0.99

if it is above the first value you have a positive expectancy within that system for the desired confidence

if this is to much post the standard deviation and I'll calculate for you...
Cheers, this is what I wanted!
Swedish steel!
  • #13
  • May 21, 2012 6:50am May 21, 2012 6:50am
Quoting grovsnus
That's exactly what it is.

Now, I studied some statistics back in uni, 10 years ago, most is gone due to vodka though.

What is best statistical test to check for nonrandomness?
Hi grovsnus, everyone,

I'll postulate two options...the easy and the hard way to check for nonrandomness.

The easy way:
Reverse your triggers. Where the code says to "buyorder", replace it with "sellorder" and do the same for the "sellorders". If the results still end up in profits, it only means you could've also made money flipping coins. i.e. The strategy has no predictive power.

The hard way:
There's no way I'm gonna be able to explain this totally but I'll give the insight as much as possible.

You need to come up with two hypotheses:

1. That your strategy is not a result of randomness i.e. It has predictive power.
2. The null hypotheses which states that the predictive power is zero.

Think of these hypothesis:

If it is a dog : then it has four legs.
If we discover the animal is a dog, then we can be sure it has four legs. But what if we discovered the object has four legs? That is not enough to say that it is a dog. It could as well be an elephant, or a goat.

Now to trading statistics:
If a strategy has predictive ability : it will be profitable.
Truly, if the strategy has predictive power, it will show profits. But showing profits is not enough to say the strategy has predictive powers. That is were the null hypothesis steps in.
If you can prove the null hypothesis to be false, then the real hypothesis will be true, i.e. The strategy truly has predictive powers.

The process to go about such test for randomness is not something I can be explaining during trading hours. Maybe during the weekends might be better.

BTW, when testing for randomness, remember to detrend the data. There's a big difference between statistics from a bull market and that of a bear market.

Cheers,
Motion
Don't just beat the markets, leave bruises.
  • #14
  • May 21, 2012 2:48pm May 21, 2012 2:48pm
Quoting Rikers
you can do a test on a lower confidence band as:

X-Φ-1(p)*(std.dev)*sqrt(n) where:

X= last value
Φ-1 = inverse standard normal cumulative distribution
std.dev = standard deviation
n = number of periods
p = P(Z<=z)=1-alpha/2
alpha = 1-confidance
confidence = desired confidence like 0.95 or 0.99

if it is above the first value you have a positive expectancy within that system for the desired confidence

if this is to much post the standard deviation and I'll calculate for you...
Rikers,

Can you explain above in more details?Where should I fit the above?In an expert advisor?Can you give an example if it is not too much to ask for?Thanks.Also good article : http://forum.mql4.com/14281
  • #15
  • May 21, 2012 3:54pm May 21, 2012 3:54pm
I was wondering after reading this thread, lets say the system IS random, what does this means going into the future? Will the randomness still be profitable like in the backtest?
  • #16
  • May 21, 2012 4:16pm May 21, 2012 4:16pm
Quoting misterboy
I was wondering after reading this thread, lets say the system IS random, what does this means going into the future? Will the randomness still be profitable like in the backtest?
The randomness could continue in the future and give profitable results. That's the thing about randomness, it's totally random.
Swedish steel!
  • #17
  • May 30, 2012 9:15am May 30, 2012 9:15am
Quoting Custos
well, then I would suggest you to buy a statistics book and see how statistical significance is calculated and how degrees of freedom are related to that. By the way, you are correct that 5.000 trades is not enough, the lowest amount of data points pointed out in statistics books is +10.000 data points (i.e. trades). The more data you have of course, the better.
Fascinating.
From reading the market wizards and market histories...i am of the opinion that : A technical/fundamental approach CAN ONLY BE solidly functional for a specific kind of market within a certain period of time(+/- 10 years...if you are lucky...15yrs). After the 10 year mark, the trading approach will start to fail unless it has undergone updating to reflect the new market reality. The trader that hasnt done this will start to see his method fail.

Logically, i dont think you can find a method that works in all market condition. I sincerely doubt such a method exists. The market is too dynamic for that. Despite its inefficiencies, i doubt the market will allow an exploitable edge in all market conditions...over decades...past, present, future.

Backtesting your methodology to oblivion is not the answer in my opinion, but acutely recognizing when that method works and when it doesnt work.

Trading history is full of carnage of methods that didnt adapt or die: gann's, elliot wave, turtle method, canslim, breakout, smas(there was a time when SMAs trading was the money maker)...etc....reading the market wizards you can see all the successful traders with longevity adjusted and adapted their methods within a period of roughly 10 years or so. Nothing last forever. The turtle method was perfect for the fundamental situation of the Nixonian era....all the way to the early 80s....afterwards...it displays a lot of fakeout without extensive modification. All the traders in Covel' TURtle book do not use the original, classic method. again: adapt or die.

i humbly dont care that my method can work over 2 million backtested trades...i care about *knowing* thoroughly when the method works and when it doesnt. As such, if the method only gives me 5 years of profit, i will take it and move on.



my two cents.
" It may mean something. It may mean nothing at all "
  • #18
  • May 30, 2012 9:28am May 30, 2012 9:28am
Quoting Skynet
Fascinating.
From reading the market wizards and market histories...i am of the opinion that : A technical/fundamental approach CAN ONLY BE solidly functional for a specific kind of market within a certain period of time(+/- 10 years...if you are lucky...15yrs). After the 10 year mark, the trading approach will start to fail unless it has undergone updating to reflect the new market reality. The trader that hasnt done this will start to see his method fail.

[b]Logically, i dont think you can find a method that works in all market condition....
that may be, I don't have an opinion towards that.
So, how do you know when the method doesn't work anymore? Hopefully not after the account is depleted.
  • #19
  • May 30, 2012 9:56am May 30, 2012 9:56am
Quoting Custos
that may be, I don't have an opinion towards that.
So, how do you know when the method doesn't work anymore? Hopefully not after the account is depleted.
How?

#1. When the fundamental nature of the econom(ies) has changed..then it is probably the case that your theories/hypothesis that gives birth to your methodology may be getting untenable. you track the fundamental trend...the fundamental serves as the foundation for the methodology. What is going on in the world right now? what are the major economic issues? what are their components and how do they affect each other? what are the various schools of thoughts with regards to these things? Which one, at this particular time best explains(in terms of financial results), what is going on? I am not a disciple of any school...i am a disciple of the school that makes me money at this time.
How do you fashion a fundamental trading approach based on the economic situation? How do you fashion a technical trading approach based on the economic situation? a good beginners book will be be trader vic's "method of a wall street master" and ashraf laidi "currency trading and intermarket analysis".

Why is it that GANN'S method worked so well when it did? Why is that elliot wave method worked so well, when it did? why did breakout used to work like a charm? same with daily pivot points? what was going on in terms of fundamentals/sentiments/types of market players when those methods were very brutally effective? What kind of market was it?

Do i understand this enough to the point where i can see how gann's method could have been derived to suit those market periods?.......then...what market period do we have now? historical analogies? what methods are best for this time/market?

#2. it doesnt hurt to simply track your equity curve and use a pre-determined level as to when to call it quit.
" It may mean something. It may mean nothing at all "
  • #20
  • May 30, 2012 10:04am May 30, 2012 10:04am
Quoting Skynet
How?

#1. When the fundamental nature of the econom(ies) has changed..then it is probably the case that your theories/hypothesis that gives birth to your methodology may be getting untenable. you track the fundamental trend...the fundamental serves as the foundation for the methodology. What is going on in the world right now? what are the major economic issues? what are their components and how do they affect each other? what are the various schools of thoughts with regards to these things? Which one, at this particular time best explains(in terms...
okay, that might be true for methods that aren't purely based on technicals.

Quoting Skynet
#2. it doesnt hurt to simply track your equity curve and use a pre-determined level as to when to call it quit.
that I can agree with. (I actually incorporate that into my trading).
0 traders viewing now