How big of a sample size is needed? | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

What kind of sample size is ideal? 6 replies

EA Testing - Sample Size? 0 replies

Stop Loss needed for Sample MACD 1 reply

Question about timeframe for sample EAs in MT4 0 replies

VantagePoint Sample System 0 replies

How big of a sample size is needed?

  • #1
  • First Post: Dec 18, 2013 3:25am Dec 18, 2013 3:25am
I was wondering if anybody knows the answer, probably mostly from the trading mathematics point of view.

I am trading a mechanical system with 100% of rules defined. All the possible back testing was done with positive results so I am trading it live for a few months now.

The trade is taken once a day. Theoretically there is a condition when no trades are taken but it is rear, so let’s say I will have 260 trades a year.

What time, or number of trades is enough, mathematically, to start really trusting that system and move it from the experimental stage to a money making machine? There should be some math involved, which I cannot seem to find.

Would it be 100 trades, 260 trades, 520 trades?
  • #2
  • Dec 18, 2013 12:58pm Dec 18, 2013 12:58pm
if you understand why your system works then number of real trades should be quite small, because you understand when/if to turn it off.

if you do not know why system should perform then you have bigger problem as nothing guarantees that system will not stop after 3200 trades.

however I do not have experience with 100 pct mechanical system.

I am discretionary and have done 50 trades with particular method and reached conviction level so risking max size every time.
  • #3
  • Dec 18, 2013 12:59pm Dec 18, 2013 12:59pm
You should know why your systems works, i mean what it is that its taking advantage of, is it volatility, current fundamentals, intermarket relations etc and if its 100% systematic than its should not be hard to figure out.

Sample size maybe an important part but success has nothing to with statistics. If you dont know why the system works, one early morning when you've just made your coffee and starting the day...

dont matter if it was profitable for last 1000 trades, there are people who loose fortunes made in years, figuring out too late that what was working before is't anymore.
.
  • #4
  • Dec 18, 2013 8:33pm Dec 18, 2013 8:33pm
any mathematical suggestions with formula?
  • #5
  • Edited 7:32am Dec 19, 2013 1:08am | Edited 7:32am
Quoting zzz
I was wondering if anybody knows the answer, probably mostly from the trading mathematics point of view. I am trading a mechanical system with 100% of rules defined. All the possible back testing was done with positive results so I am trading it live for a few months now. The trade is taken once a day. Theoretically there is a condition when no trades are taken but it is rear, so let’s say I will have 260 trades a year. What time, or number of trades is enough, mathematically, to start really trusting that system and move it from the experimental...
Depends on what type of test are you trying to do.I would suggest that first of all to calculate the expectancy of your system:



reward=(sum of the profits of each winner trade) / total number of winner trades
risk = (sum of the losses of each loser trade) / total number of loser trades
winrate = total number of winner trades / total number of all trades
expectancy = (winrate * reward) - [(1- winrate)*risk]



A positive expectancy suggests a profitable system, but because of uncertainty i would recommend that your expectancy to be atleast >1

For which i would suggest atleast 5000 trades, independent of timeframe, so you might need to get a 10 or 20 year database if you trade D1.

I would also suggest to calculate the standard deviations of the R-R and separately the WR and see how close they fluctuate from the mean, you want a very tight stdev since you dont want that your results to be unpredictable.I would suggest that a < 0.5 sigma for both can be acceptable on 5000 trades.

Also might wanna check this thread also:
http://www.forexfactory.com/showthre...95#post7063395

They explain if the (winrate * reward) - [(1- winrate)*risk] * stdev ^2 is also positive, then you are good to go, but again in my experience i would dispose all systems which dont hit atleast 1, on this test too.A positive expectancy suggests a profitable system, but because 5000 samples are still small but you dont really have data to do more (the best results you would have on >1m trade sample, but given that you trade daily that would be thousands of years of data ), so 5000 trade sample and expectancy of atleast 1 or better.

Also for safety you can also calculate the standard deviation of your expectancies also, to see how they deviate and how many times they are in the negative if they are at any given point, it would be nice that at no given point they are negative that would be the ultimate test, but thats hardly accomplishable, so if in the average they are positive and above 1 than its also a good one.
"There's a sucker born every minute" - P.T. Barnum
  • #6
  • Dec 19, 2013 4:40am Dec 19, 2013 4:40am
thanks proximus for the solid answer. will study it over the next few days.

any other ideas or places where it was discussed?
  • #7
  • Dec 19, 2013 7:34am Dec 19, 2013 7:34am
Quoting zzz
thanks proximus for the solid answer. will study it over the next few days. any other ideas or places where it was discussed?

I dont know, you can search the threads i`m sure there are many threads talking about backtesting and system evaluation.

Also i note i edited the last post, for idk what reason the formulas were not shown propertly , now i fixed it so good luck calculating those stuffs.
"There's a sucker born every minute" - P.T. Barnum
  • #8
  • Dec 20, 2013 3:48am Dec 20, 2013 3:48am
i actually was asking about the forward testing. i don't have questions regarding the backtesting, it's all done and the results and all the numbers are very positive.
  • #9
  • Dec 20, 2013 4:24am Dec 20, 2013 4:24am
Quoting zzz
i actually was asking about the forward testing. i don't have questions regarding the backtesting, it's all done and the results and all the numbers are very positive.
Well its the same right, do a forward test on a demo account, and you do the same calculations after you have a decent sample size.

Not that you could draw any conclusions from 2-3 trades, but make like 5000 trades on a demo forward test and after that you can draw a good conclusion.

Yes it may be painful for a D1 strategy so you might want to trade more pairs to reach that sample size in a year if its possible.I dont think its any way around to shorten the test time.You must have a decent sample size or else the accuracy of conclusion is pretty bad.
"There's a sucker born every minute" - P.T. Barnum
  • #10
  • Dec 20, 2013 5:33am Dec 20, 2013 5:33am
well, i thought so too. the problem is that i am trading only 1 pair this particular way, so if i follow the road you are suggesting it will take me forever, it is not real. there must be another way. that's why i asked to begin with.

there must be a formula regarding the minimum sufficient number of trades in the sample or something similar. i am trading it live anyway but if i have some clear and understandable math behind the quantity of a sample then i can start really bumping my account if math and real results confirm each other.
  • #11
  • Dec 20, 2013 4:38pm Dec 20, 2013 4:38pm
Quoting zzz
well, i thought so too. the problem is that i am trading only 1 pair this particular way, so if i follow the road you are suggesting it will take me forever, it is not real. there must be another way. that's why i asked to begin with. there must be a formula regarding the minimum sufficient number of trades in the sample or something similar. i am trading it live anyway but if i have some clear and understandable math behind the quantity of a sample then i can start really bumping my account if math and real results confirm each other.
I dont know, you can have any number of sample, but the less you have the margin of error increases almost exponentially.Here are a few links where they explain how to calculate your margin of error, but i would still recommend atleast 100 samples no matter what, below that its just not worth and even with 100 samples you will have a relative very big margin of error, but you have to live with that.

http://statistics.about.com/od/Formu...or-Formula.htm
http://www.dummies.com/how-to/conten...a-sample-.html

I just dont know why you cant trade more than 1 pair, i mean if a strategy is very good it must work on all pairs relatively good
"There's a sucker born every minute" - P.T. Barnum
  • #12
  • Dec 21, 2013 8:09pm Dec 21, 2013 8:09pm
thanks for your answers again. will do more digging.
  • #13
  • Edited 1:53pm Dec 30, 2014 1:14pm | Edited 1:53pm
Trying to look at it from a different angle.

So, I have a working system, 100% mechanical. It makes money, I would say, good money for more than a year and a half now. I am trading a decent account size. I want to increase it significantly to really make a killing and trying to somehow get a feel for – what if in a year or two it will stop working?

Those who make money with any kind of a trading system, or very strict trading rules, and I am hoping someone with such experience will chime in, how are you treating your believe in the system? To which degree you trust? When have you decided that you really trust it and devoted serious money?

If you really believe in it then you should put as much as you can money into the game to maximize the profit. If you are not doing it, then you don’t believe in it. Is it a correct approach?

As a more general question, what is the proof that any system is working and will be working in the future? Nobody has 50 years in Forex to back their experience and success, if any. Even if they did, one could argue that you need 200 years of experience. So, this is not really an answer. But what is? Number of trades? Opinion of others?

I am not talking about the people who sell how to trade and ‘teach’ others - I understand their motivation although those who buy their services have to answer the same questions. But those traders who do have their own way of making money in trading – I am interested in your opinion. How do you know that your way will work for years to come? You probably don't so how do you look at that uncertainty then?

There is no one to prove or disprove how I trade as I am probably the only one doing it this way. This is a good thing and a bad thing...

Never thought I would have this sort of questions but now I do.
  • #14
  • Jan 3, 2015 12:08am Jan 3, 2015 12:08am
If you want to approach it from a pure maths perspective then have a look at Power Analyses in statistics..

Its basically a method by which you determine your sample size so that you know its going to give a significant result.. or rather.. n isnt too small to be a factor in the result
  • #15
  • Jan 3, 2015 4:31am Jan 3, 2015 4:31am
Quoting zzz
I was wondering if anybody knows the answer, probably mostly from the trading mathematics point of view. I am trading a mechanical system with 100% of rules defined. All the possible back testing was done with positive results so I am trading it live for a few months now. The trade is taken once a day. Theoretically there is a condition when no trades are taken but it is rear, so let’s say I will have 260 trades a year. What time, or number of trades is enough, mathematically, to start really trusting that system and move it from the experimental...
My answer is, a big big number.
A trading system can be extremely profitable for the past 2-3 YEARS and then fail spectacularly the year after, it can even produce consistent profit year after year for 10 YEARS and then fail miserably the year after.

The market is always changing, always new, whatever results a system gave in the past (can be the past month or the past decade) means nothing.

But i agree as somebody said already here, if you know how and why your system "work", you may have an advantage in recognizing why and how it can stop "working".
For example if your system take advantage of a big trend, the moment the trend finish and the market start a ranging period, you can detect it and stop trading the trend-system before losses get on their way, or vice versa for a ranging system.

Not only the market do change from trending to ranging, but also the size (volatility, speed, size of waves) of trends and ranges do change from time to time, so even a trending system can stop giving profits when the size of trends changes, or a ranging system stop giving profits when the size of ranges changes, etc.
  • #16
  • Last Post: Jan 5, 2015 8:25am Jan 5, 2015 8:25am
Quoting tunera
A trading system can be extremely profitable for the past 2-3 YEARS and then fail spectacularly the year after, it can even produce consistent profit year after year for 10 YEARS and then fail miserably the year after.

Do you have any specifics about it? Was any of such systems and their failures discussed in any of the threads that you know of?
0 traders viewing now