How do you determine when a strategy is valid and profitable?

I'm building a scalping EA, and will be forward testing it on a live account with only a couple hundred dollars. Since it's a scalping EA it has the potential to make as many trades in a week as some systems do in a year.

That brings up my question of when do you know a strategy is profitable over the long run. If it is profitable after 50-100 trades on the week, is that a large enough sample set to say it will work? Or can I only say it works if it survives several months of trading; at which point it will have produced hundreds and perhaps thousands of trades?

So whats the measure?
A. Time
B. # of trades
C. Both.

If C, at what point would you trust the system? I guess I could ask the same question if the answer is A or B too.

TIA,
Matt

Well, I trust a system if it is profitable. But the ugly truth is, that all statements about a EA are just valid for the past. There is no guarantee that a nearly 100% accurat system will perform well even in the near future. Neither time nor profit nor # of successfull trades can give you confidence for the future.

I also like to see EAs just like the currency pairs they trade, they also have up and down trends. Ride the uptrends and if you encounter a down, then step aside from your EA.

Well that was kind of my question. You probably won't find many people saying a system is profitable because they made one trade and it won.

Well that's true for any statement about any system, even one that is traded manually. The results are of the past. You can't predict the future based on the results of a system, but we're much more inclined to use a system that has profitable past results vs one that doesn't.

A system isn't doomed to fail in the future just because it's doing well now. It's a little bit of a circle you're running in.

If you want to judge your live results read up on statistical confidence intervals, you want to be able to reject a null hypothesis of mean profit being greater than zero as a result of luck with 90-95% confidence. I would use 50-100 trades, the same measures can also be calculated by bootstrapping your mean adjusted returns.

If you are looking at back test results, greater numbers of trades mean you have a greater chance of profit in the future, but nothing guarantees it, if you have come to your system parameters via optimization your chances are even worse, Google 'data mining bias'.

I would use 100 trades. That is my personal benchmark.

That's not what I meant. I just wanted to lower expectations a little and emphasis on the necessity to permanently judge a system, not only once.

I'm surprised this thread has received so little attention, the question it poses is something everybody needs to determine for themselves before diving into the next system de jour, have we jumped the shark?

Yea I would have expected more of a response too.

C with real money test results.

I would also run it through simulated market with some surprises from my side.

On a similar note, when would you trust a system to retire?

Effectively, I should be willing to quit my job as soon as my income from trading equals the income of my job. Actually, I'd be willing to take a pay cut if it meant having as much time to do what I want.

That's sort of why I was asking/posting this thread. When would I be able to trust the system to have enough confidence to go in and give them my 2 weeks notice? The system I'm building could effectively replace my income in as little as 3-4 months if it makes as many trades per day as I believe it will.

When do you trust the system and when do you retire?

If this is true then why not give it 6-8 months. That way you know that if anything should go wrong you will have a back up of few months income to rely on.

Personally, I have a one year time frame in which i want my system to be profitable before i quit my day job

I guess that opens up a lot of questions.

For one, what if my system was able to produce so many trades in just 1-2 months that it could replace my income? Would you say, 'Why not trade it for 3-4 months,' or would you still say, 'why not trade it for 6-8 months?'

I would have to ask, whats the difference between analyzing a system after 1000 trades vs 2000? The both exhibit the law of large numbers and the results should be consistent.

The point this really brings across though is, how long does it take for the market climate to shift so that you know the system can weather many conditions? Is the transition from ranging to trending periods instant or gradual? If gradual, how long is the process? What about volatility? Does increased volatility come gradual or rapidly after certain world events? If it's instant, how fleeting is the condition?

In that aspect I would agree with you that trading a system for a year would probably cover all the bases for market conditions; unless your system only makes several trades. As mentioned, my system is a scalping system and has the opportunity to make thousands of trades in a year.

Where's the middle ground?

That's the tricky one. From what I see is that the market behaves like a fraktal, meaning that major changes happen on all timescales. Major changes happen more seldom than small ones, but they can happen and are still unpredictable. Just take many of EAs you find in here. For a certain time they work but most of them have severe degradations during test periods bigger than 5 years.
I also try to figure out some sort of quality measurement of an expert to track over time and to compare with competing systems. But even that is not easy. Just profit over time is not enough. You have to take bound capital, max drawback, win/loss ration etc into account. Maybe there are statistic measurements I don't remember any more that would be better suited. I like the idea of competing EAs in my portfolo that are weighted due to their success or quality regarding the actual market conditions, though I don't have a clue how to get there, yet...

Good post. Thanks for the reply. The idea of competing EA's certainly is interesting. I've always hoped to get to a point where I could have several systems running such that when a system isn't profiting, then at the least it is breaking even so the other systems don't have to make up for it. Your idea of weighting the systems is interesting. That almost seems as difficult as predicting the markets though. One would hope the trend of which system is working is longer than the time line the system is playing.

I had a thought though. As you noted, the system is always changing behavior, but really based on the timescale you're looking at. So my thought is that the ability to trust an EA, or any system, after so many trades or time is really dependent on the time scale of the system. For example, if I run my scalping system for 3-4 months then I have to expect it will have experienced ranging/trending markets, smooth/choppy markets, and however else you can describe market movement. Perhaps on a scalping system, 3-4 months is plenty of time to test the system, but a system based on daily candles could need a year or more to fully test since testing through only one season of ranging/trending still isn't enough.

I'll end this post by saying that perhaps we shouldn't judge the success of a system after so many trades or a time period per say, but by how many candles the system has run through depending on the time period of your system. I assume we don't judge it by that because for a scalping system on 1 minute candles, 2000 candles would only be 33 hours of trading. Whereas, 2000 candles of a daily system is over 5 years. I imagine there is some equation/curve that could explain how many candles would be needed to judge the validity of a system based on its time frame.

I don't think there is any magic milestone where you can assert that a system is going to remain in profit, the only constant is that the market keeps changing and you have no idea what the changes are really. So all you can do is track recent progress and draw your own line in the sand.

I think a martingale hedge is the only "proven" system, and it requires an infinite bankroll.

In fact, I believe all systems fail due to the chaos of markets. Its time to bite the bullet and realize that all your winning trades were lucky, and your losing trades were bad luck.

http://www.forexfactory.com/showthread.php?p=1915324

I am still amazed at how many people will trust their cash to a meta trader EA, And one for scalping at that, longer term is far more rewarding
also MT brokers are well lets just say less than sterling.
If you want to automate your trading go with a good broker and use their API.

By the way I am more than happy to accept your pips.........

I'm not going to use MT. I'm building it on the Ninjatrader platform to use with MBT, then eventually Dukascopy, lava, or whatever other ECN I want.

I'm not so sure martingale is really a system; perhaps it's only half a system. It's only a form of money management. It still has to rely on entry/exit signals that will produce a winning trade. A martingale method could still effectively lose if the other half of the system is bad enough.

BTW - My system is based on that chaos.

PS) A martingale hedge is an "always in the market strategy." There is a nice explanation of the strategy by FXTradePro called "FXTradePro - Semi Martingale" with good pictures, though his entry approach is discretionary.