Welcome to Testing Club, put your strategy to the test

At this stage of my forex career I've realized a few things.

1. I now realize how much I don't know.
2. Forex is a brutal game. It hurts sometimes.
3. It's also fun, and it reveals a lot about one's character.
4. Most forex websites have lots of information about trading systems, and strategies, but not very much info about correctly evaluating those systems and strategies.

Even just looking at FF, among the threads devoted to strategies/systems (or parts of systems, like indicators), the testing methodology used to evaluate those systems usually goes something like this:

1. A new system thread gathers followers based on how cool its name is, and how brazen are the claims of the system developer. The more bravado and challenging is the developer, the more followers the system gets.
2. Many people download the files related to the system (if any) and try to interpret the instructions. There are many different interpretations. At least a few people inadvertently invent a new strategy based on not understanding or following the instructions.
3. Feedback streams in slowly with no measurable data, and no standard for sharing that data so that it means the same thing to each tester. Misunderstanding and misinterpretation of results rules the day.
4. Eventually enthusiasm for the new system wanes, and a mutated system takes its place, generating new interest, or the thread slowly dies off.
5. New visitors to the thread have no idea if the system was good or not, presumably they test it on their own without feedback or move on to something else. Ultimately the system exists in a limbo where the superior merits of that system are taken (mostly) on faith by its practitioners.

That's another thing. One of the most pernicious forms of failure for new traders, is jumping from system to system, with no objective method of determining if its worth casting aside their existing strategy in favor of a new one, or if it's better to simply keep it and tweak it.

Testing your own strategies is even worse, since you are devoting time to testing, trying to improve your thingie, and possibly trading at the same time. Very time consuming. Sometimes you need a helping hand(s). The internet is good at this kind of thing.

What if there was a group where, every week(?), a new indicator, strategy, system or EA was presented for consideration, and everyone in the club, a Test Club, if you will, voluntarily tested that strategy according to a common protocol? Then at the end of the week, they would vote to decide if that system was worthy of further testing, maybe live testing, and the results of those tests would be shared. Then, the following week, a new item would be evaluated for consideration, in an attempt to unseat the existing 'king of the hill' system.

One idea. One week. One survivor.
At least let's start with that, and see how it goes.

What are the pitfalls of such an idea?
I know enough to know that often, MT4 backtesting is not sufficient to say whether something is worthy of further use. I know about LEAN testing. I know a little about genetic algorithm testing. I don't know a lot more than that. I am not a testing expert (yet!).

In order for this to succeed, the development of a Distributed Independent Standardized and Coherent (DISC) testing method would need to be developed. Everyone would use it. All testing results would be comparable. Utopia.

Alas such a thing is not easy to implement, much less document, so I think it should follow the same Battle Royale style method as the winnowing for the best trading tool/method above. A tournament. The best/most thorough/most impressive tester is crowned the Testing Champion (TC), their testing method is widely adopted (as much as possible) and as a perk of their awesomeness they are the ones who get to pick the following week's item for testing.

Does that make sense? To summarize (and this could change regularly, in fact it should)

 

1. Every week a new forex [thingie] will enter the ring and strut its stuff vying, ultimately, for consideration as a live testing candidate.
2. That thingie will be selected by the best tester of the previous round (as decided by a vote).
3. After testing, a vote will be held to decide which is the best thingie, the existing one or a new one, and whether it's worth continuing to optimize the existing or switch to a new.

So that's two votes, one thingie, and one supreme tester. It could be too complicated. We'll see.

Of course this is only going to work if everyone is forced (subtly manipulated) to participate. It will also only work if everyone stays focused on the task at hand - getting better at testing, and finding better things to test. This brings us to the rules.

 

1. Welcome to Test Club. The first rule of Test Club is talk about Test Club. Share it. Make suggestions for improving it.
2. The second rule of Test Club is DO NOT TALK ABOUT TRADING STRATEGIES in Test Club. Except to make suggestions for new things to test. If we clutter the thread with discussions of the minutiae of each trading thingie (indicator, system, strategy) then this will rapidly swell the thread and make it chaotic and unreadable. Restrict conversations about ways to improve the strategy to the forum where that strategy lives. It's OK to post links.
3. The third rule of test club is make suggestions for improving our testing abilities. Better statistics, faster iteration, simplified methods, or whatnot.
4. Fourth rule - only one test at a time. Of course you are free to test as many things as you wish, but there is only one official test at one time.
5. Fifth and (so far) final rule. If this is your first time at Test Club, you must test. Everyone who participates in discussions is expected to submit test results to the best of their ability. If you don't, the consequences are simply that we will not respect you, and reserve the right to mostly ignore you.
   
   That's good for now, will post more specifics and a new thingie for testing in the near future if there is at least some interest in this idea and a few people following the topic.

May 8 2017 - first system test - Polyjuice Potion https://www.forexfactory.com/showthread.php?p=9845689

Reserved for future use.

Reserved.

Are you talking about forward testing, because if you are, one week is hardly enough time to get any reliable results. And what about trading methods which require the total of the traders knowledge to implement.

Finding and testing of a new strategy is very difficult and time consuming. The forex market has no borders and perhaps there are many trading strategies which can be used. This is something like searching needle in a haystack . No one can predict what strategy will be more profitable now and in the future. But having a possibility to share it with others is a good idea. This may improve the strategy.

Those are both good points.

I agree forward trading needs more than a week to determine reliability, but depending on the system it might be enough time to ditch something that is uninteresting. There are also alternatives to pure live trading such as simulated (trainer) trading. I think we should be open to different methods until something gains consensus. I am not married to any of the limits I used in the original post. Like any good engineer, I would want to keep what works and change what doesn't.

As far as trading methods that require experienced skill and judgment, no amount of testing will eliminate the edge that knowledge and skill provide, but every system worthy of consideration (in my opinion) should be able to to be at least 'mimicked' by a less skilled trader, who has the (properly documented) algorithm in front of him.

There is a risk/reward consideration involved too. If a system exists that requires the dedication of a large amount of time before it can become profitable, how much more profitable does it have to be than a system that simply enters and exits on signals, that can be picked up in a week or less, and is profitable (to an extent) almost immediately? I would say the skill-based system has to be a LOT more profitable, to justify the sunk costs and opportunity costs. If such a system exists it should be tested and considered, but intuition tells me that low-hanging fruit are a better first option. I am very open to being proved wrong though.

The needle in a haystack analogy is a good one. One of my hopes for a club of testers is that many people searching the stack will find the needle faster than one trader muddling their way through several different haystacks.

Okay, here is the first test subject -a system I whipped up over the weekend. I did a minimal first check over recent history to test its viability,and when that looked good I posted it over in the trading systems forum.

https://www.forexfactory.com/showthread.php?p=9840304

I will update this thread and that one with some test results. I will post details of my testing setup in post 2.

just create a trade explorer, and share it .. so we can monitor also.

https://www.forexfactory.com/clemmo17#acct.39