DislikedHow much can you know about your strategy if it's never been in a fight?
- Tester Durden
At this stage of my forex career I've realized a few things.
- I now realize how much I don't know.
- Forex is a brutal game. It hurts sometimes.
- It's also fun, and it reveals a lot about one's character.
- Most forex websites have lots of information about trading systems, and strategies, but not very much info about correctly evaluating those systems and strategies.
Even just looking at FF, among the threads devoted to strategies/systems (or parts of systems, like indicators), the testing methodology used to evaluate those systems usually goes something like this:
- A new system thread gathers followers based on how cool its name is, and how brazen are the claims of the system developer. The more bravado and challenging is the developer, the more followers the system gets.
- Many people download the files related to the system (if any) and try to interpret the instructions. There are many different interpretations. At least a few people inadvertently invent a new strategy based on not understanding or following the instructions.
- Feedback streams in slowly with no measurable data, and no standard for sharing that data so that it means the same thing to each tester. Misunderstanding and misinterpretation of results rules the day.
- Eventually enthusiasm for the new system wanes, and a mutated system takes its place, generating new interest, or the thread slowly dies off.
- New visitors to the thread have no idea if the system was good or not, presumably they test it on their own without feedback or move on to something else. Ultimately the system exists in a limbo where the superior merits of that system are taken (mostly) on faith by its practitioners.
That's another thing. One of the most pernicious forms of failure for new traders, is jumping from system to system, with no objective method of determining if its worth casting aside their existing strategy in favor of a new one, or if it's better to simply keep it and tweak it.
Testing your own strategies is even worse, since you are devoting time to testing, trying to improve your thingie, and possibly trading at the same time. Very time consuming. Sometimes you need a helping hand(s). The internet is good at this kind of thing.
What if there was a group where, every week(?), a new indicator, strategy, system or EA was presented for consideration, and everyone in the club, a Test Club, if you will, voluntarily tested that strategy according to a common protocol? Then at the end of the week, they would vote to decide if that system was worthy of further testing, maybe live testing, and the results of those tests would be shared. Then, the following week, a new item would be evaluated for consideration, in an attempt to unseat the existing 'king of the hill' system.
One idea. One week. One survivor.
At least let's start with that, and see how it goes.
What are the pitfalls of such an idea?
I know enough to know that often, MT4 backtesting is not sufficient to say whether something is worthy of further use. I know about LEAN testing. I know a little about genetic algorithm testing. I don't know a lot more than that. I am not a testing expert (yet!).
In order for this to succeed, the development of a Distributed Independent Standardized and Coherent (DISC) testing method would need to be developed. Everyone would use it. All testing results would be comparable. Utopia.
Alas such a thing is not easy to implement, much less document, so I think it should follow the same Battle Royale style method as the winnowing for the best trading tool/method above. A tournament. The best/most thorough/most impressive tester is crowned the Testing Champion (TC), their testing method is widely adopted (as much as possible) and as a perk of their awesomeness they are the ones who get to pick the following week's item for testing.
Does that make sense? To summarize (and this could change regularly, in fact it should)
- Every week a new forex [thingie] will enter the ring and strut its stuff vying, ultimately, for consideration as a live testing candidate.
- That thingie will be selected by the best tester of the previous round (as decided by a vote).
- After testing, a vote will be held to decide which is the best thingie, the existing one or a new one, and whether it's worth continuing to optimize the existing or switch to a new.
So that's two votes, one thingie, and one supreme tester. It could be too complicated. We'll see.
Of course this is only going to work if everyone is
forced (subtly manipulated) to participate. It will also only work if everyone stays focused on the task at hand - getting better at testing, and finding better things to test. This brings us to the rules.
- Welcome to Test Club. The first rule of Test Club is talk about Test Club. Share it. Make suggestions for improving it.
- The second rule of Test Club is DO NOT TALK ABOUT TRADING STRATEGIES in Test Club. Except to make suggestions for new things to test. If we clutter the thread with discussions of the minutiae of each trading thingie (indicator, system, strategy) then this will rapidly swell the thread and make it chaotic and unreadable. Restrict conversations about ways to improve the strategy to the forum where that strategy lives. It's OK to post links.
- The third rule of test club is make suggestions for improving our testing abilities. Better statistics, faster iteration, simplified methods, or whatnot.
- Fourth rule - only one test at a time. Of course you are free to test as many things as you wish, but there is only one official test at one time.
- Fifth and (so far) final rule. If this is your first time at Test Club, you must test. Everyone who participates in discussions is expected to submit test results to the best of their ability. If you don't, the consequences are simply that we will not respect you, and reserve the right to mostly ignore you.
That's good for now, will post more specifics and a new thingie for testing in the near future if there is at least some interest in this idea and a few people following the topic.
May 8 2017 - first system test - Polyjuice Potion https://www.forexfactory.com/showthread.php?p=9845689