Is random really random?

which camp do you belong

Threats to Conclusion Validity
Threats to Conclusion Validity
A threat to conclusion validity is a factor that can lead you to reach an incorrect conclusion about a relationship in your observations. You can essentially make two kinds of errors about relationships:

1. conclude that there is no relationship when in fact there is (you missed the relationship or didn't see it)
2. conclude that there is a relationship when in fact there is not (you're seeing things that aren't there!)

Most threats to conclusion validity have to do with the first problem. Why? Maybe it's because it's so hard in most research to find relationships in our data at all that it's not as big or frequent a problem -- we tend to have more problems finding the needle in the haystack than seeing things that aren't there! So, I'll divide the threats by the type of error they are associated with.
Finding no relationship when there is one (or, "missing the needle in the haystack")
When you're looking for the needle in the haystack you essentially have two basic problems: the tiny needle and too much hay. You can view this as a signal-to-noise ratio problem.http://www.socialresearchmethods.net...s/signoise.gifThe "signal" is the needle -- the relationship you are trying to see. The "noise" consists of all of the factors that make it hard to see the relationship. There are several important sources of noise, each of which is a threat to conclusion validity. One important threat is low reliability of measures (see reliability). This can be due to many factors including poor question wording, bad instrument design or layout, illegibility of field notes, and so on. In studies where you are evaluating a program you can introduce noise through poor reliability of treatment implementation. If the program doesn't follow the prescribed procedures or is inconsistently carried out, it will be harder to see relationships between the program and other factors like the outcomes. Noise that is caused by random irrelevancies in the setting can also obscure your ability to see a relationship. In a classroom context, the traffic outside the room, disturbances in the hallway, and countless other irrelevant events can distract the researcher or the participants. The types of people you have in your study can also make it harder to see relationships. The threat here is due to random heterogeneity of respondents. If you have a very diverse group of respondents, they are likely to vary more widely on your measures or observations. Some of their variety may be related to the phenomenon you are looking at, but at least part of it is likely to just constitute individual differences that are irrelevant to the relationship being observed.
All of these threats add variability into the research context and contribute to the "noise" relative to the signal of the relationship you are looking for. But noise is only one part of the problem. We also have to consider the issue of the signal -- the true strength of the relationship. There is one broad threat to conclusion validity that tends to subsume or encompass all of the noise-producing factors above and also takes into account the strength of the signal, the amount of information you collect, and the amount of risk you're willing to take in making a decision about a whether a relationship exists. This threat is called low statistical power. Because this idea is so important in understanding how we make decisions about relationships, we have a separate discussion of statistical power.
Finding a relationship when there is not one (or "seeing things that aren't there")
In anything but the most trivial research study, the researcher will spend a considerable amount of time analyzing the data for relationships. Of course, it's important to conduct a thorough analysis, but most people are well aware of the fact that if you play with the data long enough, you can often "turn up" results that support or corroborate your hypotheses. In more everyday terms, you are "fishing" for a specific result by analyzing the data repeatedly under slightly differing conditions or assumptions.
In statistical analysis, we attempt to determine the probability that the finding we get is a "real" one or could have been a "chance" finding. In fact, we often use this probability to decide whether to accept the statistical result as evidence that there is a relationship. In the social sciences, researchers often use the rather arbitrary value known as the 0.05 level of significance to decide whether their result is credible or could be considered a "fluke." Essentially, the value 0.05 means that the result you got could be expected to occur by chance at least 5 times out of every 100 times you run the statistical analysis. The probability assumption that underlies most statistical analyses assumes that each analysis is "independent" of the other. But that may not be true when you conduct multiple analyses of the same data. For instance, let's say you conduct 20 statistical tests and for each one you use the 0.05 level criterion for deciding whether you are observing a relationship. For each test, the odds are 5 out of 100 that you will see a relationship even if there is not one there (that's what it means to say that the result could be "due to chance"). Odds of 5 out of 100 are equal to the fraction 5/100 which is also equal to 1 out of 20. Now, in this example, you conduct 20 separate analyses. Let's say that you find that of the twenty results, only one is statistically significant at the 0.05 level. Does that mean you have found a statistically significant relationship? If you had only done the one analysis, you might conclude that you've found a relationship in that result. But if you did 20 analyses, you would expect to find one of them significant by chance alone, even if there is no real relationship in the data. We call this threat to conclusion validity fishing and the error rate problem. The basic problem is that you were "fishing" by conducting multiple analyses and treating each one as though it was independent. Instead, when you conduct multiple analyses, you should adjust the error rate (i.e., significance level) to reflect the number of analyses you are doing. The bottom line here is that you are more likely to see a relationship when there isn't one when you keep reanalyzing your data and don't take that fishing into account when drawing your conclusions.
Problems that can lead to either conclusion error
Every analysis is based on a variety of assumptions about the nature of the data, the procedures you use to conduct the analysis, and the match between these two. If you are not sensitive to the assumptions behind your analysis you are likely to draw erroneous conclusions about relationships. In quantitative research we refer to this threat as the violated assumptions of statistical tests. For instance, many statistical analyses assume that the data are distributed normally -- that the population from which they are drawn would be distributed according to a "normal" or "bell-shaped" curve. If that assumption is not true for your data and you use that statistical test, you are likely to get an incorrect estimate of the true relationship. And, it's not always possible to predict what type of error you might make -- seeing a relationship that isn't there or missing one that is.
I believe that the same problem can occur in qualitative research as well. There are assumptions, some of which we may not even realize, behind our qualitative methods. For instance, in interview situations we may assume that the respondent is free to say anything s/he wishes. If that is not true -- if the respondent is under covert pressure from supervisors to respond in a certain way -- you may erroneously see relationships in the responses that aren't real and/or miss ones that are.
The threats listed above illustrate some of the major difficulties and traps that are involved in one of the most basic of research tasks -- deciding whether there is a relationship in your data or observations. So, how do we attempt to deal with these threats? The researcher has a number of strategies for improving conclusion validity through minimizing or eliminating the threats described above

Hi Hanover, can you give one example of TA that is proven to give an edge....

60mm,

I don't believe that it's possible to absolutely prove or disprove anything. I think it's more complex than that, and here’s an example which I hope explains what I mean.

Recently I volunteered to be a guinea pig for a group of traders who've developed a system that has apparently been extremely profitable for the last 3-4 years. The entry component is a mix of (1) public domain TA tools that provide a multi-TF road map around which to interpret the price action, (2) proprietary TA-based signal-generating indicators, and (3) some discretion (which pairs to trade, which signals to take, etc). Exits are based on other level-based technical tools, and more discretion (take profits early, let profits run, etc). They are in the process of releasing the system to the public, and they wanted to see how easy it would be for an ‘outsider’ like me to pick up the system, and (hopefully!) become profitable.

I've been experimenting with their system for around 4 weeks, but I started 'serious' trading only last week. On Friday I made 5 trades, 3 were profitable at 2:1 RR, and 2 lost at 1:1 RR (see attached screenshot). I also used some TA-based tools of my own (strength indicators) to help me decide which pairs to trade. I'm still very much learning the ropes, but it was a promising enough start.

I've performed a fairly extensive visual backtest, and have seen enough to conclude that the entries provide some kind of directional bias, with their distinctive way of picking up on momentum at the start of a possible move. But (as with all systems) exits are crucial, and a huge amount depends on how far away to place the SL, and when to take profit.

Every tool used is TA based. But, as far as I can tell, P/L ultimately depends on discretion: a thorough understanding of how to best apply the tools. From what I’ve learned in my discussions with other traders, that's typical of how many discretionary traders operate. In other words, the TA is only part of the overall methodology.

So now the question becomes: what constitutes the kind of proof that you're looking for? How many months or years worth of profitable trading would you need to see, before you're satisfied that the total methodology provides a real edge?

For me personally, nothing would satisfy me completely, at least in the sense that I don't believe that any strategy can offer an absolute guarantee that it will continue to be profitable X years from now. However, I believe that 'edges' somehow grounded in 'orderflow' are more likely to remain robust than the system I've been describing. And I also believe that a knowledgeable, experienced trader knows how to adapt the discretionary aspects to changing market conditions (for example, adjust SL and profit exits depending on volatility). He has performed a SWOT analysis, i.e. knows the strengths and weaknesses of his strategy, and under what market conditions it performs best, and what conditions should be avoided. So in this case the durability of any perceived 'edge' depends largely on the skill and knowledge of the trader.

David

The market revolves entirely around price discovery...when it is directional...otherwise it drifts aimlessly...or is in the hands of non-price discoverers.

Many thanks for the info. And anyway, it's always good to hear from a fellow kiwi. I live in Hamilton.

No probs. In Blenheim.

But yeah...it's a pretty complex process and could take a lifetime to perfect but that's the essence of whats driving direction.

you prove my point, you believe there is an edge even if unproven, to you the constitutes being experienced and knowledgeable. Logic says otherwise as you can not know for sure if it wasn't just luck or fundamentals. The human mind is much happier when it can work with certainties so traders subconsciously mark TA as having an edge even though it's not proven... the brain papers over the cracks for us.... I don't want to go down this route but there is a reason that intelligent people go to war over religion, and it is nothing to do with proof, experience or knowledge...

So we get to the important part - why do most traders fail using TA? because when you approach an uncertain outcome with a belief system that you are likely to be correct only to find you are often not, you are more likely to make bad decisions. If on the other hand you accept that the predictive qualities of any TA system are low then you can look for areas where we are more probable to find profit (using extreme RR etc).

TA without knowing ALL the factors affecting the outcome is a one sided equation (and this is why it has never been proven to work, because it is impossible)

p.s I think you can tell I sit firmly in the psychological trading camp - most systems can be made profitable but nearly all fail through human error, often predicting what will happen instead of executing the system.

Great post.

http://forums.babypips.com/newbie-is...anagement.html

this thread on babypips shows it is possible to make profit from random entries if the risk reward ratio is adjusted, logically we would assume that to be true.... So therefore any technical strategy could be replicated by a strategy with no TA... so therefore TA is probably no better than a coin toss strategy ...

If we add fundamentals to TA then you could argue that then we have an edge using TA but I very rarely see anyone doing that on this forum

It doesnt work,I researched very well the random entry system, and no RR adjustment gives an edge to that.It is only tends to go to positive expectancy but if you mathemally analyze it you find that its limit is actually the spread.So that curve there is the highest at higher 1:1 RR settings but the balance never gets >0.

So yeah TA alone is 100% random, no edge there, but that doesnt mean that TA is completely useless, as i told already TA is only a mean to localize trades, but for the bias/direction you must have fundamental analysis.

That is the recipe for succes in FX.

Digest what the information is saying, over a set period of time a random entry system made profit by adjusting the RR and putting stops outside the noise of the daily swings - then apply that to why TA might or might not work - his figures are there for you to read - if you wanted to publish your results and what your technique was we could compare

as for TA, I have never said it is useless but if someone wants to declare it has predictive qualities I will ask the for prrof and if they do not have any proof ask them why they believe it. Most traders fail because of what they do and do not believe.

I would rather have dental surgery than spend time trying to prove the points I have been making. Instead, here is one chart from last week that outlines my trades on the E/A (I am currently tracking 13 currency pairs each day for trades, so this is just one example). I've stripped away some of the clutter and structure analysis to just point out what should be obvious about patterns and simple, simple TA. On this pair alone, I made over 900 pips on trades 1-6, while breaking even on trade 7 and 8, and taking a small 10 pip profit on trade 9 as the week closed. Entries are not random, but planned. Targets are planned, and not a result of chance. I have faith that the patterns will continue and repeat. When they do, I trade. When they don't, I don't. My experience is all the proof I have to offer, and is the only proof I need to continue trading.

Nice trading GE . You could just be riding the fundamentals though and I think many TA traders do but give it no consideration.

I remember reading that the market is least likely to display randomness when it is being driven by fundamentals...

Well actually TA and FA are part of structure, which is what this is all about. Last week, there were no fundamental changes to structure for either the EUR or AUD, IMHO. I was trading the market that was being technically traded. A FA that changes assumptions would likely change structure, bringing into play a new technical picture. This is basic stuff.

Well said sir. I am an enormous technical trader, but I agree with you.

Well, you have more stamina for these kinds of discussion than I do.

I've already made the points I wanted to, and I'm happy to leave it at that.

The experiment was done wrong, i did exactly the same a few weeks ago and came up with totally different results, here are mine:






As you can see the balance kinda hovers around 0 or 0-spread but it never gets above it, shows that this method is totally random.

I tested 500x500 TP/SL settings and the method where SL=ATR and TP was a multiplication of that from TP=ATR*1 to TP=ATR*10 to work in a 1:1 to 1:10 RR range.It didnt worked.
The initial balance was either 100 or 10.000 in the pictures so you see how the optimization value never gets above it.

The random numbers were 100% random, and the historical data was tick data.So yeah, Myth busted

Prox,

Can you provide some details on the tests you ran to satisfy yourself that FA provides directional bias, e.g. what fundamental numbers did you test, how did you go about testing them, and what the results were?

Many thanks,
David

Haha you are basically asking me to reveal my Holy Grail system

Here is a hint: GDP, CPI, Unemployment and Interest rates

Find the connection between these 4 and you unlocked the mystery of the FX market.

I wouldn't call your stuff basic but it has limitations yes.