A Systematic Approach to Markets

This thread is about my approach to market analysis and robust system development. I'm a computer scientist by training, started out at a market making firm in 2000 implementing then later developing models, moved to a prop desk at an IB, and then set out on my own in 2005. I'm both, a discretionary and automated trader. I have a couple of fully automated models and various gray-box models (they're run at discretionary times with discretionary parameters), but 40% of my trading is still totally discretionary. These days, I mainly trade several of the index futures, a few forex pairs, and select stocks.

Over my 10 years in the industry, I've gathered a lot of experience and information, and the purpose of this thread is to share some of that. I've made a generic mental outline of what I'd like to cover and will do so over a few weeks if the thread shows to be of interest to some people. My first posts may be a little rough around the edges as I formulate how to best approach writing about these things.

PS: If you've got any questions, please don't PM me and post directly in this thread so others can benefit as well.

Well, your first post sounds like you are someone with some serious experience to share. I hope you enjoy your time here and welcome to the Factory.

Ill be checking in on this one I think...

Regards, Leon

The first thing I'd like to talk about is becoming familiar with the market of your choosing. It's always been stunning to me that a lot of traders have no real perspective on what they're trading. Aside from knowing the generics (ie: who are the big players), I think it's very beneficial to build an "image" of the market of interest. The first thing I always do is analyze the statistical behaviors. So, I've taken the time to put together a few things for the EUR/USD.


EUR/USD 2007-12 : 2009-04
Daily Range Mean: 161 pips
Daily Range Stdev: 103 pips
Daily Range Max: 549 pips








Here is a breakdown of the range exhibited by the hour (EST) for
EUR/USD 2007-12 : 2009-04


Hour Pips Stdev Max
0 25.03014 19.52018 185.6494
1 31.46894 17.77241 106.6506
2 43.31991 23.31560 146.9517
3 46.19313 25.81822 268.0004
4 40.06850 23.57592 285.7006
5 38.45960 21.47560 151.6497
6 36.62749 21.12035 143.2502
7 37.95700 22.52928 140.6002
8 53.31827 29.78925 226.3999
9 50.64993 30.01113 214.1500
10 52.78789 28.44433 181.9992
11 45.63094 26.72196 254.9005
12 36.66439 22.66366 152.1492
13 33.39430 23.86578 182.5499
14 36.14067 32.06178 331.8500
15 29.95385 27.67375 204.4499
16 23.44253 20.47866 199.0509
17 20.44553 17.57472 116.6499
18 23.80159 15.57926 106.7495
19 27.19086 18.48608 112.2499
20 30.71478 22.41102 160.2507
21 26.93573 17.66958 126.4501
22 23.60505 17.75578 140.7003
23 23.41537 17.98529 146.6501




This set of data already can give you some very interesting insight. Tomorrow I will add a few more statistics (such as market price distributions) and then do a quick summary of everything.

That's the real deal. I'm reserving a seat here with my chips waiting for the next post. Way to go Jupiter

Seems like an intelligent, alien life form has invaded FF. I am hereby subscribed to this thread.

Hi Jupiter,

Your stats show what seasoned traders already know... Europe/London and US open times produce the most pip/point moves/volume in the day. However, it is nice to see that you have laid it out here to see visually and have taken the trouble to list your current E/U findings for others.

I am curious to know where you are heading with this, so please find another subscriptor in me!

Fire away, fella...

Hello and thanks for the generosity. Hereby signed up.

Lenoxer

interesting.

It's interesting to analyze when on average a daily range will have completed. The following stats are for the EURUSD (same time range as from my prior posts).

1am -- 1.162791%
2am -- 1.860465%
3am -- 3.023256%
4am -- 3.720930%
5am -- 8.604651%
6am -- 12.325581%
7am -- 15.116279%
8am -- 24.186047%
9am -- 31.162791%
10am -- 36.511628%
11am -- 43.953488%
12pm -- 51.162791%
1pm -- 56.046512%
2pm -- 62.325581%
3pm -- 67.441860%
4pm -- 69.534884%
5pm -- 73.953488%
6pm -- 77.674419%
7pm -- 80.930233%
8pm -- 83.255814%
9pm -- 87.441860%
10pm -- 90.465116%
11pm -- 94.651163%
12am -- 100.000000%




Along with analyzing the daily range as I did in the prior posts, a chart displaying daily volatility (calculated from intraday data via Garman-Klass estimator) can be fairly revealing.




And here are the results of a GARCH model fit to the volatility data and the predicted volatility cone:


Volatility Prediction:
meanForecast meanError standardDeviation
1 0.0881038 0.06401426 0.03218258
2 0.0881038 0.06401426 0.03256767
3 0.0881038 0.06401426 0.03294827
4 0.0881038 0.06401426 0.03332451
5 0.0881038 0.06401426 0.03369656
6 0.0881038 0.06401426 0.03406454
7 0.0881038 0.06401426 0.03442859
8 0.0881038 0.06401426 0.03478883


And here's a market distribution histogram for a sample day (April 8th):



Lastly, the past few months the EUR/USD has enjoyed positive and negative correlations with the general market (I'm using the S&P500 futures for calculations). For the most part, as the market was tanking, there was a clear flight to safety into the dollar, and as it was rallying, the euro gained strength (negative correlation). Lately, the two instruments have become positively correlated, and for the most part (due to major government focus) the S&Ps have been leading.

ES EURUSD
ES 1.0000000 0.7918692
EURUSD 0.7918692 1.0000000




So, what does all of this data even mean? As Limstylz pointed out, most traders know what times of the day produce the largest moves – namely the overlapping sessions, and specifically the opening times of these overlaps, but I know most traders (from my experience, even in the institutional environment) don’t go much further than that. What I’m doing is QUANTIFYING this general knowledge to directly apply to my daily trading. I think my point would be best clarified with an example.

Let’s say that I’m trading a mean reversion strategy. Every day I will try to formulate a gameplan for that day’s trading. I’ll look at the daily range charts, volatility charts (including volatility GARCH prediction), and market price distributions. The daily range/volatility charts allow me to create a confidence interval of what kind of range/volatility I’ll be expecting for the next day. The market distribution charts help me determine areas of supply and demand (market auction theory/market profile – I’ll try to cover this in detail sometime in the thread as I find it to be tremendously useful, but it’s a large topic which I want to structure properly before doing so). Now, let’s say that at 3pm EST I get a price action based reversal signal (or an indicator signal or w/e). The range and volatility have stayed within my predicted interval and price dead stopped at a out-of-value area extreme congestion point (1.3182 for example on the market distribution histogram I posted). Combine all of this data with the fact that I know that 67% of the time the daily range will be completed by 3pm, and the fact that the S&P is highly correlated and is majorly overbought and is screaming reversal PA right now, I can get some major confluence for my EUR/USD reversal signal and mean reversion strategy.


Anyways, I'm going to take the next few posts to discuss edges, psychology, and the statistics of position sizing. I should have something together to post tomorrow. In the mean time, I'd be curious to know what any of you think an edge is and what you think the best sources for edges are (ie do you think psychology is an edge?). Also, how do you go about position sizing?

Let me start by saying that your ideas about daily trading ranges and price action fall into line with the Igrok method outlined in this forum, (a strategy that I am studying and trying to implement in my trading, with varying degrees of success, but haven't quite nailed yet.) Therefore I am keen to understand how you analyse your chosen market as I think this might be key to assist in my own trading.

Your question about edge is a difficult question to answer with any degree of certainty but I believe that intimate knowledge and in-depth chart and market analysis of a chosen pair as outlined in your second post is probably the best edge you can have. I agree that psychology is important and it is something that I struggle with on a daily basis, but I hope that as my technique improves, so too will my fear and greed.

I am experimenting with enlarging my position size on profitable trades (if price action agrees) at fib/SR levels, again with varying degrees of success. I think that where I am failing in this instance is incorrect chart analysis, but otherwise the theory is sound.

I look forward to your thoughts and next post.

Now, I was going to write some really clever comments here, but then it dawned to me that it is quite difficult to put in words, what an edge really is. Anyway I will give it a try. Speaking from my viewpoint, which is strictly quantitative, an edge has 3 components:

1) a set of Market events which has non-random characteristics
2) one or more of these characteristics are exploitable.
3) a hypothesis that explains these characteristics within a given theoretical framework of the Market.

Market here denotes everything that is relevant to trading. Far from all non-random characteristics of the markets are exploitable so 1) and 2) are strictly necessary in order to gain an edge. 3) is not strictly necessary, since some characteristics can be very, very hard to explain. But having some sort of explanation can lead to a better exploitation and to a better estimation of when the edge might appear or disappear.

And yes, psychology can be an edge, but not the way most people view it here (at least not from a quantitative perspective). Knowledge of behavioral psychology can be of value when looking for and refining edges.


I will be enjoying this thread.

Take a look at Cable in the picture below. The red lines outline the average daily trading range for the past 30 days (00:00 - 23:59 GMT). What is interesting is how the range completed this morning at midday GMT. True to the range, price retraced back into the daily average range after completing its move.

I attach both H1 and M1 pictures for condsideration.

Limstylz:

I will take a look at the Igrok method later today and let you know what I think. From a quick preliminary look at your posts and description, it looks interesting, but generally I don't like using a rolling window of simple averages (the ranges in this case), because it distorts the distribution. This is why I combine the daily range, fit a cubic spline to get an average (which guarantees equal distribution above/below mean), and the volatility/volatility forecasts. Of course, this is a lot more complex than just taking the ATR, but I've achieved much better results.


BlackMage:

I think you basically nailed this one.




I believe that an edge is simply taking advantage of the non-random occurences in the market. All of these occurences are quantifiable and persistent through time. A very simplistic example of an edge is a price "trend". Every market I've analyzed has some trending tendency, some of course more than others, but incorporating that tendency by placing orders in the direction of the trend would be a proper exploitation.

That being said, the best edges are driven by the most defined non-random occurences found external from the actual price data. When I was a trader for an IB, one of the best edges we had was knowing when mutual funds would rebalance their portfolios. Measuring the in/outflows of money into specific instruments gave us a deep insight on how these operations were investing. Given their huge capital requirements, they obviously moved the market, and we then simply were front running their orders. During the recent flight to safety in the dollar, it was a known that large players were buying in during the asian session. This continued for a couple of weeks, and formed the slow creeping trends around WAP during that time. This was very easy to quantify and exploit.

Most people believe that an edge in a specific trading strategy is anything that can give positive expectancy. Sometimes this could be the case, but in my opinion it is important that you understand if the strategy is exploiting non-random occurences in a persistent manner; if the strategy is not doing this, the short-term "fitted" model will fail when the market reverts to its true, more random, behavior. -- Just look at, I'd venture out to say, 95% of the automated models published here.

When I started my first trading job, my boss came up to me in the first week, sensing from my looks (and probably my P/L statements), that I was really really stressed out. After he gave me a little pep talk, he started walking away and loudly proclaimed: "Remember, psychology is a crutch for those not knowing what's going on." To this day I still reflect on that simple, short, and ambigious sentence. I'll leave it at this for now, until I get some more time to go further into the psychology topic.

Now you have caught my attention, please dont use terms I dont understand . By cubic spline, I presume that you are somehow curve fitting your data using three (cube?) or more statistical points?

Actually the ranges are a very minor factor in the vast Igrok thread... worth reading at some point, but looking at what you wrote in your last post, is no where near as in depth as your analysis.

Please enlighten us further...

http://en.wikipedia.org/wiki/Spline_(mathematics)

Specifically, I'm doing cubic spline smoothing. I've found splines to be very superior to standard moving averages, especially for reversion to mean strategies.

Having read the wiki (or rather tried to follow it using my schoolboy maths), I'm afraid that this subject is a little beyond me.

I will keep silent now and try to follow what is here as best I can. I dont expect special consideration, but please keep in mind that not all of us who wish to follow your thread are as mathematically inclined as you so obviously are.

Hello JupiterOGX,

First of all, many thanks for starting this thread and thank you for your hard work sharing what you gathered on this pair.

I really would like make sure I understand the approach you are presenting. I tried one year ago to "profile" EUR/USD but did not go into that depth. I am still using my findings on average range and volatility in my trading strategy.

To be honest I am a little but frustrated by my lack of knowledge: I tried to reproduce the calculation you have done in post #9 but didn’t succeed.

Is it possible for you to post the calculation you are using in an Excel spreadsheet? So we can follow the computation and refresh it latter on by ourselves?

I understand lots of work has to be done on my side; but I am a little bit stuck right now…

Thank you. Kind regards

Limstylz:

I realize that not everyone has the more advanced mathematical basis to understand some of the things that I'm discussing, that's why I kept the topic fairly short for now. I'm planning on revisiting these things and explaining the mathematical approach in more detail in the future. What I do encourage everyone to do is polish up their mathematical/statistical knowledge as it will give you a whole new perspective on trading. I'm going to continue discussing psychology, edges, position sizing, and then move onto system development and testing. Some of these topics inherently deal more with mathematics than others, but I'll try to keep things as simple as possible from now.



Deimos:

If you're interested in analysis of market distributions, I can highly recommend reading "Mind over Markets". It's an introductory book into market profiling. I'll do my best to clearly explain how I'm using the principles, but for a more in depth study, please refer to that book. Secondly, I don't use Excel for any of my calculations. I prefer using Matlab and R, and lately have only been using R because of wider use in the academic/financial/scientific community. http://www.r-project.org. I've developed my own analytical packages over the years in Matlab and R, so I can't just hand you the code, but on the weekend I can take some time to put together a few scripts so you can do these calculations yourself. As for the actual calculation, I'm assuming you're referencing how I'm creating the market distribution graphs -- basically I read in tick data for some period I want to create a distribution for and record the number of times price ticked/traded at that number. Afterwards you just need to create a histogram from this data.

Psychology

I’d like to open up with a quite from Bill James: “Twentieth century man uses psychology exactly like his ancestors used witchraft; anything you don’t understand, it’s psychology.” How much information can you find on psychology in the trading world? There’s an abundance of literature assessing psychology on everything from position management to signal discipline to trade outcome emotional reaction. In one of my earlier posts I mentioned how my former boss told me that psychology is basically a crutch for those who don’t know what they’re doing. A lot of traders truly believe that they’re not performing because they’re hindered by emotional and psychological issues, while I would claim that all of that stress and tension is caused by poor trading, not the other way around. In other words, you can have all the psychology you want, but unless you have a quantifiable edge with positive expectancy along with proper money management system, you’re going to fail. From my institutional experience, I’ve seen a lot of traders come and go, and a lot of them blamed their inadequancy on psychology, but I can honestly say that most of them had no real edge. I’m not saying that a good psychological handle can’t improve a skilled trader, but rather that most people aren’t at the level where it’s going to make much of a difference.

Having said all of that, I’ve probably made every psychological mistake before achieving my current status. At this point when I’m trading, I don’t get emotional when I lose, but I also feel nothing when I win. I think I’ve gotten as close as to emotional indifference as possible – I don’t see risk in any trade, I only see what percentage of my capital I need to lay on the line to find out if a prospective opportunity was worth my time. Sadly, this desensitized mindset has penetrated other aspects of my life, and I don’t quite enjoy the spectrum of emotions like other people do, but that is another topic.

Anways, for this post I’d like to go over a few of the psychological biases that I’ve seen in others and experienced myself over the span of my career. Self-attribution, overconfidence, information availability, short-term perspective, hindsight, and confirmation biases.

When traders experience the self-attribution bias, they take credit for the good things that happen to them, such as making a profitable trade, but then tend to attribute bad experiences, a losing trade, on others, the market, the weather, bloomberg, santa claus, or whatever. The problem here is that it becomes impossible to analyze the true situation and learn from mistakes. Additionally, it makes it easy for traders to attribute success to what really is pure luck, which leads to overconfidence.

Overconfidence is a big problem I consistently see, especially in beginning traders. Often this bias is developed through self-attribution (like random trading luck) and it fosters the trader to completely overestimate their skill level and go on to make more serious mistakes. Good traders are confident, because they know how much time and effort they’ve put into becoming good traders. There’s a clear cause and effect. Don’t mix up the two.

Having a short-term perspective will lead to setting irrational goals and making irrational trading decisions. Traders will take large risks to achieve short-term goals while ignoring the long term guaranteed outcome (95% of the forex mean reversion strategies I’ve seen in the public domain). This is a dangeous perspective to have, especially if you’re riding high on overconfidence and have just experienced major self-attribution bias.

The information availability bias basically indicates that people most often take and process information that is easy and readily available. A good example of this would be a newcomer to a trading board reading all the threads/posts of how traders are making millions, turning $1000 into $100000 within a few weeks. This can breed the other biases, but will eventually also lead to major frustration, when a newbie doesn’t experience anything near that level of success (most probably a blowout too). This illusion of excellence is fostered by the availability bias, and it masks the reality that most traders are losers, are blowing up, or have just plain quit the business.

When experiencing the confirmation bias, one reacts much more favorably to confirming evidence than to contradictory evidence. I can’t remember how many times I was stuck in a position which gave me a good signal, but I refused to acknowledge all of the contradictory signals that followed thereafter. Instead of getting out breakeven or with a small loss, I took a full stop. Part of being a proficient trader is absorbing all information, good and bad, and then systematically processing it.

Everything is easier to see in hindsight. It’s easy to go look at a chart of the EUR/USD and seeing a 100 pip move, and in hindsight seeing all the signals that indicated that particular move. It’s much easier for people to grasp a single outcome, than to imagine all the possible outcomes of some situation that did not occur. This bias is particularly nasty as it leads to all of the other ones and it encourages people to overestimate the accuracy of their predictions. In other words, it’s like fitting a model to past data.

As I’ve noted before, I’m guilty of committing all of these errors. I didn’t become a consistently profitable and well-balanced trader overnight. There’s a reason that the failure rates are so high; trading is one of the hardest and most stressful occupations. You can’t become an engineer in a few weeks, so don’t be expecting to become a master trader quickly. In my next post, I’ll go over some techniques to overcome making these mistakes.

Starting to feel an affinity with this thread and it is the first time I have posted this voraciously, but I think you struck a chord... Having read your last post, looked in the mirror, and what re-read what I wrote earlier about my own fear and greed, I can see that I am still using my emotions as a 'crutch'.

I, for one, definitely appreciate your candour...