Strategy: Inside Bar + ATR - help needed

Hello, fellow traders
I need your help in development of the IB strategy (or better say - methodology) which will be adjustable to changing market conditions. I've got a strong gut feeling that I'm close.... but some small piece of puzzle is missed.

Intro
Last couple of weeks I'm trying to implement the strategy which utilizes IB. What I've encountered that in general the idea of using inside bars as some entry signal is quite good but it requires some additional filters to skip false signals. Also to make the strategy be able to work in the long run (3-5-7 years) in the conditions when the market is constantly changing we need some methodology to adjust these filters to these new conditions. What I've come to is just adding an ATR and to measure the "parent" bar from inside bar pattern against it - i.e. if "parent" bar's body N (let's call it ATRCoeff) times bigger than ATR for some prev number of bars we are considering this as a significant signal and placing our orders.
As the result I've got a strategy based on only 4 parameters - ATR period, ATR shift, % of balance to risk (balance risk ratio) and Parent Bar Body to ATR ratio. And getting the right combination of these four parameters we can have a strategy which can be adjusted to any market conditions. But the question which I would like to raise - and which I would like you to help me to answer! - is what methodology can be used to find these combinations of parameters which more or less suitable for the current market. I've some thoughts on this - they will be at the end of this post.
Sp here we go...

Strategy
Setup:
1. Waiting for IB pattern to form
2. Checking if "parent" bar is ATrCoeff times bigger than ATR
3. If yes - pleasing SELLSTOP on the low of "parent" bar and BUYSTOP on its high
4. When one of these orders is hit - deleting opposite one

Money Management:
1. We are risking only some % of our balance - it's our RiskRatio coefficient
2. SL in pips is calculated based on RiskRatio parameter
3. TP is taken is either SL in pips or "parent" bar's (whatever is higher)
4. Once the we have opened order - after some period of time trailing the SL.

Here is the EA


Parameters:
ATRPeriod - numberof bars for ATR to calculate;
ATRShift - shift ATR for specified number of bars;
ATRCoeff - this coefficient sets what should be the ration "Parent Bar's body"/ATR;
RiskRatio - what percent of account we are risking per trade - i.e. if it's 0.05 then we are risking 5%;
LotsPer100 - this parameters sets how many lots we are setting per each $100 of balance; (set to 0.02)
TrailStopIntervalMins - time in minutes to check for SL trailing; (always set to 60 by default)
TrailStop - turn trail stop on and off; (always set to ON)
OrderShiftATR - this paameter allows to shift pending orders high/lower based on ATR value ; (for now always set to 0)

Experiments
In order to figure out - whether in general this strategy works I've performed following experiments.
For each year from 2006 to 2015 I've tried to find some combination of these four parameters - ATRPeriod, ATRShift, ATRCoeff and RiskRatio - which is close to optimal - i.e. give the best possible result in terms of balance.
Initial Balance: $3000
Pair: EURUSD

2006:
ATRPeriod = 12
ATRShift = 2
ATRCoeff = 2.5
RiskRatio = 0.07



2007:
ATRPeriod = 18
ATRShift = 1
ATRCoeff = 1.7
RiskRatio = 0.05



2008:
ATRPeriod = 20
ATRShift = 0
ATRCoeff = 2.2
RiskRatio = 0.05



2009:
ATRPeriod = 10
ATRShift = 2
ATRCoeff = 1.5
RiskRatio = 0.03



2010:
ATRPeriod = 12
ATRShift = 5
ATRCoeff = 1.4
RiskRatio = 0.06



2011:
ATRPeriod = 10
ATRShift = 3
ATRCoeff = 1.2
RiskRatio = 0.1



2012:
ATRPeriod = 10
ATRShift = 0
ATRCoeff = 1
RiskRatio = 0.06



2013:
ATRPeriod = 14
ATRShift = 4
ATRCoeff = 2.7
RiskRatio = 0.06



2014:
ATRPeriod = 12
ATRShift = 0
ATRCoeff = 1
RiskRatio = 0.1



2015:
ATRPeriod = 18
ATRShift = 5
ATRCoeff = 1.3
RiskRatio = 0.08

(didn't manage to attach - says max num of atachments reached, Total net profit - 12227.26)

Please also note that I've selected the best results I've found - but there are also lots of different parameters combinations which easily doubles and triples account YoY.
So based on these stuff I've made a conclusion that the strategy is pretty ok, it can give a significant multiplication of our deposit.... but! Here comes the problem which I need your help to solve...

Problem
And the problem is: what methodology/approach/??? etc can we use to tune these four parameter - i.e. from time to time - each 1, or 3, or 6, or 12 months - check the current market state, identify the combination of parameters which is not necessary optimal but more or less close to, and use this combination to trade. After 1-3-6-12 moths again - look at the market, tune parameters, continue. From my perspective the success criteria will be following: backtesting this methodology could allow us on average to at least double the account YoY and for at least 10 years. Why doubling?
Here are the approaches I'm testing right now:
Let's say we want to tune parameters each 1 month. So I'm trying to find how I can more or less closely predict ATR based on the 3-6 moths of previous history.
How based on previous 3-6 moths we can predict whether we should increase RiskRatio coefficient - i.e. start trading more risk - or in opposite - it should be decreased.
I aslo feel that it somehow similar to how the people adjust their trading decisions based on what they see - something like "I'm looking on the chart, checking X,Y and Z and saying to myself - now the market is in condition A(or B,C.....)" - so I need to figure out what to pay attention to when looking onto the bigger picture.

So... what do you think? In fact I'm a little bit lost in all these combinations so any ideas, suggestions, critics, pointing on my mistakes - will be highly appreciated.
I'll be also posting my results here according to the progress in my research. Also I believe that if collectively we'll figure out the approach - as the result we'll have a really really powertool (semi)automatic tool to trade with good profits and in the long run.

I would suggest you first find out which ofthe 4 parameters affects the logic the most, by keeping all others constant while varying each one, that way you can narrow down your required adjustment variable. Then a rule of thumb I read somewhere is to always check profitability over a number of trades e.g every 30 trades your settings should produce a minimum amount of wins.

Good luck

a screenshot of a trade would be much helpful to understand the logic and method...

Hi sinqua, thank you for idea - I'll try that and go back with the results.

Hi tradeforlife, yes, sure, I'll post couple of screenshots today

Hi Globus,

I like your idea. I tested IB's myself but never really followed up to a system development. The ATR component might be a good addition. Some thoughts. If the Mother Bar is a very large bar there is potential for a period of consolidation. Not always but this happens so maybe your ATR coeff can use a lower bound filter and an upper bound?

I would be careful with the parameter optimization. Soon, you can run in to the trap of overfitting. Best is to think about your parameters logically and assign a range of values that you like then test best ones between this range.

Another possible something could be to identify your most probably direction for the breakout. This way, you could limit the number of false breakouts (=Hikake patterns). Clearly you might have less trades but increased profitability could make up for this. Another possibility is to reverse when you get stopped out the first round.

The ATR lookback period. I would suggest a weighted average of ATR values with more weight to more recent values. This could be something like this:

(4*ATR(5) + 3*ATR(20) + 2*ATR(50) + ATR(100) ) / 10

This could bring your ATR values in line with current market conditions better then just one ATR value I think.

Could you explain this ATR shift a bit more perhaps?

Looking forward to more testing!

Hi tibovwh (and all other guys)!
First of all I want to apologize for not writing for the whole month - but I have good reason for that: I've got an idea on how to solve the problems with parameters values which are constantly changing from year to year (I'll write details below).

tibovwh, so first of all - thank you very much for your idea on ATR calculation and usage - it seems like very fresh, i'll perform backtests and will post results.

Regarding ATRShift - this is the parameters which shows how many periods (i.e. hours) back's value of ATR we should take - for example if the parameter is 1, then the strategy will be using ATR for the previous hour, if it's 5 - the ATR value 5 hours back, if 0 - it will use current ATR value. To be perfectly honest, I've introduced thisparameter just for the experiment, but I can see that it can significantly affect the strategy performance. Unfortunately, I don't have a strong "scientific" proof why we should use it at all - there could be the cased that by using it in the back-testing we could just take the value which are more suitable for us - so you're right, we could have just over fitting here

So going back to the problem I've outlined in the original post: as market changes over time, our strategy should adopt its parameters respectively - but how to figure out whether we should decrease ATRCoeff or on opposite - increase it?
While investigating possible solutions I've come to idea of using ANN (hope guys you know the basic concepts, if not - just read the Wikipedia or something like this - the concept itself is pretty simple to grasp). So the idea is following: I want to have a neural network, which will be trained to produce strategy parameters' value for the future periods based on the previous data. For example, each 3 months some data will be fed to the ann (let's say 1 year of data for the percent change of ATR, MAs etc) and as the result I'm expecting to get the parameters of that my coefficient which I expect should be optimal or somehow close to optimal. Then using this coeffs we are trading for 3 months, then re-training the network and getting new coefficients, and so on. So the idea here is to make the strategy constantly adopt itself.
The main question which I'm trying to solve right now is to figure out: does the history is realy guide - I mean, is there really any correlation between historical data and these parameters? Unfortunately this work takes lots of efforts but after about a month of work I managed to implement my library which uses 3rd-party ANN sdk and integrate it in one of my EA - so for example, I'm able to predict the moves of simple 5- and 10-day MAs (change of their values in %) for 1-3 days ahead with quite reasonable accuracy (about 75 (3 days ahead) to 90% (1 day ahead) of correct predictions). So what I'm going to do now is I'll try to teach this ann-s to predict my parameters and check, whether it's possible in general. Here actually we are stepping into very interesting territory

More info to come!

P.S. BTW small "philosophical" observations on strategies, optimizations etc (I decided to place it here because I just would like to share my thoughts with someone who can understand me - there is no technical details on strategy itself so you can just skip it ).
Recently I've just noticed that in fact in my MetaEditor I have lots of strategies, which actually quite fine and more or less profitable. In fact there's absolutely no problem to get any strategy which have some more or less theory behind it ("theory" I mean for example "why we should by when two moving averages acre crossing" etc). The problem is that most (if not all) of these strategies are heavily dependent on the values of the parameters. Usually when we (at least me, but I believe you too ) find some strategy (for example on this forum or somewhere else) which looks ok, it works only for some period of time - (week, month or year). But after that it just fails. And of course we understand that the issue here is the market which has changed - the strategy was optimized by it's author for some limited period of time, it works on that period, maybe even shows some good results 2-3 months after that period, then it fails, and we are starting the search of new strategy which will work for next 2-3 months again... and again and again... But all we have as the result - is just another variation of few well-known types (ma-s, stochastics, IB's - you name it). In reality there is not much stuff there, what we need to have to be successful - is maybe 2-3 (in worst case 5) strategies in our belt, and we just need a routine to adjust them from time to time. So in it doesn't matter with which strategy you trade - if it has some reasoning behind it (something different from "buy when your cat is licking his left ball, sell - when right") and we are able to adopt it constantly - then you can have a stable income YoY. If it's more or less clear with reasoning, the question is - how to perform this adoption.

As I understand it , why open 2 pending orders , if ib pattern has a clear definition of the movement. If it is broken , it does not ib pattern and this can lead to losses.

There could be different strategies - you can look here for details https://www.dailyfx.com/forex/educat...rade_them.html

So I'm currently utilizing the second - a breakout in one direction or another.

HI Globus,

Great to see you are still exploring this idea further. Im sure there is a system to be found in something as simple as an IB Breakout strategy.

The only thing that somewhat bothers me in your whole explanation is the fine line between calibrating and overfitting. I really think you need to watch out and not fall into this trap.

Im pretty interested in this whole machine learning stuff. I have not researched it yet but I will do so in the coming months. Again: fine line between optimizing and overfitting.

On a sidenote: I think you might be interested to look at GARCH modelling. forecasting volatility is certainly possible and is a regular practice in finance. The general idea is that volatility has an unconditional mean from which is deviates but we expect it to mean-revert back to this value.

Its quite busy atm but I would certainly like to find some time to discuss some things about your strategy! I will send you a PM

Best,

Hi tibovwh

Once again - sorry for pauses in response - spending all my free time on this investigation and development stuff
And first of all thanks for your interest in the topic - this really motivates me
Second, I'd like to confirm that your topic "calibration vs over-fitting" is absolutely relevant - and it's actually what I'm trying to achive with all that ANN stuff. In terms of overfitting - of course I can just try to perform strategy optimization for (let's say) 5 last years, obtain +300% gain each year, get these parameters value and say I would say it's something like this: we are driving down the road at 100 mph and want to know about all the turns and bumps for example in 500 meters in advance. In general if we'll be able to do this we can pass these turns not ideally but quite efficiently.

Also thank you for GARCH modelling - I'll definitely look into this.

And now about machine learning. I've made some initial experiments and can share some results.
So first of all - Methodology
In my strategy I've got 3 main parameters which affects the trading results. The idea is (as I've already mentioned before) is to pass some inputs to ANN and I've want to get these 3 parameters predicted, using thesevalue we re trading for next let's say 3 months, ten training network again, getting prediction, trading next 3 months etc.
Training. Pair for training - EURUSD. So first of all for the period of time from 1.1.2005 to 1.1.2016 with step of 1 month using the MT4 optimizer I've identified the ideal combinations of parameters - its our outputs.
As an inputs I'm feeding the ANN with the % of change for ATR indicator and EMA. I'm taking the data for the last 6 months. So for example: in order to figure our parameters to trade in period from 1.1.2005 to 1.2.2005 (1 month) I'm feeding the ATR and EMA data for period from 1.7.2004 to 1.1.2005.
ANN configuration. Currently I'm using simple network with one hidden layer, sigmoid activation functions for the neurons, number of neurons in hidden layer waries from 2x to 5x of inputs counts. Also I'm using the committees of network with different configurations so they perform "collective" predictions.
Results. Fail Yes, the network performs some predictions, but they are absolutely irrelevant. So it seems like either inputs are not valid or the relations between inputs and outputs are much more complex and cannot be reflected with single layer.
What I'm going to do next. 1. Try different inputs - technical indicators, economic data (inflation, gdp etc...), data from correlated pairs... 2. Different network configurations - number of layers, number of neurons in each layer, activation functions.

Unfortunately it's very slow process - it took me usually hours because I'm training hundreds of ANNs, selecting the best ones for commitees (kinda Genetic Algorithms), than backtesting - applying parameters to strategies and checking the results etc... But I'll continue sharing my results here.

Also I'll check in parallel the modelling methods you've mentioned - I believe combining different approaches could lead to the breakthru.

Stay tuned!

P.S. BTW! While playing with all that inputs I've implemented a bunch of MT4 of indicators which displays US economic data and COT reports for fin futures and options. So I can literally use these indicators in my EAs, export data for ANN feed etc. In fact it's the service which collects and aggregates info from different sources (US bureau of statistics, CFTC etc.) and passes to indicators for display. So if anyone is interested - I'll be happy to share these indicators with you. It's an official statistics so I presume all of us can benefit from using them in our trading. So again: if someone is interested - I'll share these indicators with you guys. I'm currently in the stage of final testings.

Looks like an EA Abowaik did.

Hi Soros

If you're talking about IB breakout - then probably yes I'm not the first one here with it.

I hate the fact that i get lazy to read something that seems long and complicates, without reading your whole strategy, it seems that you use fixed stops, well with constant changing market conditions as these i have seen that strategies that use fixed sl and tp usualy are curve fittn type system, a system that makes it through the test of time is a system that uses dynamic SL on previous highs and lows. thats just my 2cents.

Not 100% true My SL are identified by the RiskRatio and lots per $100 parameters. But in general I agree with you - all the system's parameters should be functions of the current market state.

I have idea about how to trade IB, but only best ones. In my opinion best IB is which demonstrate certain market structure - like
Impulse wave - corrective wave - resume of impulse wave. Please see attached picture.

How to find these moments? I can suggest is to use neural networks with image recognition features. Take some examples, train neural network on those examples and let it to find these ideal patterns around all pairs. Those patterns repeating across all pairs, not often, but periodically. If someone can wrap this knowlegde in code this can be very good system, with very small risk and big reward.

Hey Tenax, the idea is great. Something similar regarding market patterns recognition is also spinning in my mind but until your post I didn't have any specific idea on how it can be applied. Definitely worth trying.

Ok, then what we need is to let's say "strick" definition of what the impulse looks like. At the beginning even just simple description which I can use looking onto my charts. Could you share with us how do you identify them? I mean maybe you have some checklist with conditions to be met?

Some example.
Previous day closed above open, so today we look only for longs above day open. We need to look a strong bullish move from day open, then small corrective wave against initial wave, then new candle in original direction. So we look for bullish inside bar.
Previous day filter is very powerfull but sometime we can miss some good moves with it. So it can be optional.

Today on AUDUSD, not a ideal, but very good.