Old Dog with New Tricks | Page 18 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Old Dog's Taming of the Beast? 34 replies

Old Dog Fib Trading - EA Development 18 replies

Old Dog Fib Trading - EURUSD 399 replies

5 Tricks of Trend Trading, by Rob Booker 25 replies

How To Work Around A Market Maker's Tricks 0 replies

Attachments: Old Dog with New Tricks
Exit Attachments

Old Dog with New Tricks

  • #341
  • Jul 17, 2010 6:25pm Jul 17, 2010 6:25pm
There really is no way to back test the HiLo Self Optimizing EA like any other EA is there? I've tried and and under the Journal Tab in the strategy tester it says not enough bars on the screen. Is there a different way to do this?
  • #342
  • Jul 17, 2010 10:51pm Jul 17, 2010 10:51pm
Hi Ron

Not sure if I should even be positing as I have not used the EA itself yet, but I thought I would throw out a couple of ideas while they are fresh in my mind - and before I forget them!

 

  1. Weighting: What I mean by this is having an overall number of bars to optimize and then within that having a subset of bars closest to the present that are given a higher weighting. So, you could have the optimizing period set to the previous 1 year of data, but then have the bot weight the latest 3 months of results more than the previous 9. (Kind of like the EMA giving greater weight to the prices closest to the present candle, if you see what I mean.) This should mean that the bot is more likely to be in tune with market conditions as they are presently.
  2. Trading sessions: Instead of making one overall optimized set of parameters, have the EA create 3 sets based on trading conditions in 3 periods of the day. Broadly speaking, we could call them Asian session, European session and US session (ignoring the fact that there are overlaps between these in reality, or we will end up with too many sub-divisions.) This should create sets that broadly reflect the specific trading conditions in each trading session, rather than one overall "generic" set. The bot would then use each set of parameters in turn based on what time of day it was.

Pop these ideas in the recycle bin if they are not useful to you.

I am looking forward to working with this bot and seeing what it can do!

Cheers

Invisible

  • #343
  • Jul 18, 2010 1:17am Jul 18, 2010 1:17am
Quoting Invisible
Trading sessions: Instead of making one overall optimized set of parameters, have the EA create 3 sets based on trading conditions in 3 periods of the day. Broadly speaking, we could call them Asian session, European session and US session (ignoring the fact that there are overlaps between these in reality, or we will end up with too many sub-divisions.) This should create sets that broadly reflect the specific trading conditions in each trading session, rather than one overall "generic" set. The bot would then use each set of parameters...
Hello Invisible,

The bot actually breaks the week down into 7 different divisions for each day of the week and 24 different sub-divisions for the hours. Then only uses statistics from that same hour on that same day of the week to decide its strategy.
  • #344
  • Jul 18, 2010 10:44am Jul 18, 2010 10:44am
Quoting Invisible
Hi Ron

Not sure if I should even be positing as I have not used the EA itself yet, but I thought I would throw out a couple of ideas while they are fresh in my mind - and before I forget them!
[list=1][*]Weighting: What I mean by this is having an overall number of bars to optimize and then within that having a subset of bars closest to the present that are given a higher weighting. So, you could have the optimizing period set to the previous 1 year of data, but then have the bot weight the latest 3 months of results more than the previous...
I'm always looking for inputs and new ideas for this.

 

  1. How should the weighting be done? Should the weight of each trade be calculated based on the number of bars between the trade entry and the present? As for using the weighting, how should I use the weights? Trades with a higher weight contribute more to the accuracy, profitability, and risk/reward readings? Another concern with weighting that I personally have is it turns the optimizations from a "lagging system"--a trading system whose parameters lag the market--into a "leading system"--a trading system whose parameters lead the market. There are pros and cons to each, namely that leading indicators have more false signals. I'm not trying to discount weighting in any way, I'm trying to find proper justification for weighting the testing data.
  2. As rfking mentioned, this EA already uses a more granular level of sessions, and that is the hourly periods. I could add more granularity to the EA, but then you begin to approach more extreme curve-fitting and exceedingly small sample sizes for each granular period.

  • #345
  • Jul 18, 2010 11:01am Jul 18, 2010 11:01am
Ronald, your story makes me a little worried, cos i've often posted some junk code of my own...... dang, U.S. suers are everywhere.......

this HiLo EA looks very fascinating. please allow a dwarf like me to ask a question:

this NEAT led me to think that HiLo EA is cored on (neural network), and i went thru your coding, seemingly it is NOT, but it is self-adjusting to a complicated set of pre-defined (specific rules). am i right? but are you working towards to use a neural network based EA? if so, MQL can be combined with neural network library in MATLAB (which is most accessible NN tool)......

just an idea if your goal is to develop an NN based EA. i know i might totally mis-read your code and ideas, if so, sorry for that......

Quoting Ronald Raygun
I was hoping for some of the giants like SteveHopwood, Hanover, Hayseed, FerruFX, CodeMeister, et al to join together and come up with some sort of library to enable my newest EA concept which uses NeuroEvolution with Augmenting Topologies (NEAT). In short, NEAT is a program which writes a program I tell NEAT what the holy grail does/is, and NEAT tells me what the holy grail looks like. I figured I might as well be the change that I want to happen, and that's why I released this older EA.
  • #346
  • Edited 11:49am Jul 18, 2010 11:20am | Edited 11:49am
Quoting Eureka
Hi Getting,

Congratulations on your testing of the Hi Lo Trader. And thanks for posting your settings.

I attempted to set up my charts like yours and I used the 50,000 bars in the Bar Start field and I went to the History Center of MT4 and set the maximum bars on the chart to 250,000. Then I set it in motion at 9:30 yesterday.

It's still running!! How long did yours run before it was finished? Do you have any idea why it would still be running?

On the chart, the number count is over 1 M!!
I have a brand new computer. It's a quad core and 2.66 ghz.

Thanks for your help.

Carol aka Eureka
I think adjusting max bar in history and chart have no effect on the EA calculation, but I will post the result next week, I also testing HiLo-43kb EA on InstraTrader - ECN 4digit broker , and on the bar used row it shows bar used only at 2404(on EU M1) compares to 68,363(alpariUK EU M1)

have no idea why tick count and last cycle rows is still running, but the CPU consumption is very minimal compares to when the market is active.

Carol - from the time when I drag the EA onto a chart and load my saved .set file to the time it finish calculating(when a bunch of rows and paragraph blanket your left side of the screen) is probably 10-15 seconds. Is that what u getting?

I'm little unsure of maxSimultaneousTrades being 10, but thanks for adding that feature RR.
  • #347
  • Jul 18, 2010 2:22pm Jul 18, 2010 2:22pm
Quoting charvo
Ronald, your story makes me a little worried, cos i've often posted some junk code of my own...... dang, U.S. suers are everywhere.......

this HiLo EA looks very fascinating....
This EA is designed to optimize one trading system. There is very little flexibility here.

Neural networks are designed to modify and adjust one trading system. There is more flexibility in the sense that all the inputs are adjusted for the output layer.

NEAT is designed to build, modify, and optimize a trading system. This is basically one step above a standard Neural Network in the sense that I don't have to design the neural network's hidden layer anymore. All I do is give NEAT the input layer (current bar, OHLC, MA values, tick values, etc etc etc) and the output layer (buy, sell, do nothing) and NEAT fills in the hidden layer. With a traditional neural network, I need to put in the hidden layer myself.

Quoting getting
I think adjusting max bar in history and chart...
Are you sure you have all the bars in history? Check your history center (F2), I suspect you don't have 50,000
  • #348
  • Edited 4:52pm Jul 18, 2010 3:03pm | Edited 4:52pm
Thanks Ronald - I overlooked downloading M1 history on EU, GB, UC; Now the CPU going nuts calculating
now database said "database = 250000, 3,700,000 records"
glad I got this part taken care of before the week begins.

Carol- 15sec was wrong, since I didn't download M1 database, sorry.
I hope my VPS resource is up to tasks....hahaha!
http://i31.tinypic.com/jrfii1.png
http://i30.tinypic.com/1627ifc.png
  • #349
  • Jul 18, 2010 3:14pm Jul 18, 2010 3:14pm
Quoting Ronald Raygun
I'm always looking for inputs and new ideas for this.

 

  1. How should the weighting be done? Should the weight of each trade be calculated based on the number of bars between the trade entry and the present? As for using the weighting, how should I use the weights? Trades with a higher weight contribute more to the accuracy, profitability, and risk/reward readings? Another concern with weighting that I personally have is it turns the optimizations from a "lagging system"--a trading system whose parameters lag the market--into a "leading...

...

I am not sure that my suggestion would make the system a "leading system". I think all systems inherently lag as they all use past data to make future projections.

To elaborate a little, it seems that right now a user can either choose to optimize over a long market period, or a short market period. If the user chooses, say 1.5 million bars, then the bot will produce settings that would have worked very well over the last 5 years. However, as we know the market moves in phases, and what has worked in general over the last 5 years, may not have worked very well in the last 3 months. So another user might optimize for the last three months only. It is an either / or choice.

My idea to get the benefits of both long-term optimization and reflecting short-term current market conditions by weight the results from the recent period more heavily than those from longer ago.

You could do this by having 4 inputs: Number of long-term period bars, long-term weighting, number of short-term period bars, short-term weighting.

So, for example, if you say that 1 year is 200 trading days, and three months is 60 trading days, you could set up an optimization as follows to have the bot optimize for the last year, but to value what has worked best in the last 3 months one third more than what had worked in the 9 months prior to that:

Number of long-term period bars = 288,000, long-term period weighting = 1.0, number of short-term period bars = 96,400, short term weighting 1.33. In other words, the success quotient of the first 191,600 bars (the first 9 months) in the 1 year period starting 1 year prior to the current time would be multiplied by the weighting of 1.0. The final, most recent period - the 96,400 bars prior to right now - would have the success quotient multiplied by 1.33. (In other words, the most recent three months of results would be giving a third more "weight" than the previous 9 months of result.)

You could then either simply add the two scores together to give the overall success score, or you could normalize it: ((long period success quotient * long period weighting) + (short period success quotient * short period weighting)) / (long period weighting + short period weighting) Normalizing it would then mean that you could compare it with figures produced without weightings, which might prove useful in some cases.

Math was never my strong point, so I hope I have gotten this idea right!

Anyway, I hope that in some way or other this weighting idea is making sense, and if it is useful, it would be interesting to see how it could help the bot.

 



  1. Quote
  2. As rfking mentioned, this EA already uses a more granular level of sessions, and that is the hourly periods. I could add more granularity to the EA, but then you begin to approach more extreme curve-fitting and exceedingly small sample sizes for each granular period.
You already have a more sophisticated system in place than I was suggesting. Great stuff!

  • #350
  • Jul 18, 2010 4:20pm Jul 18, 2010 4:20pm
Here is another way you could use the results from two different periods of optimization:

A good manual trader can look at a chart and know right away if the pair in question seems to be behaving in a normal or abnormal way. If this trader opens his MT4 in the morning and sees patterns that look strange (unusual volatility / lack of volatility / spikes etc) he will go to check out the current business news, the stock indices, the upcoming announcements etc to see what is the deal, and at that point he may either decide not to trade for the present time, or to use a different system to the one he would usually trade.

It is not reasonable to expect a program to do that. However, your program could do something like this: Take the parameters generated by a long-term optimization and compare them with those generated by a short-term one. Then if the short-term results were within a user-defined distance from the baseline long-term results, the bot would be allowed to trade, if outside then it would not be allowed to trade at that time. If it was allowed to trade, then if would use the parameters generated by the long-term optimization to place the trades.

So a user could have the EA establish the long-term parameters for a particular pair over 1.5 million bars (5+ years), but have it continually compare the parameters generated from an optimization of the last 10 days (2 trading weeks). The user could then, for example, allow the bot to trade if the results from the last 10 trading days were within + /- 10% of the results from the 1.5 million bar long-term baseline. (In other words, the bot would be able to say whether the pair had recently been trading within the "normal" range of parameters for this pair or not. Of course, if would not be able to see the upcoming NFP / tsumani / black swan, but it would at least be able to say, "The XXXYYY pair is currently trading in a typical way, so I should be able to use my long-term optimized parameters safely at the moment.")

Recycle if this is not useful.

Cheers

Invisible
  • #351
  • Jul 18, 2010 5:27pm Jul 18, 2010 5:27pm
After trading commenced the EA waited x amount of bars before setting HiLos. However it did not take into account market movement during those bars to calculate HiLos. On pairs showing a downward trend, the hi is not as hi as it should be. This may be due to the way StrToTime is calculated.
  • #352
  • Jul 18, 2010 5:45pm Jul 18, 2010 5:45pm
What is the general practice for end of week? Let the EA run through the close with positions open or shut it down and close out everything beforehand?

Im having a blast messing around with this
  • #353
  • Jul 18, 2010 6:30pm Jul 18, 2010 6:30pm
Quoting rfking2
After trading commenced the EA waited x amount of bars before setting HiLos. However it did not take into account market movement during those bars to calculate HiLos. On pairs showing a downward trend, the hi is not as hi as it should be. This may be due to the way StrToTime is calculated.
Could you elaborate?

Quoting faze
What is the general practice for end of week? Let the EA run through the close with positions open or shut it down and close out everything beforehand?

Im having a blast messing around with this
The EA optimizer includes weekend gaps as well, I quite literally set it and mostly forget it.
  • #354
  • Jul 18, 2010 7:23pm Jul 18, 2010 7:23pm
Quoting Ronald Raygun
Could you elaborate?
I didn't have this problem with the old version. So looking at the HiLo calculation code I see they they have roughly the same logic written differently. If RangeStart = DayStart - InitialRange and HiLos are calculated from RangeStart then it seems to me that if InitialRange = x the EA should be looking x bars beyond DayStart and then also waiting x bars after to place a trade. Giving you 2x bars to calculate HiLo. However this pic shows the EA waited x bars after the market opened to begin calculating HiLos and began trading at the same time. That negates the point of using InitialRange, and runs the risk of a breakout trade being placed when it should not be.

Since the formula looks correct to me, I looked at the differences between the two EAs. Primarily the DayStart calculation. In the old version you used Period_D1 to calculate both DayStart and HiLo. So even on a Sunday, no matter what time zone you are in, when the market opens it is a new day. The new version uses StrToTime("00:00") to calculate DayStart. StrToTime Calculates local pc time rather then server or GMT time. Since it is not midnight here, that line of code may be causing the bug. Just a theory, because I did not have this issue last week at midnight. It worked as it should.
Attached Image (click to enlarge)
Click to Enlarge Name: Purple Monkey Vomit.jpg Size: 71 KB
  • #355
  • Jul 18, 2010 7:26pm Jul 18, 2010 7:26pm
Well, after trying and trying, it seems your limited to < 10,000 records for M1 and @ 2077 for M5 on any pair.

I tried downloading through a 5 decimal IBFX demo and migrating the data....NO

I tried downloading the data via demo and then logging in to my live account....NO; it just reverts back.

If anyone else is using IBFX and knows how to increase the records in the History Center, let me know.

I already have the Bars on chart and in history and all that set to the maximum.
Capital Preservation is key to long term wealth accumulation
  • #356
  • Edited 9:08pm Jul 18, 2010 8:22pm | Edited 9:08pm
EDIT2: Or.. I could just be a retard who forgot to modify the code to change reject to accept.... lol
  • #357
  • Jul 18, 2010 9:19pm Jul 18, 2010 9:19pm
Quoting MuddBuddha
Well, after trying and trying, it seems your limited to < 10,000 records for M1 and @ 2077 for M5 on any pair.

I tried downloading through a 5 decimal IBFX demo and migrating the data....NO

I tried downloading the data via demo and then logging in to my live account....NO; it just reverts back.

If anyone else is using IBFX and knows how to increase the records in the History Center, let me know.

I already have the Bars on chart and in history and all that set to the maximum.
OK, being that I'm using IBFX as well here are some "problems" I've run into.

First, being that I am using Windows 7, windows likes to "lock" programs out from editing files in their default installation directory of "Crogram FilesInterbank FX". This "Could" be preventing MT4 from being able to save the downloaded data. The solution I have found is simply to uninstall and then reinstall into a different "unprotected" directory, IE: C:MT4 Trader. Why windows treats programs installed in a "Program Files" subfolder differently then if you installed them somewhere else I'll never know, but it does and has solved a LOT of my issues with applications and games not saving their data properly.

Second, IBFX doesn't provide historical data, if you pay close attention to the message that pops up when you click download in the history center, its telling you it will download the data from "Metaquotes Software Corp" and not IBFX. This could be why your having issues importing/exporting the data. (also MT4 seems to store different history data with different profiles, so when you log out of the "demo" profile its data is no available to the "live" profile. You will need to log on demo, export, log on live, import.. though you mention you already tried that)

Third, I seem to be limited to 65655 records on my IBFX account even though I set the max bars to 250k and the history to 512k. No idea how to get around this as I have tried manually typing in numbers and saving the setting, but no matter what I put in there it "caps" out at 65655. Limitation of IBFX servers possibly?
  • #358
  • Jul 18, 2010 9:53pm Jul 18, 2010 9:53pm
Quoting MuddBuddha
Well, after trying and trying, it seems your limited to < 10,000 records for M1 and @ 2077 for M5 on any pair.

I tried downloading through a 5 decimal IBFX demo and migrating the data....NO

I tried downloading the data via demo and then logging in to my live account....NO; it just reverts back.

If anyone else is using IBFX and knows how to increase the records in the History Center, let me know.

I already have the Bars on chart and in history and all that set to the maximum.
To download more bars: http://www.easyexpertforex.com/backt...imization.html

Use this source: http://www.forextester.com/data/datasources.html

Quoting rfking2
I didn't have this problem with the old version....
I think you're right. It's mentioned here: (http://forum.mql4.com/33023). I wonder how I didn't have any trouble with this... If anyone is good with timezone math, I'm in the New York timezone (Eastern).
  • #359
  • Jul 18, 2010 10:38pm Jul 18, 2010 10:38pm
I'm in Seattle so you are a few hours ahead of me. I think this bug only happens on Sunday open. So it could be easy to overlook if your not paying attention. But that also means that everyone using this EA has different HiLos depending on their timezone.
  • #360
  • Jul 19, 2010 12:10am Jul 19, 2010 12:10am
OK, so after reading and re-reading how to get this EA working, I "believe" I have everything set up correctly. So after dropping it on a chart I've been waiting now for several hours for it to finish optimizing, but nothing has changed since I started it except the cycle count. This is what the EA is showing me.

Attached Image


All settings are default except for lots, I changed it from 0.1 to 0.01.

So should I keep waiting? Reset and try again?
6 traders viewing now