codification of the market / Genetic Algorithms | Page 2 | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Statistically testing every indicator with my Genetic Algoritm 132 replies

Trendline EA which rates trendlines by formulas/algorithms 2 replies

Has anyone had luck with high frequency using either neural nets or genetic Pr? 3 replies

Genetic Algorithm for Metatrader 1 reply

Designing of algorithms and creation of complex indicators and advisers for MT4 0 replies

codification of the market / Genetic Algorithms

  • #21
  • Aug 14, 2005 11:00am Aug 14, 2005 11:00am
Quoting merlin
orii, what "math wiz gamblers" are you referring to? any person good enough at math to be considered a wiz will know that you can not make money with a negative expectancy game.
I use that term in a sarcastic way. I've read plenty of gambling websites wherein some self proclaimed "math wiz" (some of whom do seem to have substantial educational background) details a money management system (typically dealing with streaks and trends in what I consider to be a random string of events) that he/she believes can return profits from a random system. I add always include the term "gamblers" because, IMO, a true math wiz probably wouldn't fall for such irrational beliefs, while a math wiz "gambler" falls for the same misconceptions as all other gamblers.

I do agree with you though... in a game that is both negative or zero expectancy and completely random, no "money management" can extract long term profits.
  • #22
  • Aug 14, 2005 11:15am Aug 14, 2005 11:15am
Quoting fijitrader
The question posed was whether a negative sum game with a 50/50 expectancy could succeed by applying good money management. Merlin said that he didn't think so and cited why. I cited something that may contradict that.
Two thoughts and a question.

Did Van Tharpe's returns remove the yearly trend of the market(s) he applied his system to (i.e., did his system do better than the market - commissions)?

I see a big difference between random systems like a series of coin tosses and something like the market. While both might be zero expectancy (actually, the market is positive historically, but after commissions let's call it zero expectancy), the market may not be completely random. In a non-random system, you could still extract profits by carefully choosing your exit point even when playing on random entry points. This isn't "money management" of the same form as gamblers use, because you are watching for real trends in the system and basing your exit at least partially on that, whereas with gambling you make exit decisions solely based on your own performance in a random game. For example, say in a "market equivalent" game of coin tossing (that isn't completely random) you notice that anytime a string of at least five heads or tails is reached, the string always continues to at least ten (which obviously isn't the case in real coin tossing). Your entry can be random, but if you watch for these strings of five and take advantage of them when you see them, you can still make a net profit in the long run with your non-ramdom exits (note that the number of heads and tails over long time spans will still be even).

Finally, I assume that any mechanical system (GA or otherwise) will have to generate exits. As for the GA framework I described, each organism should generate an entry, and exit, and a position size for each, and the fitness function simply evaluates each on some combination of time averaged profits and drawdown.
  • #23
  • Aug 15, 2005 3:29pm Aug 15, 2005 3:29pm
Quoting fijitrader
The question posed was whether a negative sum game with a 50/50 expectancy could succeed by applying good money management.
i dont see why this question is important because there is no place where you can play a negative sum game with exactly 50/50 expectancy. placing random entry and exits will NOT give you a 50/50 expectancy. it may give you a good expectancy, maybe bad. in the end its random so you cant even measure the expectancy.

forex is a negative sum game (ie when you win, someone else loses, and the broker always wins), but it doesnt have a 50/50 expectancy. the expectancy depends on what system you are using. if your system has a negative expectancy, there is NO WAY you can win, no matter what MM system you are using. i dont think there is anyone disagreeing with this statement here, but if you do, you have to look no further than Ed Thorp's "Elementary Probability" to find the mathematical proof of this.
Relax and be happy.
  • #24
  • Edited 7:40am Aug 16, 2005 7:27am | Edited 7:40am
Quoting merlin
i dont see why this question is important because there is no place where you can play a negative sum game with exactly 50/50 expectancy. placing random entry and exits will NOT give you a 50/50 expectancy. it may give you a good expectancy, maybe bad. in the end its random so you cant even measure the expectancy.

forex is a negative sum game (ie when you win, someone else loses, and the broker always wins), but it doesnt have a 50/50 expectancy. the expectancy depends on what system you are using. if your system has a negative expectancy, there is NO WAY you can win, no matter what MM system you are using. i dont think there is anyone disagreeing with this statement here, but if you do, you have to look no further than Ed Thorp's "Elementary Probability" to find the mathematical proof of this.
Because I make too long of posts anyways I was hoping to avoid the obvious explanation. Anyway randomness, if it exists, implies an equal distribution of results. I was simply using the idea of 50/50 expectancy to illustrate that. Once you add spread, interest, and commissions, to the game it becomes a negative sum game at that time. Until expenses are applied it is theoretically a break even game -- if it is ever, in fact random. So my point was not that it is not a negative sum game. In fact I was actually saying that it is because I mentioned both slippage and commissions. Hense the return of the detailed explanations... yawn...

Regardless I understand the math of a negative sum game and have no argument against the theoretical math models that supposedly prove it. I don't know the details of Van Tharpes system that makes money (using random entries) after slippage and commissions are deducted. He claims to have developed a system that with his application of clever money management has apparantly been beating the market for some time.

Now I agree that it is also hard to argue that random entries in a negative sum expectancy -- with managed exits can possibly win. My point in posting this was to see if you had an explanation. I already know that you don't believe it is possible because most people don't. I can argue the negative sum math argument a lot more convincingly than I can argue Van Tharpes. But the guy seems to be trying to make the opposite point of what the math models supposedly prove.

The problem is the statistical arguments are made by better statisticians than me so if there are flaws they are not so easy to find. I have taken enough statistics to know that not all statisticians agree what the probabilities are with various scenarios but have noticed that all have convincing arguments. Anyway I'm thinking "is it possible that his (VT's) approach to money management applies exits in such a way that the expectancy of the system is altered sufficiently that the random exits are compensated for? If not then we could also conclude that perhaps exits are not more important in trading than entries. Clearly system expectancy is altered by altering exits. The question is whether expectancy can be altered sufficiently by exits to overcome the negative expectancy found in a situation. I know the statistical argument against it is very solid but does that make it foolproof? Van Tharpe would not be the first to intelligently argue that money management can alter expectancy.

Personally I've never bothered to investigate the arguments against Ed Thorpes probability theory because it has always seemed so clearly to be correct. However I also try to keep in mind that his application of probability theory, while seeming correct, does not actually prove his thesis absolutely -- just convincingly. It is not necessarily irrefutable. (I should mention think it is undoubtably proof enough that it is a waste time trying to make money with anything that falls within the definition of negative expectancy. However I don't tend to hold too tightly to anything that I cannot establish to be an absolute. Once someone like Van Tharpe comes along and provides good evidence to the contrary I cannot, withouth being dogmatic, hold fast to the former conclusion unless I can find an adequate explanation for the contradictory evidence. The existance of a convincing, though unexplained, contradiction may hold information that is important but not contained within the former (apparantly air tight) argument.

Anyway the intention of my post was to see if you had an explanation. So far the only thing I'm hearing is that Ed Thorp is right. While a noble gesture indeed it doesn't do much for explaining the seeming contradiction emanating from Van Tharpes system.

FT
  • #25
  • Edited Aug 17, 2005 2:52am Aug 16, 2005 8:09am | Edited Aug 17, 2005 2:52am
[quote=Orii]
Quote
Two thoughts and a question.

Did Van Tharpe's returns remove the yearly trend of the market(s) he applied his system to (i.e., did his system do better than the market - commissions)?
Yes after all costs.

Quote
I see a big difference between random systems like a series of coin tosses and something like the market. While both might be zero expectancy (actually, the market is positive historically,

What market is positive historically? Equities? If so that is only true if you are net long over the entire life of the market. Othewise how can anyone say that the market has a positive expectancy? You can't make a generalization like that which has any meaning.

Quote
but after commissions let's call it zero expectancy), the market may not be completely random.
What are you talking about?

Quote
In a non-random system, you could still extract profits by carefully choosing your exit point even when playing on random entry points. This isn't "money management" of the same form as gamblers use, because you are watching for real trends in the system and basing your exit at least partially on that, whereas with gambling you make exit decisions solely based on your own performance in a random game.
The system is mechanical so none of this applies.

Quote
Finally, I assume that any mechanical system (GA or otherwise) will have to generate exits. As for the GA framework I described, each organism should generate an entry, and exit, and a position size for each, and the fitness function simply evaluates each on some combination of time averaged profits and drawdown.
And your point is?

FT
  • #26
  • Aug 23, 2005 9:17am Aug 23, 2005 9:17am
Quoting fijitrader
What market is positive historically? Equities? If so that is only true if you are net long over the entire life of the market. Othewise how can anyone say that the market has a positive expectancy? You can't make a generalization like that which has any meaning.
Yes, I was referring to a net long position, and more specifically to stocks. There is more than one way to invest in the market, you know.

Quote
What are you talking about?
About the possibility of non-randomness in the market? An assumption practically everyone here stakes their money on? And more to the point how non-randomness can allow profits even in zero sum games.

Quote
The system is mechanical so none of this applies.
What? The example I provided is perhaps the simplest sort of mechanical rule one could use. No "instinct" required. I'm really not sure what you mean by this.

Quote
And your point is?
The same point(s) I made in the first post.
  • #27
  • Sep 2, 2005 1:16pm Sep 2, 2005 1:16pm
Very interesting post!!!!...

I like to keep trading simple: You win some, you lose less - If you get that the other way around, you're screwed.

Trading is like the insurance business - your premiums are based on risk & probabilities.

The human emotions of fear & greed are pretty predictable in the midst of Market chaos.

Pipbull
  • #28
  • Edited Oct 14, 2007 6:56pm Oct 13, 2007 11:26pm | Edited Oct 14, 2007 6:56pm
OK guys, here I am two years later, and I need your help. You can see by the number of posts under my name that I've been doing things other than conversing on forex forums.

One of those things has been programming. I have created a fairly sophisticated program that implements all of the ideas in my original post, except for accepting input of "fundamentals" in the market (things like earnings forecasts, interest rates, etc.). I'll work on that later.

The program has the ability to read in a historical data file, virtually evaluate existing organisms (trading rule sets) over a trial of a couple of virtual weeks, rate them for fitness, breed and mutate them in highly configurable ways, and step forward through the data one generation at a time. A new generation is created at each new data entry in the historical file, and the reigning champion from the previous generation is allowed to dictate trading decisions on the "real" account for that generation. For now, I'm working with twenty years of open/close data for a couple of pairs. The concept is the same if applied to shorter time frames for scalping, but this data was easier to acquire in bulk.

The program seems to work in the technical sense. Organisms work on a specified chunk of the data for any one time increment looking for patterns. The director program allows them to try out their algorithms virtually before ranking them. Mutations types mimic our DNA with point mutations, frame shifting, and both asexual and sexual transfer of larger chunks of the code.

I must point out that this program is not using genetic algorithm concepts to simply backfit or curvefit data... i.e. it does not use canned rules and simply converge on the best coefficients. Nor does it run through exactly the same data over and over trying to evolve better algorithms to fit it. Each generation sees a different set of data. Actually, for any one time index, an organism looks at a large chunk of data to search for patterns; all organism within a generation virtually step through many days (from the past to the current date) for evaluation; and the "current date" advances with each generation - thus there is obviously overlap as each generation is looking at the same data plus an additional point and minus the oldest, with the important difference being the current date has shifted and the algorithm must be robust enough to work with that.

The idea is to never run algorithms through the same data twice. Since the spans evaluated are largely overlapping, the program naturally does its backfitting, and that is the point. The most current data is always being added to the range with the oldest becoming obsolete. If this was working in real time, it would be forced to stay current (or fail).

So where do I need your help? Well, the organisms evolve, but they are starting from plain junk. I've yet to attempt to put in decent starting trading rules. Everything so far has been complex garbage to test the robustness of the code for the ugliest ruleset I could create. Needless to say, evolving hasn't helped the programs find patterns.

I'd like suggestions on some pretty basic, sound trading rules to code in as the starting gene pool. A variety of moving average schemes come to mind, but I'd like more diversity. I've been reading plenty on forex basics and trading strategies, but there almost seems too many esoteric ideas to get a grasp on what is really important. Think in terms of larges species groups, not individuals. I want an organism to represent a large subset of trading styles.

What are some of the simpler candlestick schemes that you think would be fairly to code by hand? Resistance lines? Im all ears...
  • #29
  • Oct 15, 2007 4:45pm Oct 15, 2007 4:45pm
Well, I'll continue on with my own reading and research. I was hoping to at least get some suggestions that would highlight the major categories of trading strategies; as any one book or website or thread only deals with a couple at most, it seems easy to overlook a completely different take on strategy and I would prefer for the starting population for any serious research be as diverse as possible.

This will be my last update. For those that have the programming fortitude and are interested in the ideas discussed here I'll offer a few whistle wetters...

- the program currently runs slow, taking about an hour to evaluate each data point. The code is messy - all procedural, but easy to modify and quickly interpret. Significant gains can be realized in optimizing mutation types and rates, code optimization, and especially in increased computing power (the platform currently running the program is about equivalent to a cheap laptop of today).

- so far 17 have been evaluated, with over 4000 offspring organisms evaluated. Even with crappy starting rules (not even rules, just junk), small population sizes, and just 17 generations, all surviving organisms show consistent net gain over a span of 50 data points. This could be due to efficient curvefitting due to the significant overlap in data evaluated from one generation to the next. I'm implementing a master virtual account now to track real performance of the compilation of winners...
  • #30
  • Nov 13, 2007 9:00pm Nov 13, 2007 9:00pm
Hello Orii,

Are you still working with this? It actually sounds like the branch of GA known as Genetic Programming, created by a guy named Koza I think. It's an interest of mine, but I have never been able to devote any serious time to it. My approach would have been to use a recursive language like LisP for the evolutionary process, but I would have to go back and relearn it.

On the specific question you asked, one main division of trading strategy is trend following (and its subset of breakout trading) vs. contrarian strategy. I'm not sure if that's the kind of thing you were looking for.

That being said however, I think you may be better off starting with the "junk" rules and letting them recombine as they will over the generations. This way, your genetic soup doesn't start off with a bunch of preconceptions which may be wrong. Seed the initial population with only the most basic of concepts (higher, lower, rising falling, larger, smaller, crossing, touching, between, outside, etc.) and let the process take its course. That's what GP is all about.

Am I to understand that it takes around 50 hours of computational time per generation? That's disheartening; I had envisioned dozens of generations going by in an hour.
  • #31
  • Nov 13, 2007 9:37pm Nov 13, 2007 9:37pm
Organisms on earth evolves due to their surrounding , the organism identify its own weakness and mutate itself to fits into the nature
Past has shown when a species takes excessively from nature , nature destroys such species and creates new one , the process continues till the species and nature are in equilibrium ...
how does nature destroy its own species ? by evolving ... the important part is this , nature evolves too ...
same way i ve the feeling , an evolving system would get destroyed by the evolving market over time ...
unless the system knows how the market would evolve and evolved before the market itself evolves to destroy the system ...
if thats achieved ... the system and market would exchange its places , the system becomes the market ...
To Live is the rarest thing in the world , Most people exist , that is all
  • #32
  • Nov 13, 2007 10:04pm Nov 13, 2007 10:04pm
Quoting maheswara
Organisms on earth evolves due to their surrounding , the organism identify its own weakness and mutate itself to fits into the nature
Past has shown when a species takes excessively from nature , nature destroys such species and creates new one , the process continues till the species and nature are in equilibrium ...
how does nature destroy its own species ? by evolving ... the important part is this , nature evolves too ...
same way i ve the feeling , an evolving system would get destroyed by the evolving market over time ...
unless the system knows how the market would evolve and evolved before the market itself evolves to destroy the system ...
if thats achieved ... the system and market would exchange its places , the system becomes the market ...

OK, I'll buy your view that the market is evolving over time. I firmly believe that there are some aspects of market behavior that change in an evolutionary vs. a revolutionary fashion. One example I've heard about is the increase in false breakouts in response to successful range expansion systems. I've even written some articles in support of this market evolution theory (which I won't link here because I don't have enough posts). So I agree with you to that point.

However, there is a huge difference of time scale here which gives the genetic programmer a vast advantage over the market when it comes to evolution. The market evolves in real time. It may take weeks, months or years for a certain type of behavior to change noticeably. However, the genetic programs evolve over a series of computer simulated generations, where each generation of trading ruleset "organisms" is born, tested, mated, and replaced by the next generation in just a few minutes.

So the evolution in the real market is taking place at much too slow a speed to be any kind of a problem for the idea of genetic programming.
  • #33
  • Nov 14, 2007 6:24am Nov 14, 2007 6:24am
Quoting capitalist88
OK, I'll buy your view that the market is evolving over time.
got the point ... but if the system was simulated within the past behavior of the market , wouldnt it be susceptible to the next evolution of the market ... and there is a chance that evolution of the market would accelerate when such system is introduced into it ...
i see such system as an analogy to virus in our system ...
To Live is the rarest thing in the world , Most people exist , that is all
  • #34
  • Nov 14, 2007 2:56pm Nov 14, 2007 2:56pm
Quoting maheswara
got the point ... but if the system was simulated within the past behavior of the market , wouldnt it be susceptible to the next evolution of the market
That's an excellent point and it's exactly why people came up with the idea of Genetic Programming. Since the systems evolved through GP have survived through many of those past evolutions, the surviving rulesets should be robust enough to survive future ones too.

One of the goals of GP though is to evolve robustness within a trading ruleset, which means that the ruleset is resistant or adaptable to changes in its environment. Evolution does the same with real organisms; the most successful can survive under many different environmental conditions (as long as those conditions don't change too quickly or radically).


Quote
... and there is a chance that evolution of the market would accelerate when such system is introduced into it ...
i see such system as an analogy to virus in our system ...

I don't know about this. From the "market's viewpoint" a ruleset evolved through GP is just another system introduced by just another random trader. This happens every day. The market doesn't care what process we use to devise our various trading rules. It evolves at the same pace that the behavior of the participants does (because our behavior is the driver of the market's evolution).

Although you could be right. If a GP user came up with some new and improved ruleset every few weeks or so, and more traders started to use GP, then you'd have kind of an evolutionary "arms race" between the market and the trading population.

This is all pretty theoretical of course, but fun to think about!
  • #35
  • Nov 14, 2007 5:44pm Nov 14, 2007 5:44pm
Quoting capitalist88

This is all pretty theoretical of course, but fun to think about!
Yeah i know !
To Live is the rarest thing in the world , Most people exist , that is all
  • #36
  • Oct 19, 2016 5:05am Oct 19, 2016 5:05am
For first post

What mean genetic algorithms?

How many of experience do you have?

Did you fund the best (in general terms about logs, exponents, derivatives, integrals, etc.?

What means GA?
  • #37
  • Last Post: Oct 19, 2016 5:11am Oct 19, 2016 5:11am
For second post

Probably is smart

But is necessary to be hard not easily to invest, is relative.

I think that someone could predict the market behavior, i In theory, i do it

And is necessary to remove the human emotions, like me
0 traders viewing now