Neural Networks: Myths And Reality

By Dima Vonko

In this age of previously unheard of technological progress many technology-related things either come unnoticed as they appear or, vice versa, are vastly extolled and turned into totems that inevitably attract a following of ardent worshippers. If such a popular technology-related phenomenon can make a difference to your business, it is, sometimes, vital to learn about this phenomenon as much as possible before you start with it so that you know what to expect from the selected technology and what to beware of while using it. For a modern trader, one of such potentially important phenomena is neural nets.

So what is the neural network technology, what should and what shouldn’t a trader expect from it if he selects to use it to achieve his trading goals?


Dispelling the Myths

Myth 1: Supernatural Intelligence

One of the commonly held misconceptions about neural networks is that they represent the kind of Artificial Intelligence which is it not only capable of fully replacing the human brain, but which also possesses some nearly supernatural power, thus enhancing the capacity and functions of this brain to the point when any kind of task can be solved almost miraculously without any effort on the part of the proud owner of this Holy Grail. This vulgar and frequently occurring notion of an undoubtedly valuable trading tool is dangerous in several ways. Let us see why.

First of all, neural networks are not all things to all traders. To understand what neural networks can and cannot do one should look into what they are. Neural networks are algorithms, loosely based on the nervous systems of humans and animals. Neural networks can detect and use to advantage the numerous interdependencies in data that are hidden from the human eye due to the data’s complexity and non-linearity. This has been proven by the broad experience of neural networks’ application in a wide array of industries, and trading is no exception. However, on no account should you consider neural networks to be something that will think or decide for you.


Myth 2: Magic Software

The second as, if not more, dangerous, misconception about neural networks emanates directly from the first one: somewhere out there, there is a heaven-sent trading software that basically works as a minting machine and all you need to do is find it. This misconception is not dangerous only because you will lose time and money while looking for what doesn’t simply exist, but also because your delusions are well-known to those who crank out one-magic-button, slipshod software programs and fob them off on the seekers of the neural Holy Grail. Normally, those who try to exploit others’ delusions make poor professionals and, thus, poor software too. Remember – neural network software can only do what neural networks themselves can do, and they can do a lot if you know how to apply them and what software to purchase. However, no neural network software can tell you the exact time and the type of action you should take at this particular time to profit.


Myth 3: Neural Networks Can Predict Precise Figures

The third frequent misconception is that by using a neural network you will be able to predict the future prices. Many traders believe that their networks are capable of telling them when to buy and when to sell. If you understand that those people are wasting their time and money you will probably be a success with neural networks. No neural network, no matter how sophisticated or well-built, will be able to precisely inform you about the future price or, at the push of a button, tell you, and you alone, when it’s best to buy or sell (for, otherwise, there would no longer be a market). However, you can, undoubtedly, predict the likelihood of other important things happening, which will help you make better trading decisions. Therefore, even with what neural networks really can do, they remain the most powerful market analysis tool ever in situations, involving noisy data or non-linear dependencies. In other situations, using neural networks may be inexpedient. We will dwell on the predicting ability of neural networks and on what and how they can actually forecast later in this article.


Myth 4: Some Nets Are Significantly Better than Others

Many traders who want to employ AI for making their trading solutions mistakenly believe that the quality of the neural network capabilities of the different trading applications on the market varies significantly, and there is some special neural network somewhere that will eclipse all the rest in terms of the quality of the forecasting results. However, practice and experience show that the quality of different neural networks, no matter how much touted for, differs within the range of 10%, and even so it varies for different tasks and data sets. Of course, while selecting a trading software program one should look at the AI background of its developers (building a good neural network takes a great deal of skill and experience), but, at the same time, the application must provide the rest of the required functionality (such as, for example, the charting functionality) with excellent quality. In other words, one should look for a successful combination of neural network functionality and other vital functionality.

Looking for the only magic net is much like looking for one magic technical indicator. Aside from that, this quest often feeds those who are after a quick buck.


Myth 5: The Quality of the Forecasting Result Depends Solely on the Quality of the Network Used

The quality of the forecasting results does depend on the quality of the network you apply, but for not more than 10-15 %. The rest depends on how well the trader has prepared the data sets the network works with. The data sets must be sufficiently representative. They must include all the important influencing factors. Besides, the application of a neural network must be combined with Money Management and classical filters.


What Neural Networks Can Do for You and What You Need to Know to Make Them Work

Neural networks are definitely not a solution to all problems and they shouldn’t be regarded so. What they are is a most powerful, technology-based method of technical analysis that can become an inestimable addition to your trading arsenal. Just like any other method, neural networks have their advantages and limitations, but their unique ability to track even the most subtle interdependencies in the available data no other method can establish, as well as build patterns based on this analysis, definitely make neural nets stand out from the rest of the existing methods and tools.

You can effectively use neural nets to:

* estimate the likelihood of a trend continuing;
classify market phases;

* produce time estimates of highs and lows for various timeframes and combine results into a committee;

* predict the probability of a new, strong upswing after an uptrend, followed by a classic correction;

* track inter-market dependencies.


In other words, you receive a TA tool which will be a lot more efficient than classic TA methods anywhere where there is too much noise or where the interdependencies in the data are floating and significantly non-linear. For example, if after analyzing a number of charts you have discovered that the closer an uptrend is to a pivot point, the closer the bar’s Close is to the bar’s High, and you are planning to create an oscillator to anticipate reversal, you should use classic math as was done by the inventor of the Stochastic oscillator George Lane. But if you are trying to find a formula for the inter-relationship between S&P, InterestRates, $/Euro, Oil prices, and so on, you will make sure that the classical correlation or ratios won’t be any use, since although interdependencies do exist, they are not stationary or linear. These interdependencies oscillate, ”float” through time and are influenced by noise. In this case, neural networks can solve the task better than the classical statistics.

When used in a combination with other technical analysis methods, and when sufficient attention is paid to the preparation of data sets (this procedure is, actually, central to success with neural networks), neural networks will undoubtedly provide the punch you need to success on the market. After all, this has been proven by both time and experience.


Dima Vonko is the creator of Tradecision trading software (www.tradecision.com) for advanced technical analysis, which boasts a unique combination of a powerful neural network capability and full-fledged, feature-rich charting functionality.

He has contributed to several well-known trading publications, such as Traders magazine, Investopedia, Trade2Win, and others. This article is based on Mr.Vonko’s more than five years’ experience in providing consulting services related to technical analysis and financial modeling.

He can be reached at [email protected].

This thread was started in Nov 3, 2006. There are very few other threads on the same topic. This means that despite the overwhelming ignorance among the traders population some people were ahead of the curve even 14 years ago.
Unfortunately there was no further development on any of the threads.
I assume they gave up quickly once the complexity of the task exceeded their own abilities to deal with the problematic nature of general AI.

So this is the reality: The neural network is not the final solution to our problems. The neural network is only one part of the AI machine. Just like your brain or your PC contains many different parts. You need all parts to work together. You cant make a working computer if you only have the CPU and a bunch of cables. There are many other components. And you have to assemble them correctly. But first you have to make sure that all of the components are compatible with each other.
Then you need power to start the machine. And then you need operating system. And finally you need applications on top of the operating system. Each application is made to solve specific tasks.

Now in 2020, most of the problems from 14 years ago are solved for us. So we don't have to reinvent the wheel. We can work smarter by adapting available solutions from other industries. And this is exactly what I'm going to do. If there are some non random repeating patterns in any of these FX charts am going to find them. I will find them all!

Yes my dear friends! Today is Nov 2, 2020. And tomorrow is Nov 3, 2020. Can you even believe it? Is this just another coincidence? Is this even real? Maybe.
But what if there are no coincidences? What if everything runs according to some script. Like a movie. What if there is some code? A pattern. Hidden inside the data. How can you find this pattern? What if there are many patterns? How to find them? That's a lot of questions!

Let's think about it!

If you want to find something first your have to search in the right place at the right time!
You see, that something has location in the spacetime continuum!
You also have location in the spacetime continuum.
The next trade has location in the spacetime continuum.
The next price has location in the spacetime continuum.
Your profit and your loss both have locations in the space-time continuum.

You start to understand now?

These locations are not static. They are in constant motion. They are possibilities. They are probabilities inside the bell curve. They are billions! They are trillions of possible combinations.
At each point on the spacetime continuum 1(THE ONLY ONE) of all these trillions of combinations will become a reality. And then the state of that code will change to static! This code is fixed on the timeline permanently forever. We call this history.

We assume that to know the future you have to study the history.
But what if there is another way!?
What if everything you know is wrong?
What if you can study the future first?
What if you can travel in time?
What if you can visit all possible locations in the spacetime continuum?
But HOW?
We build the future now!
We don't have to travel in time.
Time can travel around us.
We can bend the spacetime continuum like a piece of paper in order to read the information on the other side.

Is this possible?
Is this science fiction or science fact?

Yes my dear friends! This is what we're up against.
And yes - it is possible.

Before I go any further, let me tell you something.

For this type of simulations we will need a lot of processing power.
In this case power=electricity!
And electricity=heat!
And heat can generate new electricity.
But I digress here.....Anyway.
Keep in mind that when I say a lot of power I mean A LOT OF POWER!
To be more precise 600 GWh per year! Or 1.64 GWh every 24 hours. Or 1 640 000 KWh PER DAY.
То get some sense of scale, a typical household consumes around 7 400 KWh PER YEAR.

BTW Do you want to see my home now?



Yes my dear friends! YES! The picture is real! This is where we are now in the spacetime continuum! 2020! Perfect vision! Do you have it? Or you are still asleep? Don't worry! You will wake up soon!
Stay tuned!

Reality Now!

Reality Now!

Think of universe as Cinema where is going trillions movies and their trillions variations at the same time. All is predefined by scenarious. And if you believe what you see then you are catched by particular scenario and you cant get out of it. You can switch movies by doing something different. Choose different route to works/home. Talk to stranger etc... Then will receive new ideas to your AI maybe..

The one challenge to using AI in the FX market is the lack of high-quality data. While there is a large volume of price data, there is an extremely limited semantic network. For example, the order book and volume are missing/misleading concepts, so the relationships among orders, price, and volume cannot be fulfilled with the available data. So it doesn't matter how good the algo is - garbage in results in garbage out.

I'm not at this stage yet. I want to start at lower level and to build it up gradually. I want to keep it "simple" at this level. Just to lay good foundation.
For now I want to know 1 thing - is it possible to beat the market using only the information available to the retail trader?
Is it possible to beat the market by using only technical patterns?
For this we need simple AI. Or as I call it Level 0

Here is one example of a simple AI.

And this is the source code: https://github.com/Code-Bullet/SnakeFusion

Are you aware of this?
https://www.forexfactory.com/thread/...ex-system-mlfx

Interesting project, but this is not exactly what I'm Looking for.
These guys are stuck with the old way of thinking that was invented in the 1970's. Their logical proceeding is using the standard direction where first you study the past and then you make actions in the present. The snake learning algo in my previous post is using the same type of logic. Each new generation learns from the mistakes of the previous generations.

What I'm trying to do is different. I want to study the future first!
There are 2 ways to study the future.

1. We can travel in time to a different timespace location. (We can't do this right now, or at least I don't have access to such technology)
2. We can simulate the future now by using building blocks characters from the past to recreate all possible timespace locations.

In our case simulating the future is not too hard because we have to build only 2D XY spacetime. After all, at this level 0 we are interested only in 2D technical patterns and not in the whole reality of existence.

If we study only the past we have limited access to information because the past is only 20 years. I mean the EURO was created in 1999. There is no tick data before that.

But to study the future is another story! We can build millions of price curves with perfect quality tick by tick data 1000 years into the future.
Then we can train our AI with this future data. Just think about this!

You see, the past = limited! The future = all possibilities!
Do you want to be limited in your understanding or you want to know all possibilities and probabilities?

This snake is much better!
https://github.com/Chrispresso/SnakeAI
I'm going to use some parts of this NN code in my experiments.
Wow. Look how fast it learns the optimal strategy! And people still think they can beat AI? Sorry guys, you have no chance!

What I'm trying to do is different. I want to study the future first!
There are 2 ways to study the future.
For your future data for an AI, you need to consider the AI's perspective of "future", not your perspective. You can collect a large data set of price. The AI does not have any of this data set: The data will be made available to the AI at some point in its future, at which moment it becomes present data and immediately past data for the AI. But it is ALL past data from your perspective.

So, you now have the ability for the AI to peek into the future and see the data that WILL BECOME its present then past data. So it is merely up to you as to how you will have the AI process this data from the peek into the future. Perhaps you will have it peek X number of time units into the future and simply store this value in order to "predict" the value when the X number of time units have passed by simply recalling the stored value.

You could do essentially the same training of the AI as when past data are used as training input but instead, the future data from the peeking are used as the training input.

But as I see it, your AI will always have some kind of training bias: The bias introduced by the set of training data. So, I fail to see that there would be a worthwhile advantage to train the AI with future data versus past data.

DrDave No bickering Please

While trading-related debate is encouraged, members should not engage in mindless arguing, battling, or trolling. Not only does bickering distract others from trading, but it wastes energy for the bickering participants. Unresolvable personality conflicts should be managed using the 'Ignore' () feature.

DrDave No bickering Please
LOL

Hasib, you need to learn the difference between bickering and having a discussion to clarify a topic and/or to offer some alternatives to consider.

The puzzle was solved by seeking rules by fine tuning. Rote "learning" by virtue of a genetic algorithm process, which is a highly efficient grid search for rules to an optimum path. Similar to a multitude of other problems easily solved by a genetic algorithm. The neural network is just the mechanics, like a spreadsheet. Choosing the correct NN architecture is imperative - that one in the demo (feed forward, basic backpropagation, cuppla slabs) may/would be unsuitable for financial markets/time series where you need feedback loops/memory and so on - I'm a little rusty on the various names of architectures.

You will note that right near the start a mathematical deterministic fitness function was specified - the rules of the game - this is the intelligent part, and in real problems, quite a few other things that might require a high understanding of maths, econometrics and philosophy/logic.

Once everything is sufficiently defined that describes what is required for a solution to be found, then, yes, a solution may be easily and efficiently discovered with a genetic algorithm within a neural net.

For those familiar with solver in Excel, kinda similar, except that a genetic algorithm can solve problems too complex for a conventional mathematical algorithm which can easily get stuck at a sub-optimum solution, or less.

Outside of dedicated NN software (the stuff I had won awards. It was a bit of a grey box, though, and I had issues with it, because in the interests of ease of use it actually was wrong, PCA was ??? non existent?? beggars belief- next para - variables must not be correlated and be normalised - excuse the bold, it is important, or you want your NN to fail? Again? And blame the NN?), I have some good experience with building neural nets in Excel with an off-the-shelf genetic algorithm add-in. Substantially better than the expensive retail dedicated NN software for financial market data. We all make mistakes, a learning experience. Maybe they fixed it.

With time series data such as financial data, it is possible to include too much. The most relevant data is the most recent. Discriminate analysis (principal component analysis (PCA)) reduces the number of variables to just a few uncorrelated ones that you won't recognise, anyway, if there are more than a handful that you think might be relevant. Decades of data is positively bad. 5 years of daily is generally sufficient. Plus one at the start that is not included to get the indicators settled in. You should really re-optimise with each new week of data - this is not tweaking or curve fitting, just a necessity with NNs because of maths.

I just re-read the first post/s here. Some of what I said is covered there, good. Some is totally way off the mark, and that's nearly OK, as it was 2006 and before many adults now trading with phenomenal success were even born. Like I would *never* use something like stochastics in a NN, they are a visual thing, open to interpretation and abuse, and can sit at extremes by nature of its calculation, perhaps why traders like them - complicated and can be made to fit whatever you are looking at. Would confuse a NN forever and it won't like you or your friends. NNs have high moral standards.

Same with "patterns". How would you code a pattern? No two are the same. Divergence? Tricky as. I saw one bloke try and it was about 30 lines of C code. Start doing the pattern thing and all bets are off. You don't know the power of NNs. You can look at a chart and just see it. After all, you have to tell the NN what to look for, so do it yourself or code a blip into your trading software. Big green and red arrows.

Things need to be expressed/normalised either as 0 or 1 (categories, eg, up, down - easy, some NNs get off on simple pleasures, true), or usually continuous between 0 and 1.

Having vented about stochastics, use meaningful oscillators which are relevant. Perhaps the *only* thing relevant. Like RSI or MACD, other momentum indicators, acceleration, etc. Use different lengths, formulas, etc - they are alll usually correlated, which PCA will take care of. Oscillators are sort of stationary, being a difference of prices- some combination of low pass and high pass moving averages.

To get really snazzy, you'd divide the oscillator by its root mean square - then is close to N~(0,1). (not really, one more step, but that'll do). Done correctly, oscillators have zero lag. Like the RSI, for example.

Do a simple momentum of the oscillator. An RSI of 80 going up is vastly different to an RSI of 80 going down. If the period of the RSI is 14, do a simple 14 period momentum on it. Or 28.

MACD isn't theoretically zero lag but is quite close sometimes. Experiment. Its fun experimenting, naturally.

Use a lot of imagination if its not naturally an oscillator, like say volatility, or Kaufman's efficiency ratio, or the Sevcik fractal formula (the latter two are sort of correlated at some level). Much other stuff, each different but may convey unique information.

Keep things simple (but not too simple) and robust. Things you understand. Things that make sense.

Like, that intermarket stuff that was heavily promoted by a bloke and now his son - is pretty easy (relatively. Nothing is actually easy with NNs. If its "easy" then you are truly blessed). It makes sense. A day doesn't go by when there's not some discussion about what effect will this have on that, and it depends. You have indicators for the market you are looking at. Now chuck into the cake mix the same ones from other markets, press enter and go visit your therapist.

If there's nothing, there's nothing. No big deal. Re calibrate, think.

Nobody gets things exactly right and its easy to give up if you think it should be easy, but you might be onto something with some thought. Maybe incorporate correlation into the actual indicator itself.

Last month there was positive correlation, this month negative. Or, last month strong positive, this month weak positive. Things stay correlated for a reason. I have found good results carefully incorporating correlation - should his be a surprise? Doesn't it make sense? If the correlation is strong (+ve or -ve) the NN may ignore other stuff that is important at other times. This shouldn't really be done with two variables in a NN unless you are a person with a proclivity for believing conspiracy theories like the one going around that Fed is benevolent.

Optional, but related and also to PCA, google multicollinearity for tricks and understanding, and in the end its actually quite easy if you are keen. More than two neurons required if you get into ANOVA and other heavy mental stuff.

So, you have to babysit a NN - it cannot tell whether something now has a positive correlation when previously it had negative without you telling it. They are dumb. There should be jokes about them.

Now if NN fails in its duties, its not really her fault, is it? It is you who jokes should be about. You may have not correctly specified a variable. OK, there's the maths, but if that scares you then what are you doing with NN? - NN is maths, and you will enjoy it.

Excuse any double entendres.

There are lot of tutorials about predicting stock prices using Python based on Neural networks. I also tried it but it cannot be used to forecast future prices. Accuracy is around 50%. So I gave up using neural networks. I tried LSTM model (Long Short Term Memory). Is there anyone who is really successful using neural networks?

Are python based neural networks even advanced enough to be used in stock price prediction? I think we'll have to wait for neural networks to become far more advanced before common folk like us can even contemplate using it to our advantage. There are some sources online like this https://realpython.com/python-ai-neural-network/ that talk about python based neural networks but all of it looks pretty meh from where I'm sitting.

Python neural networks? No forget it. Besides it's a dumbed down language for children. My attempt to forecast price action by adapting existing tech from other industries was unsuccessful overall. I found later that we need completely different type of technology to match the dynamics of the market. I have found the right solutions for the problems. But I don't think it's suitable for the retail trader because proper development and operation of such AI system will require substantial capital investment way beyond the typical trader net worth.

You absolutely right. The main problem for retail traders is that it is very complicated and cost much.

But I see one more problem: magic. Everything about trading is covered with magic. Check the threads. There are so many lucky traders. They are making 5% daily, 100% per week and so on. They are sharing their successful just another moving average strategy and so on. They are teaching, mentoring, creating youtube channels and selling indicators. Also look at the ea market. There are even no ea that can calculate the lot size properly depending on the pair. All that we still have (since 2006, the start of that thread) is just another one martingale/grid system.

Everyone want to get rich fast drawing random lines on the chart or placing fibos. Every found thread has a word "simple". Trading is simple. Do not overcomplicate it - the main message of 99.99%threads and posts. And we are talking about forex. If you will look at the crypto market the degree of idiocy is much higher. There are pump&dump schemes, pyramids and other stuff.

About neural networks. I think it is the future. It IS possible to predict the future price movements with high accuracy. And I think there are a lot of traders around who use and develop it. But they will not post and will not share. Because the knowledge is the power.

About language to use. I don't think it is the problem to use python. I'm using php and only now moving to javascript and many people laugh at me. But I don't see anything bad using it. Just because it is the question of measurement and scale. You can start with anything you are familiar to and then dig harder.

To create the team to develop you need a lot of money. And of course like-minded people. And this is the biggest problem. For example my personal dream is no earn enough and then create a company to develop such a things. But first I need to have some kind of passive income to cover all the expenses.

It's "simple" haha. But seriously, digging in the wrong places will never give you positive results.
Not possible to create sustainable trading strategy based on chart data. The prices move randomly from our point of view. Some patterns seem to repeat but they have no relation to the future and have absolutely zero predictive power. In other words, the predictive power is exactly 50%. It's random.
The patterns are garbage and noise! And you know the main rule with neural networks is - Garbage in, garbage out!

Another data source is needed! You have to mine not the charts but the internet. Primarily the social networks, the news networks and the statistics from search engines. Some correlations have strong predictive power. It's well know stuff in the high circles of finance. There are positive and negative trends in crowd behavior and they directly correlate with the markets. Also fear seems to be the primary driving factor in the markets and it's directly correlated with market volatility and liquidity gaps. You can see this AFTER the fact on your charts as large long bars. Global negative news and fear will rock the markets every time without fail. If your robot is fast enough to get the news and to analyze the internet activity before most traders you can make some good money from this by front-running the crowd. But, as I said this is not the type of job for the retail traders. It's complicated stuff.