Artificial Intelligence over The Capital Market

Dear Members,

I live in Romania, am married, have three children and am in love with AI. Over the past 18 months a friend of mine and I started working on a project that regards the use of AI on the capital market.

In this project (that we now work on with 4 people) my role is the role of the product owner (product owner as in a SCRUM team).

I read a lot of research papers during those 18 months and still haven't found in any paper the approach that we are going to pursue.

We don't claim to find the “holy grail”, even though we do dream of that . But we do believe that we can get very close to it.

We know we are not the only ones that are attempting to forecast the market; a lot of bright minds are or have been working on this already.

Imagine having two soccer teams: one with a high budget and top players and one with a modest budget, unknown players but with a brilliant coach. On which team would you place your bet? (Think of Bertha Berlin who now plays for the title in Germany and defeat all big budget teams).
What I'm trying to say is that the training stage of AI is the most important things: We should be careful with what data we show and what data we don't show to the AI for learning.

Most of the studies I've read about would simply take time series and technical indicator data for the market and train neural networks or Genetic Programs on all that data and nothing more than that.

On the internet thanks to people like Klaus Meffert with JGAP, Eibe Frank and Mark Hall from WEKA, Paolo Marrone from JOONE and Steffen Nissen from FANN, and many others who have developed libraries and made their work public everyone is able to experiment with AI.

You may say that what we are doing has been done before; but we plan to filter our training data well according to certain specifications and rules. Think about what Ford, Google and McDonalds have in common.. they all took something that excited already and turned it into something better. Google didn't invent the search engine and Ford didn't invent the car.

This leads us to a well known expression: Our limits are merely the limits of our imagination.

What we would like to do is to get all those AI components together under one roof and use all of them to forecast the market.

What we would like to do is to get all those applications together under one roof and start to use in forecast. why to bring all those applications under one roof? because everyone in part have very heavy very nice good stuff and in our opinion it would do much much better forecast.

We've already made quite some progress: We've setup a server with SVN on which we work and we've almost completed data collection, (multiplexer) server and database modules and are well on our way with the AI modules.

Currently though, we are little bit short on people. We are looking for good programmers that have experience in the AI field.

We will keep you informed of our progress in this thread.

With respect,
Kerosen

before this new approach presented in intelligentics-design.pdf file, we did try the easy way as all do and is recommended like design some ANNs probabilistic neural networks, radial basis function network back propagation network and so on.

here you can find some screen shots with our previous ANNs, API (because we don't want to depend on one kind of platform the GUI where to show the results and the test result xls file where there is shown just two of the ANN.

The test ware over 10000 so the percentage accuracy should be a good one

Can you provide source code or more detailed instruction how is it working?

Hi Graff

The system that i presented here (the screen shoots) uses ANN' with 10000 of inputs, and one output. The inputs are tick data because we consider that the most accurate data is tick. It shows the direction and not the price. as I said before, we setup an DB the API takes the input and throw in the DB from where the ANN start and analyze. As I said this was our first system and even if we got like 60% accuracy over some pairs wasn't what we looking for. in this way we start to develop another one who will search for hide correlations between the pair you watch and things like calendar, fundamental analyze and technical analyze, the system will extract it self the conditions and give an forecast. remember that even if ANN are good for solving general problems that wouldn't be enough.

I forgot to put the test results from our previous work, and I put it here now.

much about to say about the system right now is earlier.

we are short to people who do now programming (we use java) and AI, but with this post is a good a opportunity for us and people who love AI and would like to join such a project

As I said before, we run over 10000 tests

will be PosgreSQL and it should store the following(future inputs to store is subject to discuss):
All major Indexes
All major commodities
Economic Calendar
All major currency ticks

The way those data will be collected must not under any circumstances to be broken data (e.g. blackout power or accidental internet connection interrupt)

Is very important to be this way having in view that for AI (artificial intelligence) DM (Data mining) and ML (Machine Learning) modules, the inputs should be as accurate as it can.

Here an external solution like having a script on a 99,9% online server who will collect and record all the data required in a file and the file to be send it to the DB (Data Base) would be a solution. Any others solutions are welcome. or when is done this part, put the module over a few servers on different locations with different ISP and to be synchronized

some tasks to be done, are:

Implement Communication Thread

Introduce JAXB

Configure and Add HIbernate and mappings

DB persistence layer

Investigate how to control Replay Thread

Implement Replay

Test

OK i can help out on the hibernate and jaxb stuff.

Regarding the thread call back, have you looked at something like quartz? means you would have to introduce spring into your project tho?

We consider that as long time the movement of the price depend on more factors, first is to be analyzed the impact of those factors(e.g. Fundamental analysis , Technical analysis, Economic calendar) in correlation with the pair to be study and only after that analysis is done to start and extract some basics conditions.
We all know that if we discuss about economic calendar, very important factor is the forecast (which at the time for release of that –important-announcement the forecast is already included in the price) as important is release vs forecast.

Trying to apply AI over the capital market should means some strictly rules to follow such as:

Each currency has it’s own characteristics
One pair of currency, has both currencies characteristics
One pattern(e.g H&S) does not have the same impact/accuracy on all currencies pairs
The genetic program should extract the trading rules(long, short or range)
Each currency should be analyzed separately and together with it’s pair
Not always, if a long position isn’t to be taken means short one (it could mean sideways-range)
Fibonacci levels should be taken in consideration.
And so on

We should consider that being a dynamic market with dynamic rules small to medium time frames would be ok to investigate
__________________

here are some shoots about the work we already have done it http://www.forex-tsd.com/images/smilies/smile.gif

crawler for major indexes and the economic calendar

and the DDE application

also we have finished the Technical Indicators module


we are still short to programmers over AI http://www.forex-tsd.com/images/smilies/smile.gif

any help would be great as ideas too

Ok… we already start to work on the AI module although this is not our first priority(here you have an attachment over our AI module work). Our first priority is to get done the communication way between the modules and start to gather data into our data base, to put the gathering system on multiple servers, and synchronize them in order the data to be as accurate as it can be.
Beside this, one of the most important thing, as I said it before, is how the AI module and Data Mining module should be used because in our opinion is not at all enough just to take some Time Series and throw it into any kind of AI technique and say “this is The forecast”.

If you look at “Swarm Intelligence in Data Mining” at first chapter you will understand for sure why collecting data is important and why we don’t want to take just simple Time Series. On this book you have also a schema about how is/should to be apply.
So, coming back at my example about a soccer team with high budget and top players (here I call soccer team the AI techniques) and a soccer team with modest budget but a great coach (here I call great coach the strategy to be apply in order to get your AI techniques to work at high percentage), I would prefer the second one.
I don’t say AI techniques doesn’t matter(it matter a lot) but the strategy you use those AI techniques, is above.

We’ll keep you up to date

WE ARE STILL SHORT TO COLLEAGUES

Any ideas or/and suggestions, are more than welcome.

That part makes or breaks the whole system imo.

Make sure you always control the 'kill switch' and can switch over to semi-automated at any point, and take the pilots seat. There's -always- lots of foggy mountains on the horizon.

Run realtime, and test -while- you are trading and combining and weighting modules, even if you can't save yet and have to dump tick data.

To say the least.... Rgrds.

thx for answering kd3726

I am agree with you that the communications way is very important and that why it takes us so long.

before to run real time, we do need to gather data into data base (e.g. Economic Calendar where I repeat by the time an announcement is about to to be release, the forecast figure is already included into the price of that specific currency which means that we do have to try understand how currency price evolve from the forecast announcement until and before the new release ).

imho is necessary first the Data Mining module to try and extract some rules on the specific moments of the market(I won't say yet those to be study )

regarding dynamic market with dynamic rules, those rules when they do change, they stays (the rules) in the market for some period of time.

also I would like to say that we don't have to be "The Big Shark" we want to become the "pilot fish" (the one who stays under and react in the same direction like the shark)

I do appreciate and wait for more discussions

for those who are familiar with FANN you will find a very interesting article HERE I think Emsi, the one who wrote the article have done a very good job

enjoy it

I am not familiar with FANN, though I do have experience with AI in gaming applications. I also have experience with some mathematical/statistical tools that are used for predictive analysis.

If I were to implement an AI system for trading forex, I would do the following:

* I will gather as many Trading Systems as possible such as Moving Average Crosses on different Time Frame (from 1 Minute all the to Daily). This alone would be a huge library of system some of which are composite with each other (ie, MA + MACD, or MA + MACD + RSI, or multiple MA+Sto, etc...)

CLARIFICATION: A Trading System based on Moving Average Crosses on M1 is different/counted separately if used on a different timeframe like M5, M15, M30, H1, H4, daily. This moving Average crosses is counted as 7 trading systems. Now, why even bother with M1 or Daily? because over time, these lower timeframe or higher timeframe will be mathematically (by statistics) considered as entry and trend.

Pattern Recognition (such as double tops, flags, etc) are also considered as trading system.

* From the above, each system will then be graded based on probability of profit/loss. over time, we get a fairly accurate data of each Setups based on different Time Frame.

We now have a weight (probability ratio) and when we encounter a particular setup (ie, on M1 we get a BUY from a Moving average Cross, a BUY from a MACD zero crossing, a SELL from a RSI, a SELL from Bollinger Band upper bound limit, etc) we can use Bayesian mathematical model for predictive analysis to predict whether to go LONG or SHORT or stay away.

Some might think that all these will require an ultra fast CPU. No not at all and I believe it can even be programmed using MQL alone!

If anyone is in the same page as I am as far as AI is concern and is interested to join forces, let me know...

Nice to see a fellow Romanian guy pasionated about forex and knowing so much about programming and neural networks. I will be pleased to help if I can. I don't know much about NN but I have 10 years on real money trading in different markets and I can write code in MQ4.

Good luck in your quest!

ragnakore

having so many Technical Indicators result of many Trading Systems... for example you can combine them starting from like Combination of N(where N represent the number of all technical indicators) taken by 3 and so on until you get to Combination of N taken by N this can be done without Artificial Intelligence

indeed ANN (artificial neural networks) have an advantage by the fact that they are known to have very well perform into pattern recognition and also for memorize those patterns (especially hidden and complicated patterns)

is true that those days you are not anymore CPU depended

regarding FANN in order to understand it much better, you can download Steffen Niessen thesis from HERE and some research regarding genetic programming for discovering trading rules from here and here and here
you can find a lot of white papers


coldrecs

thank you youre experience I am quiet sure is a great one (10 years into the market and still breath and grow )

imho MT4 is just for amateurs like all of us(this include me too) that why we try the system not to be dependent for any trading platform regarding your your experience, I would be glad to have some discussion regarding capital market

You still have to weight this together in some manner, while considering that not all signals will perform at all times.

I know... that's why we have a Data Mining module

nice. but still the same questions 'whos the pilot' Rgrds.

Using Paco Hernandez PNN implementation, it is possible to self learn for a specified bars backwards (look in mql forum on this topic). However FANN seems to be faster and perhaps using similar self learning algorithm of PNN self learn, self learning algorithm can be constructed using FANN...Is this possible?

HERE you can download Steffen Niessen Thesis where he explain what is all about in FANN .

@kd3726 it should be on automatic pilot