Potential new machine learning style software.

All data from MT4 since 2001 has been extracted... from it I conclude that the pattern holds all the way back to 2001 at the least! This is great news! Sorry, but I'm going to stop posting updates here. I'll answer any last Qs if there are any.

Oh and it looks like it works for all currency pairs. I haven't tested commodities or indexes yet, but it'll probably work for those too.

I would show you guys a graph since 2001, but I've got about 30MB worth of data and over 0.5 Million data points. Anyone know where I can graph this many points? lol

Thanks in Advance

hey sloof,

take excell or rapidminer. there you can visualize your data's. maybe you could give us a little hint about your work. you told that you use an own-written cycle indicator with fixed cycle frequency. you then measure the amplitude. what are you taking for this measurement? the distance from a MA? or the zig-zag? and if the cycle frequency is fixed, did you scale the chart in time to overlap the peaks? but how you get back the right time in the chart where those bottoms and tops happens? did you rescale? if yes how do you know what scale-ratio you have to use, because the ratio must be different from one swing to the other. what about phasemodulation. for different timeframes? what about the mainfractalfrequency which have to be in every timeframe?

sorry for interrupting..

questions removed

What I wrote was never intended to be a cycle indicator at all! That's the crazy part about all this... it just turned out to have one and I never noticed before. I can tell you most of the indicator is based on the MA with a period of 120. I haven't tried with other periods (I was thinking that other periods may give better patterns... haven't tested it yet) but there is a chance that 120 is the best. I'm still looking into the different TFs... I think 1M has too much noise to really follow any pattern, but 5M+ patterns can be found. So far this only applies to 5Min. By "mainfractalfrequency" are you asking if there is a relationship to the # of fractals in each cycle?

Phasemodulation.... I suspect this may be involved with higher TFs

"did you scale the chart in time to overlap the peaks"
Not yet and the peaks may be important, it certainly looks like the peaks are variable... I still haven't discovered the pattern there, but with the data since 2001 I should be able to figure it out with a good graphing software. Thanks for the heads up about dataminer. Excel has a 60,000 or something limit. I've got some 680,000 data points.

Excel 2007 can handle 1 million rows.

but it probably can't graph it all. I'd like to create a HUGE graph such that I can zoom in and out as well as navigate left and right. I think Wolfram Mathematica can do it... I'll just have to take the data with me to school sometime this week.

Matlab!

hahaha oh yeah! that would work

somebody recommended rapidminer, i used this tool too until i discovered knime (http://www.knime.org/). knime has also fully integrated support for R (i strongly suggest learning R too and i am sure you will love knime).

No need for tools that were never designed for this kind of Job (Excel). And before you start saving money for a matlab license i would suggest exploring the potential of R first (knime and R both are open source).

I would've used matlab at school (they have it on all campus comps.), but I'll check out knime now! Thanks!

Duuuuude! This is awesome! 10 Min into d/l it and I've already gotten used to it lol

Nope no Problem with Excel. Even on one of my shitty old PCs with only 192 MB RAM. I've loaded over 1.5 Million Datarows without problems.

Have you already finished C?

Sloof

how is your predictive analysis working so far?

Pretty good =)

I'm just waiting to do some live tests

Hello Sloof,

Just stumbled on this thread, before diving in was wondering how the live testing is going,and by any chance do you have any live trading results for us to see?

Regards

Man, did my day just got better...R integration...and libsvm...and a delicious Eclipse based GUI...

Seriously thank you for pointing out this little gem. Funny btw to see two German universities competing on making the best open source visual data mining tool.

I have only toyed with RapidMiner in the past, thus I don't have a strong basis for comparing it with KNIME. Any comments on KNIME's stability/maturity compared to RapidMiner?

Hi there,

both, RapidMiner and KNIME, are freely available open source data mining tools and both are quite mature high-quality software tools. Both include LibSVM. In addition, RapidMiner also provides JMySVM, EvoSVM, PSO-SVM, and 100+ other modelling techniques. I never used KNIME for financial time series analysis and forecasting, but I have used RapidMiner a lot for this task. So I will focus my answer mainly on RapidMiner and demonstrate that RapidMiner is mature and stable enough for professional financial data mining and forecasting:

 

1. RapidMiner provides standard evaluation schemes like cross-validation as well as time series oriented evaluation schemes like back-testing (sliding window validation). KNIME also provides simple like cross-validation for static problems, but I am not sure, if it supports back-testing for time-varying problems, which I consider essential for evaluating automated machine learning based trading systems.
2. RapidMiner provides a free extension for advanced value series and time series analysis and processing including windowing, moving functions like moving averages, transformations like Fourier transformations and phase space transformations, and a lot of automated time series feature generation and selection optimizers. I do not know any tool that comes close to this in terms of automated time series processing and representation optimization.
3. RapidMiner can handle large data sets, even if they do not fit into main memory. You have the choice between in-memory and on-database data mining.
4. RapidMiner can read and write Excel sheets, CSV files, all common relational databases (Oracle, IBM DB2, Teradata, Ingres, MySQL, PostgreSQL, ODBC/JDBC, Microsoft Access, Microsoft SQL Server, etc.), web pages, and many other data formats.
5. RapidMiner and KNIME both also provide text mining functionality. RapidMiner has already been used for web mining and sentiment anaylsis on financial stocks from financial news and from internet discussion groups like Yahoo finance and Seeking Alpha. This type of information can be nicely combined with time series data in order to improve the time series forecasting and the risk analysis.
   
   
6. RapidMiner is used by many financial institutions and investment banks. Among others, Rapid-I lists Allianz (Germany), comdirect bank (Germany), Ineas (France), Landesbank Berlin (Germany), Merrill Lynch (USA), PentaSecurity (Chile), and aiinvesting.com (USA):
   http://rapid-i.com/content/view/8/56/
   
   
7. There are a lot of free resources to get started with RapidMiner quickly (see below) as well as special training courses and webinars on financial time series analysis and forecasting with RapidMiner for the most effective use of RapidMiner for financial data mining (see further below).

Resources to get started with RapidMiner:

 

1. RapidMiner Community Edition: free download:
   http://rapid-i.com/content/view/26/84/
   
   
2. Free RapidMiner short video (3 minutes): The main features:
   German: http://rapid-i.com/videos/rm_5_demo/rm_5_demo.html
   English: http://rapid-i.com/videos/rm_5_demo_EN/rm_5_demo_EN.html
   
   
3. Free RapidMiner video tutorials: 6 short 3 minutes videos in English: GUI, Data Import and Repositories, Modelling and Scoring, Visualization, Evaluation and Performance Estimation, Data Pre-Processing, Data Transformations, Meta-Data Transformations, Quick Fixes:
   http://rapid-i.com/content/view/189/198/
   
   
4. Free RapidMiner video tutorials at NeuralMarketTrends.com:
   Building predictive models and AI financial market models with RapidMiner, automated genetic feature selection with RapidMiner, etc.:
   http://www.neuralmarkettrends.com/tutorials/
   
   
5. Free RapidMiner tutorials for web mining:
   http://nemoz.org/joomla/content/blogcategory/21/53
   
   
6. Free RapidMiner tutorial on sentiment analysis:
   http://kmandcomputing.blogspot.com/2008/06/opinion-mining-with-rapidminer-quick.html
   
   
7. Tuesday, February 9th, 2010 from 15:00-16:30h Central European Time (= 3:00 PM to 4:30PM CET): RapidMiner Live-Webinar "Financial Data Mining with RapidMiner" (English, 90 minutes, via WebEx):
   http://rapid-i.com/component/page,shop.product_details/flypage,garden_flypage.tpl/product_id,35/category_id,18/option,com_virtuemart/Itemid,180/
   
   
8. Wednesday, February 10th, 2010 from 17:00-17:45h CET (= 5:00PM to 5:45PM CET): free 45-minutes RapidMiner Live-Webinar "Introduction to Data Mining with RapidMiner" (English, 45 minutes, via WebEx):
   http://rapid-i.com/component/page,shop.product_details/flypage,garden_flypage.tpl/product_id,5/category_id,20/option,com_virtuemart/Itemid,180/
   
   
9. Wednesday, February 24th, 2010: RapidMiner 5 Intro Day: 1-day workshop introduction to RapidMiner 5:
   http://rapid-i.com/content/view/188/
   
   
10. 2-days RapidMiner training course "Data Mining in Finance and Financial Forecasting":
    http://rapid-i.com/component/page,shop.product_details/flypage,garden_flypage.tpl/product_id,45/category_id,13/option,com_virtuemart/Itemid,180/
    
    
11. Current RapidMiner Data Mining training courses and webinars:
    http://rapid-i.com/component/option,com_virtuemart/Itemid,180/
    
    
12. Other financial discussion groups, forums, and blogs covering financial data mining with RapidMiner:
    1. Investox forum (German):
       http://www.investoxforum.de/index.ph...&threadID=5618
    2. NeuralMarketTrends.com (English):
       http://www.neuralmarkettrends.com/
    3. Knowledge Discovery and Opinion Mining (English):
       http://kmandcomputing.blogspot.com/
    4. RapidMiner Forum (English):
       http://forum.rapid-i.com/

I guess, this should be more than enough to demonstrate that RapidMiner has the maturity and stability and scalability for professional financial data mining and forecasting and also enough to get you started with RapidMiner. I hope you find this collection of resources and references helpful.

Best regards,
Frank

http://www.marketwatch.com/story/dep...k=MW_news_stmp

Is it possible to get this version of Eureqa from you? The software is not for free anymore...

Thanks...