Going Platform independent! Going pyhon!

Going Platform independent!


I'm involved in trading related activities for some time/years.
Started manual trading mainly stocks, a little bit with options and CFDs.
I did direct investments in real-estates which were the most lucrative investments I ever did till now


A couple of years ago I found out about Forex: and started to trade manually but always knew that in the end I want to trade automatized.


I'm a self-thought programmer and in the last years I invested a lot of my spare time in learning MT4
starting automated trading (next to manual ect..) and developing quite a good MT4 code base for my purpose.


I have been away for some month for personal reason and discovered that Metatrade is (at least in the progress of)
introducing major changes to MT4. Whatever the outcome for backwards compatibility it made me really think.




Here is my conclusion at this point in time:


WHERE DO I STAND:
- I'm quite sufficient in my programming skills (mt4 and some others) but I'm also very much aware that there are much smarter people out there (even at FF)
- some time ago I gave up to think I need ultra fast automated systems, based on tick data ect..
I will not be able to succeed with my limited programming skills,
a moderate internet connecting and as a private person to compete with the big companies who have fast resources, ultra-fast computers, pay some of the smartest programmers ect..)
- I also concluded that for me I see only 2 ways to automated trading entries
a. random based entries
b. backtested probabilities entries (which is my preferred one)


BACKTESTED PROBABILITIES ENTRIES
- backtests will not and do not need to be 100% but (at least good enough)
because backtests are somehow anyway limited: e.g. spread differences, broker differences they do not need to be 100% but good enough to do the job
- some time ago, because of this I decided that for me I do not need tick data but 1 minute or higher bar data (depending on the system) are good enough
(I might change that in the future: e.g. for renko bars generation I believe tick data are necessary)
advantage there are more freely available long term historical data, less download time, less needed storage space but most of all less backtest time required
my current choice are dukascopy 1 minute data as the base (I did write some tools for MT4 http://www.forexfactory.com/showthread.php?t=426120)
- In the future I might also be looking into generating artificial backtest data (not sure if this has some value - never gave it much though) ANY IDEAS/INFO?
- I also concluded that it for my needs it is best to base any trade entry decision only on closed bars


GOING PLATFORM INDEPENDENT
- previously I had implemented most and more of this in MT4 (coding around some of the problems I found ect..)
Main trigger is that MT4 is changing in a way I do not like: (Not saying all are bad - but only saying there are "recently" a lot of changes in MT4 - I assume because MT5 did not take of as hoped for)
- first the automated, forced Live Updates
- many recent changes like One click trading, and the strong focus on implementing signals following options
- and now the current move to backport MT5 into MT4 http://www.forexfactory.com/showthread.php?p=7142542


The second trigger is a previous experience: I had programed an independent trading application but very broker specific (they changed some things and I was left with nothing) one reason I moved to MT4 later on




MY CONCLUSION:
1. I do not necessarily need my backtest environment also trade live. (even-though I liked that a lot in my MT4 stuff)
many platforms/brokers seem to focus more recently on things like trade copy, signal provider ect...
- MT4
- September 24, 2013 OANDA Announces Acquisition of Forex Copy Trading Network http://www.currensee.com/OANDA-Annou...rading-Network


2. In case I should continue to use MT4 I only will implement a final trading system without all the extras I did till now.
3. there is a high probability that in a future MT4 version or as such in many other platforms there are ways to pass on the most important data to an external program
- because I base all my main trading decision on closed bars I do not need too much data to pass around and ultra high speed I gave up anyhow.
e.g. Oanda my current broker they work on an OANDA REST API https://github.com/oanda/apidocs
Not sure about other brokers but I guess Tradestation, many of the java based platforms like Interactive Brokers, dukascopy ect will allow similar things



I LOOKED FOR OPTIONS AND I DECIDED TO START LOOKING INTO THIS:
- build all myself based mainly on opensource available libraries, apps
- mainly in python: with MT4 this is where I'm most sufficient and in the last years a lot of good libraries have come out.
- also mainly looking for Linux options






Going pyhon!


- data storage in : HDF5 (.h5) Hierarchical Data Format or/and database


1. main libraries:
Python Data Analysis Library — pandas: Python Data Analysis


TA-Lib (includes 200 indicators) No experience This seems to be a bit old: Version 0.4 (September 2007) but maybe it is also a good sign?
ta-lib (PyhonWrapperBasedOn Cython) This is a Python wrapper for TA-LIB based on Cython instead of default one with SWIG.
http://www.pyqtgraph.org/



2. plotting Not sure about plotting or for a start if I output only in spreadsheet
matplotlib


Chaco Chaco is a Python plotting application toolkit that facilitates writing plotting applications at all levels of complexity

pyqtgraph Scientific Graphics and GUI Library for Python: see post from corrugatedit

1. Post 22

3. Machine Learning not sure if I do later on something like this no experience at all
scikit-learn: Machine Learning Built on NumPy, SciPy, and matplotlib


PyBrain: is a modular Machine Learning Library for Python


skdata Skdata is meant to interoperate with other Python machine learning software (such as scikit-learn, PyBrain, or custom algorithms) but skdata does not aim to provide machine learning algorithms.




4. Bundled distributions
anaconda Enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing.


Enthought Scientific Computing Solutions


pythonxy Scientific-oriented Python Distribution based on Qt and Spyder Windows (Not for my consideration: I want to stay on linux)
pythonxy-linux Scientific-oriented Python Distribution for GNU/Linux based on Qt and Eclipse


WinPython Portable Scientific Python 2/3 32/64bit Distribution for Windows (Not for my consideration: I want to stay on linux)


Pyzo provide a computing environment aimed at doing science and building professional applications




5. Some other related projects: who knows maybe something useful
PyTradeLib Python financial trading, research and backtesting library


ultra-finance It aims to be a fully featured event-driven based backtesting system.


PyaAlgoTrade It's coded to allow for distributed testing of strategies on Google's cloud infrastructure. It incorporates the open source TA-Lib technical analysis library.


TradeProgrammer It also uses the TA-Lib library. The package is free to use for backtesting, but its live trading version is commercial.


Zipline, a Pythonic Algorithmic Trading Library (is currently used in production as the backtesting engine powering Quantopian (https://www.quantopian.com) -- a free, community-centered platform that allows development and real-time backtesting of trading algorithms in the web browser.)
Zipline Python Opensource Backtester google group


QuantSoftware Toolkit


QuantLib A free/open-source library for quantitative finance
PyQL : a new set of Python wrappers for QuantLib based on Cython original is with Swig


iTrade The iTrade project is an open source initiative to provide a charting and trading system written in Python language


ProfitPy ProfitPy is a set of libraries and tools for the development, testing, and execution of automated trading systems.


tradingmachine pandas ta-lib matplotlib (git source seems to be gone)




===== others =====
docker: https://www.docker.io/ MAYBE: looks interesting as a base block for coding

LCONF L(ight) CONF(iguration): A simple human-readable data serialization format for dynamic configuration.


Any related tips/helpful past experience are welcomed


Cheers

Ati

I'm running my own (live) platform that directly interfaces with mt4 and commands mt4 to do execution and report back.

From this perspective I can tell you that you can export any data from, and pass on any command to, mt4, easily. At least with some windows (os) internal mechanisms.

So the (comming - or not) changes to mt4 are in fact a benefit in this regard, and mt5 is even better, because of event handling and stuff.

hi rockit,

may I ask in which language you coded your platform and also for how long do you use it for actual trading (demo/live)

Thanks

Ati

I need speed, so, it be c++ of course. I use it for real trading probably since one year now. Before that I have used basically the same gui, based off java, on a java platform (Dukas) for also at least one year. Once you have done the basic work it is no big prob to connect it to any platform/api, or transfer to any suitable lang.

MT4/5 comes handy because it can use low level OS stuff via c/c++ dll, when one needs speed. Of course direct apis are more interesting.

I'm behind my plan and must extend: *** ROADMAP *** 1. == Preparation of basic coding environment:till 15th Jan 2014

Currently I am tinkering with Spotware's Connect API (cTrader platform): https://sandbox-connect.spotware.com/draftref/

The good thing about it is that it can be used with all brokers that offer cTrader platform, and all in one place, so not broker dependent but platform dependent.

It has the potential to become a big thing, especially for paid services. Python (v2) can be used to access the API, too. It is not feature complete yet however.

rockit,
Thanks for the info

hi everyone,

I'm also building my own software for trading. I was using Java but in the last few months I switched to C++ and Qt.
I think that the best way is to build your software as independent as possible and not rely on MT4 or other platforms. This gives you great flexibility.

Ati Baia: "- In the future I might also be looking into generating artificial backtest data (not sure if this has some value - never gave it much though) ANY IDEAS/INFO?"

Are you referring on generating new price paths ? If yes, you could read about Monte Carlo/brownian generator. You could generate new paths by modeling your own distributions and other interesting things. Definitely has value if you want to test your strategies to new data generated from certain assumptions. just my opinion!

I already built my self a backtesting platform from scratch which deals with tick data and in the future I want to implement an artificial price generator.

hi Reverse,

thanks for stopping by and all the best for your coding.


UPDATE: have been very busy in working on preparing my base coding environment which is boring but needs to be done. (got also a bit sidetracked)

1. Post 5

I'm back. After nearly 1 month doing other things I will be starting with part2:

2. == coding the price data storage

plan: because price data are mostly in csv format available (or if not they can be most of the time be exported to csv)
I will base my price data input on csv format and the final storage in hdf5 (Hierarchical Data Format)

Hello Ati,

Compliments on the awesome breakdown of Python resources, this has already become valuable and provided me inspiration for new coding ideas. This forum is one to watch and quite timely with the frustrating MT4 changes.

You may find this post interesting, it is from the blog portion of a forex development blogger/forum I track, the author has had it with Metatrader and wrote his own trading software in Python: http://mechanicalforex.com/2013/06/i...solutions.html

Granted it is not as ambitious as your project, but it was successfully integrated with the Oanda API and the intent is to write trading systems in Python to then link to this same trading application to handle execution. That post had caught my eye for potential, yours has furthered the concept.

Good luck with your progress, I eagerly await your next updates...

Hello dlazor,

sorry for a later reply, at the moment I'm not actively trading - so I do not check forexfactory as much as I used to.

Also thanks for the link: will have to check that out some times.

I'm very glad my little python resources breakdown was of some value to you.



There is one update:
I played around with anaconda/conda for a month+ and recently decided against it.
Couple of reason for that:

- I do not need all the features of having different python enviroments: but I liked the idea of some kind of isolation of the main system
- for this new project I will switch straight away to python3 - conda does support it but some of the packages I was looking to use where not available or not uptodate
- so I had to build them myself (no problem) and digged a bit deeper and found a couple of issues which in the end I think is not worthy to spend more time on.

Since a couple of days I'm looking into: docker: https://www.docker.io/
it sounds even more interesting to me to use for my coding effort: but not yet sure if I want to go this way to setup my coding platform.

Cheers

Ati

Hi.I am an amateur trader from China. I used to backtest forex strategy using mt4. Now i intend to try python-pandas. I have read some articles in quantstart and learned a lot, but still facing a problem: I don't konw how to use my forex data. The forex data is from http://www.forextester.com/data/datasources.html. 1M accuracy. I downloaded the txt file, but I do not know how to import it. (it's easy to import it into mt4) Would you help me? Or is there any tutorial available? Thanks.

Thank you so much, great links. I'm in the same boat...labouring away on Python for 4-5 weeks now.

I have some HTML/CSS/PHP knowledge, but even without it, I have to admit I'm liking Python a lot. It's actually readable compared to a lot of the other stuff.

Time past quickly. Still working on brushing up my python with some smaller sub projects.

You might want to have a look at FIX. It's a protocol/interface shared by many brokers. There are also some python and java libraries for it.

I'm looking into doing something similar this year, but I will use java (I'm developing java software since ~10 years for germanys chemical and food industry, so it became my second native language, kind of).

The string date parser is the bottleneck here.
10 years of m1 (4 million records) take (on my machine)

40 seconds to load & parse from csv text

Once you've got them in a dataframe it only takes

4 seconds to save in hdf5

< 1 second to load from hdf5

Thanks a lot! I'll try to overcome it.

words of caution from personal experience

as much as i would like to see this space progress and have other people do the required work, i feel obligated to offer a bit of advice about writing your own trading structure - don't

UNLESS
unless you already have proven profitable systems that you want to implement live in your own trusted code with custom features. this clear aim makes the whole project realistic and attainable
OTHERWISE

before even taking on this project, switch for a moment to a bigger perspective and ask yourself:
AM I IN THE PLATFORM BUILDING BUSINESS OR THE TRADING BUSINESS ?

writing your own platform may be a form of procrastination and 'greener grass syndrome' because

realistically right now creating profitable trading systems takes work and navigating a narrow path full of real practical troubles you experience in mt4 everyday

compared to

your platform idea - refreshing idealised clean clear concept full of excitement hope potential - you notice you have many good ideas on how it can work and even if takes some work, you only need the core features in clean fast code. it's not real

"hey, i can execute on ticks with real bidask, make it super fast with cython, pandas already has basic indicators and charts the whole thing with one line, i can easily track signals in one array, size in another, fifo makes position tracking easy, maybe i'm not smarter than metaquotes or ninjatrader guys but i can focus exactly on what i need without all the bs features and bugs, surely i can run much slimmer and more powerful without the myriad frustrations that mt4 gives me, and wow python has super powerful ready made libraries: machine learning, cv, neural networks - the sky's the limit"

yes writing your own testing/trading platform from scratch gives you flexibility and power and feels fun and easy fueled by the hype (python clarity & toolspan, java portability, c++ speed), but

IT IS NOT HERE READY NOW

you will still have to solve most of the problems you have to solve now - except from scratch

for example consider that you have to write your own charting
- matplotlib is utter shit speedwise & interactivity, you're forced to peek at static samples
- chaco sucks at managing mt4'style graphics objects, even a simple horizontal line
- pyqtgraph (out of the box) can't draw more than few candles with float ohlc without artifacts

do you want to contribute to open source projects by studying & debugging charting libraries or do you want to build trading systems ?

and then you have to write order tracking, stops, limits, poll live data, communicate with the broker, recognize handle error fills, (btw zipline's shit too)

it's an extra layer of consumed time and brain cycles and after months or years of debugging your platform, you still have to write the actual trading systems

otoh ninjatrader, mt4, amibroker, tradestation, jtrader, whatever as buggy and limiting as they are, they are here now ready to use and man up to the real challenge - BUILD THE ACTUAL TRADING SYSTEMS

mt4 is shit inaccurate and metaquotes is untrustworthy. but it's ready to use NOW. it's brute metal fast through dlls. it has historical data already in it, and demo feed to live test out of the can. the historical data is gritty sketchy without bidask, and still useful to quick and dirty prototype and backtest systems _robust_ enough to overcome the adversities.

you can already interface python with any platform. use dirty hacks, communicate through text files if you have to - it maybe dirty messy but it's DOABLE NOW

WORK ON THE ACTUAL TRADING SYSTEM - any platform will not do it for you

Hello All- this is somewhat a follow up to ati's thread on search for an open source application comparable to trade station easy-language? Looking for something I can use for backtest ing. It has been 3 yes. Since using trade station. Any thoughts on where to start?