Disclaimer: Building a testing and execution infrastructure is a long haul, and should only be considered by those willing to dedicate time to optimization and debugging as much as developing the actual trading algorithms.
Dear FF,
It has now been since 2017 I have not been active on the forum, only watching from far.
I only made time to answer the most relevant emails I received on my passed projects (ADXB / Autop / Horizon break / Keops...).
There are many reasons for my stepping back, not to belabor it was mostly due to studies and other side hustles.
Under-capitalized, I could not afford to go live with high MCR / low margin strategies (such as the grid and multi-asset ones I put so much time researching).
Times have changed, I started here as curious amateur in 2014 but now coming back as software engineer for an institutional asset management firm, I am more aware of our capabilities as retail players and about the FX market microstructure.
About the project:
This platform will be an all-in-one simulation and execution software, being able to:
- fetch raw ticks/spreads/depths (L1/L2) from selected brokers/exchanges/liquidity providers on demand (through API adapters)
- generate ticks based on an instrument statistical behaviour (i.e. extracting EURUSD statistical features and volatility profile to simulate synthetic EURUSD ticks) using brownian motion or 1D perlin noise
- aggregate ticks / order books in custom time series (e.g. renko, ohlc and aggregated order book)
- expose custom hooks for developing algorithms: on-tick, on-time, on-open, on-close, on-profit, on-loss, on-vol
- allow multi-assets and multi-timeframe strategies to be tested and ran
- run trained ML models under the ONNX runtime (ability to import and use Pytorch / TensorFlow models)
- run high precision back or forward test using aforementioned live / passed or synthetic data, and real fees / spread data
- route orders to brokers for live trading with as little latency as possible (through API adapters)
- monitor execution through a dedicated dashboard (connected to all accounts / brokers / exchanges)
- analyze and report P&L through a risk-reward monitoring and reporting dashboard
This is not not a first iteration, as I already worked on tick-based backtesters in C and C++, and modified to my use cases third party libraries such as Quantopian's Zipline and TAlib.
For the sake of diversity (to trade not only Forex but Stocks / Indices / Commo / Crypto CFDs, Futures and/or Options) and to decouple EAs from a commercial software, the platform should be able to retrieve data and route orders from/to third party brokers / exchanges / liquidity providers through two types of "connectors":
- the "API connectors", wrapping HTTP or websocket, or FIX APIs (e.g. for IB / IG / OANDA / Binance)
- the "bridge connectors", wrapping tcp socket / named pipe interco to deal with platforms (i.e. MT4/MT5/NinjaTrader/ToS/TWS)
These will be developed one at a time using open source connectors available on github and public API documentation, and act as drivers.
Each broker / exchange elected here as execution candidates will be reviewed and get allocated development time.
Reliability / stability of these connectors is crucial to profitability.
The platform will be command-line based while I am still working on the front-end (which will be a progressive web-app)
The end goal is to make the platform accessible online, so you guys can track my performance.
Hardware stack:
- Computing: 4 * Dell Optiplex with i5 > 8th gen + SSD + 8Gb DDR4 > 2400Hz
- Switch: > 2.5Gbe switch with > 10 slots
- Network: 1Gb optical fiber
- Wiring: RJ45 Cat 5e cables (will change to Cat6 SFP+ DAC or OAC when possible and if required)
- Storage: NAS with > 4 bays for scalable storage + RAID chip for RAID5, serving NFS over ex4 or ZFS through 10Gbe network card, starting with 4 * 2Tb drives, scalable to 4 * 18Tb on 4 bays or more with bay extensions
Software stack:
- Virtualization and cluster management: Proxmox + K3S (light kubernetes)
- Back-end / simulation / execution: Golang
- Analytics / research: Python3 (Pandas / Scikit-learn Pytorch or TensorFlow + Keras)
- Front-End: TypeScript (with Vue 3 + TradingView Lightweight Charts + apache Echarts) / SCSS / HTML
- DBMS: PostgreSQL + TimescaleDB
- R&D: Python3 (Pandas / Scikit-learn Pytorch or TensorFlow + Keras)
Milestones:
- infrastructure setup: ...
- backtest first algo: ...
- execute first algo through MT4 bridge or any compatible Broker's HTTP/WS based API: ...
- visualize performances and monitor execution: ...
But why?
The platform aims at being future-proof, adaptative (all instruments / asset classes as underlyings), and broker agnostic.
Developing a custom platform for high reliability backtesting / simulation and fast execution takes time, which I am happily putting in.
As proof of work I will port two of my historical FF published EAs, Keops and Horizon Break to the platform, and share the sample code and performances here.
I personally have trust and performance issues with existing platforms such as Quantopian / Quantconnect and others, and am happy to build my own infra, suited to my own use. Many will prefer spending time on the strategy and not on the infrastructure, but experience taught me both matter much.
Not to say how frustrating are the backtesters / optimizers widely used such as MT4/MT5's ones.
But why this thread?
Such endeavor comes at the cost of loneliness and moments of doubt.
This thread will be my journal / showcase, and maybe keep me accountable for the state of the platform !
Willing to get involved?
Pm !
Just curious about the result and public availability?
Sub !
See you very soon
Edo
Edit: Typo
Dear FF,
It has now been since 2017 I have not been active on the forum, only watching from far.
I only made time to answer the most relevant emails I received on my passed projects (ADXB / Autop / Horizon break / Keops...).
There are many reasons for my stepping back, not to belabor it was mostly due to studies and other side hustles.
Under-capitalized, I could not afford to go live with high MCR / low margin strategies (such as the grid and multi-asset ones I put so much time researching).
Times have changed, I started here as curious amateur in 2014 but now coming back as software engineer for an institutional asset management firm, I am more aware of our capabilities as retail players and about the FX market microstructure.
About the project:
This platform will be an all-in-one simulation and execution software, being able to:
- fetch raw ticks/spreads/depths (L1/L2) from selected brokers/exchanges/liquidity providers on demand (through API adapters)
- generate ticks based on an instrument statistical behaviour (i.e. extracting EURUSD statistical features and volatility profile to simulate synthetic EURUSD ticks) using brownian motion or 1D perlin noise
- aggregate ticks / order books in custom time series (e.g. renko, ohlc and aggregated order book)
- expose custom hooks for developing algorithms: on-tick, on-time, on-open, on-close, on-profit, on-loss, on-vol
- allow multi-assets and multi-timeframe strategies to be tested and ran
- run trained ML models under the ONNX runtime (ability to import and use Pytorch / TensorFlow models)
- run high precision back or forward test using aforementioned live / passed or synthetic data, and real fees / spread data
- route orders to brokers for live trading with as little latency as possible (through API adapters)
- monitor execution through a dedicated dashboard (connected to all accounts / brokers / exchanges)
- analyze and report P&L through a risk-reward monitoring and reporting dashboard
This is not not a first iteration, as I already worked on tick-based backtesters in C and C++, and modified to my use cases third party libraries such as Quantopian's Zipline and TAlib.
For the sake of diversity (to trade not only Forex but Stocks / Indices / Commo / Crypto CFDs, Futures and/or Options) and to decouple EAs from a commercial software, the platform should be able to retrieve data and route orders from/to third party brokers / exchanges / liquidity providers through two types of "connectors":
- the "API connectors", wrapping HTTP or websocket, or FIX APIs (e.g. for IB / IG / OANDA / Binance)
- the "bridge connectors", wrapping tcp socket / named pipe interco to deal with platforms (i.e. MT4/MT5/NinjaTrader/ToS/TWS)
These will be developed one at a time using open source connectors available on github and public API documentation, and act as drivers.
Each broker / exchange elected here as execution candidates will be reviewed and get allocated development time.
Reliability / stability of these connectors is crucial to profitability.
The platform will be command-line based while I am still working on the front-end (which will be a progressive web-app)
The end goal is to make the platform accessible online, so you guys can track my performance.
Hardware stack:
- Computing: 4 * Dell Optiplex with i5 > 8th gen + SSD + 8Gb DDR4 > 2400Hz
- Switch: > 2.5Gbe switch with > 10 slots
- Network: 1Gb optical fiber
- Wiring: RJ45 Cat 5e cables (will change to Cat6 SFP+ DAC or OAC when possible and if required)
- Storage: NAS with > 4 bays for scalable storage + RAID chip for RAID5, serving NFS over ex4 or ZFS through 10Gbe network card, starting with 4 * 2Tb drives, scalable to 4 * 18Tb on 4 bays or more with bay extensions
Software stack:
- Virtualization and cluster management: Proxmox + K3S (light kubernetes)
- Back-end / simulation / execution: Golang
- Analytics / research: Python3 (Pandas / Scikit-learn Pytorch or TensorFlow + Keras)
- Front-End: TypeScript (with Vue 3 + TradingView Lightweight Charts + apache Echarts) / SCSS / HTML
- DBMS: PostgreSQL + TimescaleDB
- R&D: Python3 (Pandas / Scikit-learn Pytorch or TensorFlow + Keras)
Milestones:
- infrastructure setup: ...
- backtest first algo: ...
- execute first algo through MT4 bridge or any compatible Broker's HTTP/WS based API: ...
- visualize performances and monitor execution: ...
But why?
The platform aims at being future-proof, adaptative (all instruments / asset classes as underlyings), and broker agnostic.
Developing a custom platform for high reliability backtesting / simulation and fast execution takes time, which I am happily putting in.
As proof of work I will port two of my historical FF published EAs, Keops and Horizon Break to the platform, and share the sample code and performances here.
I personally have trust and performance issues with existing platforms such as Quantopian / Quantconnect and others, and am happy to build my own infra, suited to my own use. Many will prefer spending time on the strategy and not on the infrastructure, but experience taught me both matter much.
Not to say how frustrating are the backtesters / optimizers widely used such as MT4/MT5's ones.
But why this thread?
Such endeavor comes at the cost of loneliness and moments of doubt.
This thread will be my journal / showcase, and maybe keep me accountable for the state of the platform !
Willing to get involved?
Pm !
Just curious about the result and public availability?
Sub !
See you very soon
Edo
Edit: Typo
You'll always miss 100% of the shots you don't take.