FIX API | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

How do I enable my MT4 EA to trade using a FIX API? 30 replies

FIX API Traders 1 reply

Need a NET Programmer with FIX API experience 8 replies

Need some advice/information or even a developer for FIX API 7 replies

Hire a Coder. C++, TCP, FIX API 0 replies

Attachments: FIX API
Exit Attachments

FIX API

  • #1
  • First Post: Aug 16, 2018 1:39pm Aug 16, 2018 1:39pm
Hi Everybody,
from time to time I am being asked about FIX API as I have dropped some posts about it here on FF.
I started to learn MQL and faced the necessity to become trading platform independent by migrating to FIX API.
That was quiet difficult as I did not study IT and was not able to code any language but MQL.
There is a lot of stuff about FIX API and great help for any problem you simply do not understand when the first steps are missing.
These first steps were the biggest barrier for me and it took me very long to make them. And without the help of skillful coders I would not have been able to solve it.
I am going to help everybody who is clueless how to make the next step to code his own FIX API engine. For sure I need free time for this so if there are too many questions it might take a while. But tbh I do not believe that this will be the case.
I am simply getting started by posting the last PM I got about it and my recent answer.
  • #2
  • Aug 16, 2018 1:40pm Aug 16, 2018 1:40pm
Hello. I noticed that you trade using FIX. I am just starting with that, trying to make FIX engine. Which language are you using for FIX engine? C++, Java? Thanks

Hi,
I coded my first FIX bot in Java. Then did it again in C++ to increase speed. Now I rolled it up from scratch in Java including all my experience I made with the first attempts and I can say that Java is suitable for me. You can come down very low level too so there is no big speed difference compared to C++ and I can make changes quickly. And that is really important. Even more than saving another 5 Microseconds to compose an order message or something like that.

Best regards,
Jan
  • #3
  • Aug 16, 2018 1:40pm Aug 16, 2018 1:40pm
Thank you for the answer.
This is a good news for me because I've decided as well to use Java because it is easier to learn and code.
What is in your opinion a good enough execution? Is 10 milliseconds to slow?
Thanks

From what point of time do you start to measure execution? When the order has arrived at the brokers trade server? If so everything about 10 ms is pretty good and you will not find that very often. Almost all brokers performing last look or using speed bumps to delay execution.
  • #4
  • Edited 2:08pm Aug 16, 2018 1:52pm | Edited 2:08pm
Hello,
I meant time from placing order to the broker's server.
With MT4 the best I've achieved was:
Ping less than 2 ms (brokers and VPS in London), MT4 system 30 ms, and system (processor, memory) 5-90 ms. In average I've had total of 50 ms.
I hope with FIX (but don't know):
to still have ping 2 ms (with dedicated server and cross-connection 0.5 ms). Using FIX to eliminate MT4 and 30 ms loss. And probably my app and server would take less than 10 ms. I thought that I would be able to have order executed within 10 ms after my software calculates opportunity. That's why I was asking if 10 ms is too slow. is it realistic or we can go faster? Do we need faster? What is your experience and thought about that?
Thank you for the help.


Hi,

You have the following latencies:

1. Trade server to TCP stack on your server
2. TCP stack to your algo
3. Parsing the incoming message within your algo
4. Trade logic
5. Composing the order message
6. Sending the order message through your TCP stack
7. TCP stack to the trade server

With my current version my latencies on a 2 core VPS with Linux Cent OS for about 60 € per month are:
1. With a cross connection you can come down to 0.2 milliseconds if you believe what hosters are saying. You cannot really measure it by yourself as pinging trade servers mostly is not working. They do not answer.

2. I donīt really know. If you rent a dedicated server with a solaflare card you can do kernel bypassing and save like 10 to 20 micro seconds. If someone knows it would be interesting

3. I spend months on my parser. If you do not use inbuild Java functions you can become very fast here. Also only parse what you really need to parse. I have a latency about 5 to 10 microseconds here. But I also do a lot of stuff within the parser that is more efficiently done directly when parsing a message so it does not need to be done later e.g. within the trade logic.

4. Like 5 to 20 microseconds. Depends on your logic.

5. Composing takes also about 5 microseconds.

6. Writing the composed message into the TCP stack takes about 15 to 50 microseconds. I cannot say how long it then takes until the message is on its way to the trade server

7. same as 1

Compared to the duration the connection between the trade server and our server or VPS (0.2 milliseconds per direction) the latency of the algo can be reduced very low.

Edit:
I measured my latencies when running my algo live on my VPS. There are some vendors of FIX API engines that claim to be super fast and lowest latency possible. What they do is to make measurements under conditions that are never met in live conditions. E.g. they are parsing a message 100 thousand times, measure the time it took and divide the time by 100,000. When I do this I need like 900 nanoseconds (0.9 microseconds) simply because a procedure that is repeated so frequently stays in the upper memory and / or gets such a high priority from the OS that is is progressed that fast.
When running your algo live there are many competing procedures and steps that are being performed so live latencies differ heavily from those micro benchmarks that are only good for making someone buy something...
  • #5
  • Aug 16, 2018 8:04pm Aug 16, 2018 8:04pm
Quoting JT4xPro
Hello, I meant time from placing order to the broker's server. With MT4 the best I've achieved was: Ping less than 2 ms (brokers and VPS in London), MT4 system 30 ms, and system (processor, memory) 5-90 ms. In average I've had total of 50 ms. I hope with FIX (but don't know): to still have ping 2 ms (with dedicated server and cross-connection 0.5 ms). Using FIX to eliminate MT4 and 30 ms loss. And probably my app and server would take less than 10 ms. I thought that I would be able to have order executed within 10 ms after my software calculates...
Hi JT4xPro,
I am glad that you started this thread.
It looks like you managed to achieve total speed of less than 1 millisecond!
Is Linux Cent OS significantly faster than Windows? Does it make much difference on such quick execution?
Regards
  • #6
  • Aug 17, 2018 2:07am Aug 17, 2018 2:07am
Quoting zosoforever
{quote} Hi JT4xPro, I am glad that you started this thread. It looks like you managed to achieve total speed of less than 1 millisecond! Is Linux Cent OS significantly faster than Windows? Does it make much difference on such quick execution? Regards
Cent OS is a lightweight compared to windows. It does not need so much resources and it runs less tasks in the background. So you have less disturbances for your algo what means it will run more consistent.
Something you can do is to pin the OS to one core and you algo to another one (Probably at windows too)
I am also not using any GUI. It is just the black command prompt we also know from windows.
There are some pretty awesome free tools to manage the VPS very handy without GUI. You need WinSCP for up and downloads and putty to run your algo on your VPS. Both are for free.

But you can also use windows. Especially in the beginning to keep things simple.
  • #7
  • Aug 17, 2018 6:32am Aug 17, 2018 6:32am
Did yo connect your algo to Metatrader?
  • #8
  • Edited 5:50am Aug 20, 2018 1:35am | Edited 5:50am
Quoting zosoforever
Did yo connect your algo to Metatrader?
I have a bridge from FIX API to MT4 but found no use for it that makes sense.
  • #9
  • Aug 21, 2018 7:42am Aug 21, 2018 7:42am
Quoting JT4xPro
{quote} I have a bridge from FIX API to MT4 but found no use for it that makes sense.
Do you mind if I ask you, what kind of bridge is that? Some DLL?
Thanks
  • #10
  • Aug 21, 2018 2:58pm Aug 21, 2018 2:58pm
Quoting masterx584
{quote} TCPAckFrequency = 1 & TCPNoDelay =1 on x64 Win7 machine ------> 5 ms to my broker & 0 ms on my lan network.VPS is a myth.My broker is in London but my orders are running into a Switzerland much closer server.Our orders never go into our broker server but into diff servers around the world he rented and he run his software on. ...."Using FIX to eliminate MT4 and 30 ms loss" ----> must be a joke ...eliminate useless laggy indicators from your charts an u will be just fine.MT4 gives u 0 latency.Set correctly your OP & router.Good luck....
Hello masterx584,

Let me disagree with you. MT4 does slow down execution. Even MQL5 will admit that their Metatrader will slow down execution, mainly on broker's servers and bridge. And this latency is on top of EAs and Indicators. I estimate latency due to MT4 to 25 ms, that's what I am getting after eliminating MT4. I did extensive testing on 4 VPSs and I found out that bigger problem (latency) was caused by VPS provider's older infrastructure, overload etc. I saved more time by selecting a good VPS than location.

I don't know what do you mean by VPS is a myth. Unless you live in the same city as your broker's servers, VPS is necessity for news, arb, HFT traders.

Also a broker doesn't have servers all over the world. Servers are located in one of a few big data centers like Equinox in London.
  • #11
  • Aug 21, 2018 5:11pm Aug 21, 2018 5:11pm
Quoting masterx584
{quote} U speak like a bilionaire hedge fund manager looking to gain 1ms against other mkt participants ...but (no ofense) u send HFT nanoseconds orders through a retail platform to a retail broker and u also use very laggy indicators that kill your pc with too much nonsense and useless calculations that BTW have only a bunch of code lines ...instead of building your own proprietary software and to trade 1:1 in the real market ...u maybe know institutions doesn't have any deals with retail brokers ...your speed is useless inside a closed network.If...
Are you talking to me?
  • #12
  • Aug 21, 2018 5:23pm Aug 21, 2018 5:23pm
Quoting masterx584
{quote} U speak like a bilionaire hedge fund manager looking to gain 1ms against other mkt participants ...but (no ofense) u send HFT nanoseconds orders through a retail platform to a retail broker and u also use very laggy indicators that kill your pc with too much nonsense and useless calculations that BTW have only a bunch of code lines ...instead of building your own proprietary software and to trade 1:1 in the real market ...u maybe know institutions doesn't have any deals with retail brokers ...your speed is useless inside a closed network.If...
Even with an empty ontick() function and no indicators, MT4 has about 8-10ms lag after each tick due to all the behind the scenes processing going on. i.e updating all the built in variables, shifting arrays, etc. I built an asynchronous order system for MT4 and had to use offline charts to run my workers on because of the ontick delay. Even if you had a perfectly efficient front-end the true bottleneck is the MT pipeline. Client->MT4-Server->Process incoming aggregate orders->Send message to FIX bridge->LP. I've been on a colocated VPS to the MT trade server and I still never saw round-trip trade times under 250ms. What's even worse is not knowing your place in the queue when the SHTF, and you're also at the mercy of where your MT broker sets their slippage params. Trading via FIX isn't always about participating in the submillisecond market microstructure, rather, it's more about participating in the real liquidity pool - and not some retail construct designed to protect and enrich brokers.
  • #13
  • Aug 22, 2018 12:22am Aug 22, 2018 12:22am
Quoting zosoforever
{quote} Do you mind if I ask you, what kind of bridge is that? Some DLL? Thanks
I used
- memory maps via winapi in mql
- sockets with a dll
- sockets with a library (think it was this one: https://www.mql5.com/en/blogs/post/706665)
- sharing a file
- named pipes

I would recommend to take a look at sockets with library as it worked pretty easy and well (files did as well but only on the same computer)
  • #14
  • Aug 22, 2018 1:06am Aug 22, 2018 1:06am
Quoting JT4xPro
{quote} I used - memory maps via winapi in mql - sockets with a dll - sockets with a library (think it was this one: https://www.mql5.com/en/blogs/post/706665) - sharing a file - named pipes I would recommend to take a look at sockets with library as it worked pretty easy and well (files did as well but only on the same computer)
How is this related to a FIX bridge? After this response I have serious doubts you know what you are talking about.
It takes blood, sweat and enough tears to wash away the blood.
  • #15
  • Aug 22, 2018 3:44am Aug 22, 2018 3:44am
Quoting emmzett
{quote} How is this related to a FIX bridge? After this response I have serious doubts you know what you are talking about.
This is related to a bridge between my FIX API trading algorithm (app 1) and the MT4 Terminal (app 2).

To let both apps communicate with each other I was e.g. using sockets. So if the trade logic of app 1 decided to send an order to app 2 it did this via the established socket connection between both apps. App 2 then sent the order to the MT4 broker.
All other kinds of information exchange between both apps were progressed on the same way (quotes, order status, history and so on)

Instead of sockets I also used the other mentioned methods to let both apps communicate with each other.

Maybe you understood something different under the term 'FIX bridge' than I did?
  • #16
  • Edited 3:55pm Aug 22, 2018 2:20pm | Edited 3:55pm
I was right. You don't know what you are talking about. FIX is a protocol between an application and a (typically) institutional execution system (it may be a datafeed too). You claim your algorythm is a FIX trading algorythm which is an oxymoron. Your execution is on MT4, so exactly NOT FIX. In your post I quoted you describe mechanisms for inter-process communication and nothing related to FIX.

Maybe you look for some freely available information before you mis-lead people. A good start is here:
https://en.wikipedia.org/wiki/Financ...ation_eXchange

More free materials and tutorials:
https://www.fixtrading.org/beginners-resources/

FIX open-source version:
http://www.quickfixengine.org/

An example FIX message looks like this:
Inserted Code
8=FIX.4.2 | 9=176 | 35=8 | 49=PHLX | 56=PERS | 52=20071123-05:30:00.000 | 11=ATOMNOCCC9990900 | 20=3 | 150=E | 39=E | 55=MSFT | 167=CS | 54=1 | 38=15 | 40=2 | 44=15 | 58=PHLX EQUITY TESTING | 59=0 | 47=C | 32=0 | 31=0 | 151=15 | 14=0 | 6=0 | 10=128 |

Good luck

PS: If you really want to play with a free FIX implementation (maybe to connect your strategy to) you may try the free FIX implementation endpoint Spotware offers with cTrader. Usually you have to pass a compliance test before you can put an application using the FIX protocol into production. Spotware is an exception. You will quickly realize it has nothing to do with anything you did here.
It takes blood, sweat and enough tears to wash away the blood.
  • #17
  • Edited 5:01pm Aug 22, 2018 4:47pm | Edited 5:01pm
Quoting VEEFX
{quote} HI there again, I am desperate to get out of the MT4/5 retail brokerage space and go the FIX route...
OK, this is going now completely off-topic here but I'll try to give some general answers.

Language is irrelevant, simplicity and ROI are important (KISS). If you'r already familiar/professional with one language don't switch (as long as it's not VB or Excel). Plan and measure in *your* time, that's the resource you can control.

FIX is the standard for institutions but it's an unimportant detail for retail. Good to know but not important. You don't need FIX to become consistently profitable and you will not increase your chances to do so by using it. The opposite is true. What really matters is what stage you are on with your trading.

Becoming broker/platform independant is a natural wish, especially as MetaQuotes will cease to exist as we knew it. What you need for this depends on your strategy (if you have one). The options we have in the future in retail space are platforms like cTrader, Quantower etc. You can setup a broker independant MT4 for charting and market analysis if you want to, even if there's no MetaQuotes anymore. But if you don't need charting but only market scanning it will be completely different. May be you understand now that there can be no general answer as it really depends on your needs. Let's say we don't have a strategy and need a good charting module. OK, we might get used to MT5 for this (this is all MT5 is good for). Retail more than others needs to be able to adapt.

The real issue with retail automation is not finding a profitable strategy. The real issue is managing the infrastructure and potential infrastructure errors. A realistic backtest showing 7% return per month over 5 years will not work out as expected in a live account, as your account will crash once or twice a year because you have a network outage or made a stupid coding mistake. There is not a single code example on mql5.com or forexfactory.com with proper error handling. Think about this if you have experience in IT. On the other hand there is always room for unexpected error. Problem here: you can test your strategy but you cannot test in a backtester (whatever platform) for those unexpected errors.

So in summary the real issue is not what platform/language do I use. The real issue is how do I prepare my trading for the...let's call it Black Swan events. They will always come and they are the real threat. It might be your fault, it might be a 15th of January 2015. It might turn out that I better control such risk by manual trading (this is what institutions do). Manual trading also makes you independant of platform changes. But it requires completely different skills which a retail trader can aquire only in a very difficult way.

So focus on what you already can do and simplify that. Don't complicate it by introducing more abstractions (e.g. FIX). At the current moment in time there is no need for FIX in the retail space. I say "need"...
It takes blood, sweat and enough tears to wash away the blood.
  • #18
  • Aug 22, 2018 4:50pm Aug 22, 2018 4:50pm
Quoting VEEFX
{quote} I agree. It's the single threading limitation... and I use no charts and no technical indicators so I know it is not my graphics adapter. I learnt C in the 80s in high school and then some more for my post-graduate project to develop a compiler for a language. Is there a mq4 to C++ convertor that I can play with?
Only the hard way as i know.U need to add Winsock for connections , a draw engine for C++ environment ,to rewrite all the algorithms or to create new multithreaded ones.. to allocate memory etc. c++ have alot of elasticity and can compile assembly language inside ,i have a few functions in machine code for more speed.It's easy for me to speak now but i worked almost 2 years on the entire thing ...mostly on my multithreaded algorithms.Anyway if u start to build your C++ system i can help u with things like multithreaded clients,graphics ,how to build charts or how to transform functions like NormalizeDouble which is pain..u can almost create a copy of MT4 ...how to completely eliminate the netwotk lag.
It's show time
  • #19
  • Aug 22, 2018 4:50pm Aug 22, 2018 4:50pm
Quoting VEEFX
Its the 9 levels deep nested recursive logic which hogs my CPU on every tick
Never do this. A 9 level deep loop will kill every performance, even if it's written in Assembler.
It takes blood, sweat and enough tears to wash away the blood.
  • #20
  • Aug 22, 2018 5:31pm Aug 22, 2018 5:31pm
OK, we already hijacked the thread. Sorry to the OP.

Key to becoming better/more independant/more successful is not improving our platform or an indicator or a strategy.

Key is awareness. Awareness *why* the natural consequence of our trading approach as a retail trader is for the vast majority failure. If we gain that awareness we are able to make the required changes. Without that awareness we are doomed. A platform or language will not help us.

Attached Image (click to enlarge)
Click to Enlarge Name: ft71.png Size: 951 KB


As Morad Askar says: The statistics about 99% of failing retail traders are correct. But it is a useless statistic. Online trading has no entry barriers, a 14 year old can open an account, upload the ID of his sister which lays around in her room all day long and fund the account with the credit card of his mother which lays around in the kitchen.

Take a chirurg with proper study and years of experience. Take 100 people you find randomly in the streets and give each one of them a scalpell. What percentage of the 100 people do you expect to become successfull chirurgs?

None. 100 percent failure. Is this statistic correct? Yes. Does it correctly depict your chances to become a successfull chirurg? Of course not.

It's the same with trading. Awareness is key.

Inserted Video
It takes blood, sweat and enough tears to wash away the blood.
1 trader viewing now