General tech-talk

This is thread is for discussing general tech related algo-trading topics.

Hijacking this thread with MQL programming solicitations will get you banned!

Continuation from:
HI there again,

I am desperate to get out of the MT4/5 retail brokerage space and go the FIX route. I am zero'ng on Python to replace my EA automation so research to build a solution in parallel has begun. Is there a better alternative to Python? I came across this but haven't played around much.

I would also like some GUI frontend to make things easier. My order execution piece is pretty straightforward but the administrative piece (management functions like DD, margin, trade, RM, exposure etc) is quite advanced and complicated to transform that into a brand new language. I am looking for something that comes with a "building block" approach that I can plug and play. I am looking for a quicker alternative as I have no desire to spend next two years replacing mt4. I am looking for a multi-broker, multi-market/exchanges/DMA type solution....away from the likes of Oanda/Forex.com/retail servers.

EDIT: My automation is extremely CPU intensive... it goes exponential the more pairs are trade and goes even higher the more orders I have open. MT4 (demo) simply clocks when I reach the 200+ open trades

 

1. Post 18
2. Cleanup
3. Quote
4. Aug 22, 22:09 (2 hr 24 min ago)

1. https://cdn-assets.forexfactory.net/...r459520_11.gif masterx584
2. | Joined Apr 2016 | Status: LiquidatorSoft | 16 Posts | Invisible

Quoting VEEFX
{quote} HI there again, I am desperate to get out of the MT4/5 retail brokerage space and go the FIX route. I am zero'ng on Python to replace my EA automation so research to build a solution in parallel has begun. Is there a better alternative to Python? I came across this but haven't played around much. I would also like some GUI frontend to make things easier. My order execution piece is pretty straightforward but the administrative piece (management functions like DD, margin, trade, RM, exposure...
The entire MQ4 code can be converted to C++.On my system under C++ the CPU runs 0% with 350 threads at 1ms sleep time.MT4 is killing CPU's because using WinApi for draw functions not because of the math calculations and isn't even allowing multithreading.More colors text or draw on your charts more CPU needed.
Quantitative algorithmic to build smart data packets.
Liquidator Return This Week: 1.8% AUD/USD

 

1. Post 19
2. Cleanup
3. Quote
4. Aug 22, 22:16 (2 hr 17 min ago)

1. VEEFX
2. | Joined Jun 2006 | Status: Member | 1,713 Posts | Invisible

Quoting masterx584
{quote} The entire MQ4 code can be converted to C++.On my system under C++ the CPU runs 0% with 350 threads and 1ms sleep time.MT4 is killing CPU's because using WinApi for draw functions not because of the math calculations and isn't even allowing multithreading.More colors text or draw on your charts more CPU needed.
I agree. It's the single threading limitation... and I use no charts and no technical indicators so I know it is not my graphics adapter. I learnt C in the 80s in high school and then some more for my post-graduate project to develop a compiler for a language. Is there a mq4 to C++ convertor that I can play with?

EDIT: Its the 9 levels deep nested recursive logic which hogs my CPU on every tick
Attached Image
https://www.forexfactory.com/attachm...1&d=1534973088

 

1. Post 20
2. Cleanup
3. Quote
4. Aug 22, 22:47 (1 hr 46 min ago) | Edited Aug 22, 23:01

1. emmzett
2. | Joined Apr 2008 | Status: Member | 58 Posts

Quoting VEEFX
{quote} HI there again, I am desperate to get out of the MT4/5 retail brokerage space and go the FIX route...
OK, this is going now completely off-topic here but I'll try to give some general answers.

Language is irrelevant, simplicity and ROI are important (KISS). If you'r already familiar/professional with one language don't switch (as long as it's not VB or Excel). Plan and measure in *your* time, that's the resource you can control.

FIX is the standard for institutions but it's an unimportant detail for retail. Good to know but not important. You don't need FIX to become consistently profitable and you will not increase your chances to do so by using it. The opposite is true. What really matters is what stage you are on with your trading.

Becoming broker/platform independant is a natural wish, especially as MetaQuotes will cease to exist as we knew it. What you need for this depends on your strategy (if you have one). The options we have in the future in retail space are platforms like cTrader, Quantower etc. You can setup a broker independant MT4 for charting and market analysis if you want to, even if there's no MetaQuotes anymore. But if you don't need charting but only market scanning it will be completely different. May be you understand now that there can be no general answer as it really depends on your needs. Let's say we don't have a strategy and need a good charting module. OK, we might get used to MT5 for this (this is all MT5 is good for). Retail more than others needs to be able to adapt.

The real issue with retail automation is not finding a profitable strategy. The real issue is managing the infrastructure and potential infrastructure errors. A realistic backtest showing 7% return per month over 5 years will not work out as expected in a live account, as your account will crash once or twice a year because you have a network outage or made a stupid coding mistake. There is not a single code example on mql5.com or forexfactory.com with proper error handling. Think about this if you have experience in IT. On the other hand there is always room for unexpected error. Problem here: you can test your strategy but you cannot test in a backtester (whatever platform) for those unexpected errors.

So in summary the real issue is not what platform/language do I use. The real issue is how do I prepare my trading for the...let's call it Black Swan events. They will always come and they are the real threat. It might be your fault, it might be a 15th of January 2015. It might turn out that I better control such risk by manual trading (this is what institutions do). Manual trading also makes you independant of platform changes. But it requires completely different skills which a retail trader can aquire only in a very difficult way.

So focus on what you already can do and simplify that. Don't complicate it by introducing more abstractions (e.g. FIX). At the current moment in time there is no need for FIX in the retail space. I say "need"...

1

 

1. Post 21
2. Cleanup
3. Quote
4. Aug 22, 22:50 (1 hr 43 min ago)

1. https://cdn-assets.forexfactory.net/...r459520_11.gif masterx584
2. | Joined Apr 2016 | Status: LiquidatorSoft | 16 Posts | Invisible

Quoting VEEFX
{quote} I agree. It's the single threading limitation... and I use no charts and no technical indicators so I know it is not my graphics adapter. I learnt C in the 80s in high school and then some more for my post-graduate project to develop a compiler for a language. Is there a mq4 to C++ convertor that I can play with?
Only the hard way as i know.U need to add Winsock for connections , a draw engine for C++ environment ,to rewrite all the algorithms or to create new multithreaded ones.. to allocate memory etc. c++ have alot of elasticity and can compile assembly language inside ,i have a few functions in machine code for more speed.It's easy for me to speak now but i worked almost 2 years on the entire thing ...mostly on my multithreaded algorithms.Anyway if u start to build your C++ system i can help u with things like multithreaded clients,graphics ,how to build charts or how to transform functions like NormalizeDouble which is pain..u can almost create a copy of MT4 ...how to completely eliminate the netwotk lag.
Quantitative algorithmic to build smart data packets.
Liquidator Return This Week: 1.8% AUD/USD

 

1. Post 22
2. Cleanup
3. Quote
4. Aug 22, 22:50 (1 hr 42 min ago)

1. emmzett
2. | Joined Apr 2008 | Status: Member | 58 Posts

Quoting VEEFX
Its the 9 levels deep nested recursive logic which hogs my CPU on every tick
Never do this. A 9 level deep loop will kill every performance, even if it's written in Assembler.

 

1. Post 23
2. Cleanup
3. Quote
4. Aug 22, 22:56 (1 hr 36 min ago) | Edited Aug 22, 23:26

1. https://cdn-assets.forexfactory.net/...r459520_11.gif masterx584
2. | Joined Apr 2016 | Status: LiquidatorSoft | 16 Posts | Invisible

Quoting VEEFX
{quote} I agree. It's the single threading limitation... and I use no charts and no technical indicators so I know it is not my graphics adapter. I learnt C in the 80s in high school and then some more for my post-graduate project to develop a compiler for a language. Is there a mq4 to C++ convertor that I can play with? EDIT: Its the 9 levels deep nested recursive logic which hogs my CPU on every tick {image}
U have a 9 level big problem ...break it into multiple small problems ...go multithreading.
Attached Image (click to enlarge)
https://www.forexfactory.com/attachm...1&d=1534975667

Right now i am with only 123 threads ,8 clients..16 threads *client,used to connect 20 clients before and 340+ threads.I calculate tick by tick data no timed bars.My threads are setted to 1ms sleep.They keep pushing.The algorithm alone divided in multiple cells running through threads is 3000+ code lines totally.It's more about how u divide & conquer.
Quantitative algorithmic to build smart data packets.
Liquidator Return This Week: 1.8% AUD/USD

 

1. Post 24
2. Cleanup
3. Quote
4. Aug 22, 23:31 (1 hr 2 min ago)

1. emmzett
2. | Joined Apr 2008 | Status: Member | 58 Posts

OK, we already hijacked the thread. Sorry to the OP.

Key to becoming better/more independant/more successful is not improving our platform or an indicator or a strategy.

Key is awareness. Awareness *why* the natural consequence of our trading approach as a retail trader is for the vast majority failure. If we gain that awareness we are able to make the required changes. Without that awareness we are doomed. A platform or language will not help us.
Attached Image (click to enlarge)
https://www.forexfactory.com/attachm...1&d=1534976458


As Morad Askar says: The statistics about 99% of failing retail traders are correct. But it is a useless statistic. Online trading has no entry barriers, a 14 year old can open an account, upload the ID of his sister which lays around in her room all day long and fund the account with the credit card of his mother which lays around in the kitchen.

Take a chirurg with proper study and years of experience. Take 100 people you find randomly in the streets and give each one of them a scalpell. What percentage of the 100 people do you expect to become successfull chirurgs?

None. 100 percent failure. Is this statistic correct? Yes. Does it correctly depict your chances to become a successfull chirurg? Of course not.

It's the same with trading. Awareness is key.
Inserted Video

 

1. Post 25
2. Cleanup
3. Quote
4. Edit
5. Aug 22, 23:47 (45 min ago)

1. https://cdn-assets.forexfactory.net/...atar2942_7.gif Nicholishen
2. Joined Jul 2005 | Status: Member | 997 Posts | Invisible

Quoting masterx584
{quote} U have a 9 level big problem ...break it into multiple small problems ...go multithreading. {image} Right now i am with only 123 threads ,8 clients..16 threads *client,used to connect 20 clients before and 340+ threads.I calculate tick by tick data no timed bars.My threads are setted to 1ms sleep.They keep pushing.The algorithm alone divided in multiple cells running through threads is 3000+ code lines totally.It's more about how u divide & conquer.
Multithreading aids in IO bound, flat, and asynchronous operation. For CPU intensive tasks, multiprocessing is required. If the algorithm (actually) requires nested loops then neither would be useful since the each inner loop is interdependent on the iteration outcome of its outer loop. I've rarely ran up against a problem that couldn't be flattened, but I won't presume to know the logic in question, and assume that OP knows what he/she is doing.

 

1. Post 26
2. Cleanup
3. Quote
4. Aug 22, 23:48 (45 min ago)

1. https://cdn-assets.forexfactory.net/...r459520_11.gif masterx584
2. | Joined Apr 2016 | Status: LiquidatorSoft | 16 Posts | Invisible

Quoting emmzett
OK, we already hijacked the thread. Sorry to the OP. Key to becoming better/more independant/more successful is not improving our platform or an indicator or a strategy. Key is awareness. Awareness *why* the natural consequence of our trading approach as a retail trader is for the vast majority failure. If we gain that awareness we are able to make the required changes. Without that awareness we are doomed. A platform or language will not help us. {image} As Morad Askar says: The statistics about 99% of failing retail traders are correct. But it...
hahah, u think i was so stupid to loose 2 years to build and put my algos into a completely isolated and secured network if they weren't valid?..away from any broker spy plugins ... with OP hijacked most of the services disabled completely invisible against Kali Linux or Parrot Security tested in the same network...i think who will work harder will be more succesfull in life.Who want to take it professionally need to cut brokers from this ecuation with all the services they offering and all the preparation or webinars...What u put is what u get...as allways.. and i am able to take trades without never watching the Bid...only the spread.Peace .
Quantitative algorithmic to build smart data packets.
Liquidator Return This Week: 1.8% AUD/USD

 

1. Post 27
2. Cleanup
3. Quote
4. Aug 23, 0:03 (29 min ago)

1. VEEFX
2. | Joined Jun 2006 | Status: Member | 1,713 Posts | Invisible

Quoting emmzett
{quote} Never do this. A 9 level deep loop will kill every performance, even if it's written in Assembler.
Thanks emmzett and masterx for the responses. I really appreciate it. Level 9 deep is only on one pair..... the depth also exponentially increases depending on when my signal confluence happens which triggers a recursive crawl if you know what I mean but I have capped it at 32 levels. It's is designed as a sequential crawl and cannot be done via parallel processing so to mimic multi-thread, I run it across several terminals plus one terminal to handle all portfolio/trade mgmt functions but still my CPU clocks continuously non stop at 50%-60%. Ultimately, I need to use Spot to trade regulated futures, indices, sub-sector ETFs etc.... any market that is a derivative of underlying instruments as this model is quite expandable to other markets (because it is purely data driven and contained within the weekly candle so not a whole lot of volume of data involved) so yeah... i do have a strategy. I just do not want reliance on a broker-provided platform. Order Execution Speed/latency/slippage etc is not my concern at all as they have no or minimum impact on my overall outcome. What's important is the timely processing (computations) to complete within one minute for the signal to remain valid.

 

1. Post 28
2. Cleanup
3. Quote
4. Aug 23, 0:09 (24 min ago)

1. zosoforever
2. | Joined Sep 2010 | Status: Member | 55 Posts | Invisible

Hi Guys,
This must be very interesting conversation, but it is not related to FIX API.
It would be more beneficiary for you to start your own thread. Also for us who want to discuss FIX API.
Thanks

 

1. Post 29
2. Cleanup
3. Quote
4. Aug 23, 0:14 (19 min ago) | Edited Aug 23, 0:26

1. https://cdn-assets.forexfactory.net/...r459520_11.gif masterx584
2. | Joined Apr 2016 | Status: LiquidatorSoft | 16 Posts | Invisible

Quoting VEEFX
{quote} Thanks emmzett and masterx for the responses. I really appreciate it. Level 9 deep is only on one pair..... the depth also exponentially increases depending on when my signal confluence happens which triggers a recursive crawl if you know what I mean but I have capped it at 32 levels. It's is designed as a sequential crawl and cannot be done via parallel processing so to mimic multi-thread, I run it across several terminals plus one terminal to handle all portfolio/trade mgmt functions but still my CPU clocks continuously non stop at 50%-60%....
My initial computation takes 30 sec too but i am able to pack that data in a way that allows me to save it on HDD & to reutilize it with the new arrived ticks as it is without any other computations.I am only making ticks based calculations no timeframed bars.I create my own "smart packets" that can be utilised as it is.I pack and create my own history ... don't know the mkt price for none of my trading pairs... am only based on the market value *strength, no price action,high lows or whatever.I don't care what price it has but only if the market is into a value*strength accumulation range or is going to explode after being ranging for a while.I can wait 2-3 weeks to take a pair outside of the range going to replace a top trending one who's weakening and ready to get into accumulation.Is allways an exchange.
Quantitative algorithmic to build smart data packets.
Liquidator Return This Week: 1.8% AUD/USD

1

 

1. Post 30
2. Cleanup
3. Quote
4. Last Post: Aug 23, 0:30 (3 min ago)

1. https://cdn-assets.forexfactory.net/...r459520_11.gif masterx584
2. | Joined Apr 2016 | Status: LiquidatorSoft | 16 Posts | Invisible

Quoting zosoforever
Hi Guys, This must be very interesting conversation, but it is not related to FIX API. It would be more beneficiary for you to start your own thread. Also for us who want to discuss FIX API. Thanks
That's true.I'm very sorry.I'm done.See u guys.

Subscribed
"I am desperate to get out of the MT4/5 retail brokerage space and go the FIX route. I am zero'ng on Python to replace my EA automation so research to build a solution in parallel has begun. Is there a better alternative to Python? I came across this but haven't played around much."
I got out ..it's a totally diff world.Escaped from MT4Alcatraz.

Me too...but I am ex-geek and a old dawg who left systems engineering about two decades ago to go on the business side of the equation :-

Regarding your last post on stoding data set on HDD, I went that route too but disk IO was killing me more and I could not risk missing new ticks or getting data stale...plus my EA can run on one pair one timeframe if I really want to spit out the overhead but I still like a single EA approach to free up my time without worrying about multiple instances.

Could you expand on what you mean by 'market value"

In my vision the entire mkt stocks,currencies etc. ,all the mkt participants have something in common ==>USD .This lead me to build a system where i can give a certain level of value *strength to any mkt participant. I can compare for example Apple or Amazon against NZD or CAD even if apparently they have nothing in common.I almost can smell the buying selling accumulation pressure inside my "value accumulation period"... am focusing right now on the currencies exclusively. I am bridging all the mkt participants through USD and i bet most of the Big Players algos do the same thing...after all nobody wants to be on the wrong side.I completely ignore news or quotes, i never watch charts or fibo etc..I notice the Bid_Ask quotes only when taking a trade ...i completely ignore them.My goal was to reduce the entire info i need to know in order to take a trade to a SINGLE NUMBER and to PACK it aside other brother$ & $ister$--->'S.M.A.R.T PACKETS'

Interesting concept. USD is indeed King. I could ask more questions on how you measure value and relative strength but OP will probably want to focus on tech talk.

Did you mean range instead of period in "value accumulation period" ?

Value*strength range at micro micro level.Just close to JP Morgan bots.Every fire need to start from somewhere.I'm looking to find that spot + the direction of the wind.All the info is tick by tick based,encoded.. but we do our best.

Packing on the HDD from time to time.All the data is loaded in memory.I run multiple buffers with 100k ++ packed elements and i'm not talking about mkt ticks but allready elaborated data.When i restarting the software i reupload the buffers from the HDD.Am live 24*5.

Elaborated data for the past 45 ticks range/duration/scale as shown in your screenshot? I think I understand. What's your threshold for directional trigger ?

What u saw on my screen are 15-17 days of data.The red rectangles around GBP packets from 13 to 40 represent 500++ pips drop.I'm packing 2-3 cells daily ,sometime 5 if the mkt moves...all the mkts on my screen are perfectly synchronized. With the right arrow key i go back in history, upto 1000 packets .Every packet have 100 'ticks'-->mkt value*strength etc etc..Red or blue rectangles with the number 100 stamped on the cell represent the max of the max power a mkt can have in uptrend or downtrend.Anyway, inside these cells i also measure micro trends. The 0 cells are inside the accumualtion range.Inside the accumulation we also have uptrends and dwntrends.But they are smaller and less profitable,is the place where most traders let theyr money.I don't trade this area , i wait for the pressure to build and more burnt money ==more pressure ==> lead to more explosion on trend days. "If u can't measure it u can't trade it". The threshold is +1 -1 ,it can go to 4.5 -5 on gold or GBP pairs where is the most volatility.When the red rectangles or blue are over ...i'm out with a nice catchAttachment G

{ copyrighted attachment deleted by staff }

"I am desperate to get out of the MT4/5 retail brokerage space and go the FIX route. I am zero'ng on Python to replace my EA automation so research to build a solution in parallel has begun. Is there a better alternative to Python? I came across this but haven't played around much."

I will make you a simple question. Are you in this business for earn money or for burn your life time?

This question doesn't translate well to English. What is this suppose to mean?

Just if spend a very long time in that kind of project compensate you. If you want earn money trading don't seem the best option

LOL

I think it's a problem with the translation. I think he means to say that if you want to make money in trading, then porting your strategy over to a new language isn't worth the time investment.

Something like that...

Hi braintheboss,
I am very interested for FIX. Did you mean that it takes too much time to setup everything for FIX trading?
Thx

Complex projects are expensive in time. But i wonder if his strategy works ( the current tools he is using ) then really he needs replace a full platform?

I see, you are talking about VEEFX's strategy, not overall for FIX trading.

About OP thread topic. Not idea about VEEFX. I never read his posts...