MT4 mirroring & a Disaster Recovery Plan

I’m thinking of a Disaster Recovery Plan (DRP) for any post-crash open trades.
Say I have an MT4 handling a handful of open trades – and the “system” crashes (MT4 itself, the hardware, the network, the power falls on the VPS farm - you name it).

Now, assuming it can take quite sometime until the system comes back, an alternative “system” needs to pick up the open trades and manage them instead of the original one. I’m talking mirroring of two client terminals – located in separate geographical locations.

I wonder has anybody had any experience in mirroring client terminals (essentially MT4 - but it can be any platform).

The part that I’m missing is a ping utility where a “passive” MT4 client routinely pings a Master MT4 client to test it is live and whenever it loses connection it assumes the Master responsibility, or something like that.

Any experience, thoughts or ideas would be greatly appreciated.

Pdat100

Good idea. You probably successful if you care about your trades so much...lol...
As I know there is no connection between MT4 platforms. MT5 platforms do.
What I would do is to keep sending data to flash drive, so when a computer is crushed,
you stick to another computer and it resumes all the going.
Am curious to hear any more ideas.

I have 5 servers colocated throughout the US.

All the servers have a heartbeat to each other. They can detect when one dies. If a server dies for any reason, the other 4 try to wake it up with Wake On LAN commands, and I get a text message. If only 1 server is left standing, all trades are closed and I get a phone call with a TTS-message telling me the error code.

In live trading, I've only had 2 servers offline simultaneously due to some sort of failure (broken network switch and power outage).

COOL RR!

Can you elaborate on the heartbeat mechanism? Do you also monitor the trading software or only the hardware platform?

Just the hardware.

It's simple really. Each server is a node as part of an extended mesh network. All servers ping each other every 30 seconds. If one server doesn't respond for 10 minutes, it's declared 'dead' and resuscitation attempts begin. While the other servers are resuscitating the dead server, another server's MT4 terminals are activated, and trading resumes.

I have a batch script running on each server pinging its siblings. It's scheduled as a windows service.

Recorded mouse gestures manipulate the MT4 terminals as needed. The week's EA pre-set files are distributed via DropBox, so anything I put on my main trading computer is propagated to the 'slave' computers. EAs, indicators, scripts, and presets are updated every weekend. And if there's a major code change needed, I can force updates.

Each box is a 4U rackmount ATX monster running standard consumer components. All the operating systems are virtualized so each physical box is supporting the trading platform as well as 'snapshots' of that specific node so I can SSH into any physical computer and revert back to an older snapshot if necessary.

There's far more to my trading operation than a simple coin toss =)

Very nice, and well thought out.
Is there an elegant way to analyze the pinging batch script results? Better yet, is there a utility (dll?) that does that for you?

It's a simple command...


It then sends 10 ping requests, and checks how many are returned. If all 10 are returned, then all clear. If all 10 are not returned, then set status to 'dead'

I know that much :-)...

My question was how do you analyze the text results of the ping command? Anything simple?

RR... that's pretty kick @ss. i was thinking of doing something similar.

It's a boolean value.

Quoting Ronald Raygun

Disliked

It's a boolean value.

Ignored

Cool, RR you are absolutely correct. It is indeed simpler than I thought. Seems these three batch lines do the trick:

set state=alive
ping -n 1 %1
if errorlevel 1 set state=dead

Yet, it still leaves uncovered an MT4 failure (hardware / network continues to function but the master client terminal is dead).

I believe running an availability test between the two end points (the EAs running on the MT4 clients) would cover the chain end-2-end. For this to happen there needs to be some direct connection between the EAs via an external function call/dll.

Anyone?

Not true.

Each hardware node monitors its own terminal. I wrote the EAs and the terminal management so there is no need for the terminals to communicate with each other.

I suppose you could try to coordinate the terminals so orders are broken down through them, e.g. you want to send a 10 lot order to the broker, so you have server A send a 5 lot order and server B send a 2 lot order and server C send a 3 lot order. But the time you spend having servers coordinate each other and sending orders offsets any benefits.

RR that sounds cool!

So, what are you using to monitor the terminals? Are these the "recorded mouse gestures" you mentioned previusly or a third party software?

Recorded mouse gestures is a 3rd party program which runs independently of the batch scripts.

http://www.winautomation.com

Just about any mouse recorder with image recognition will work. Set it to look for the "Common Error" "No Connection" etc pictures.

I should probably note, this setup is nowhere near enterprise-quality. I took a google-style approach where I use relatively inexpensive hardware and made it as redundant as possible. I expect my servers to fail, and I build my infrastructure with that assumption in mind.

Impressive, RR!



The architecture I have in mind is very similar with the exceptions that: a) I do not like to monitor the terminal / EA from “outside” (via recorded mouse gestures.) I prefer each Client Terminal / EA (both the Master and the Backup) to routinely flag (each n seconds) their status via time stamping a local file (ideally it can write to the file a complete status report but it is not a must). And b) I'd like a single piece of software to monitor and sync the whole thing - installed on both computers (master and backup), monitoring:

1. The local terminal is up and running (via the flag file timestamp and potentially content).
2. The other computer well being (via pinging it).

FWIW, I have reviewed a few alternatives, I found that ServersALive and PRTG Network Monitor seem to provide a comprehensive solution for the monitoring and synch/reaction needs. PRTG Network Monitor seem to be stronger on the reaction side (including being able to send eMail, SMS, Network Broadcast, SNMP Trap, ICQ, MSN, as well as execute an executable and an HTTP request.)

Thanks much for sharing your architecture, RR.

Well, it's time to implement..


Cheers, Pdat100

I used to do this, but the overhead required on the metatrader terminal itself I found was hampering my trading. My solution to this was to use an external program which uses different thread from metatrader so the metatrader thread can spend more time actually processing trade data. I used to have metatrader terminals reporting to webservers running in a VM in each physical machine. I'm not sure how exactly you plan to implement your version, but keep in mind the overhead that you put on the actual metatrader terminal.

RR, et. al,

you guys are way smarter than me. I too would like to have a CPDR process for my trading. However, I am wondering if there is another way for me (the non-techie) to have CPDR.

I would like to make sure that I have a continuity and disaster recovery plan for my MT4 trade station. I am running a robot and would like to have multiple locations using different power and broadband providers. I would like to have each server running mt4 on the same broker and same account number live. In the event of a server crash or power/broadband outage at location #1, I want location #2's server to take the lead without my robot skipping a beat. This will give me greater assurance that the robot will run but my ea's at both locations need to be live. If location #1 goes down and then I bring location #2 up live it will have different settings and my money management will be changed. I didn't know if I could set up a system whereby location #1 is default and location #2 is on a slight delay, but is also live. The delay will keep location #2 from entering or existing trades because it will just pick up location #1 transactions and show it in the terminal. Does anybody have any information on:
1) if this can be done with mt4 with more than one location mirroring the default location mt4 and everyone running live?
2) if yes, can you provide the instructions on how to do it
3) perhaps I could use a service like logmein two link all of my computers together and have remote monitoring capability from any location?

Any info you guys could provide would be greatly appreciated. This is a major flaw in my trade system so I have to have CPDR. THANKS!

Thanks.