MT4 DLL Memory Pollution | Forex Factory

Options

Go to
Permalinks
Print Thread

Similar Threads

Wininet.dll & user32.dll. Can we use it in Macintosh MT4/MT5? 0 replies

Multiple EAs using single DLL with DLL Windows - Using Delphi 9 replies

2009.03.11 15:41:15Tester memory handler: tester stopped because not enough memory 19 replies

MT4 price data and memory size 0 replies

Memory leak in MT4 3 replies

Attachments: MT4 DLL Memory Pollution
Exit Attachments

MT4 DLL Memory Pollution

  • #1
  • First Post: Feb 22, 2010 3:54pm Feb 22, 2010 3:54pm
Hi Guys,

Im having an intermittent(gota love those ones) with an EA and a DLL ive written. The EA runs on 5 different pairs, but the information passed to the DLL is essentially the same allbeit different actual values.

The problem im having is that data from one pair is being picked up by the EA on another pair. For example, i send in to the EA data to calculate the correct LotSize, if they are run individually then they work fine, sometimes when i run them all together, they all end up with the same lotsize.

I read somewhere on here that when accessing a single DLL like this from multiple charts, you are actually sharing the same memory space, which was the conclusion i reached, its almost like random as to which one writes the data i need into the array to be picked up by the DLL. Oh on that point im using pointers. here is the DLL declaration:

#import "xxx.dll"
bool f1 (string& arr1[10], double& arr2[40], double& rates1[2000][6], double& rates2[2000][6]);
double f2 (string& arr1[10], double& arr2[40], double& rates1[2000][6], double& rates2[2000][6]);

am i correct in my idea of what the problem is, if so, is there any reasonable workaround other than duplicating the DLL for each pair?

thanks in advance
  • #2
  • Feb 22, 2010 10:55pm Feb 22, 2010 10:55pm
I think your diagnosis is dead on. I suppose writing data to separate .CSV files is no better than having separate DLLs.

Another idea would be to have a boolean parameter value act like a semaphor and if it is set to busy another EA is blocked. Once the first EA is finished, it would reset to ready and the other one could access the DLL. MT4 isn't real good at handling events but I suppose there might be some kludgey way to implement this too. It all depends on how often the DLL code is called (like for every new bar or new tick or once per activation) and the likelihood of multiple access.
  • #3
  • Feb 23, 2010 12:12am Feb 23, 2010 12:12am
Use a c++ .exe that sets up a "shared" memory space, then access the shared memory space via your dll call from metatrader.

For multiple pairs, you will probably have to set up multiple shared memory space arrays, each accessed via seperate dlls.

I have done this, and it worked for me.
  • #4
  • Feb 23, 2010 3:45am Feb 23, 2010 3:45am
Unfortunately you are unclear about your implementaion. As I understand it :

1) EA passes data to DLL (by reference ? (pointers))
2) Your DLL copies? this data into its own variables / allocated memory
3) your DLL uses this data now in its own variables to calculate values to return to MT4
4) Your DLL returns values to your EA (by reference ?(pointers))

If this is the case why not just code your dll to use separate sets of arrays.. after all you cant expect a single location to do 5 jobs when those jobs are running from different threads!

so in your dll

1) add another dimension to your arrays (one for each pair (+ a few)-or dynamically manage them)

2) add an initialisation function whereby your MT4 EA requests and is returned a unique ID which can be used to reference its own array set.


EG.

EA code

#import#
double f2 (Int ID, string& arr1[10], double& arr2[40], double& rates1[2000][6], double& rates2[2000][6]);

//Global
int myid

..
..
myid = Request_DLL_ID();
if myid = -1 then error


f2 (myid, string& arr1[10], double& arr2[40], double& rates1[2000][6], double& rates2[2000][6]);



The ID number your DLL returns maybe just a unique number with an associated look up list to allocate a real array dimension or be the actual array dimension itself.
  • #5
  • Feb 23, 2010 9:24am Feb 23, 2010 9:24am
Quoting GreatSamps
am i correct in my idea of what the problem is, if so, is there any reasonable workaround other than duplicating the DLL for each pair?
There are two solutions to this problem, none of them involves duplicating the dll.

1.) pass all parameters to all functions *always* as function arguments to all functions in the dll so that there are no global varibles inside the DLL at all. Pass around only pointers to bigger data structures, things like double &foo[] from mql, so you dont have too much data to push and pop from the stack on each function call.

2.) put all your functions of the dll and the global data it shares inside a class and on init() of your EA call a function in the dll that will create a new object instance from this class and returns a pointer to this instance. You can safely cast this 32 bit pointer into a 32 bit int variable to be returned to mql4 and stored in your EA.

The API of your dll should now consist of functions that accept as first parameter this integer number. For every method you want to call on your object you will write and export a wrapper that accepts the pointer to the object as first parameter. Your EA will call these wrapper functions with its own int number and the wrapper functions in the dll will cast the int back into the pointer to the object belonging to that EA and then call the corresponding methods on that object.

The first one (no global variables at all) is easier to implement. The second one is recommended for more complex things, you will use OOP in these cases anyways. Given that this will only ever be used on 32bit windows it is completely safe to pass the pointers around.

If for whatever reason you don't want to cast and pass around pointers between dll and EA you can also use some sort of hash table or dynamic array or whatever your runtime library or toolkit has to offer for such things to store the pointer to the object in the dll itself and pass something more portable (a string or a handle) as key to and from the EA. This would of course impact the performance.
  • #6
  • Edited 4:07pm Feb 23, 2010 10:08am | Edited 4:07pm
If you decide to go the first route (passing around all paramters always as function arguments) it is good practice to do the following from the very beginning on:

Imagine your EA has the following (fictional) external settings:

Inserted Code
extern double ex_foo;
extern double ex_bar;
extern double ex_period;
extern double ex_gamma;
extern double ex_delta;
and the following global variables that may change during operation and are also needed for many of the functions:
Inserted Code
double last_high;
double last_low;
then do the following: define named constants for the indices like this:

Inserted Code
#define I_FOO       0
#define I_BAR       1 
#define I_PERIOD    2
#define I_GAMMA     3
#define I_DELTA     4
#define I_LAST_HIGH 5
#define I_LAST_LOW  6

[color=SeaGreen]// the max. size of the array[/color]
#define I_SIZE      7
and a global array
Inserted Code
double global[I_SIZE];

int init(){
[color=SeaGreen]   // externals[/color]
   global[I_FOO] = ex_foo;
   global[I_BAR] = ex_bar;
   global[I_PERIOD] = ex_period;
   global[I_GAMMA] = ex_gamma;
   global[I_DELTA] = ex_delta;

[color=SeaGreen]   // initialize other globals[/color]
   global[I_LAST_HIGH] = 0;
   global[I_LAST_LOW] = 0;
}
call your functions in the dll
Inserted Code
int result = dllfunc(globals, ...);
and in the dll you have defined the same constants as above you can simply do the following.
(for this example we will assume the dll is in pascal so it doesn't get too boring here . It doesn't influence the problem we are trying to solve, it applies to all languages. If you already can read C++ code you can intuitively read and understand the much simpler (object-)pascal code as well and translate the concept into your preferred language without any problems)

Inserted Code
[b]library[/b] myholygrail;

[b]const[/b]
   I_FOO =       0;
   I_BAR =       1; 
   I_PERIOD =    2;
   I_GAMMA =     3;
   I_DELTA =     4;
   I_LAST_HIGH = 5;
   I_LAST_LOW =  6;
   I_SIZE =      7;
   
[b]type[/b]
   TGlobals = array[0..I_SIZE-1] of double;

[b]function[/b] dllfunc([b]var[/b] global: TGlobals; ...): integer; stdcall;
[b]var[/b]
   foo:  double;
   high: double;
[b]begin[/b]
   foo := global[I_FOO];
[color=SeaGreen]   // do some stuff[/color]
   global[I_LAST_HIGH] := high;
[color=SeaGreen]   // do some more stuff[/color]
[color=SeaGreen]   // ...[/color]
[b]end;[/b]

[b]exports[/b]
   dllfunc;
[b]
begin[/b][b]
end.
[/b]
The reason to to this is you can later add more parameters, more global variables to this array and the only thing you have to change is the definition of these constants. The code will always be readable, you will never use literal numbers like global[42], always global[I_SOMETHING] and don't have to care whether it is actually at position 42 or somewhere else.
  • #7
  • Feb 23, 2010 1:51pm Feb 23, 2010 1:51pm
Hi Guys,

Thanks for all your responses, it is great to have a community with such knowledgeable people contributing.

7bit: If i understand you correctly, you are saying the real problem is the fact that i am calling the arrays by position, for example arr2[25] rather than explicitly defining each element individually and referencing it that way? the DLL does not have any global variables within its self, all variables that are passed to it are local to the function only.

RangeBound: What im doing is creating a double array within MT4, this is being given access to by the DLL by the way of a pointer. The function looks at some elements of it, does some math, then writes to another element in that same array the output value, in this case lotsize.

That same output value (arr2[19]) is used for all pairs, and what im seeing is that a lot size destined for another pair ending up elsewhere.

I did think about using multiple arrays for each, but wanted to be clear exactly where the problem is before complicating it further. at the outset it would seem that 7bits solution is simpler, but i would welcome comments regarding this.

Thanks once again.
  • #8
  • Edited 4:24pm Feb 23, 2010 3:35pm | Edited 4:24pm
Quoting GreatSamps
7bit: If i understand you correctly, you are saying the real problem is the fact that i am calling the arrays by position, for example arr2[25] rather than explicitly defining each element individually and referencing it that way?
No, not really, it is all about readability (and thus avoiding errors). What i wanted to say is that using arr[MEANINGFUL_NAME] instead of arr[25] is only a convenient way to keep the code readable and maintainable. This way you could easily stuff ALL your globals into ONE array and pass only one pointer around and still have something easily readable without producing programming errors by accidently confusing the array indices or forgetting to change the numbers in all places when you decide to change the structure of that array later.

consider this code:
Inserted Code
  scale[I_SCALE] := (price_range[I_HIGH] - price_range[I_LOW]) /  (pattern_size[I_HIGH] - pattern_size[I_LOW]);
  scale[I_OFFSET] := price_range[I_LOW] - pattern_size[I_LOW] *  scale[I_SCALE];
and compare it with this:
Inserted Code
  scale[0] := (price_range[1] - price_range[0]) /  (pattern_size[1] - pattern_size[0]);
  scale[1] := price_range[0] - pattern_size[0] *  scale[0];
It does the same thing, compiles to the same machine code but which one is more readable? The same code appears later with an inverted pattern, low becomes high and high becomes low, what do you think how long must I sit there and stare at these two lines if i accidently confused 0 and 1 somewhere if i didn't use named constants?

It's all about reducing possible causes of programming errors.

Quoting GreatSamps
the DLL does not have any global variables within its self, all variables that are passed to it are local to the function only.
Then your initially posted problem should not occur, and if it occurs it is not caused by the fact that the functions live in a dll. Variables that are passed as arguments and those declared local to the function will always be on the stack, meaning each thread using these functions at the same time will have its own space for its variables (because each thread will come along with its very own stack and each EA instance can be regarded as a different thread for our problem). Furthermore, when a function is called then it will always reserve new space on the stack and when a function returns the stack is always returned into exactly the same state like it was immediately before the function was called. Local variables simply CAN not accidently persist and reappear beyond the current function call.

There must be another cause for the behaviour you are observing. There must be some well hidden nasty programming error somewhere (not necessarily in the dll) in your code.
  • #9
  • Feb 23, 2010 4:05pm Feb 23, 2010 4:05pm
Thanks for the reply. Just to 100% confirm that im not getting local/global mixed up, here is the DLL function thats causing the concern:

bool get_lotsize(MqlStr* arr1, double *arr2, const RateInfo* rates1, const RateInfo* rates2, double stop_loss) // returns lot size
{
doubleriskvalue,
pipvalue,
lotsize;

riskvalue = (arr2[10] * (arr2[7] / 100)) / arr2[11];

pipvalue = arr2[8];

if(arr2[1] == 5 || arr2[1] == 3) pipvalue *= 10;

lotsize = riskvalue / (stop_loss * pipvalue);
lotsize = floor (lotsize / arr2[4]) * arr2[4];

if ( lotsize < arr2[5]) lotsize = arr2[5]; // min lot
if ( lotsize > arr2[6]) lotsize = arr2[6]; // max lot

arr2[19] = lotsize;
return(true);
}

just looking at this actually stop_loss IS a DLL global variable. its declared in the header of the DLL, would this variable be persistant between all calls to that DLL ? If so then we have found the problem
  • #10
  • Edited 7:16pm Feb 23, 2010 5:37pm | Edited 7:16pm
Quoting GreatSamps
just looking at this actually stop_loss IS a DLL global variable. its declared in the header of the DLL, would this variable be persistant between all calls to that DLL ? If so then we have found the problem
a variable declared in the dll outside of the functions would be shared with all function calls from all threads (all EAs).

What makes me wonder here is what happens with the same variable declared in the function's argument list. Normally the local variable should then have prevalence over the same variable outside the function but my C knowledge is limited. However, whether your C compiler will handle this correctly or not, such a naming collision is always calling for some kind of trouble, making programming errors more likely and their detection more difficult. You should get rid of all global variable declarations outside the functions in the dll.
  • #11
  • Feb 23, 2010 5:45pm Feb 23, 2010 5:45pm
BTW: did you already see my thread about creating dlls with FPC/Lazarus? http://www.forexfactory.com/showthread.php?t=219576

I should add a chapter dedicated to global variables, how to avoid them and how to use them if really necessary.
  • #12
  • Feb 24, 2010 1:42am Feb 24, 2010 1:42am
As a rule Global vars in your DLL will be created the instant the DLL is loaded (first call to the DLL). and changed by ALL access. Local variables within each function are allocated when the function is called and pointers to them are stored on the stack, they are only accessed by the function call that created them, and they are destroyed when the function exits.

The problem your'e having is a Multi Threading issue EG :-

E1 and E2 are 2 EAs each running in a different thread.
D1 your DLL
F1 your DLL function

Inserted Code
// Your DLL code

//Global Var
int x;

void F1();
{
  int y; // local var

  x = 1000;
  while (x>0)
 {
  y= 2000;
   while (y>0) {y--;}
   x--;
 }
}

So when it runs the following could and will happen :

E1 calls F1, and a new 'temp' var Y is created. It then stops running halfway through the function because the OS says its timeslice is up. Everything is working well. If E2 did not exist there would be no problems at all.

E2 now calls F1 and a new 'temp' var Y is created. F1 Immediately sets x to 1000 ... You have now broken E1's call to the function because the global variable has changed, and when E1 gets the processor back it will find int x to be different than when it left off. Of course E1 wont know, or care about this.


You can easily solve this without OOP you just have to be aware that you are calling the same function(s) from multiple threads so you need to be thread safe. Either with or without OOP you need to have a way to alot a separate int X to each EA. Hence the need to provide an Init function to allocate a separate 'bank' of global variables to each EA. In the above example this could easily be achieved by making Int X a suitably sized array, and providing init/deinit functions to allocate an array dimension element to each unique caller to the init function. E.G.

E1
int myID = DLLinit('GBPJPY');
// returns 1

F1(myID);
// now F1 can use MyID to reference the correct X var in the global array.


E2
int myID = DLLinit('EURUSD');
// returns 1



OOP is often a far more elegant way of sovling this problem but will only work if you understand the problem and then code it correctly. With OOP you'd be looking to create a new object (one for each EA) when they call a DLL init function. This would work only if you encapsulate a separate set of 'Global' variables within each new object.
  • #13
  • Feb 24, 2010 3:30am Feb 24, 2010 3:30am
7bit: i have changed the code to not have those couple of global variables, i had not even noticed them until i pasted the code, also with your confirmation that they do persist between calls makes perfect sense as to why it was doing what it was. i will let you know how i get on with demo testing of it.

i will take a look at your article now, i may have read it, you read so much these days! thanks once again
  • #14
  • Feb 24, 2010 3:34am Feb 24, 2010 3:34am
Rangebound: Thanks for your post, i can see the problem i went wrong was that i presumed that each load of an EA would load a separate instance of the DLL, nicely separated from other calls that might be going on. Completely agree with your example.

Funnily enough knowing this has actually given me a solution to another problem i have had, the answer was there all along.

Thanks for your help
  • #15
  • May 3, 2010 6:55am May 3, 2010 6:55am
I also would like to say "thank you!" to 7bit and rangebound for their assistance here. It has been very helpful.

I am using Delphi 2009 (Object Pascal) to write my DLLs. Using the advice in this thread and also this one, I have eliminated all crashes by moving all global variables to a class and passing a pointer to the object between MT4 and the DLL. This solved any memory collisions.

However, it seems there is still something I'm not understanding. In order to avoid ALL crashing, I needed to use a TCriticalSection to ensure that only one chart was accessing the DLL at a time. For example:

Inserted Code
// Delphi (Object Pascal) DLL Code:
var
  csTickLock: TCriticalSection;

function  DLLGetLock: Boolean;
begin
  csTickLock.Enter;

  Result := True;
end;


procedure DLLFreeLock;
begin
  csTickLock.Leave;
end;



initialization
  csTickLock := TCriticalSection.Create;

finalization
  csTickLock.Free;


Inserted Code
//EA Code:
#import "MyLogic.dll"
  bool DLLGetLock            ( );
  void DLLFreeLock           ( );
#import


int init() {
  DLLGetLock( );

  int iPointer = DLLGetNewObject( Symbol());

  ...Other initialization...

  DLLFreeLock( );

  return(0);
}


int start() {
  static int iPrevOrderCount = 0;
  
  DLLGetLock( );

...multiple DLL function calls...

  DLLFreeLock( );

  return(0);
}

Again, I have no global vars in the DLL; everything (except the TCriticalSection object) is wrapped in the class, and the class reference is passed to the DLL with every function call. But I'm still getting some sort of collision/corruption because I need to use the thread semaphore (TCriticalSection) to make it work properly.

Here is the reason I'm bringing all this up. There is still one minor problem: if deinit() gets called for a chart, MT4 breaks out of the start() function immediately, which sometimes means DLLFreeLock() doesn't get called and all other charts are locked out of the DLL.

So in summary my question is: why do I need to use the TCriticalSection semaphore? If everything is in a class, what other collisions could there be?

Thank you!
  • #16
  • May 3, 2010 7:54am May 3, 2010 7:54am
My bad .

I did find a global var in an included unit. I removed that var and all is fine now - without the need for the semaphores!

Again, thank you for your help.
  • #17
  • May 4, 2010 4:26am May 4, 2010 4:26am
You shouldnt need to use critical sections for most DLLs so this points to some other underlying coding issue, which could be one of many things.

The best you could do is .... Cut code out of your DLL till you have the most basic DLL that still Crashes. Then post the source here and I'll take a look.

Bearing in mind you dont state when a crash occurs ...
If you dont want to go down that route then my best guesses are :

1) If your EA does a deinit / Init then your DLL may try to create a second object for the same currency pair. Have you coded a trap to stop duplication.
Dont assume your DLL is guaranteed to get a DeInit call. You must check for duplication of same currency initialisations.


2) Is you DLL storing the pointers to, and managing the destruction of these Objects ?
  • #18
  • May 5, 2010 7:30am May 5, 2010 7:30am
Thank you Rangebound! Everything is working fine now, with no crashes or memory leaks, but your post raises a couple questions:

Quoting rangebound
1) If your EA does a deinit / Init then your DLL may try to create a second object for the same currency pair. Have you coded a trap to stop duplication.
Dont assume your DLL is guaranteed to get a DeInit call. You must check for duplication of same currency initialisations.
The way I have it written, if deinit() does not get called, the object will not get freed. But aside from the memory leak that shouldn't cause crashing because the second object will have a new memory location, correct?

Quoting rangebound
2) Is you DLL storing the pointers to, and managing the destruction of these Objects ?
Normally I would use a TThreadList (a thread-blocking list of pointers) in the DLL to do something like this - is there another way? In other words, how should I store the pointers in the DLL without using a global var or some blocking mechanism?

This is very helpful - thanks for your input.
  • #19
  • Edited 9:26am May 5, 2010 9:15am | Edited 9:26am
Quoting amckeen
Normally I would use a TThreadList (a thread-blocking list of pointers) in the DLL to do something like this - is there another way? In other words, how should I store the pointers in the DLL without using a global var or some blocking mechanism?
Don't store the pointers in the dll at all. Cast them into 32 bit integers, store them in the EA, treat them like resource handles of some sort, and pass them as parameters to wrapper-functions in the dll (which then will cast them back, dereference them and then call the actual method of the object instance).

I'm doing this in my Python binding. All Python objects in the python26.dll are actually pointers but I can easily treat them like integers to be used by the mql code. All my python API wrapper functions that need a pointer to a python object simply accept an integer, for the MQL code all python objects are just integer handles.
  • #20
  • May 5, 2010 10:09am May 5, 2010 10:09am
Quoting 7bit
Don't store the pointers in the dll at all. Cast them into 32 bit integers, store them in the EA, treat them like resource handles of some sort, and pass them as parameters to wrapper-functions in the dll (which then will cast them back, dereference them and then call the actual method of the object instance).

I'm doing this in my Python binding. All Python objects in the python26.dll are actually pointers but I can easily treat them like integers to be used by the mql code. All my python API wrapper functions that need a pointer to a python...
Yup, that's exactly what I'm doing, but RangeBound raised a concern about the object not getting freed because DeInit() might not get called, and suggested I store a list of pointers in the DLL so it can manage the memory.
0 traders viewing now