dealing with an unmanaged dll with a memory leak

Question

I have a c# application that depends on a third-party unmanaged assembly to access certain hardware.

The unmanaged code has a memory leak that will increase the memory consumption by ~10mb after each access. The problem is known; no bugfix is available.

Is there a way I can continue to use this assembly without regular restarts?

I tried to create a separate AppDomain, load the offending code into that AppDomain via appDomain.CreateInstanceAndUnwrap() and then later unloading the domain via AppDomain.Unload(). However this apparently does not free the unmanaged memory used by that domain, only the managed memory.

I could also split the application into two independent parts, and restart only the part with the unmanaged dll. However this would mean a major redesign, and probably cause a lot of slowdown, since large amounts of data would have to be exchanged between these two parts.

Is there another way to tame this leaky assembly and force it to release its memory without restart.?

Answer 1

The way you describe it, the dll is allocating non-managed memory. This kind of memory will not be impacted by the act of unloading an appdomain, unfortunately, as you've already found out.

You got a couple of options, but I don't think any of them are appealing:

1. You can keep using the dll in your main application. You can advise your users/clients that they need to have lots of memory available, but will need to restart the application from time to time. You would probably want to detect out-of-memory conditions before the program actually starts crashing, to gently prod the user to restart it, instead of just crashing hard with an exception.
2. You can try to dig into the data structures of the dll. If the dll is actually changing (ie. new versions are coming out), I would strongly suggest you lean hard on the author to get the memory leak fixed. However, this seems unlikely since I'm pretty sure the leak would be fixed if a new version came out. As such, tying your code directly to the innards of that dll might be a solution. Having access to the actual pointer that references the unmanaged memory might allow you to manually free it as you see fit.
3. You can isolate the problem to a separate process. More on that below.
4. You could reimplement the functionality of the dll.

There might be a 5th or a 6th option here, but think the above 4 covers the things I came up with off the top of my head.

About isolating it into a separate process, here's what I would try to do first:

I would spin up a process, and pump requests to it using the fastest intra-process communication channel you can find. Pipes seems a good fit, or memory-mapped files.

You would then, in that separate process, detect the out-of-memory condition, hopefully a bit early, so that you could advise the main program that it should think about spinning up a replacement process.

The main process could then do that, but instead of waiting for that other process to fully spin up, it could keep pumping a few more requests to the soon-to-be-dead instance, filling it up a bit more, before switching over to the new instance and asking the old to terminate.

This would minimize the downtime at the expense of temporarily having an extra process alive during transitions.

All of this depends, a lot, on the actual scenarios you have. If you need to call this dll 100s or 1000s of times a second, doing intra-process communication might not be doable in any case.

Answer 2

The DLL allocates unmanaged memory in your process and does not free it. Unloading the application domain will obviously not help in this case because the memory is allocated within the process, not within the application domain.

In order to get rid of the memory leak you have to free the unfreed memory and there are two ways to do it - terminate the process and let the operating system do it or free it yourself. In order to free it yourself you have to get hold of the handle to the memory by either obtaining it via the public interface of the DLL, by private implementation details of the DLL or by inspecting the heap of the process. It is probably doable but I guess it will be neither fun, nor easy and maybe not even robust.

In comparison the solution with two processes and regularly restarting the leaking one seems easy and robust. I would not be to worried about performance because you can for example use memory mapped files to share your data and avoid copying it around.

The easiest solution would of course be getting the memory leak fixed or maybe even fixing it yourself either by fixing the source if available or patching the DLL.