Unload a module in Python

Question

TL/DR:

import gc, sys  print len(gc.get_objects()) # 4073 objects in memory  # Attempt to unload the module  import httplib del sys.modules["httplib"] httplib = None  gc.collect() print len(gc.get_objects()) # 6745 objects in memory 

---

UPDATE I've contacted Python developers about this problem and indeed it's not going to be possible to unload a module completely "in next five years". (see the link)

Please accept that Python indeed does not support unloading modules for severe, fundamental, insurmountable, technical problems, in 2.x.

---

During my recent hunt for a memleak in my app, I've narrowed it down to modules, namely my inability to garbage collect an unloaded module. Using any method listed below to unload a module leaves thousands of objects in memory. In other words - I can't unload a module in Python...

The rest of the question is attempt to garbage collect a module somehow.

Let's try:

import gc import sys  sm = sys.modules.copy()  # httplib, which we'll try to unload isn't yet                           # in sys.modules, so, this isn't the source of problem  print len(gc.get_objects()) # 4074 objects in memory 

Let's save a copy of sys.modules to attempt to restore it later. So, this is a baseline 4074 objects. We should ideally return to this somehow.

Let's import a module:

import httplib print len(gc.get_objects()) # 7063 objects in memory 

We're up to 7K non-garbage objects. Let's try removing httplib from sys.modules.

sys.modules.pop('httplib') gc.collect() print len(gc.get_objects()) # 7063 objects in memory 

Well, that didn't work. Hmm, but isn't there a reference in __main__? Oh, yeah:

del httplib gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

Hooray, down 300 objects. Still, no cigar, that's way more than 4000 original objects. Let's try restoring sys.modules from copy.

sys.modules = sm gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

Hmmm, well that was pointless, no change.. Maybe if we wipe out globals...

globals().clear() import gc # we need this since gc was in globals() too gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

locals?

locals().clear() import gc # we need this since gc was in globals() too gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

What the.. what if we imported a module inside of exec?

local_dict = {} exec 'import httplib' in local_dict del local_dict gc.collect() print len(gc.get_objects())  # back to 7063 objects in memory 

Now, that's not fair, it imported it into __main__, why? It should have never left the local_dict... Argh! We back to fully imported httplib. Maybe if we replaced it with a dummy object?

from types import ModuleType import sys print len(gc.get_objects())  # 7064 objects in memory 

Bloody.....!!

sys.modules['httplib'] = ModuleType('httplib') print len(gc.get_objects())  # 7066 objects in memory 

Die modules, die!!

import httplib for attr in dir(httplib):     setattr(httplib, attr, None) gc.collect() print len(gc.get_objects())  # 6749 objects in memory 

Okay, after all attempts, the best is +2675 (nearly +50%) from starting point... That's just from one module... That doesn't even have anything big inside...

Ok, now seriously, where's my error? How do I unload a module and wipe out all of it's contents? Or is Python's modules one giant memory leak?

Full source in simpler to copy form: http://gist.github.com/450606

Answer 1

Python does not support unloading modules.

However, unless your program loads an unlimited number of modules over time, that's not the source of your memory leak. Modules are normally loaded once at start up and that's it. Your memory leak most likely lies elsewhere.

In the unlikely case that your program really does load an unlimited number of modules over time, you should probably redesign your program. ;-)

Answer 2

I'm not sure about Python, but in other languages, calling the equivalent of gc.collect() does not release unused memory - it will only release that memory if/when the memory is actually needed.

Otherwise, it makes sense for Python to keep the modules in memory for the time being, in case they need to be loaded again.