Python not freeing up memory in Linux

Question

I've seen a few related questions posted like two years ago, but I would like to know if any solution has come up recently.

I have a huge dictionary of dictionaries. There are about 4 dictionaries (each of 500 MB size) in my memory. As I keep running the program, I need to delete one of those 4 dictionaries and free up the memory to the OS. So, it is not possible for me to begin a new subprocess for memory allocation as was mentioned in some of the previous posts.

Here's some code to illustrate the problem:

import cPickle
import resource
import gc
import time

mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print "memory usage:", mem
test_dict = {}
for i in range(100000):
    test_dict[i] = "AAAAAAAA"
    if i%10000 == 0:
        mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        print "memory usage:", mem

mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print "memory usage: (dict created): ", mem
del test_dict
mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print "mem usage: (dict deleted)", mem
gc.collect()
mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print "mem usage (garbage collection)", mem
print "sleeping for a few seconds"
time.sleep(30)
gc.collect()
mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print "memory usage after sleeping ", mem

Here's the result. The memory is reported in KB.

memory usage: 5152
memory usage: 8316
memory usage: 9176
memory usage: 9176
memory usage: 12076
memory usage: 12076
memory usage: 12076
memory usage: 12076
memory usage: 12076
memory usage: 12076
memory usage: 17548
memory usage: (dict created):  17548
mem usage: (dict deleted) 17548
mem usage (garbage collection) 17548
sleeping for a few seconds
memory usage after sleeping  17548

As you can see, the memory does not seem to freeing up at all. I tried this on my Ubuntu 11.10 machine with Python 2.7.2

Answer 1

According to man getrusage:

ru_maxrss (since Linux 2.6.32)
    This is the maximum resident set size used (in kilobytes).

If I understand it correctly it means peak usage rather then current usage.

EDIT:

Also it is worth to look at Memory Management article from Python's official docs.

Answer 2

As Ihor Kaharlichenko points out, ru_maxrss is the peak usage of the program. Consider the following program which is very similar to yours:

import time
time.sleep(10)
string = ' ' * int(5e8) # 500 MB string
time.sleep(10)
string = None # the huge string is automatically GC'd here
time.sleep(10)

If you watch the memory usage of this in top or whatever, you'll see it is very small for the first 10 seconds, then spikes to ~500 MB for a little while, and then drops again. Your program exhibits the same behaviour.