Menu
If I execute the Python interpreter it needs roughly 111 MByte:
>>> import psutil
>>> psutil.Process().memory_info()
pmem(rss=19451904, vms=111677440, shared=6905856, text=4096, lib=0, data=12062720, dirty=0)
After importing django it uses 641 MByte
>>> import django
>>> django.setup()
>>> psutil.Process().memory_info()
pmem(rss=188219392, vms=641904640, shared=27406336, text=4096, lib=0, data=284606464, dirty=0)
And the WSGI process (which has already executed some http requests) 919 MByte:
>>> psutil.Process(13843).memory_info()
pmem(rss=228777984, vms=919306240, shared=16076800, text=610304, lib=0, data=485842944, dirty=0)
I think that's too much.
What can I do to investigate this in more detail? What occupies the memory?
Background: From time to time memory on the server is running low and the oom-killer terminates processes.
416
In computing, resident set size (RSS) is the portion of memory occupied by a process that is held in main memory (RAM). The rest of the occupied memory exists in the swap space or file system, either because some parts of the occupied memory were paged out, or because some parts of the executable were never loaded.
RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory.
You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.
Those numbers can easily fit in a 64-bit integer, so one would hope Python would store those million integers in no more than ~8MB: a million 8-byte objects. In fact, Python uses more like 35MB of RAM to store these numbers. Why? Because Python integers are objects, and objects have a lot of memory overhead.
You're looking at the wrong attribute:
rss is the Resident Set Size, which is the actual physical memory the process is usingvms is the Virtual Memory Size which is the virtual memory that process is usingKernels allow a process to get a different view of the memory where the process thinks like it is the only program running in the system, that's why the virtual address space is for. While in reality kernel uses memory management to synchronize memory usage between processes. Also note that, the shared libraries between processes play a part in memory consumption as well.
Regarding your OOM incident, see which process is being killed and see what's the process was doing. For example, Linux uses /proc/PID/oom_score to keep track of each processes OOM score to find which process to kill in OOM situations -- higher value indicates a higher probability of selection. Linux sets this value based on different heuristics e.g. number of children, how long it's running, CPU usage, niceness and so on. And you can tweak this for the process by writing to /proc/PID/oom_score_adj.
But don't influence the OOM score, try to debug the actual problem in the process. A memory profiler like valgrind might be helpful in this regard.
101
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
