Menu
I'm developing an application in Python on Ubuntu and I'm running external binaries from within python using subprocess. Since these binaries are generated at run time and can go rogue, I need to keep a strict tab on the amount of memory footprint and runtime of these binaries. Is there someway I can limit or monitor the memory usage of these binary programs at runtime? I would really hate to use something like "ps" in subprocess for this purpose.
662
You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.
Mem usage: Memory usage by the Python interpreter after every execution of the line. Increment: Difference in memory consumption from the current line to the last line. It basically denotes the memory consumed by a particular line of Python code. Occurrences: Number of times a particular line of code is executed.
Python doesn't limit memory usage on your program. It will allocate as much memory as your program needs until your computer is out of memory. The most you can do is reduce the limit to a fixed upper cap. That can be done with the resource module, but it isn't what you're looking for.
Subprocess in Python is a module used to run new codes and applications by creating new processes. It lets you start new applications right from the Python program you are currently writing. So, if you want to run external programs from a git repository or codes from C or C++ programs, you can use subprocess in Python.
You can use Python's resource module to set limits before spawning your subprocess.
For monitoring, resource.getrusage() will give you summarized information over all your subprocesses; if you want to see per-subprocess information, you can do the /proc trick in that other comment (non-portable but effective), or layer a Python program in between every subprocess and figure out some communication (portable, ugly, mildly effective).
131
Having a PID number of your subprocess you can read all info from proc file-system. Use:
/proc/[PID]/smaps (since Linux 2.6.14) This file shows memory consumption for each of the process's mappings. For each of mappings there is a series of lines as follows:
or
/proc/[PID]/statm Provides information about memory usage, measured in pages.
Alternatively you can limit resources which subprocess can aquire with :
subprocess.Popen('ulimit -v 1024; ls', shell=True)
When given virtual memory limit is reached process fails with out of memory.
6
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
