Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I automatically kill a process that uses too much memory with Python?

Tags:

python

linux

The situation: I have a website that allows people to execute arbitrary code in a different language (specifically, an esolang I created), using a Python interpreter on a shared-hosting server. I run this code in a separate process which is given a time limit of 60 seconds.

The problem: You can do stuff like (Python equivalent) 10**(10**10), which rapidly consumes far more memory than I have allotted to me. It also, apparently, locks up Apache - or it takes too long to respond - so I have to restart it.

I have seen this question, but the given answer uses Perl, which I do not know at all, hence I'd like an answer in Python. The OS is Linux too, though.

Specifically, I want the following characteristics:

  1. Runs automatically
  2. Force-kills any process that exceeds some memory limit like 1MB or 100MB
  3. Kills any process spawned by my code that is more than 24 hours old

I use this piece of code (in a Django view) to create the process and run it (proxy_prgm is a Manager so I can retrieve data from the program that's interpreting the esolang code):

prgmT[uid] = multiprocessing.Process(
    target = proxy_prgm.runCatch,
    args = (steps,),
    name="program run")

prgmT[uid].start()
prgmT[uid].join(60) #time limit of 1 minute

if prgmT[uid].is_alive():
    prgmT[uid].terminate()
    proxy_prgm.stop()

If you need more details, don't hesitate to tell me what to edit in (or ask me questions).

like image 600
El'endia Starman Avatar asked Dec 01 '15 06:12

El'endia Starman


People also ask

How do you kill a process in Python?

Alternatively, you can use the top command to find the python process. Simply enter k (for kill) and the top program will prompt you for the PID of the process to kill.

Why Python memory consumption is high?

Those numbers can easily fit in a 64-bit integer, so one would hope Python would store those million integers in no more than ~8MB: a million 8-byte objects. In fact, Python uses more like 35MB of RAM to store these numbers. Why? Because Python integers are objects, and objects have a lot of memory overhead.

What is OOM killer?

The Out Of Memory Killer or OOM Killer is a process that the linux kernel employs when the system is critically low on memory. This situation occurs because the linux kernel has over allocated memory to its processes. When a process starts it requests a block of memory from the kernel.


1 Answers

Another approach that might work; using resource.setrlimit() (more details in this other StackOverflow answer). It seems that by doing so you can set a memory limit on a process and it's subprocesses; you'll have to figure out how to handle if the limit is hit though. I don't have personal experience using it, but hopefully doing so would stop Apache from locking up on you.

like image 183
John O'Brien Avatar answered Sep 17 '22 15:09

John O'Brien