Determining if a python subprocess segmentation faults

Question

I am writing a program that grades student programs, and as I am sure you can imagine, they sometimes segmentation fault. The problem I am having is that when the student programs segmentation fault, there is no indication that is what happened.

proc = subprocess.Popen(student_command, 
                        stdout=subprocess.PIPE, 
                        stderr=subprocess.PIPE)
self.stdout, self.stderr = proc.communicate()
self.returncode = proc.returncode

I pick up the stderr, stdout, and the return code from the subprocess, but if the program segmentation faults, stderr is empty, stdout is empty, and the return code is -11. Now I could look for the -11 exit code and assume that if that is the return code there was a segmentation fault, but there is also nothing to prevent a student's code from having -11 as a return code just because the student felt like returning -11.

How do you tell if a subprocess segmentation faults, as opposed to just feeling like returning -11? I don't really care all that much about what is in stderr and stdout, and to that end have seen a number of posts including this that deal with picking up the output, but I don't care all that much about the output, although it would be nice to get the "Segmentation Fault" string out of stderr, but what I really need is a way to definitively tell what happened to the subprocess.

Answer 1

Well, in fact, on UNIX, a process that attempts to return -11 will usually end up returning a positive integer instead. This is because the return status from the wait series of functions is actually a set of bitfields, with a field for the signal that ended the process and a separate field for the return value. Python decodes the wait return value from these bitfields.

On most systems, these fields are unsigned and 8 bits in size, so you will probably see something like this:

>>> import subprocess
>>> subprocess.Popen(['python','-c','import os; os.kill(os.getpid(),11)']).wait()
-11
>>> subprocess.Popen(['python','-c','exit(-11)']).wait()
245

In the former case, the process "segfaults" (by killing itself with SIGSEGV), and so wait returns -11. In the latter case, the process exits with a return code of -11, and the resulting wait value is 245 (256-11). You can therefore rest assured that any negative return value from wait must represent a fatal signal, as opposed to a normal return. Note, though, that processes may kill themselves to fake a fatal error.