Deadlock in Python's subprocess popen

Question

I'm having a problem where popen is deadlocking. Specifically, the thread (not the main thread) that runs the popen is stuck at:

File: "/usr/lib/python2.7/subprocess.py", line 679, in __init__
  errread, errwrite)
File: "/usr/lib/python2.7/subprocess.py", line 1224, in _execute_child
  data = _eintr_retry_call(os.read, errpipe_read, 1048576)
File: "/usr/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call
  return func(*args)

With ps, I can see that the python process has forked itself, but it has not yet run os.execvp. This can be seen by checking ps, where I see that there is a child process of my main process with the same name. Killing that process causes the stuck thread in the parent to resume.

As far as I can tell, the child process (that will execvp) is locking up somewhere in the 50 lines between its creation and running os.execvp. To complicate matters, it is rare; maybe 1 in 100 popens. How can this lock-up occur? How do I fix it? (I cannot reproduce this behavior in a simple python program).

I should note that while this program is incredibly multithreaded (hundreds of threads running), no other threads fork or use popen. Many other threads are working with file descriptors (mostly sockets) though.

Technical notes:

• Running on Amazon EC2
• Linux 3.0.0-14-virtual x86_64 in ubuntu 11.10
• python 2.7.2 interpreter.
• Using several libraries including paramiko and boto

Answer 1

Looks like it is a recently filed python 2.7.2 bug:

http://bugs.python.org/issue13817