poll method of subprocess.popen returning None for long process

Question

I am executing curl command through subprocess. This curl command starts video processing at another server, and waits for the response. Once the process is done, the remote server returns json object. I am checking the status of the subprocess using poll() value, which is None - process not completed, 0- process completed successfully and 1- for error.

I am getting the correct response if the processing takes around 30 mins/or less on remote server, but if the processing is taking more time, I am getting just None value , even though I can see that the remote server has finished processing and already returned the json object.

Can anyone tell me, what could be the possible reason for poll() returning only None after certain time. Thank you in advance.

My Popen object is :

object = subprocess.Popen(str(curlCmd), shell=True,
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE)

and I am calling object.poll() after every 2 seconds to check if the process is successfully completed or not.

Answer 1

.poll() is None means that the child is still running.

The curl process may hang as soon as it fills its stdout or stderr OS pipe buffer (around 64K on my Linux box) unless you read from your end of the pipe.

That is while you are waiting for the child to finish, the child waits for you to drain the pipe buffer -- deadlock. From the subprocess docs:

This[.wait()] will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

You could use threads, async.io to consume the pipes concurrently.

Answer 2

In python 2.7 and possibly 3.x can use:

object._internal_poll(_deadstate=127)

instead of regular Popen.poll() as a workaround. This will return 127 instead of None if the process is terminated.

Of course this is an internal module's method and there's no guarantee it will work after Python's library update.