Debugging a subprocess.Popen call

Question

I have been using subprocess.Popen successfully in the past, when wrapping binaries with a python script to format arguments / customize etc...

Developing a nth wrapper, I did as usual... but nothing happens.

Here is the little code:

print command
p = subprocess.Popen(command, shell = True)
result = p.communicate()[0]
print vars(p)
return result

And here is the output:

/usr/bin/sh /tmp/run/launch.sh
{'_child_created': True, 'returncode': 0, 'stdout': None, 'stdin': None, 'pid': 21650, 'stderr': None, 'universal_newlines': False}

As you can see, the goal is to create a shell script setting up everything I need, and then executing it. I would prefer to use real python code, but unfortunately launch.sh call 3rd party shell scripts that I have no wish to try and replicate (though I've been insisting for a python api for over a year now).

The problem is that:

• the shell script is not executed (it should spawn process and output some little things)
• no python exception is raised
• there is nothing in the p object that indicates that an error occurred

I have tried check_call without any success either...

I am at a loss regarding what I should do, and would be very glad if someone could either point my mistake or direct me toward resolution...

EDIT:

• Trying to run this on Linux (sh)
• shell is necessary for variable substitution in the scripts invoked

EDIT 2:

Following badp suggestion, I tweaked the code and added

subprocess.Popen('ps', shell = True).communicate()

Right after p = ... line that creates the process, here is the output:

/usr/bin/sh /tmp/run/launch.sh
  PID TTY          TIME CMD
29978 pts/0    00:00:01 zsh
 1178 pts/0    00:00:01 python
 1180 pts/0    00:00:00 sh <defunct>
 1181 pts/0    00:00:00 ps
None

Apparently the process is launched (even though <defunct>) and one should also note that I have a little problem passing the parameters in...

Thanks.

Answer 1

I've finally found the answer to my question, thanks to badp and his suggestions for debugging.

From the python page on the subprocess module:

The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable. The only reason you would need to specify shell=True on Windows is where the command you wish to execute is actually built in to the shell, eg dir, copy. You don’t need shell=True to run a batch file, nor to run a console-based executable.

Since I am on Linux and using shell=True, my command is in fact a list of arguments to be executed by executable, which defaults to /bin/sh. Thus the full command executed was: /bin/sh /usr/bin/sh /tmp/run/launch.sh... which did not work so well.

And I should have used either:

subprocess.Popen('/tmp/run/launch.sh', shell=True)

or

subprocess.Popen('/tmp/run/launch.sh', executable = '/usr/bin/sh', shell=True)

It's tricky that shell=True would actually modify the default executable value on Linux only...

Answer 2

Try this:

p = subprocess.Popen(command,
                     shell = True, #is this even needed?
                     stdin = subprocess.PIPE,
                     stdout = subprocess.PIPE,
                   # stderr = subprocess.STDOUT #uncomment if reqd
                    )

Tested working on Windows with the ping command. This lets you communicate, which might help you find out why the script isn't launched in the first place :)