Passing Variables to Subprocess.Popen

Question

I have a script which calls another Python script by subprocess.Popen. But since I have arguments stored in variable(s)

servers[server]['address']
servers[server]['port']
servers[server]['pass']

I am unable to perform the command

p = subprocess.Popen(
    ["python mytool.py -a ", servers[server]['address'], "-x", servers[server]['port'], "-p", servers[server]['pass'], "some additional command"],
    shell=True,
    stdout=subprocess.PIPE
)

Answer 1

Drop shell=True. The arguments to Popen() are treated differently on Unix if shell=True:

import sys
from subprocess import Popen, PIPE

# populate list of arguments
args = ["mytool.py"]
for opt, optname in zip("-a -x -p".split(), "address port pass".split()):
    args.extend([opt, str(servers[server][optname])])
args.extend("some additional command".split())

# run script
p = Popen([sys.executable or 'python'] + args, stdout=PIPE)
# use p.stdout here...
p.stdout.close()
p.wait()

Note that passing shell=True for commands with external input is a security hazard, as described by a warning in the docs.

Answer 2

When you call subprocess.Popen you can pass either a string or a list for the command to be run. If you pass a list, the items should be split in a particular way.

In your case, you need to split it something like this:

command = ["python",  "mytool.py", "-a", servers[server]['address'], 
           "-x", servers[server]['port'], 
           "-p", servers[server]['pass'], 
           "some",  "additional", "command"]
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

This is because if you pass in a list, Popen assumes you have already split the command line into words (the values that would end up in sys.argv), so it doesn't need to.

The way you're calling it, it will try to run a binary called "python mytool.py -a", which isn't what you meant.

The other way to fix it is to join all of the words into a string (which Popen will then split up - see subprocess.list2cmdline). But you're better off using the list version if possible - it gives simpler control of how the commandline is split up (if arguments have spaces or quotes in them, for example) without having to mess around with quoting quote characters.

Answer 3

Your problem in type str for first Popen argument. Replace it to list. Below code can work:

address = servers[server]['address']
port = servers[server]['port']
pass = servers[server]['pass']

command = "python mytool.py -a %s -x %d -p %s some additional command" % (address, port, pass)
p = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
#        it is a list^^^^^^^^^^^^^^^  shell=False

If command arguments get from a trusted source you can construct command and use it with shell=True to such manner:

import pipes as p
command = "python mytool.py -a {} -x {} -p {} some additional command".format(p.quote(address), p.quote(port), p.quote(pass))
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

Note 1: constructed command with shell=Trueis potentially insecure. Use pipes.quote() for reduce injection possibility.
Note 2: pipes.quote() deprecated since python2; for python3 use shlex module.