Python subprocess call with whitespaces in arguments doesn't work on Windows

Question

I am running a java command which takes the classpath and other file locations which have a white space. Windows doesn't seem to like it.

I have the program running from C:\Program Files\Splunk , which has a white space

Here is my command

c1 = os.path.join(appdir, "bin", "apache-flume-1.3.1-bin", "lib", "*")
c2 = os.path.join(appdir, "bin", "apache-flume-1.3.1-bin", "lib", "flume-ng-node-1.3.1.jar")
c3 = os.path.join(appdir, "bin", "dtFlume.jar")

classpath = c1 + os.pathsep + c2 + os.pathsep + c3
log4j = os.path.join(appdir,"bin", "apache-flume-1.3.1-bin", "conf", "log4j.properties")
flumeconf = os.path.join(appdir,"bin","flume-conf.properties");

Variation 1 - doesn't work

cmdline = "java  -Xmx20m -Dlog4j.configuration=file:" + log4j + " -cp " + classpath + " org.apache.flume.node.Application -f " + flumeconf + " -n agent1"


try:
    p = subprocess.Popen("%s" %(cmdline),shell=True,stdout=subprocess.PIPE)

Variation 2 - doesn't work

cmdline_1 = "java  -Xmx20m -Dlog4j.configuration=file:"
cmdline_2 = log4j 
cmdline_3 = " -cp " 
cmdline_4 = classpath
cmdline_5 = " org.apache.flume.node.Application -f " 
cmdline_6 = flumeconf
cmdline_7 = " -n agent1"


try:
    p = subprocess.Popen('""%s" "%s" "%s" "%s" "%s" "%s" "%s""' %(cmdline_1,cmdline_2,cmdline_3,cmdline_4,cmdline_5,cmdline_6,cmdline_7),shell=True,stdout=subprocess.PIPE)

Variation 3 - doesn't work

cmdline_1 = "java  -Xmx20m -Dlog4j.configuration=file:"
cmdline_2 = log4j 
cmdline_3 = " -cp " 
cmdline_4 = classpath
cmdline_5 = " org.apache.flume.node.Application -f " 
cmdline_6 = flumeconf
cmdline_7 = " -n agent1"


try:
    p = subprocess.Popen("%s %s %s %s %s %s %s" %(cmdline_1,cmdline_2,cmdline_3,cmdline_4,cmdline_5,cmdline_6,cmdline_7),shell=True,stdout=subprocess.PIPE)

Every time it gives me the same/similar error, couldn't find main class, and the path is truncated at the C:\Program

"C:\Program Files\Splunk\etc\apps\APM_dynatrace\bin\runFlume.py"" Error: Could not find or load main class Files\Splunk\etc\apps\APM_dynatrace\bin\apache-flume-1.3.1-bin\conf\log4j.properties

Answer 1

Don't use shell=True. The docs state you only need shell=True on Windows if using a command actually built-into the shell:

The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.

If you aren't using shell=True, you can pass the arguments as a list, and not worry about spaces being treated incorrectly by the shell.

p = subprocess.Popen(['java', '-Xmx20m', '-Dlog4j.configuration=file:%s' % log4j,
                      '-cp', classpath, 'org.apache.flume.node.Application', 
                      '-f', flumeconf, '-n', 'agent1'], stdout=subprocess.PIPE)