subprocess.call() arguments ignored when using shell=True w/ list [duplicate]

Question

I am trying to get python's subprocess.call method to accept some args commands through a list (consisting of a sequence of strings) as advised in the python documentation. To explore this behavior before putting it into my actual script, I opened up IPython, ran some commands involving different combinations of shell settings and args commands and got the following behavior:

In [3]: subprocess.call(['ls', '-%sl' %'a'])
total 320
drwxr-xr-x  20 Kohaugustine  staff   680 Oct 15 16:55 .
drwxr-xr-x   5 Kohaugustine  staff   170 Sep 12 17:16 ..
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 a.out
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct  3 10:28 ex1-6
-rw-r--r--@  1 Kohaugustine  staff   204 Oct  3 10:28 ex1-6.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Oct  3 10:15 ex1-7
-rw-r--r--@  1 Kohaugustine  staff    71 Oct  3 10:15 ex1-7.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:22 hello
-rw-r--r--@  1 Kohaugustine  staff    58 Sep 12 16:27 hello.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@  1 Kohaugustine  staff   185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 21 10:06 temperature.o
-rw-r--r--@  1 Kohaugustine  staff   406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@  1 Kohaugustine  staff   582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@  1 Kohaugustine  staff   178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 23 17:21 temperature_ex1-5.o
Out[3]: 0

In [4]: subprocess.call(['ls', '-%sl' %'a'], shell=True)
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[4]: 0

In [6]: subprocess.call(['ls', '-al'])    
total 320
drwxr-xr-x  20 Kohaugustine  staff   680 Oct 15 16:55 .
drwxr-xr-x   5 Kohaugustine  staff   170 Sep 12 17:16 ..
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 a.out
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct  3 10:28 ex1-6
-rw-r--r--@  1 Kohaugustine  staff   204 Oct  3 10:28 ex1-6.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Oct  3 10:15 ex1-7
-rw-r--r--@  1 Kohaugustine  staff    71 Oct  3 10:15 ex1-7.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:22 hello
-rw-r--r--@  1 Kohaugustine  staff    58 Sep 12 16:27 hello.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@  1 Kohaugustine  staff   185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 21 10:06 temperature.o
-rw-r--r--@  1 Kohaugustine  staff   406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@  1 Kohaugustine  staff   582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@  1 Kohaugustine  staff   178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 23 17:21 temperature_ex1-5.o
Out[6]: 0

In [7]: subprocess.call(['ls', '-al'], shell = True)
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[7]: 0

It seems like whenever shell=True, the output seems to be the same as:

In [9]: subprocess.call(['ls'])
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[9]: 0

I'm puzzled; what happened to the '-a' option when I set shell=True? Didn't the shell read it? I've read the Docs and that it says that when shell=True, it means that my specified command will be executed through the shell, so it should mean that ls -a was fed to the shell and acted upon by the shell. Then why the behavior in [4] and [7] ? Also the pydocs doesn't explain it directly (although it does say what subpprocess will NOT DO when we set shell=False); what does it mean when we let shell=False? Is a new process spawned in the OS without having the shell actually control it?

Also, in case it might seem really awkward that I'm using a format string in [3] and [4], its because in my actual script where I'll be using subprocess.call, I will have to rely on these format strings to substitute in the appropriate command options. I cannot hardcode some of the command line options. Using a pure string for args is out of the question too because in my script there will be a method that has to do list operations on the commands. I don't know if there might be a better way to go about this though, so if will really help if anyone can suggest something different.

Thank you very much!

Answer 1

When shell is True, the first argument is appended to ["/bin/sh", "-c"]. If that argument is a list, the resulting list is

["/bin/sh", "-c", "ls", "-al"]

That is, only ls, not ls -al is used as the argument to the -c option. -al is used as the first argument the shell itself, not ls.

When using shell=True, you generally just want to pass a single string and let the shell split it according the shell's normal word-splitting rules.

# Produces ["/bin/sh", "-c", "ls -al"]
subprocess.call("ls -al", shell=True)

In your case, it doesn't see like you need to use shell=True at all.