Python's subprocess module returning different results from Unix shell

Question

I'm trying to get a list of the CSV files in a directory with python. This is really easy within unix:

ls -l *.csv

And, predictably, I get a list of the files that end with .csv in my directory. However, when I attempt the Python equivalent using the Subprocess module:

>>> import subprocess as sp
>>> sp.Popen(["ls", "-l", "*.csv"], stdout = sp.PIPE)
<subprocess.Popen object at 0xb780e90c>
>>> ls: cannot access *.csv: No such file or directory

Can somebody please explain what's going on?

Edit: Adding shell = True removes the error, but instead of getting a list of just CSV files, I get a list of all the files in the directory.

Answer 1

If you want it to behave as it does at the shell, you need to pass shell=True (your mileage may vary here, depending on your system and shell). In your case the problem is that when you do ls -l *.csv, the shell is evaluating what * means, not ls. (ls is merely formatting your results, but the shell has done the heavy lifting to determine what files match *.csv). Subprocess makes ls treat *.csv literally, and look for a file with that specific name, which of course there aren't any (since that's a pretty hard filename to create).

What you really should be doing is using os.listdir and filtering the names yourself.

Answer 2

Why not use glob instead? It's going to be faster than "shelling out"!

import glob
glob.glob('*.csv')

This gives you just the names, not all the extra info ls -l supplies, though you can get extra info with os.stat calls on files of interest.

If you really must use ls -l, I think you want to pass it as a string for the shell to do the needed star-expansion:

proc = sp.Popen('ls -l *.csv', shell=True, stdout=sp.PIPE)