Python on Windows: path as subprocess argument gets modified and generating error

Question

Am using subprocess on Windows and Python 2.6 as follows. I am trying to parse a text file using a legacy parser application (assume parser.py) as follows:

import subprocess
k = subprocess.Popen(['python', 'parser.py', '-f C:\Report1\2011-03-14.txt'],
                     shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print k.communicate()

The issue here is with the way filename gets passed to the legacy application where I cannot change the code but only can access it using Python.

It generates with the following error:

IOError: [Errno 22] invalid mode (\'r\') or filename: C:\\Report1\\2011-03-14.txt

When I copy the modified filename(with double forward slashes) from the traceback to check the existence, the system is not able to find it.

Question: How can I pass the path as argument so that it gets treated without getting changed to double slashes so that the system can read the file?

NOTE: os.sep also does not resolve the issue.

EDIT: Executing using os.system works perfectly, but the issue there is to grab the output for later use. Am currently using os.sytem in a module(run_parser.py) and then using subprocess in another module(get_parse_status.py) that Popens run_parser.py to grab the output. Would appreciate anything that is better than this.

Thanks for the time.

Answer 1

Change your parameter list to encode the path as a raw string:

k = subprocess.Popen(['python', 'parser.py', '-f', r'C:\Report1\2011-03-14.txt'],
                     shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

a simple program that reads a file and reports the length:

import sys
import os
userinput = sys.argv[1]
data = open(userinput, 'rb').read()
datalength = len(data)
fname = os.path.basename(userinput)
print "%s datasize = %s" % (fname, datalength)

Then to call it through the interpreter:

>>> k = subprocess.Popen(['python', 'test2.py', 'w:\bin\test2.py'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> k.communicate()
5: ('Traceback (most recent call last):\r\n  File "w:\\bin\\test2.py", line 4, in <module>
data = open(userinput, \'rb\').read()
IOError: [Errno 22] invalid mode (\'rb\') or filename: 'w:\\x08in\\test2.py', None)
>>> k = subprocess.Popen(['python', r'w:\bin\test2.py', r'w:\bin\test2.py'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> k.communicate()
6: ('test2.py datasize = 194\n', None)

Answer 2

"C:\Report1\2011-03-14.txt" isn't the same as the path C:\Report1\2011-03-14.txt. It's actually some bytestring, 'C:\\Report1\x811-03-14.txt'. Strangely enough it doesn't sound like this is your issue, but it might be related. r"C:\Report1\2011-03-14.txt" fixes this.

But be aware that double backslashes in the printed representation doesn't necessarily mean that there are actually two backslashes. '\\' is a Python string of length 1.