python StringIO doesn't work as file with subrpocess.call()

Question

I'm working with subprocess package to call some external console commands from a python script, and I need to pass file handlers to it to get stdout and stderr back separately. The code looks like this roughly:

import subprocess

stdout_file = file(os.path.join(local_path, 'stdout.txt'), 'w+')
stderr_file = file(os.path.join(local_path, 'stderr.txt'), 'w+')

subprocess.call(["somecommand", "someparam"], stdout=stdout_file, stderr=stderr_file)

This works fine and txt files with relevant output are getting created. Yet it would be nicer to handle these outputs in memory omitting files creation. So I used StringIO package to handle it this way:

import subprocess
import StringIO

stdout_file = StringIO.StringIO()
stderr_file = StringIO.StringIO()

subprocess.call(["somecommand", "someparam"], stdout=stdout_file, stderr=stderr_file)

But this doesn't work. Fails with:

  File "./test.py", line 17, in <module>
    subprocess.call(["somecommand", "someparam"], stdout=stdout_file, stderr=stderr_file)
  File "/usr/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python2.7/subprocess.py", line 1063, in _get_handles
    c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'

I see that it's missing some parts of the native file object and fails because of that.

So the question is more educational than practical - why these parts of file interface are missing from StringIO and are there any reasons why this cannot be implemented?

Answer 1

As you said in your comment, Popen and Popen.communicate are the right solution here.

A bit of background: real file objects have file descriptors, which is the fileno attribute StringIO objects are missing. They're just ordinary integers: you may be familiar with file descriptors 0, 1 and 2, which are stdin, stdout and stderr, respectively. If a process opens more files, they're assigned 3, 4, 5, etc.. You can take a look at a process's current file descriptors with lsof -p.

So, why can't StringIO objects have file descriptors? In order to get one, it'd need to either open a file or open a pipe*. Opening a file wouldn't make sense, since not opening files is the whole point of using StringIO in the first place.

And opening a pipe also wouldn't make sense, even though they live in memory like StringIO objects do. They're for communication, not storage: seek, truncate, and len have no meaning at all for pipes, and read and write behave very differently than they do for files. When you read from a pipe, the returned data is deleted from the pipe's buffer, and if that (relatively small) buffer is full when you try to write, your process will hang until something reads from the pipe to free up buffer space.

So if you want to use a string as stdin, stdout or stderr for a subprocess, StringIO won't cut it but Popen.communicate is perfect. As stated above (and warned about in subprocess's docs), reading from and writing to pipes correctly is complicated. Popen handles that complexity for you.

* I guess I could theoretically imagine a third kind of file descriptor corresponding to a memory region shared between processes? Not really sure why that doesn't exist. But eh, I'm not a kernel developer, so I'm sure there's a reason.