Can I pipe a io.BytesIO() stream to subprocess.popen() in Python?

Question

I'm trying to pipe a io.BytesIO() bytetream to a separate program using subprocess.popen(), but I don't know how or if this is at all possible. Documentation and examples are all about text and newlines.

When I whip up something like this:

import io
from subprocess import *

stream = io.BytesIO()
someStreamCreatingProcess(stream)

command = ['somecommand', 'some', 'arguments']  
process = Popen(command, stdin=PIPE)
process.communicate(input=stream)

I get

Traceback (most recent call last):
  File "./test.py", line 9, in <module>
    procOut         = process.communicate(input=stream)
  File "/usr/lib/python2.7/subprocess.py", line 754, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1322, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib/python2.7/subprocess.py", line 1384, in _communicate_with_poll
    chunk = input[input_offset : input_offset + _PIPE_BUF]
TypeError: '_io.BytesIO' object has no attribute '__getitem__'

I think popen() is only for text. Am I wrong?
Is there a different way to do this?

Answer 1

As @falsetru said you can't stream BytesIO() object directly; you need to get a bytestring from it first. It implies that all content should be already written to stream before you call stream.getvalue() to pass to process.communicate().

If you want to stream instead of providing all input at once then you could drop BytesIO() object and write to the pipe directly:

from subprocess import Popen, PIPE

process = Popen(['command', 'arg1'], stdin=PIPE, bufsize=-1)
someStreamCreatingProcess(stream=process.stdin) # many `stream.write()` inside
process.stdin.close() # done (no more input)
process.wait()

someStreamCreatingProcess() should not return until it is done writing to the stream. If it returns immediately then it should call stream.close() at some point in the future (remove process.stdin.close() in your code):

from subprocess import Popen, PIPE

process = Popen(['command', 'arg1'], stdin=PIPE, bufsize=-1)
someStreamCreatingProcess(stream=process.stdin) # many `stream.write()` inside
process.wait() # stream.close() is called in `someStreamCreatingProcess`

Answer 2

According to subprocess.Popen.communicate:

The optional input argument should be a string to be sent to the child process, or None, if no data should be sent to the child.

---

To get (bytes) string value from BytesIO object, use getvalue:

process.communicate(input=stream.getvalue())