Pass binary data between python processes

Question

I need to transfer binary data between python processes on linux efficiently. However I can't find a way to pass it with pipe. The closest I got was to pass it as string but it keeps its binary form for some reason so it causes me problems.

ciphertext, tag = cipher1.encrypt_and_digest(input().encode())
print(ciphertext)

ciphertext is binary but when I test its type at the other process

print(type(sys.stdin.read()))

I get

<class 'str'>

Any ideas?

Answer 1

In Python 3 you can't use print or input for binary data. They're designed for handling text (Unicode), not binary. You can use file .write and .read calls to sys.stdout.buffer and sys.stdin.buffer, which are the underlying binary buffers to stdin and stdout. You can't use sys.stdout and sys.stdin since they're for text. There's a brief Note about this at the end of the docs for sys.stdin and sys.stdout.

Here's a short demo.

In "send_bytes.py" we create a bytes string data that contains all the possible byte values and write it to sys.stdout.buffer. We pipe that output to "get_bytes.py" where we read it and check to make sure it has all the right bytes in the right places.

send_bytes.py

#! /usr/bin/env python3

''' Write some binary data to stdout '''

import sys

# Make a bytes string containing all possible byte values
data = bytes(range(256))

#Send it as binary to stdout
out = sys.stdout.buffer
out.write(data)

get_bytes.py

#! /usr/bin/env python3

''' Read some binary data from stdin '''

import sys

#Read binary data from stdin
infile = sys.stdin.buffer
newdata = infile.read()

print(newdata)

# Make a bytes string containing all possible byte values
data = bytes(range(256))

#Check that the read data is correct
print(newdata == data)    

We run the programs using this command line:

$ python3 ./send_bytes0.py | python3 ./get_bytes0.py   

Here's the output printed by "get_bytes0.py"

b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'
True