Sending binary data over sockets with Python

Question

I'm looking to script some basic requests over the SPDY protocol. The protocol defines the frames you send as being comprised of binary data of very specific length and byte order.

I've only ever written small programs that send strings across sockets (HTTP). How do I go about implementing a SPDY control frame header for example? I've tried using the bitstring library and numpy to control the size of all the different sections of a control frame header for example but nothing is really working. The current SPDY library for python uses cython and C data types and i've found it to be very unpredictable. I was wondering how I can go about building simple requests with pure python or very simply how do I go about building something precisely like the protocol defines and sending it across a socket?

Answer 1

Generally bytearray class will be your friend (if I understand your question correctly). You can send it via socket:

my_bytes = bytearray()
my_bytes.append(123)
my_bytes.append(125)

// my_bytes is b'{}' now

s.send(my_bytes)

Follow the protocol specification and create byte after byte. This also works when you receive data:

data = s.recv(2048)
my_bytes = bytearray(data)

I don't know much about SPDY protocol but for example the control bit is the first bit (not byte) in the message. You can retrieve it from my_bytes via binary AND for example:

control_frame = my_bytes[0] & 128

this is because 128 is 10000000 in binary and thus binary AND will give you the first bit only (remember that each byte has 8 bits that's why we have 7 zeros).

That's how things are done manually. Of course I suggest using some library because writing a proper protocol handler will take lots of time, you may find it quite difficult and it might not be efficient (depending on your needs of course).

Answer 2

You can also use the struct module to define the header format with a string and parse it directly.

To generate a packet:

fmt = 'B I 4b'
your_binary_data = pack(fmt, header_data)
sock.sendall(your_binary_data)

Where fmt indicates the header format ('B I 4b' is just a, clearly not working for your SPDY header, example). Unfortunately, you will have to deal with non-byte-alligned header fields, probably by parsing bigger chunks and then dividing them according to your format.

Aside that, to parse the header:

unpacker = struct.Struct('B I 4b')
unpacked_data = unpacker.unpack(s.recv(unpacker.size))

unpacked_data will contain a tuple with the parsed data.

The struct module performs conversions between Python values and C structs represented as Python strings. I have no guarantees about the efficiency of this approach, but it helped me to parse different protocols just by adjusting the fmt string.