Idiom for socket receive in Python

Question

I have some experience with socket programming using the Berkeley socket API in C. Generally, any socket programming requires a strategy that enables the receiving socket to know how much data it should receive. This can be accomplished with either header length fields, or delimiter characters. Generally, I prefer a header field which contains the length.

Of course, we also need to know the size of the length header field itself, which is simply a fixed size value that must be agreed upon by both sender and receiver. In C, this is easy to implement because native integer types are fixed size and in binary format, so you can just say something like:

uint16_t bytes_to_receive;
recv(sock, &bytes_to_receive, sizeof(bytes_to_receive), 0);
bytes_to_receive = ntohs(bytes_to_receive);
// Now receive 'bytes_to_receive' bytes...

But how is this sort of idiom accomplished using Python sockets? In Python, integers are objects and pickled integers are variable length byte arrays. So we can't use a pickled integer as a length header field, because we can't be sure of its size in bytes.

Of course, I could always send a byte array of known size containing a binary integer, like b'\x05\x00' to create a 16-bit binary integer with a value of 5 in little endian format, but this really doesn't seem like the right approach.

So, how is this usually accomplished in Python?

Answer 1

You can use the struct module convert Python integers to and from strings/byte arrays. Just read the number of bytes that correspond to the size of the type header and convert it with the struct module and you should be good to go. (note: be sure to use the right endian-flags when encoding/decoding)