How can I reliably read exactly n bytes from a TCP socket?

Question

Context:

It is common that a binary protocol defines frames of a given size. The struct module is good at parsing that, provided everything has been received in a single buffer.

Problem:

TCP sockets are streams. A read from a socket cannot give more bytes than requested but can return less. So this code is not reliable:

def readnbytes(sock, n):
    return sock.recv(n)   # can return less than n bytes

The naive workaround:

def readnbytes(sock, n):
    buff = b''
    while n > 0:
        b = sock.recv(n)
        buff += b
        if len(b) == 0:
            raise EOFError          # peer socket has received a SH_WR shutdown
        n -= len(b)
    return buff

may not be efficient, because if we ask a large number of bytes, and the data if very fragmented, we will repeatedly re-allocate a new byte buffer.

Question:

How is it possible to reliably receive exactly n bytes from a stream socket with no risk of re-allocation?

References:

Those other questions are related, and do give hints, but none give a simple and clear answer:

• How to receive certain size of data in socket programming?
• How do I use ctypes to recv_into a C buffer multiple times?

Answer 1

You can use socket.makefile() to wrap the socket in a file-like object. Then reads will return exactly the amount requested, unless the socket is closed where it can return the remainder. Here's an example:

server.py

from socket import *

sock = socket()
sock.bind(('',5000))
sock.listen(1)
with sock:
    client,addr = sock.accept()
    with client, client.makefile() as clientfile:
        while True:
            data = clientfile.read(5)
            if not data: break
            print(data)

client.py

from socket import *
import time

sock = socket()
sock.connect(('localhost',5000))
with sock:
    sock.sendall(b'123')
    time.sleep(.5)
    sock.sendall(b'451234')
    time.sleep(.5)
    sock.sendall(b'51234')

Server Output

12345
12345
1234