python 3.6 socket pickle data was truncated

Question

I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated

My server : I create a numpy array and I want to send in my client with pickle (it's work)

import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2


while(True):
    HOST = 'localhost'
    PORT = 50007
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
    s.bind((HOST, PORT))
    s.listen(1)
    conn, addr = s.accept()
    print ('Connected by', addr)

    arr = np.array([[0, 1], [2, 3]])
    printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
    img = np.array(printscreen_pil) ## Transform to Array
    
    data_string = pickle.dumps(img)
    conn.send(data_string)

    msg_recu = conn.recv(4096)
    print(msg_recu.decode())

    conn.close()

My client He has my numpy array, but I can not load with pickle. I have this error.

import socket, pickle
import numpy as np

HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)


while 1:
    data = s.recv(4096)
    if not data: break
    data_arr = pickle.loads(data)
    print (data_arr)
s.close()

Answer 1

the problem is that if the size of the pickled data is > 4096 you only get the first part of the pickled data (hence the pickle data was truncated message you're getting)

You have to append the data and pickle it only when the reception is complete, for example like this:

data = b""
while True:
    packet = s.recv(4096)
    if not packet: break
    data += packet

data_arr = pickle.loads(data)
print (data_arr)
s.close()

increasing a bytes object is not very performant, would be better to store the parts in a list of objects, then join, though. Faster variant:

data = []
while True:
    packet = s.recv(4096)
    if not packet: break
    data.append(packet)
data_arr = pickle.loads(b"".join(data))
print (data_arr)
s.close()

Answer 2

In the most Simple words. the file that you are trying to load is not complete. Either you have not downlaoded it correctly or it's just that your pickle file is corrupt. You can create a new pickle to solve this issue