What is the proper way of sending a large amount of data over sockets in Python?

Question

Recently I wrote some code (client and server) to send an image - the client simply uploads the image to the server, just using the socket module: Sending image over sockets (ONLY) in Python, image can not be open.

However, the image sending part is now what I am concerned with. This is the original image I'm using:

In my server code (which receives the images), I have these lines:

myfile = open(basename % imgcounter, 'wb') myfile.write(data)  data = sock.recv(40960000) if not data:      myfile.close()      break myfile.write(data) myfile.close()  sock.sendall("GOT IMAGE") sock.shutdown() 

But I don't think this is the best way of doing it. I think I should instead implement the server such that it receives the data in chunks:

#!/usr/bin/env python  import random import socket, select from time import gmtime, strftime from random import randint  imgcounter = 1 basename = "image%s.png"  HOST = '127.0.0.1' PORT = 2905  connected_clients_sockets = []  server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) server_socket.bind((HOST, PORT)) server_socket.listen(10)  connected_clients_sockets.append(server_socket)  while True:      read_sockets, write_sockets, error_sockets = select.select(connected_clients_sockets, [], [])      for sock in read_sockets:          if sock == server_socket:              sockfd, client_address = server_socket.accept()             connected_clients_sockets.append(sockfd)          else:             try:                  data = sock.recv(4096)                 txt = str(data)                  if data:                      if data.startswith('SIZE'):                         tmp = txt.split()                         size = int(tmp[1])                          print 'got size %s' % size                          sock.sendall("GOT SIZE")                      elif data.startswith('BYE'):                         sock.shutdown()                      else :                          myfile = open(basename % imgcounter, 'wb')                         myfile.write(data)                          amount_received = 0                         while amount_received < size:                             data = sock.recv(4096)                             amount_received += len(data)                              print amount_received                              if not data:                                 break                             myfile.write(data)                         myfile.close()                          sock.sendall("GOT IMAGE")                         sock.shutdown()             except:                 sock.close()                 connected_clients_sockets.remove(sock)                 continue         imgcounter += 1 server_socket.close() 

But when I do this, the server prints:

got size 54674 4096 8192 12288 16384 20480 24576 28672 32768 36864 40960 45056 49152 50578 

And then seems to hang, and the client hangs too. However, at the server's side I can see only a piece of the image the client wanted to send:

It seems like there are some bytes missing. What is the proper way of sending a huge amount of data (images, other type of file) using ONLY sockets?

Answer 1

I'm assuming that you have a particular reason for doing this with naked sockets, such as self-edification, which means that I won't answer by saying "You accidentally forgot to just use HTTP and Twisted", which perhaps you've heard before :-P. But really you should look at higher-level libraries at some point as they're a lot easier!

Define a protocol

If all you want is to send an image, then it can be simple:

1. Client -> server: 8 bytes: big endian, length of image.
2. Client -> server: length bytes: all image data.
3. (Client <- server: 1 byte, value 0: indicate transmission received - optional step you may not care if you're using TCP and just assume that it's reliable.)

Code it

server.py

import os from socket import * from struct import unpack   class ServerProtocol:      def __init__(self):         self.socket = None         self.output_dir = '.'         self.file_num = 1      def listen(self, server_ip, server_port):         self.socket = socket(AF_INET, SOCK_STREAM)         self.socket.bind((server_ip, server_port))         self.socket.listen(1)      def handle_images(self):          try:             while True:                 (connection, addr) = self.socket.accept()                 try:                     bs = connection.recv(8)                     (length,) = unpack('>Q', bs)                     data = b''                     while len(data) < length:                         # doing it in batches is generally better than trying                         # to do it all in one go, so I believe.                         to_read = length - len(data)                         data += connection.recv(                             4096 if to_read > 4096 else to_read)                      # send our 0 ack                     assert len(b'\00') == 1                     connection.sendall(b'\00')                 finally:                     connection.shutdown(SHUT_WR)                     connection.close()                  with open(os.path.join(                         self.output_dir, '%06d.jpg' % self.file_num), 'w'                 ) as fp:                     fp.write(data)                  self.file_num += 1         finally:             self.close()      def close(self):         self.socket.close()         self.socket = None          # could handle a bad ack here, but we'll assume it's fine.  if __name__ == '__main__':     sp = ServerProtocol()     sp.listen('127.0.0.1', 55555)     sp.handle_images() 

client.py

from socket import * from struct import pack   class ClientProtocol:      def __init__(self):         self.socket = None      def connect(self, server_ip, server_port):         self.socket = socket(AF_INET, SOCK_STREAM)         self.socket.connect((server_ip, server_port))      def close(self):         self.socket.shutdown(SHUT_WR)         self.socket.close()         self.socket = None      def send_image(self, image_data):          # use struct to make sure we have a consistent endianness on the length         length = pack('>Q', len(image_data))          # sendall to make sure it blocks if there's back-pressure on the socket         self.socket.sendall(length)         self.socket.sendall(image_data)          ack = self.socket.recv(1)          # could handle a bad ack here, but we'll assume it's fine.  if __name__ == '__main__':     cp = ClientProtocol()      image_data = None     with open('IMG_0077.jpg', 'r') as fp:         image_data = fp.read()      assert(len(image_data))     cp.connect('127.0.0.1', 55555)     cp.send_image(image_data)     cp.close() 

Answer 2

A simple way is to send data size as the first 4 bytes of your data and then read complete data in one shot. Use the below functions on both client and server-side to send and receive data.

def send_data(conn, data):     serialized_data = pickle.dumps(data)     conn.sendall(struct.pack('>I', len(serialized_data)))     conn.sendall(serialized_data)   def receive_data(conn):     data_size = struct.unpack('>I', conn.recv(4))[0]     received_payload = b""     reamining_payload_size = data_size     while reamining_payload_size != 0:         received_payload += conn.recv(reamining_payload_size)         reamining_payload_size = data_size - len(received_payload)     data = pickle.loads(received_payload)      return data 

you could find sample program at https://github.com/vijendra1125/Python-Socket-Programming.git