Sending live video frame over network in python opencv

Question

I'm trying to send live video frame that I catch with my camera to a server and process them. I'm usig opencv for image processing and python for the language. Here is my code

client_cv.py

import cv2 import numpy as np import socket import sys import pickle cap=cv2.VideoCapture(0) clientsocket=socket.socket(socket.AF_INET,socket.SOCK_STREAM) clientsocket.connect(('localhost',8089)) while True:     ret,frame=cap.read()     print sys.getsizeof(frame)     print frame     clientsocket.send(pickle.dumps(frame)) 

server_cv.py

import socket import sys import cv2 import pickle import numpy as np HOST='' PORT=8089  s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) print 'Socket created'  s.bind((HOST,PORT)) print 'Socket bind complete' s.listen(10) print 'Socket now listening'  conn,addr=s.accept()  while True:     data=conn.recv(80)     print sys.getsizeof(data)     frame=pickle.loads(data)     print frame     cv2.imshow('frame',frame) 

This code gives me end of file error, which is logical because the data always keep coming to the server and pickle doesn't know when to finish. My search on the internet made me use pickle but it doesn't work so far.

Note: I set conn.recv to 80 because that's the number I get when I say print sys.getsizeof(frame).

Answer 1

Few things:

• use sendall instead of send since you're not guaranteed everything will be sent in one go
• pickle is ok for data serialization but you have to make a protocol of you own for the messages you exchange between the client and the server, this way you can know in advance the amount of data to read for unpickling (see below)
• for recv you will get better performance if you receive big chunks, so replace 80 by 4096 or even more
• beware of sys.getsizeof: it returns the size of the object in memory, which is not the same as the size (length) of the bytes to send over the network ; for a Python string the two values are not the same at all
• be mindful of the size of the frame you are sending. Code below supports a frame up to 65535. Change "H" to "L" if you have a larger frame.

A protocol example:

client_cv.py

import cv2 import numpy as np import socket import sys import pickle import struct ### new code cap=cv2.VideoCapture(0) clientsocket=socket.socket(socket.AF_INET,socket.SOCK_STREAM) clientsocket.connect(('localhost',8089)) while True:     ret,frame=cap.read()     data = pickle.dumps(frame) ### new code     clientsocket.sendall(struct.pack("H", len(data))+data) ### new code 

server_cv.py

import socket import sys import cv2 import pickle import numpy as np import struct ## new  HOST='' PORT=8089  s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) print 'Socket created'  s.bind((HOST,PORT)) print 'Socket bind complete' s.listen(10) print 'Socket now listening'  conn,addr=s.accept()  ### new data = "" payload_size = struct.calcsize("H")  while True:     while len(data) < payload_size:         data += conn.recv(4096)     packed_msg_size = data[:payload_size]     data = data[payload_size:]     msg_size = struct.unpack("H", packed_msg_size)[0]     while len(data) < msg_size:         data += conn.recv(4096)     frame_data = data[:msg_size]     data = data[msg_size:]     ###      frame=pickle.loads(frame_data)     print frame     cv2.imshow('frame',frame) 

You can probably optimize all this a lot (less copying, using the buffer interface, etc) but at least you can get the idea.

Answer 2

After months of searching the internet, this is what I came up with, I have neatly packaged it into classes, with unit tests and documentation as SmoothStream check it out, it was the only simple and working version of streaming I could find anywhere.

I used this code and wrapped mine around it.

Viewer.py

import cv2 import zmq import base64 import numpy as np  context = zmq.Context() footage_socket = context.socket(zmq.SUB) footage_socket.bind('tcp://*:5555') footage_socket.setsockopt_string(zmq.SUBSCRIBE, np.unicode(''))  while True:     try:         frame = footage_socket.recv_string()         img = base64.b64decode(frame)         npimg = np.fromstring(img, dtype=np.uint8)         source = cv2.imdecode(npimg, 1)         cv2.imshow("Stream", source)         cv2.waitKey(1)      except KeyboardInterrupt:         cv2.destroyAllWindows()         break 

Streamer.py

import base64 import cv2 import zmq  context = zmq.Context() footage_socket = context.socket(zmq.PUB) footage_socket.connect('tcp://localhost:5555')  camera = cv2.VideoCapture(0)  # init the camera  while True:     try:         grabbed, frame = camera.read()  # grab the current frame         frame = cv2.resize(frame, (640, 480))  # resize the frame         encoded, buffer = cv2.imencode('.jpg', frame)         jpg_as_text = base64.b64encode(buffer)         footage_socket.send(jpg_as_text)      except KeyboardInterrupt:         camera.release()         cv2.destroyAllWindows()         break