howto program protocol buffer in python to send messages via socket

Question

I'm trying to use protocol buffer in python to send a message from one to another computer. I learned from a few online examples, and tried to make a minimum example out of them. However, in my code, the server does not print out the correct values of the variables the client sends. Any help is very much appreciated.

First, test_msg.proto is as follows:

message test_msg {
    required int32 param1 = 1;
    required int32 param2 = 2;
}

Second, client code (test_client.py)

from test_msg_pb2 import test_msg
import socket
import sys
import struct

address = ('localhost', 6005)
client_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client_socket.connect(address)
data = test_msg()

num_retransmits = 0
while(num_retransmits < 10): # send the same message 10 times
    num_retransmits = num_retransmits + 1

    data.param1 = 100
    data.param2 = 5
    s = data.SerializeToString()

    totallen = 4 + len(s) 
    pack1 = struct.pack('>I', totallen) # the first part of the message is length
    client_socket.sendall(pack1 + s)

    print "[client] param1: ", data.param1, " param2: ", data.param2

Third, the server code (test_server.py)

from test_msg_pb2 import test_msg
import socket
import sys
import struct

address = ('localhost', 6005)
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(address)

while(1):
    print "Listening"

    totallen = server_socket.recv(4)
    totallenRecv = struct.unpack('>I', totallen)[0]
    messagelen = totallenRecv - 4 
    message = server_socket.recv(messagelen)

    msg = test_msg()
    msg.ParseFromString(message)

    print "[server] param1:", msg.param1, "param2:", msg.param2

Then, after starting the server, I executing the client and it print out the following 10 lines as it send the param 10 times

[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5
[client] param1: 100 param2: 5

However, on the server side, it prints out

Listening
[server] param1: 0 param2: 0
Listening
[server] param1: 0 param2: 0
Listening
[server] param1: 0 param2: 0
Listening
[server] param1: 0 param2: 0
Listening
[server] param1: 0 param2: 0
Listening

So, the unpacked number param1 and param2 are incorrect, and it received the message only half of the times. Thanks so much for your help.

Answer 1

First advice: simplify. Lets first see if we can get this thing working without protocol buffers:

client.py

import socket
import sys
import struct

address = ('localhost', 6005)
client_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client_socket.connect(address)

messages = ["foobar", "barbaz", "bazquxfad", "Jimmy Carter"]

for s in messages:

    totallen = 4 + len(s) 
    pack1 = struct.pack('>I', totallen) # the first part of the message is length
    client_socket.sendall(s)

server.py

import socket
import sys
import struct

address = ('localhost', 6005)
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(address)

while True:
    print "Listening"

    # totallen = server_socket.recv(4)
    # totallenRecv = struct.unpack('>I', totallen)[0]
    # messagelen = totallenRecv - 4 
    message = server_socket.recv(1000)

    print message

Now, I'm not an expert on sockets (this might be the first question I've answered about them), but if you run this example, you get the expected output. What that tells me is that each send corresponds to a single .recv. Presumably, the sockets library is handling the details of the length. Of course, on the recieving side, it's quite possible that you want to know the length so that you can pass a proper maxlen. If this is the case, then I think you probably need to send 2 messages. The first message will have length 4 and be the integer length. The second message should have the data. e.g.:

client2.py

import socket
import sys
import struct

address = ('localhost', 6005)
client_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client_socket.connect(address)

messages = ["foobar", "barbaz", "bazquxfad", "Jimmy Carter"]

for s in messages:

    totallen = len(s) 
    pack1 = struct.pack('>I', totallen) # the first part of the message is length
    client_socket.sendall(pack1)
    client_socket.sendall(s)

server2.py

import socket
import sys
import struct

address = ('localhost', 6005)
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(address)

while True:
    print "Listening"

    totallen = server_socket.recv(4)
    totallenRecv = struct.unpack('>I', totallen)[0]

    message = server_socket.recv(totallenRecv)

    print message

Now the serialization should be pretty easy. After all, proto buffers serialize to strings just fine.

Answer 2

Protocol buffers allow you to define services for remote procedure calls (RPCs). This allows you to define the interface between server and client in the .proto file in a declarative, language agnostic manner. Once the services have been defined and you've compiled the protos, you can use an RPC library to abstract away all of the serialization/networking boiler-plate code. Take a look at gRPC, which is an open source version of Google's internal RPC library and provides a really clean way to define servers and clients.