Sending data chunks over named pipe in linux

Question

I want to send data blocks over named pipe and want receiver to know where data blocks end. How should I do it with named pipes? Should I use some format for joining and splitting blocks (treat pipe always as stream of bytes) or are there any other method?

I've tried opening and closing pipe at sender for every data block but data becomes concatenated at receiver side (EOF not send):

for _ in range(2):
     with open('myfifo', 'bw') as f:
         f.write(b'+')

Result:

rsk@fe temp $ cat myfifo 
++rsk@fe temp $

Answer 1

You can either use some sort of delimiter or a frame structure over your pipes, or (preferably) use multiprocessing.Pipe like objects and run Pickled Python objects through them.

The first option is simply defining a simple protocol you will be running through your pipe. Add a header to each chunk of data you send so that you know what to do with it. For instance, use a length-value system:

import struct

def send_data(file_descriptor, data):
    length = struct.pack('>L', len(data))
    packet = "%s%s" % (length, data)
    file_descriptor.write(packet)

def read_data(file_descriptor):
    binary_length = file_descriptor.read(4)
    length = struct.unpack('>L', binary_length)[0]

    data = ''
    while len(data) < length:
        data += file_descriptor.read(length - len(data))

As for the other option - You can try reading the code of the multiprocessing module, but essentially, you just run through the pipe the result of cPickle.dumps and then you read it into cPickle.loads to get Python objects.