Python unpacking binary stream from tcp socket

Question

ok, So I thought it would be a good idea to get familiar with Python. (I have had experience with Java, php, perl, VB, etc. not a master of any, but intermediate knowledge)

so I am attempting to write a script that will take a the data from a socket, and translate it to the screen. rough beginning code to follow:

my code seems to correctly read the binary info from the socket, but I can't unpack it since I don't have access to the original structure.

I have the output for this stream with a different program, (which is terribly written which is why I am tackling this)

when I do print out the recv, it's like this...

b'L\x00k\x07vQ\n\x01\xffh\x00\x04NGIN\x04MAIN6Product XX finished reprocessing cdc XXXXX at jesadr 0c\x00k\x07vQ\n\x01\xffF\x00\x06CSSPRD\x0cliab_checkerCCheckpointed to XXXXXXXXXXXXXXXX:XXXXXXX.XXX at jesadr 0 (serial 0)[\x00l\x07vQ\n\x00\xff\x01\x00\x05MLIFE\x06dayendBdayend 1 Copyright XXXX XXXXXXX XXXXXXX XXXXX XXX XXXXXX XXXXXXXX.

from looking at this, and comparing it to the output of the other program, I would surmise that it should be broken up like..

b'L\x00k\x07vQ\n\x01\xffh\x00\x04NGIN\x04MAIN6Product XX finished reprocessing cdc XXXXX at jesadr 0'

with corresponding info

04-23
00:00:43
10
1
NGIN
MAIN
255
104
Product XX finished reprocessing cdc XXXXX at jesadr 0

Now, based on my research, it looks like I need to use the "struct" and unpack it, however I have no idea of the original structure of this, I only know what info is available from it, and to be honest, I'm having a hell of a time figuring this out.

I have used the python interpreter to attempt to unpack bits and pieces of the line, however it is an exercise in frustration.

If anyone can at least help me get started, I would very much appreciate it.

Thanks

Answer 1

Okay. I think I've managed to decode it, although I'm not sure about the intermediate 16-bit value.

This Python 2.7 code...

from cStringIO import StringIO
import struct
import time

def decode(f):

    def read_le16(f):
        return struct.unpack('<h', f.read(2))[0]

    def read_timestamp(f):
        ts = struct.unpack('<l', f.read(4))[0]
        return time.ctime(ts)

    def read_byte(f):
        return ord(f.read(1))

    def read_pascal(f):
        l = ord(f.read(1))
        return f.read(l)

    result = []

    # Read total length
    result.append('Total message length is %d bytes' % read_le16(f))

    # Read timestamp
    result.append(read_timestamp(f))

    # Read 3 x byte
    result.append(read_byte(f))
    result.append(read_byte(f))
    result.append(read_byte(f))

    # Read 1 x LE16
    result.append(read_le16(f))

    # Read 3 x pascal string
    result.append(read_pascal(f))
    result.append(read_pascal(f))
    result.append(read_pascal(f))

    return result

s = 'L\x00k\x07vQ\n\x01\xffh\x00\x04NGIN\x04MAIN6Product XX finished reprocessing cdc XXXXX at jesadr 0c\x00k\x07vQ\n\x01\xffF\x00\x06CSSPRD\x0cliab_checkerCCheckpointed to XXXXXXXXXXXXXXXX:XXXXXXX.XXX at jesadr 0 (serial 0)[\x00l\x07vQ\n\x00\xff\x01\x00\x05MLIFE\x06dayendBdayend 1 Copyright XXXX XXXXXXX XXXXXXX XXXXX XXX XXXXXX XXXXXXXX.'

f = StringIO(s)
print decode(f)
print decode(f)
print decode(f)

...yields...

['Total message length is 76 bytes', 'Tue Apr 23 05:00:43 2013', 10, 1, 255, 104, 'NGIN', 'MAIN', 'Product XX finished reprocessing cdc XXXXX at jesadr 0']
['Total message length is 99 bytes', 'Tue Apr 23 05:00:43 2013', 10, 1, 255, 70, 'CSSPRD', 'liab_checker', 'Checkpointed to XXXXXXXXXXXXXXXX:XXXXXXX.XXX at jesadr 0 (serial 0)']
['Total message length is 91 bytes', 'Tue Apr 23 05:00:44 2013', 10, 0, 255, 1, 'MLIFE', 'dayend', 'dayend 1 Copyright XXXX XXXXXXX XXXXXXX XXXXX XXX XXXXXX XXXXXXXX.']

The timestamps are out by 5 hours, so I'm assuming it's a timezone thing.