How to separate TCP socket messages

Question

I've experimented a bit with async TCP socket messages between two programs, for passing data, numbers and/or text. What I've done is to use a keyword in the start of each message, and then seperate the values with the "|" character. So a message may look like this:

"DATA|490|40517.9328222222|1|6|11345|11347|11344|11345|106|40517.8494212963"

I set the read buffer size to 1024, as most of the messages will be within that length. However sometimes I may send rapidly many short messages where several together are less than 1024 characters, and it seems then it will be read in one go. And if I send a message longer than 1024 characters, it will be split. So I'm looking for some advice on how to handle this. Should I use some special characters to start and/or end each message? Would appreciate some tips on how you do this..

Answer 1

The simplest way would be to send the message length at the beginning of each message, serialized in such a way that it will work on little-endian and big-endian hardware.

This could help your receiver preallocate its receive buffer efficiently too.

Answer 2

The easiest way would be to send the size of the message at the beginning of the packet. This way you'd know how much data to read. So it would look like:

00015MESSAGE|1|2 ...

It's important for the size field to have a fixed size.

You can also have this size field be binary, but it seems you are sending plain text so this way you'd have a humanly readable size field.

Answer 3

There are several approaches.

1. A length word prefixed to each message.

2. An STX/ETX-style wrapping of each message so you can see where it starts and finishes. This requires escaping of ETX bytes that occur in the data, and that in turn requires escaping of ESC bytes too.

3. A self-describing protocol, for example XML, or a type-length-value based protocol.