What to send over serial ports

Question

I'm sorry if this is a very vague question, but I can't seem to formulate it properly to find anyone else with this issue. The main question is, once you have a serial connection established between two devices, how do you use that connection to implement two way communication?

An example might help. Suppose you have a temperature sensor as an embedded device using a microcontroller and firmware written in C. You have a serial port connection from that sensor to a computer and some software on the computer to interface with it, say a C++ application. I understand how to setup the serial ports on both sides and read and write single bytes of data between the two devices. The real question is what convention do you use to communicate between the two devices?

Assume your requirements are as follows:

1.) You need to be able to send a command to take a single temperature reading from the embedded device and send it to the computer for display purposes.

2.) You need to send commands to make the sensor start and stop streaming temperature values.

3.) You need a set of commands to set various aspects in the firmware such as streaming rate, stream on startup, blink leds, etc.

4.) You need some sort of structure to send complicated forms of data to the computer, perhaps an array of battery voltage readings.

Ways to Accomplish This

It seems there are a couple ways people tend to do this:

Simple String API:

The most common, from dealing with third party sensors, seems use a simple string based API, such that the commands for starting and stopping streams might be "SS,1\r" and "SS,0\r" respectively. In this case you would have to read from the serial port until you get the "\r" character and then parse the data you got to see if it has a command (left of comma) and parameters (right of comma). This works for scenarios 1 to 3 above but doesn't make scenario 4 very easy.

JSON String API:

This works the same as the one above but instead of passing your parameters as simple values, you pass JSON objects that can represent complicated structures of data. Hence you could send the array of battery voltages as a JSON array. This method seems to cover all use cases 1-4 above. But JSON sends strings and it is harder to parse using embedded c. It would work wonders for the computer side which could use a higher level language such as Java that has libraries for reading JSON data.

Packet Style API:

This is the solution we accepted that I am now kind of regretting. It involves sending a structured packet convention of bytes for each piece of data we send. The packet structure is shown below.

[0xFF][0xFF][ID][CMD][D0][D1][D2][D3][D4][D5][D6][D7][0xEE][0xEE][0xEE]

With this structure we send a header and footer (0xFF's and 0xEE's) for validating packet completeness, an id for sending sequential packets (for transmitting an array of data), a data array which we can use to pack longs, floats, ints, etc, and a command byte (CMD) which can be used by the device to determine how to parse the data payload (D0-D7).

So I ask, what is the most preferred way to communicate over a serial port? Is there any other ways I am missing? I have been doing a lot of web development lately and it seems JSON is a nice abstract transmission system, but has its limitations because you have to do a lot more string parsing, which is a bit complicated on the firmware side.

Answer 1

The biggest problem here is, that there is a lack of standards. Almost everyone implements their own protocol for constraint devices or when it comes to low level transports like UART / Serial Line. But with a growing IoT trend this hopefully starts to change.

Please have a look at SLIP and SLIPMUX to send packet oriented protocols over serial line.

No matter if you send JSON, XML or anything else, you most likely need some stop flag to terminate your messages. Often \n is used for sake of readability in combination with ASCII encoded content, but that's not the best for binary encoded machine 2 machine communication.

SlipMux is backwards compatible to SLIP (for IPv4 and IPv6 packets) but also supports new message types like CoAP and Diagnostic (human readable) messages. You can simply implement your own packet types like "JSON" or "XML" messages.

As a protocol I can recommend CoAP in combination with SlipMux. CoAP is very similar to HTTP but much more lightweight and thus easy to deal with for most developers.

• CoAP: https://www.rfc-editor.org/rfc/rfc7252
• Slip: https://www.rfc-editor.org/rfc/rfc1055
• SlipMux: https://datatracker.ietf.org/doc/html/draft-bormann-t2trg-slipmux-00

Answer 2

The binary packet format is attractive because of its apparent simplicity and efficiency. Furthermore, some coomunication links are already sending data as packets of fixed size (USB, TCP/IP, file systems, etc.) but, if I had to do it differently, I would not go that way again because of the following drawbacks:

• not human readable (unless you want to obfuscate things and protect your protocol :-) )
• computer architecture dependant: think about endianness problems
• toolchain dependancies: ex., how compiler X packs data structures versus compiler Y?
• link dependencies: if the link changes, what if there is an odd fit between protocol packet size and the new link packet size?

JSON would be my way now: ASCII data transmissions are less efficient than binary transmissions, but its convenience and portability offset that, unless the application has severe bandwidth limitation. I personnally wrote a JSON parser for Arduino Uno board and it works in less than 2 kb of data RAM, Including system stack. OK, it may choke on deeply nested JSON transmissions but it was good enough to strip key elements from a Twitter tweet packet, and proved that it could work on tiny systems.

Yves McDonald