Is a checksum needed for application data of a custom protocol?

Question

Since packets travel over the wire have checksums on different layers, Ethernet and IPv4 have checksums for their headers, TCP's checksum even covers the entire segment.

I know it is not impossible that a corrupted packet, from the standpoint of the application layer, can slip in without being discarded by Ethernet/IP/TCP, because there are chances that their checksums are correct, only the probability is low.

I am designing a custom binary protocol for an IM application. My question is do I need to add a checksum to ensure the integrity of my application data? Is a checksum really needed in practice?

Answer 1

There's actual research on this subject. It's old, but very relevant to the question at hand.

The paper, from 2000, is called "When the CRC and TCP checksum disagree" by Jonathan Stone and Craig Partridge, which investigate packet and frame errors, and look how often the TCP checksum is wrong, but the Ethernet CRC is fine. You can find the PDF here. Here are the important bits.

From the abstract:

Traces of Internet packets from the past two years show that between 1 packet in 1,100 and 1 packet in 32,000 fails the TCP checksum, even on links where link-level CRCs should catch all but 1 in 4 billion errors.

From the conclusion (with some of my highlighting)

In practice, the checksum is being asked to detect an error every few thousand packets. After eliminating those errors that the checksum always catches, the data suggests that, on average, between one packet in 10 billion and one packet in a few millions will have an error that goes undetected. The exact range depends on the type of data transferred and the path being traversed. While these odds seem large, they do not encourage complacency. In every trace, one or two 'bad apple' hosts or paths are responsible for a huge proportion of the errors. For applications which stumble across one of the `bad-apple' hosts, the expected time until a corrupted data is accepted could be as low as a few minutes. When compared to undetected error rates for local I/O (e.g., disk drives), these rates are disturbing. Our conclusion is that vital applications should strongly consider augmenting the TCP checksum with an application sum.

I don't know of any newer research into that question (enlighten me if you know otherwise!), so the Internet could have become more reliable since then, and the numbers in the paper might be irrelevant.

However, and this is important, 17 years have passed, and the amount of Internet traffic simply exploded since that paper was written. At 1Gbps, which is not an uncommon connection speed nowadays, you're sending about 81K full TCP segments, with 1460 bytes of data, per second (or a lot more if the packets are smaller). That's a million big packets every 12.5 seconds, a billion in about 3.5 hours (or again, a lot more if the packets are small).

So to answer your question - that depends. For transferring large files or other data, I'd definitely add additional checks if the data itself isn't protected in any way. For messaging, which pushes very little data into the network, you'll probably be fine with TCP's checksum, with maybe some sanity checks on the input you're getting to make sure that it's in the correct format, and various parameters and fields make sense.

Answer 2

I would not bother with a checksum because of packets getting corrupted in the network.

However, since you are working on a protocol that would presumably be used on the open internet, you will need to prepare for rare cases of an unintended application sending udp packets or making tcp connections to your receiving/listening ports. Also there will be maybe less port scans and hackers / script kiddies knocking on your gates.

So you should make your protocol such that it is easy to discard this kind of traffic. Using a checksum in every transmission would imho be one sensible way of doing that.