udp file transfer project - is error checking necessary?

Question

I have been given the classical task of transferring files using UDP. On different resources, I have read both checking for errors on the packets (adding CRC alongside data to packets) is necessary AND UDP already checks for corrupted packets and discards them, so I only need to worry about resending dropped packets.

Which one of them is correct? Do I need to manually perform an integrity check on the arrived packets or incorrect ones are already discarded?

Language for the project is Java by the way.

EDIT: Some sources (course books, internet) say checksum only covers the header, therefore ensures sender and receiver IP's are correct etc.. Some sources say checksum also covers the data segment. Some sources say checksum may cover data segment BUT it's optional and decided by the OS.

EDIT 2: Asked my professors and they say UDP error checking on data segment is optional in IPv4, defauld in IPv6. But I still don't know if it's in programmer's control, or OS's, or another layer...

Answer 1

First fact:

UDP has a 16 bit checksum field starting at bit 40 of the packet header. This suffers from (at least) 2 weaknesses:

• Checksum is not mandatory, all bits set to 0 are defined as "No checksum"
• it is a 16 bit check-sum in the strict sense of the word, so it is susceptible to undetected corruption.

Together this means, that UDP's built-in checksum may or may not be reliable enough, depending on your environment.

Second fact:

An even more realistic threat than data courruption along the transport is packet loss reordering: USP makes no guarantees about

• all packets to (eventually) arrive at all
• packets to arrive in the same sequence as sent

indeed UDP has no built-in mechanism at all to deal with payloads bigger than a single packet, stemming from the fact, that it wasn't built for that.

Conclusion:

Appending packet after packet as received without additional measures is bound to produce a receive stream differing from the send stream in all but the very favourablest environments., making it a less than optimal protocol for direct file transfer.

If you do want or must use UDP to transfer files, you need to build those parts, that are integral to TCP but not to UDP into the application. There is a saying though, that this will most likely result in an inefrior reimplementation of TCP.

Successfull implementations include many peer-to-peer file sharing protocols, where protection against connection interruption and packet loss or reordering need to be part of the apllication functionality anyway to defeat or mitigate filters.

Implementation recommendations:

What has worked for us is a chunked window implementation: The payload is separated into chunks of a fixed and convenient length, (we used 1023 bytes) a status array of N such chunks is kept on the sending and receiving end.

On the sending side:

• A UDP message is inititated, containing such a chunk, its sequence number (more than once) in the stream and a checksum or hash.
• The status array marks this chunk as "sent/pending" with a timestamp
• Sending stops, if the complete status array (send window) is consumed

On the receiving side:

• received packets are checked against their checksum,
• corrupted packets are negativly acknowledged if all copies of the sequence number agree, dropped else
• OK packets are marked in the status array as "received/pending" with a timestamp
• Acknowledgement works by sending an ack packet if either enough chunks have been received to fill an ack packet, or the timestamp of the oldest "receive/pending" grows too old (some ms to some 100ms).
• Ack packets need checksumming, but no sequencing.
• Chunks, for which an ack has been sent, are marked as "ack/pending" with timestamp in the status array

On the sending side:

• Ack packets are received and checked, corrupted packets are dropped
• Chunks, for which an ack was received, are marked as "ack/done" in the status array
• If the first chunk in the status array is marked "ack/done", the status array slides up, until its first chunk again is not maked done.
• This possibly releases one or more unsent chunks to be sent.
• for chunks in status "sent/pending", a timeout on the timestamp triggers a new send for this chunk, as the original chunk might have been lost.

On the receiving side:

• Reception of chunk i+N (N being the window width) marks chunk i as ack/done, sliding up the receive window. If not all chunks sliding out of the receive window are makred as "ack/pending", this constitutes an unrecoverable error.
• for chunks in status "ack/pending", a timeout on the timestamp triggers a new ack for this chunk, as the original ack message might have been lost.

Obviously there is the need for a special message type from the sending side, if the send window slides out the end of the file, to signal reception of an ack without sending chunk N+i, we implemented it by simply sending N chunks more than exist, but without the payload.