How to send UDP packets of size greater than 64 KB

Question

I am unable to send IP multicast datagrams of size greater than 64 KB (which I need to, for my experiments). I am transferring packets between computers connected directly by 10 Gigabit ethernet links (without any hops in between, confirmed by traceroute).

All computers have Ubuntu 12.04 installed. I changed the limits of read and write buffers by adding the following lines in /etc/sysctl.conf :

net.core.rmem_max=12582912
net.core.wmem_max=12582912
net.core.rmem_default=12582912
net.core.wmem_default=12582912

and verified the changes using sysctl -a (after running sysctl -p). Do I need to restart so that I can see the changes? (I am sharing the machines, so a restart is not always possible).

The MTU of the interface used to send and receive is 9000 Bytes in all the computers. I have been successful in sending packets of size around 60 KB, and for 100 KB packets, capturing using tcpdump reveals that the packets are not even sent and probably dropped by the kernel (I don't see them in the tcpdump trace).

What more do I need to do to be able to transfer large packets (preferably of sizes of the order of 100 MBs)?

Answer 1

A UDP datagram has to fit inside a single IP datagram. The Total Length field in the IP header is 16 bits, so the maximum length (including the IP and UDP headers) is 65535 bytes. The UDP header also has a 16-bit Length field. The UDP Length field includes the UDP header, not the IP header, but since the entire UDP datagram has to fit in the payload of an IP packet, it's constrained by the IP length.

So it's not possible to send a UDP datagram larger than 64KB. Since the minimum sizes of the IP and UDP headers are 20 and 8 bytes, respectively, the actual maximum amount of payload is at most 65507 bytes.

If you need to send larger messages, you need to break it up into multiple datagrams. Or perhaps you should consider using a different transport protocol, such as TCP (unfortunately, this is not possible if you're doing multicast).

IPv6 supports Jumbograms that are larger than 64K. But you can't do it in IPv4.

Answer 2

At the lowest level, it's impossible.

According to RFC 768 - User Datagram Protocol, the structure of an UDP packet looks something like...

              0      7 8     15 16    23 24    31
             +--------+--------+--------+--------+
             |     Source      |   Destination   |
             |      Port       |      Port       |
             +--------+--------+--------+--------+
             |                 |                 |
             |     Length      |    Checksum     |
             +--------+--------+--------+--------+
             |
             |          data octets ...
             +---------------- ...

                  User Datagram Header Format

This means that the Length field, holding the packet's size, is 16-bits long. Then, we have that 2^16-1 equals 65535, thus that number being the maximum value that the Length field may contain. And, of course, 65535 is just one byte away from being exactly 64KiB. Maybe there are extensions to UDP that may overcome this issue, but I'm not aware of any. Anyway, it happens that the Length field counts the header's size too, so the minimum value for it is 8, and the maximum payload size is 65535-8=65527.

Anyway, I'm not sure if you're using the right protocol, or at least the right data manipulation model. UDP is surely your (only?) best bet for multicasting purposes, but UDP is meant to transfer small packets of data without providing the connection illution that the Transmission Control Protocol creates to overcome the Internet Protocol's unreliable, connection-less way of transferring information. UDP is just a small improvement that adds ports on top of that.

Hope this helps you!