What causes udp reception delay?

Question

I have noticed that when I am sending packets at even intervals from a udp socket, the first packet sent seems to be delayed. For example, if I am sending the packets every 100 ms, I find the delay between receiving the packets to be normally distributed with mean 100ms and average standard deviation of 4, on my network. However, the gap between the receive time for the first and second packets is usually around 10 to 40 ms - as you can see, that's clearly a statistically significant difference, and so my question is, what's causing it?

I'm using the sendto function from C on linux. Someone suggested the delay might be caused by arp resolution preventing the packet from being sent until the destination ip has been converted to a mac address - is this likely? If I restart the sending program the first packet again takes too long though, and the delay is inconsistent - 10 to 40 ms is a pretty big range.

I need to find out why this first packet is taking too long, and how to work around it.

Edit: Further analysis with pcap indicates that the sending program is sending the packets at the right interval. Problem must be with the receiver, which is using select() to wait for a readable socket, then calling recvfrom and printing the packet. Is there some sort of buffering going on there I might not know about?

Answer 1

It is most likely the time required for the ARP address resolution. This is the protocol which resolves MAC addresses to IP addresses.

For working around this, try using static entries in the arp cache with arp -s ip-address hw_address.

Answer 2

Speculation will get us nowhere, fire up Wireshark and it will tell you all you need to know.