How to scale UDP read throughput?

Question

Setup: Two linux (CentOS 6) servers connected back-to-back over a dedicated GigE link. Each server with 24 cores and 32GB RAM

Client: Simulator shooting UDP packets as fast as it could in one thread. Each packet size is 256 bytes. I see that the maximum throughput is ~200,000 packets/sec.

Server: Receive packets on the UDP socket in one thread and perform light-weight parsing. I see that the maximum throughput is ~200,000 packets/sec, with CPU one 1 core about at 85% utilization during processing. There is no packet loss, the receive buffer is set to 128M just in case.

Now I have 23 additional cores i would like to use, but as soon as i add one more thread for receiving data on the server side and a client thread for sending data on the client side over a dedicated socket, I see a lot of packet loss on the server side.

The server threads are completely independent of each other and do not block except for I/O. They are not contending on the socket either as each one of them sucks packets off of their own UDP sockets.

I see the following behavior:

1. The CPU is about 85% on each core and a process called ksoftirqd/xx using about 90% if an additional core.
2. I am loosing packets, the throughout drops down on each thread to about 120,000 packets/sec

Google says that ksoftirqd uses CPU to handle soft interrupts and heavy system calls (UDP read) can be one of the reason for high CPU usage for this kernel thread.

I repeated the experiment by pinning my process to a bunch of cores on the same physical socket and I see that the performance improved to 150,000 packets/sec with still some considerable packet loss.

Is there a way i can improve the scalability of my program and minimize packet loss?

Answer 1

First off, at 200,000 packets per second with 256 (data?) bytes per packet, once you consider UDP, IP, and Ethernet overhead, you're running at nearly half the capacity, counting bandwidth alone, of your gigabit link. At the packet-per-second rates you're pushing, many switches (for example) would fall over.

Second, you're probably being killed by IRQs. Better network cards have tunables to let you trade off fewer IRQs for increased latency (they only interrupt once every N packets). Make sure you have that enabled. (One place to check is the module parameters for your ethernet card.)