User space bridging - Linux vs. FreeBSD

Question

Background:

I'm working on porting a packet parsing/manipulation program from FreeBSD to Linux (specifically Debian - jessie). This program implements a bi-directional bridge between two physical interfaces while performing parsing/manipulation prior to TX operations.

In FreeBSD, this application used Pcap to RX/TX. In Linux, so far I've tested with Pcap, PACKET_MMAP (using TPACKET_V2) and Vanilla PF_RING.

Test Results:

Using the same hardware and laboratory environment, I observed the following (approximate) throughput results from several tests performed with iperf:

• FreeBSD Kernel bridge: 880Mb/s
• FreeBSD Pcap (user space) bridge: 700Mb/s
• Debian Kernel bridge: 880Mb/s
• Debian Pcap (user space) bridge: 120Kb/s
• Debian Vanilla PF_RING (user space) bridge: 980Kb/s
• Debian PACKET_MMAP (user space) bridge: 480Kb/s

Thoughts and Question:

The Debian user space speeds seem ridiculous to me. They're unusable - and I suspect I'm missing something. Is there a system option I need to enable ("go fast", lol)?

Or is this just how it is with Linux user space bridging?

Edit / Update

I have a lingering suspicion that there's an administrative limitation affecting the Debian tests. I'm searching system documentation for (something / anything) to no avail. Any ideas on what could be limiting this?

Answer 1

Obscure questions find obscure answers...

After much research I found the "problems" with Linux in this case is actually the result of optimizations. By default, apparently, the systems are optimizing for large sustained loads (so YMMV with the below "fix").

I resolved the speed issues and found the same throughput as the FreeBSD systems by using ethtool to adjust the optimizations on my two bridge interfaces:

ethtool -K eth1 gso off gro off tso off ufo off lro off

After the above on both interfaces, PF_RING and PACKET_MMAP met the Kernel bridge speed (which is the maximum speed for my test lab) and Pcap tests resulted in around 300Mb/s.