InfiniBand explained

Question

Can anybody explain what is InfiniBand? What is the key differences in comparison with Ethernet, how these differences allow for it to be faster than Ethernet?

In the official description from mellanox it is written that

Introduce InfiniBand, a switch-based serial I/O interconnect architecture operating at...

What does it mean that Infiniband is a switch-based interconnect? I found this description, but it does not explain what happens if several inputs want to write to a single output, how is the collision resolved?

It is also said that Infiniband has end-to-end flow control. Does it mean that there is no (need) for any other (in-between) flow control? Why?

Answer 1

Key difference between Ethernet and Infiniband, which makes Infiniband faster, is RDMA (Remote Direct Memory Access). DMA (in networking) is an operation which access the memory directly from the NIC (Network Interface Controller), without involving the CPU. RDMA is the same idea, but the direct memory access is done by a remote machine.

More differences:

1. Communication is done between QPs (Queue Pairs) instead of channels.
2. Data flow to/from user space straight to/from HW instead of going thru the kernel stack.

A basic RDMA flow between a requestor and a responder would consist of:

1. Handshake - exchange details between requestor and responder (mainly allocated memory addresses and access keys).
2. Create a READ/WRITE/ATOMIC request on the requestor side.
3. Send the request to the responder.
4. Directly access the memory on the responder side.
5. If READ/ATOMIC - send the data read from responder's memory back to the requestor.

Main benefits:

1. No CPU access on the responder side - throughput is limited by the HW (NIC & PCI) only.
2. No SW is running on responder side - allows much lower latency (~10 times less than typical TCP/UDP latency).
3. Supports "polling mode" for completion on requestor side, meaning the SW knows immediately once HW finished transmitting. Allows for lower latency and higher throughput, on the expense of high CPU utilization.

For more information please refer to the Infiniband spec (sorry it is very long).

Related traffic protocols:

• RoCE (RDMA over Converged Ethernet), which implements RDMA over Ethernet fabric by wrapping Infiniband packets with L2/L3/L4 Ethernet headers.

• IPoIB (IP over Infiniband), which implements regular networking (thru the kernel stack) over Infiniband fabric by wrapping L3/L4 packets with Infiniband headers.

Hope this helps.

Answer 2

To learn basics of InfiniBand I suggest you to visit Mellanox Academy Web-Site and after registration take InfiniBand Essentials or InfiniBand Fundamentals course (in a section Technologies).

In my opinion "switch-based architecture" means that switches are part of fabric (see picture below, where I have shown switch by blue shape).

End-to-end flow control, aka message level flow control, is a feature (capability) for reliable connections. This can be used by a responder to optimize the use of its receive resources. Essentially, a requester cannot send a request message unless it has appropriate credits to do so. Please, refer to InfiniBand specification for details.