What are the PCIe operations involved in Infiniband verbs?

Question

Here are some specifics.

When a process calls ibv_post_send(), what happens at the PCI interface to the HCA? Is the WQE encapsulated inside the PCIe doorbell and written via Programmed IO? Or is the WQE fetched in a separate DMA read by the HCA?

What happens When a process calls ibv_poll_cq()? How does the HCA push CQEs to the system memory? Or, if it's pull based, how does the CPU detect new CQEs in the HCA?

Answer 1

Is the WQE encapsulated inside the PCIe doorbell and written via Programmed IO? Or is the WQE fetched in a separate DMA read by the HCA?

It can be either. The usual way is writing WQE and ringing the doorbell, in which case the HCA will fetch the WQE through DMA. The less common way is what is called "Blue Flame" - part of the PCIe BAR is used for WQEs, and the WQE is written to the HCA the same way the doorbell is written.

How does the HCA push CQEs to the system memory?

HCA uses DMA to write CQE to system memory. Your application has two ways to know about the new completion: poll-based or event-based. ibv_poll_cq() is used in a poll-based approach.