Spark and InfiniBand

Question

I am trying to use Spark in a HPC focused cluster that has infiniband interconnections. This cluster does not provide support for IPoIB. I saw the Spakr-RDMA project from ohio state university in here. I cannot find anyone else working on this, or if apache spark is going to support IB in the future. The question is is there any other solution to get more updated version of spark in HPC environments with only IB as network?

Answer 1

You can check the reference guide for deploying RDMA over Ethernet (RoCE) to accelerate Apache Spark 2.2.0 over Mellanox 100 GbE Network https://community.mellanox.com/docs/DOC-3068