OSDI '23

Currently Playing: OSDI '23 - Flor: An Open High Performance RDMA Framework Over Heterogeneous RNICs

OSDI '23 - Flor: An Open High Performance RDMA Framework Over Heterogeneous RNICs Qiang Li, Alibaba Group, Yixiao Gao, Nanjing University, Xiaoliang Wang, Nanjing University, Haonan Qiu, Alibaba Group, Yanfang Le, AMD, Derui Liu, Alibaba Group, Qiao Xiang, Xiamen University, Fei Feng, Alibaba Group, Peng Zhang, Alibaba Group, Bo Li, Alibaba Group, Jianbo Dong, Alibaba Group, Lingbo Tang, Alibaba Group, Hongqiang Harry Liu, Alibaba Group, Shaozong Liu, Alibaba Group, Weijie Li, Alibaba Group, Rui Miao, Alibaba Group, Yaohui Wu, Alibaba Group, Zhiwu Wu, Alibaba Group, Chao Han, Alibaba Group, Lei Yan, Alibaba Group, Zheng Cao, Alibaba Group, Zhongjie Wu, Alibaba Group, Chen Tian, Nanjing University, Guihai Chen, Nanjing University, Dennis Cai, Alibaba Group, Jinbo Wu, Alibaba Group, Jiaji Zhu, Alibaba Group, Jiesheng Wu, Alibaba Group, Jiwu Shu, Xiamen University Datacenter applications have been increasingly applying RDMA for the ultra-low latency and low CPU overhead. However, RDMA-capable NICs (RNICs) of different vendors and different generations from the same vendors do not cooperate well, which causes bandwidth imbalance in the production network. Our observation of the heterogeneous RNICs is that though the data path functions of these RNICs follow the same RoCEv2 specifications, their control path functions are vendor and version specific. To this end, we propose Flor, an open framework that provides a flexible control plane in software and a unified hardware plane by adopting heterogeneous RNICs. The hardware plane requires no changes of current specifications. The software plane can run in NPU of RNICs, DPUs and host CPUs, following which we build up strengthen reliable transport over the large-scale lossy Ethernet. We implemented and evaluated Flor in both testbed and production clusters over Intel E180, Mellanox CX-4 and CX-5 and Broadcom RNICs. Experiments show that Flor achieves comparable performance to vanilla RDMA in many scenarios including 1/4096 packet loss, 6000:1 incast, and large-scale cross-pod communication. Flor mitigates the performance gap of CX-4 and CX-5 RNICs from 24.3% to 1.3% when they are deployed together without PFC dependency. View the full OSDI '23 program at https://www.usenix.org/conference/osdi23/technical-sessions