Remote Direct Memory Access (RDMA)
Remote Direct Memory Access (RDMA) is a technology that enables direct memory access from one computer to another without involving either computer's operating system.
It provides high-throughput, low-latency networking, which is crucial for modern high-performance computing (HPC), data centres, and cloud environments.
History
RDMA has its roots in the late 1990s and early 2000s, with technologies like Virtual Interface Architecture (VIA) and InfiniBand.
These early implementations aimed to reduce the overhead of network communication by bypassing the operating system kernel and enabling direct memory access between computers.
In the mid-2000s, the RDMA Consortium was formed to standardise RDMA over Ethernet, leading to the development of iWARP (Internet Wide Area RDMA Protocol) and RoCE (RDMA over Converged Ethernet).
These technologies allowed RDMA to be used over standard Ethernet networks, expanding its reach beyond specialised fabrics like InfiniBand.
Architecture
RDMA operates by using a specialised network adapter called an RNIC (RDMA Network Interface Card).
The RNIC has direct access to the computer's memory and can read from or write to it without involving the CPU or operating system.
Key components of the RDMA architecture include
Queue Pairs (QPs): Communication channels between two computers, consisting of a Send Queue and a Receive Queue.
Completion Queues (CQs): Used to signal the completion of RDMA operations.
Memory Regions (MRs): Registered memory areas that can be accessed remotely.
Work Requests (WRs): Operations posted to the Send or Receive Queue, such as Send, Receive, Write, or Read.
RDMA supports two main types of operations
Two-sided operations: Involve both the sender and receiver, such as Send and Receive.
One-sided operations: Initiated by one computer and can directly access memory on the remote computer without its involvement, such as Read and Write.
Competing Technologies
RDMA competes with traditional networking technologies like TCP/IP, which rely on the operating system kernel for data transfer.
While TCP/IP is widely used and compatible with most networks, it has higher overhead and latency compared to RDMA.
Other competing technologies include:
RDMA over Converged Ethernet (RoCE): Enables RDMA over Ethernet networks, using UDP/IP as the underlying transport.
iWARP: Implements RDMA over TCP/IP networks, providing compatibility with existing network infrastructure.
Omni-Path: Intel's proprietary high-performance interconnect that supports RDMA-like functionality.
Problems Solved
RDMA addresses the need for high-performance, low-latency networking in data-intensive applications.
By bypassing the operating system kernel and enabling direct memory access, RDMA reduces the overhead associated with traditional networking protocols.
This is particularly important in scenarios such as:
High-Performance Computing (HPC): RDMA enables fast communication between nodes in HPC clusters, improving the performance of parallel applications.
Storage Systems: RDMA is used in storage technologies like NVMe over Fabrics (NVMe-oF) to provide high-speed access to remote storage.
Machine Learning and AI: RDMA facilitates fast data transfer between GPUs and other accelerators, which is critical for training large-scale machine learning models.
Recent Advancements and Applications
GPUDirect RDMA: Enables direct data transfer between GPUs and RDMA-capable network adapters, eliminating the need for memory copies and reducing latency.
Persistent Memory: RDMA can be used to access persistent memory on remote computers, enabling possibilities for distributed storage and computing.
Blockchain: RDMA is being explored as a means to improve the performance and scalability of blockchain networks by enabling fast, direct communication between nodes.
In summary, RDMA is a technology for enabling high-performance, low-latency networking in modern computing environments.
By bypassing the operating system kernel and providing direct memory access, RDMA significantly reduces the overhead of data transfer, making it ideal for applications that require fast, efficient communication between computers.
Last updated
