> For the complete documentation index, see [llms.txt](https://training.continuumlabs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://training.continuumlabs.ai/infrastructure/data-and-memory/remote-direct-memory-access-rdma.md).

# Remote Direct Memory Access (RDMA)

<mark style="color:blue;">**Remote Direct Memory Access (RDMA)**</mark> is a technology that enables direct memory access from one computer to another *<mark style="color:yellow;">**without involving either computer's operating system**</mark>*.&#x20;

It provides high-throughput, low-latency networking, which is crucial for modern high-performance computing (HPC), data centres, and cloud environments.&#x20;

### <mark style="color:purple;">History</mark>

RDMA has its roots in the late 1990s and early 2000s, with technologies like <mark style="color:blue;">**Virtual Interface Architecture (VIA)**</mark> and <mark style="color:blue;">**InfiniBand**</mark>.&#x20;

These early implementations aimed to reduce the overhead of network communication by bypassing the operating system kernel and enabling direct memory access between computers.

In the mid-2000s, the <mark style="color:blue;">**RDMA Consortium**</mark> was formed to standardise <mark style="color:blue;">**RDMA over Ethernet**</mark>, leading to the development of <mark style="color:blue;">**iWARP (Internet Wide Area RDMA Protocol)**</mark> and <mark style="color:blue;">**RoCE (RDMA over Converged Ethernet)**</mark>.&#x20;

These technologies *<mark style="color:yellow;">**allowed RDMA to be used over standard Ethernet networks**</mark>*, expanding its reach beyond specialised fabrics like InfiniBand.

### <mark style="color:purple;">Architecture</mark>

RDMA operates by using a specialised network adapter called an <mark style="color:blue;">**RNIC (RDMA Network Interface Card)**</mark>.&#x20;

The <mark style="color:yellow;">**RNIC has direct access to the computer's memory**</mark> and can read from or write to it without involving the CPU or operating system.

#### <mark style="color:green;">**Key components of the RDMA architecture include**</mark>

1. <mark style="color:blue;">**Queue Pairs (QPs):**</mark> Communication channels between two computers, consisting of a Send Queue and a Receive Queue.
2. <mark style="color:blue;">**Completion Queues (CQs):**</mark> Used to signal the completion of RDMA operations.
3. <mark style="color:blue;">**Memory Regions (MRs):**</mark> Registered memory areas that can be accessed remotely.
4. <mark style="color:blue;">**Work Requests (WRs):**</mark> Operations posted to the Send or Receive Queue, such as Send, Receive, Write, or Read.

#### <mark style="color:green;">RDMA supports two main types of operations</mark>

1. <mark style="color:blue;">**Two-sided operations:**</mark> Involve both the sender and receiver, such as Send and Receive.
2. <mark style="color:blue;">**One-sided operations:**</mark> Initiated by one computer and can directly access memory on the remote computer without its involvement, such as Read and Write.

### <mark style="color:purple;">Competing Technologies</mark>

RDMA competes with traditional networking technologies like <mark style="color:blue;">**TCP/IP**</mark>, which rely on the operating system kernel for data transfer.&#x20;

While TCP/IP is widely used and compatible with most networks, it has higher overhead and latency compared to RDMA.

Other competing technologies include:

1. <mark style="color:blue;">**RDMA over Converged Ethernet (RoCE):**</mark> Enables RDMA over Ethernet networks, using UDP/IP as the underlying transport.
2. <mark style="color:blue;">**iWARP:**</mark> Implements RDMA over TCP/IP networks, providing compatibility with existing network infrastructure.
3. <mark style="color:blue;">**Omni-Path:**</mark> Intel's proprietary high-performance interconnect that supports RDMA-like functionality.

<figure><img src="/files/osMiMZkMjHRkNS3Gljjc" alt=""><figcaption></figcaption></figure>

### <mark style="color:purple;">Problems Solved</mark>

RDMA addresses the need for high-performance, low-latency networking in data-intensive applications.&#x20;

By bypassing the operating system kernel and enabling direct memory access, RDMA reduces the overhead associated with traditional networking protocols.

This is particularly important in scenarios such as:

<mark style="color:blue;">**High-Performance Computing (HPC):**</mark> RDMA enables fast communication between nodes in HPC clusters, improving the performance of parallel applications.

<mark style="color:blue;">**Storage Systems:**</mark> RDMA is used in storage technologies like [<mark style="color:blue;">**NVMe over Fabrics (NVMe-oF)**</mark>](/infrastructure/networking-and-connectivity/nvme-over-fabrics-nvme-of.md) to provide high-speed access to remote storage.

<mark style="color:blue;">**Machine Learning and AI:**</mark> RDMA facilitates fast data transfer between GPUs and other accelerators, which is critical for training large-scale machine learning models.

### <mark style="color:purple;">Recent Advancements and Applications</mark>

[<mark style="color:blue;">**GPUDirect RDMA**</mark>](/infrastructure/networking-and-connectivity/nvidia-gpudirect.md)<mark style="color:blue;">**:**</mark> Enables direct data transfer between GPUs and RDMA-capable network adapters, eliminating the need for memory copies and reducing latency.

<figure><img src="/files/dp2Ag4dW0udhKEuNbNyj" alt=""><figcaption></figcaption></figure>

<mark style="color:blue;">**Persistent Memory:**</mark> RDMA can be used to access persistent memory on remote computers, enabling possibilities for distributed storage and computing.

<mark style="color:blue;">**Blockchain:**</mark> RDMA is being explored as a means to improve the performance and scalability of blockchain networks by enabling fast, direct communication between nodes.

In summary, RDMA is a technology for enabling high-performance, low-latency networking in modern computing environments.&#x20;

By bypassing the operating system kernel and providing direct memory access, RDMA significantly reduces the overhead of data transfer, making it ideal for applications that require fast, efficient communication between computers.