# Remote Direct Memory Access (RDMA)

<mark style="color:blue;">**Remote Direct Memory Access (RDMA)**</mark> is a technology that enables direct memory access from one computer to another *<mark style="color:yellow;">**without involving either computer's operating system**</mark>*.&#x20;

It provides high-throughput, low-latency networking, which is crucial for modern high-performance computing (HPC), data centres, and cloud environments.&#x20;

### <mark style="color:purple;">History</mark>

RDMA has its roots in the late 1990s and early 2000s, with technologies like <mark style="color:blue;">**Virtual Interface Architecture (VIA)**</mark> and <mark style="color:blue;">**InfiniBand**</mark>.&#x20;

These early implementations aimed to reduce the overhead of network communication by bypassing the operating system kernel and enabling direct memory access between computers.

In the mid-2000s, the <mark style="color:blue;">**RDMA Consortium**</mark> was formed to standardise <mark style="color:blue;">**RDMA over Ethernet**</mark>, leading to the development of <mark style="color:blue;">**iWARP (Internet Wide Area RDMA Protocol)**</mark> and <mark style="color:blue;">**RoCE (RDMA over Converged Ethernet)**</mark>.&#x20;

These technologies *<mark style="color:yellow;">**allowed RDMA to be used over standard Ethernet networks**</mark>*, expanding its reach beyond specialised fabrics like InfiniBand.

### <mark style="color:purple;">Architecture</mark>

RDMA operates by using a specialised network adapter called an <mark style="color:blue;">**RNIC (RDMA Network Interface Card)**</mark>.&#x20;

The <mark style="color:yellow;">**RNIC has direct access to the computer's memory**</mark> and can read from or write to it without involving the CPU or operating system.

#### <mark style="color:green;">**Key components of the RDMA architecture include**</mark>

1. <mark style="color:blue;">**Queue Pairs (QPs):**</mark> Communication channels between two computers, consisting of a Send Queue and a Receive Queue.
2. <mark style="color:blue;">**Completion Queues (CQs):**</mark> Used to signal the completion of RDMA operations.
3. <mark style="color:blue;">**Memory Regions (MRs):**</mark> Registered memory areas that can be accessed remotely.
4. <mark style="color:blue;">**Work Requests (WRs):**</mark> Operations posted to the Send or Receive Queue, such as Send, Receive, Write, or Read.

#### <mark style="color:green;">RDMA supports two main types of operations</mark>

1. <mark style="color:blue;">**Two-sided operations:**</mark> Involve both the sender and receiver, such as Send and Receive.
2. <mark style="color:blue;">**One-sided operations:**</mark> Initiated by one computer and can directly access memory on the remote computer without its involvement, such as Read and Write.

### <mark style="color:purple;">Competing Technologies</mark>

RDMA competes with traditional networking technologies like <mark style="color:blue;">**TCP/IP**</mark>, which rely on the operating system kernel for data transfer.&#x20;

While TCP/IP is widely used and compatible with most networks, it has higher overhead and latency compared to RDMA.

Other competing technologies include:

1. <mark style="color:blue;">**RDMA over Converged Ethernet (RoCE):**</mark> Enables RDMA over Ethernet networks, using UDP/IP as the underlying transport.
2. <mark style="color:blue;">**iWARP:**</mark> Implements RDMA over TCP/IP networks, providing compatibility with existing network infrastructure.
3. <mark style="color:blue;">**Omni-Path:**</mark> Intel's proprietary high-performance interconnect that supports RDMA-like functionality.

<figure><img src="https://1839612753-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpV8SlQaC976K9PPsjApL%2Fuploads%2Fb2YuVKgmc0UlDyk323rW%2Fimage.png?alt=media&#x26;token=6a84c8e3-2c2f-4399-9be3-95e14fbebc12" alt=""><figcaption></figcaption></figure>

### <mark style="color:purple;">Problems Solved</mark>

RDMA addresses the need for high-performance, low-latency networking in data-intensive applications.&#x20;

By bypassing the operating system kernel and enabling direct memory access, RDMA reduces the overhead associated with traditional networking protocols.

This is particularly important in scenarios such as:

<mark style="color:blue;">**High-Performance Computing (HPC):**</mark> RDMA enables fast communication between nodes in HPC clusters, improving the performance of parallel applications.

<mark style="color:blue;">**Storage Systems:**</mark> RDMA is used in storage technologies like [<mark style="color:blue;">**NVMe over Fabrics (NVMe-oF)**</mark>](https://training.continuumlabs.ai/infrastructure/networking-and-connectivity/nvme-over-fabrics-nvme-of) to provide high-speed access to remote storage.

<mark style="color:blue;">**Machine Learning and AI:**</mark> RDMA facilitates fast data transfer between GPUs and other accelerators, which is critical for training large-scale machine learning models.

### <mark style="color:purple;">Recent Advancements and Applications</mark>

[<mark style="color:blue;">**GPUDirect RDMA**</mark>](https://training.continuumlabs.ai/infrastructure/networking-and-connectivity/nvidia-gpudirect)<mark style="color:blue;">**:**</mark> Enables direct data transfer between GPUs and RDMA-capable network adapters, eliminating the need for memory copies and reducing latency.

<figure><img src="https://1839612753-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpV8SlQaC976K9PPsjApL%2Fuploads%2Fa0aEypWyHl5NIYO67SOT%2Fimage.png?alt=media&#x26;token=86b70757-eef9-4da9-8012-a261193cadc5" alt=""><figcaption></figcaption></figure>

<mark style="color:blue;">**Persistent Memory:**</mark> RDMA can be used to access persistent memory on remote computers, enabling possibilities for distributed storage and computing.

<mark style="color:blue;">**Blockchain:**</mark> RDMA is being explored as a means to improve the performance and scalability of blockchain networks by enabling fast, direct communication between nodes.

In summary, RDMA is a technology for enabling high-performance, low-latency networking in modern computing environments.&#x20;

By bypassing the operating system kernel and providing direct memory access, RDMA significantly reduces the overhead of data transfer, making it ideal for applications that require fast, efficient communication between computers.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://training.continuumlabs.ai/infrastructure/data-and-memory/remote-direct-memory-access-rdma.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.