# NVMe over Fabrics (NVMe-oF)

<mark style="color:blue;">**NVMe over Fabrics (NVMe-oF)**</mark> is a protocol specification that enables the use of NVMe commands and data transfers over various network fabrics, *<mark style="color:yellow;">**extending beyond the traditional PCIe bus**</mark>*.&#x20;

It allows for high-performance, <mark style="color:yellow;">**low-latency access to NVMe storage over a network**</mark>, enabling efficient data sharing and improved resource utilisation in data centres.

### <mark style="color:purple;">Integration within the Broader Architecture of Computer Systems</mark>

NVMe-oF is integrated into the broader architecture of computer systems as a <mark style="color:blue;">**storage networking protocol**</mark>.&#x20;

It enables the *<mark style="color:yellow;">**separation of storage resources from compute resources**</mark>*, allowing for a more flexible and scalable architecture.&#x20;

In an NVMe-oF setup, the NVMe-oF <mark style="color:blue;">**host**</mark> (initiator) is typically a server or compute node that <mark style="color:yellow;">**requires access to storage**</mark>, while the NVMe-oF <mark style="color:blue;">**target**</mark> is a storage system or device that <mark style="color:yellow;">**provides the storage resources**</mark>.

The <mark style="color:blue;">**NVMe-oF host and target**</mark> <mark style="color:yellow;">**communicate over a network fabric**</mark>, such as <mark style="color:blue;">**Ethernet**</mark> or <mark style="color:blue;">**InfiniBand**</mark>, using the <mark style="color:blue;">**NVMe-oF protocol**</mark>.&#x20;

This allows for the <mark style="color:yellow;">**creation of a distributed storage architecture**</mark>, where multiple hosts can access shared storage resources over the network.&#x20;

The NVMe-oF protocol is implemented in the host's operating system or as a software-defined storage solution, and it interacts with the host's applications and file systems to provide high-performance, low-latency access to the remote NVMe storage.

### <mark style="color:purple;">History and Architecture</mark>

<mark style="color:blue;">**NVMe-oF**</mark> was developed as an extension to the NVMe protocol, which was initially designed for PCIe-based solid-state drives (SSDs).&#x20;

The need for NVMe-oF arose from the desire to leverage the benefits of NVMe, such as low latency and high throughput, in <mark style="color:yellow;">**networked storage environments**</mark>.

The NVMe-oF specification defines a common architecture and command set for accessing NVMe storage over various network fabrics, including:

1. <mark style="color:blue;">**RDMA**</mark> (InfiniBand, RoCE, iWARP)
2. <mark style="color:blue;">**Fibre Channel**</mark>
3. <mark style="color:blue;">**TCP**</mark>

<details>

<summary><mark style="color:green;"><strong>Understanding TCP/IP: The Backbone of Internet Communications</strong></mark></summary>

<mark style="color:green;">**What is TCP/IP?**</mark>&#x20;

<mark style="color:blue;">**Transmission Control Protocol/Internet Protocol (TCP/IP)**</mark> is a suite of communication protocols used to interconnect network devices on the internet. TCP/IP can also be used as a communications protocol in a private network (an intranet or an extranet).

<mark style="color:green;">**How TCP/IP Works**</mark>

TCP/IP provides end-to-end data communication specifying how data should be packetized, addressed, transmitted, routed, and received at the destination.&#x20;

TCP/IP has become the foundational protocol suite for the internet, ensuring reliable communication and data integrity between diverse systems.

#### <mark style="color:green;">Components of TCP/IP</mark>

1. <mark style="color:blue;">**TCP (Transmission Control Protocol)**</mark>
   * **Role**: Ensures reliable delivery of data across a network.
   * **Functionality**: TCP provides error checking, guarantees the order of data delivery, and provides flow control and congestion control.
   * **Process**: Before transmitting data, TCP creates a connection between the source and destination. It divides data into manageable packets, sequences them, and ensures each packet is acknowledged upon receipt. If packets are missing or in error, TCP is responsible for retransmission.
2. <mark style="color:blue;">**IP (Internet Protocol)**</mark>
   * **Role**: Defines IP addresses and routes data packets.
   * **Functionality**: IP is responsible for addressing and routing packets of data so that they can travel across networks and arrive at the correct destination.
   * **Process**: Data packets include both the data being transmitted and the destination IP address, enabling routers to forward the packets along the path to the destination.

#### <mark style="color:green;">How They Work Together</mark>

* **Cooperation**: TCP and IP operate together where IP handles the delivery of the packet and TCP ensures the reliability of the message being sent. IP takes care of the external aspect by routing the data to the correct machine, while TCP handles the internal management of ensuring the data is correctly received and assembled.

#### <mark style="color:green;">The TCP/IP Model</mark>

TCP/IP communication is based on a <mark style="color:yellow;">four-layer model</mark>.&#x20;

Each layer in the TCP/IP model corresponds to one or more layers in the seven-layer <mark style="color:blue;">**Open Systems Interconnection (OSI) model**</mark> proposed by the International Organisation for Standardisation (ISO).

1. <mark style="color:purple;">**Link Layer**</mark><mark style="color:purple;">:</mark> This layer includes the networking hardware and drivers that operate on the physical network connection.
2. <mark style="color:purple;">**Internet Layer (IP)**</mark><mark style="color:purple;">:</mark> Handles the movement of packet across the network including the packet routing.
3. <mark style="color:purple;">**Transport Layer (TCP)**</mark><mark style="color:purple;">:</mark> Ensures reliable transmission of data across the network.
4. <mark style="color:purple;">**Application Layer**</mark><mark style="color:purple;">:</mark> Contains all protocols that operate at a higher level, such as HTTP, FTP, etc.

#### <mark style="color:green;">Practical Applications of TCP/IP</mark>

* <mark style="color:purple;">**Web Browsing:**</mark> Uses HTTP or HTTPS protocols which rely on TCP/IP for data delivery.
* <mark style="color:purple;">**Email:**</mark> Protocols like SMTP, POP3, and IMAP use TCP/IP to send and receive mail.
* <mark style="color:purple;">**File Transfers:**</mark> Protocols like FTP and SFTP for sharing files over the internet utilize TCP/IP.
* <mark style="color:purple;">**Streaming and Communication**</mark><mark style="color:purple;">:</mark> Services like VoIP and video conferencing applications use TCP/IP to ensure data packets are correctly sequenced and error-free.

#### <mark style="color:green;">FAQs About TCP/IP</mark>

* **Is TCP/IP secure?**
  * By itself, TCP/IP does not include robust security measures. Security protocols like HTTPS, SSL/TLS are built on top of TCP/IP to ensure data encryption and secure identification of network hosts.
* **What is my TCP/IP Address?**
  * A TCP/IP address, commonly known as an IP address, can be discovered through your device's network settings, using command-line tools like `ipconfig` (Windows) or `ifconfig` (Unix/Linux), or by visiting websites that display your public IP.
* **Can I change my TCP/IP settings?**
  * Yes, TCP/IP settings can be manually configured or automatically obtained through <mark style="color:blue;">**DHCP (Dynamic Host Configuration Protocol) i**</mark>n your device's network settings.

Understanding TCP/IP is important for networking professionals and is foundational for anyone working with internet-connected devices. It ensures that no matter the type of data or the devices involved, communication can occur reliably and efficiently across the globe.

</details>

The architecture consists of an <mark style="color:blue;">**NVMe-oF host (initiator)**</mark> and an <mark style="color:blue;">**NVMe-oF target**</mark> (storage device or subsystem).&#x20;

The host communicates with the target using NVMe commands and data transfers, which are encapsulated and transmitted over the chosen network fabric.

<figure><img src="https://1839612753-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpV8SlQaC976K9PPsjApL%2Fuploads%2F4U9I5nnFR84K0kmLqLl2%2Fimage.png?alt=media&#x26;token=10c2ff62-9e65-4a8e-90d9-63c3ddef3467" alt=""><figcaption></figcaption></figure>

### <mark style="color:purple;">Relation to NVIDIA's Offerings and Products</mark>

NVIDIA has been actively involved in the development and adoption of NVMe-oF for high-performance computing (HPC) and AI workloads.&#x20;

NVIDIA's products and technologies that leverage or support NVMe-oF include:

<mark style="color:blue;">**NVIDIA Mellanox ConnectX SmartNICs:**</mark> These network adapters support NVMe-oF over RDMA (RoCE) and TCP, enabling high-performance, low-latency access to networked NVMe storage.

<mark style="color:blue;">**NVIDIA GPUDirect Storage:**</mark> This technology allows GPUs to directly access NVMe storage over the network using NVMe-oF, bypassing the CPU and reducing latency for GPU-intensive workloads.

<mark style="color:blue;">**NVIDIA Magnum IO:**</mark> A suite of IO optimisation technologies that includes NVMe-oF support, enabling high-performance, scalable storage solutions for AI and data analytics workloads.

### <mark style="color:purple;">Key Applications</mark>

NVMe-oF is particularly relevant for applications and use cases that require high-performance, low-latency access to <mark style="color:yellow;">**shared storage resources**</mark>, such as:

#### <mark style="color:blue;">**High-Performance Computing (HPC)**</mark>

In an HPC environment, NVMe-oF can be used to create a high-performance, distributed storage system that enables efficient data sharing among compute nodes.&#x20;

For example, a scientific simulation application running on multiple compute nodes can access a shared NVMe-oF storage system to read input data and write output results, achieving high throughput and low latency.

NVMe-oF enables efficient data sharing and high-speed storage access for HPC clusters, improving overall system performance and scalability.

#### <mark style="color:blue;">**AI and Machine Learning**</mark>

&#x20;In AI and ML workloads, NVMe-oF can be used to create a fast, scalable storage infrastructure that can keep up with the demands of GPU-based compute servers.&#x20;

For instance, an AI training platform can use NVMe-oF to store and access large datasets, enabling fast data loading and efficient data sharing among multiple GPU servers.

#### <mark style="color:blue;">**Distributed Databases and Data Analytics**</mark>

NVMe-oF can be used to create a high-performance storage backend for distributed databases, such as Apache Cassandra or MongoDB.&#x20;

By using NVMe-oF, the database can achieve low-latency access to data, enabling faster query processing and improved scalability. The distributed nature of NVMe-oF also allows for the creation of a resilient, highly available storage infrastructure for the database.

### <mark style="color:purple;">Summary</mark>

<mark style="color:blue;">**NVMe over Fabrics (NVMe-oF)**</mark> is a technology that enables the creation of high-performance, scalable, and efficient storage architectures for modern data centres.&#x20;

By extending the NVMe protocol over various network fabrics, NVMe-oF allows for the <mark style="color:yellow;">disaggregation of storage resources from compute resources</mark>, enabling flexible and cost-effective infrastructures that can adapt to the ever-changing needs of data-intensive applications.

With its low latency, high throughput, and support for a wide range of network fabrics, NVMe-oF is well-suited for demanding workloads in areas such as high-performance computing, artificial intelligence, and distributed databases.&#x20;

As organisations continue to grapple with the challenges of managing and processing ever-growing volumes of data, NVMe-oF will play an increasingly critical role in enabling the next generation of storage and data management solutions.