NVMe over Fabrics (NVMe-oF)

A protocol that enables high-performance, low-latency access to shared storage resources over various network fabrics

NVMe over Fabrics (NVMe-oF) is a protocol specification that enables the use of NVMe commands and data transfers over various network fabrics, extending beyond the traditional PCIe bus.

It allows for high-performance, low-latency access to NVMe storage over a network, enabling efficient data sharing and improved resource utilisation in data centres.

Integration within the Broader Architecture of Computer Systems

NVMe-oF is integrated into the broader architecture of computer systems as a storage networking protocol.

It enables the separation of storage resources from compute resources, allowing for a more flexible and scalable architecture.

In an NVMe-oF setup, the NVMe-oF host (initiator) is typically a server or compute node that requires access to storage, while the NVMe-oF target is a storage system or device that provides the storage resources.

The NVMe-oF host and target communicate over a network fabric, such as Ethernet or InfiniBand, using the NVMe-oF protocol.

This allows for the creation of a distributed storage architecture, where multiple hosts can access shared storage resources over the network.

The NVMe-oF protocol is implemented in the host's operating system or as a software-defined storage solution, and it interacts with the host's applications and file systems to provide high-performance, low-latency access to the remote NVMe storage.

History and Architecture

NVMe-oF was developed as an extension to the NVMe protocol, which was initially designed for PCIe-based solid-state drives (SSDs).

The need for NVMe-oF arose from the desire to leverage the benefits of NVMe, such as low latency and high throughput, in networked storage environments.

The NVMe-oF specification defines a common architecture and command set for accessing NVMe storage over various network fabrics, including:

1. RDMA (InfiniBand, RoCE, iWARP)

2. Fibre Channel

3. TCP

Understanding TCP/IP: The Backbone of Internet Communications

What is TCP/IP?

Transmission Control Protocol/Internet Protocol (TCP/IP) is a suite of communication protocols used to interconnect network devices on the internet. TCP/IP can also be used as a communications protocol in a private network (an intranet or an extranet).

How TCP/IP Works

TCP/IP provides end-to-end data communication specifying how data should be packetized, addressed, transmitted, routed, and received at the destination.

TCP/IP has become the foundational protocol suite for the internet, ensuring reliable communication and data integrity between diverse systems.

Components of TCP/IP

1. TCP (Transmission Control Protocol)

   • Role: Ensures reliable delivery of data across a network.

   • Functionality: TCP provides error checking, guarantees the order of data delivery, and provides flow control and congestion control.

   • Process: Before transmitting data, TCP creates a connection between the source and destination. It divides data into manageable packets, sequences them, and ensures each packet is acknowledged upon receipt. If packets are missing or in error, TCP is responsible for retransmission.

2. IP (Internet Protocol)

   • Role: Defines IP addresses and routes data packets.

   • Functionality: IP is responsible for addressing and routing packets of data so that they can travel across networks and arrive at the correct destination.

   • Process: Data packets include both the data being transmitted and the destination IP address, enabling routers to forward the packets along the path to the destination.

How They Work Together

• Cooperation: TCP and IP operate together where IP handles the delivery of the packet and TCP ensures the reliability of the message being sent. IP takes care of the external aspect by routing the data to the correct machine, while TCP handles the internal management of ensuring the data is correctly received and assembled.

The TCP/IP Model

TCP/IP communication is based on a four-layer model.

Each layer in the TCP/IP model corresponds to one or more layers in the seven-layer Open Systems Interconnection (OSI) model proposed by the International Organisation for Standardisation (ISO).

1. Link Layer: This layer includes the networking hardware and drivers that operate on the physical network connection.

2. Internet Layer (IP): Handles the movement of packet across the network including the packet routing.

3. Transport Layer (TCP): Ensures reliable transmission of data across the network.

4. Application Layer: Contains all protocols that operate at a higher level, such as HTTP, FTP, etc.

Practical Applications of TCP/IP

• Web Browsing: Uses HTTP or HTTPS protocols which rely on TCP/IP for data delivery.

• Email: Protocols like SMTP, POP3, and IMAP use TCP/IP to send and receive mail.

• File Transfers: Protocols like FTP and SFTP for sharing files over the internet utilize TCP/IP.

• Streaming and Communication: Services like VoIP and video conferencing applications use TCP/IP to ensure data packets are correctly sequenced and error-free.

FAQs About TCP/IP

• Is TCP/IP secure?

  • By itself, TCP/IP does not include robust security measures. Security protocols like HTTPS, SSL/TLS are built on top of TCP/IP to ensure data encryption and secure identification of network hosts.

• What is my TCP/IP Address?

  • A TCP/IP address, commonly known as an IP address, can be discovered through your device's network settings, using command-line tools like ipconfig (Windows) or ifconfig (Unix/Linux), or by visiting websites that display your public IP.

• Can I change my TCP/IP settings?

  • Yes, TCP/IP settings can be manually configured or automatically obtained through DHCP (Dynamic Host Configuration Protocol) in your device's network settings.

Understanding TCP/IP is important for networking professionals and is foundational for anyone working with internet-connected devices. It ensures that no matter the type of data or the devices involved, communication can occur reliably and efficiently across the globe.

The architecture consists of an NVMe-oF host (initiator) and an NVMe-oF target (storage device or subsystem).

The host communicates with the target using NVMe commands and data transfers, which are encapsulated and transmitted over the chosen network fabric.

Relation to NVIDIA's Offerings and Products

NVIDIA has been actively involved in the development and adoption of NVMe-oF for high-performance computing (HPC) and AI workloads.

NVIDIA's products and technologies that leverage or support NVMe-oF include:

NVIDIA Mellanox ConnectX SmartNICs: These network adapters support NVMe-oF over RDMA (RoCE) and TCP, enabling high-performance, low-latency access to networked NVMe storage.

NVIDIA GPUDirect Storage: This technology allows GPUs to directly access NVMe storage over the network using NVMe-oF, bypassing the CPU and reducing latency for GPU-intensive workloads.

NVIDIA Magnum IO: A suite of IO optimisation technologies that includes NVMe-oF support, enabling high-performance, scalable storage solutions for AI and data analytics workloads.

Key Applications

NVMe-oF is particularly relevant for applications and use cases that require high-performance, low-latency access to shared storage resources, such as:

High-Performance Computing (HPC)

In an HPC environment, NVMe-oF can be used to create a high-performance, distributed storage system that enables efficient data sharing among compute nodes.

For example, a scientific simulation application running on multiple compute nodes can access a shared NVMe-oF storage system to read input data and write output results, achieving high throughput and low latency.

NVMe-oF enables efficient data sharing and high-speed storage access for HPC clusters, improving overall system performance and scalability.

AI and Machine Learning

In AI and ML workloads, NVMe-oF can be used to create a fast, scalable storage infrastructure that can keep up with the demands of GPU-based compute servers.

For instance, an AI training platform can use NVMe-oF to store and access large datasets, enabling fast data loading and efficient data sharing among multiple GPU servers.

Distributed Databases and Data Analytics

NVMe-oF can be used to create a high-performance storage backend for distributed databases, such as Apache Cassandra or MongoDB.

By using NVMe-oF, the database can achieve low-latency access to data, enabling faster query processing and improved scalability. The distributed nature of NVMe-oF also allows for the creation of a resilient, highly available storage infrastructure for the database.

Summary

NVMe over Fabrics (NVMe-oF) is a technology that enables the creation of high-performance, scalable, and efficient storage architectures for modern data centres.

By extending the NVMe protocol over various network fabrics, NVMe-oF allows for the disaggregation of storage resources from compute resources, enabling flexible and cost-effective infrastructures that can adapt to the ever-changing needs of data-intensive applications.

With its low latency, high throughput, and support for a wide range of network fabrics, NVMe-oF is well-suited for demanding workloads in areas such as high-performance computing, artificial intelligence, and distributed databases.

As organisations continue to grapple with the challenges of managing and processing ever-growing volumes of data, NVMe-oF will play an increasingly critical role in enabling the next generation of storage and data management solutions.