NVMe over Fabrics (NVMe-oF)
A protocol that enables high-performance, low-latency access to shared storage resources over various network fabrics
NVMe over Fabrics (NVMe-oF) is a protocol specification that enables the use of NVMe commands and data transfers over various network fabrics, extending beyond the traditional PCIe bus.
It allows for high-performance, low-latency access to NVMe storage over a network, enabling efficient data sharing and improved resource utilisation in data centres.
Integration within the Broader Architecture of Computer Systems
NVMe-oF is integrated into the broader architecture of computer systems as a storage networking protocol.
It enables the separation of storage resources from compute resources, allowing for a more flexible and scalable architecture.
In an NVMe-oF setup, the NVMe-oF host (initiator) is typically a server or compute node that requires access to storage, while the NVMe-oF target is a storage system or device that provides the storage resources.
The NVMe-oF host and target communicate over a network fabric, such as Ethernet or InfiniBand, using the NVMe-oF protocol.
This allows for the creation of a distributed storage architecture, where multiple hosts can access shared storage resources over the network.
The NVMe-oF protocol is implemented in the host's operating system or as a software-defined storage solution, and it interacts with the host's applications and file systems to provide high-performance, low-latency access to the remote NVMe storage.
History and Architecture
NVMe-oF was developed as an extension to the NVMe protocol, which was initially designed for PCIe-based solid-state drives (SSDs).
The need for NVMe-oF arose from the desire to leverage the benefits of NVMe, such as low latency and high throughput, in networked storage environments.
The NVMe-oF specification defines a common architecture and command set for accessing NVMe storage over various network fabrics, including:
RDMA (InfiniBand, RoCE, iWARP)
Fibre Channel
TCP
The architecture consists of an NVMe-oF host (initiator) and an NVMe-oF target (storage device or subsystem).
The host communicates with the target using NVMe commands and data transfers, which are encapsulated and transmitted over the chosen network fabric.
Relation to NVIDIA's Offerings and Products
NVIDIA has been actively involved in the development and adoption of NVMe-oF for high-performance computing (HPC) and AI workloads.
NVIDIA's products and technologies that leverage or support NVMe-oF include:
NVIDIA Mellanox ConnectX SmartNICs: These network adapters support NVMe-oF over RDMA (RoCE) and TCP, enabling high-performance, low-latency access to networked NVMe storage.
NVIDIA GPUDirect Storage: This technology allows GPUs to directly access NVMe storage over the network using NVMe-oF, bypassing the CPU and reducing latency for GPU-intensive workloads.
NVIDIA Magnum IO: A suite of IO optimisation technologies that includes NVMe-oF support, enabling high-performance, scalable storage solutions for AI and data analytics workloads.
Key Applications
NVMe-oF is particularly relevant for applications and use cases that require high-performance, low-latency access to shared storage resources, such as:
High-Performance Computing (HPC)
In an HPC environment, NVMe-oF can be used to create a high-performance, distributed storage system that enables efficient data sharing among compute nodes.
For example, a scientific simulation application running on multiple compute nodes can access a shared NVMe-oF storage system to read input data and write output results, achieving high throughput and low latency.
NVMe-oF enables efficient data sharing and high-speed storage access for HPC clusters, improving overall system performance and scalability.
AI and Machine Learning
In AI and ML workloads, NVMe-oF can be used to create a fast, scalable storage infrastructure that can keep up with the demands of GPU-based compute servers.
For instance, an AI training platform can use NVMe-oF to store and access large datasets, enabling fast data loading and efficient data sharing among multiple GPU servers.
Distributed Databases and Data Analytics
NVMe-oF can be used to create a high-performance storage backend for distributed databases, such as Apache Cassandra or MongoDB.
By using NVMe-oF, the database can achieve low-latency access to data, enabling faster query processing and improved scalability. The distributed nature of NVMe-oF also allows for the creation of a resilient, highly available storage infrastructure for the database.
Summary
NVMe over Fabrics (NVMe-oF) is a technology that enables the creation of high-performance, scalable, and efficient storage architectures for modern data centres.
By extending the NVMe protocol over various network fabrics, NVMe-oF allows for the disaggregation of storage resources from compute resources, enabling flexible and cost-effective infrastructures that can adapt to the ever-changing needs of data-intensive applications.
With its low latency, high throughput, and support for a wide range of network fabrics, NVMe-oF is well-suited for demanding workloads in areas such as high-performance computing, artificial intelligence, and distributed databases.
As organisations continue to grapple with the challenges of managing and processing ever-growing volumes of data, NVMe-oF will play an increasingly critical role in enabling the next generation of storage and data management solutions.
Last updated
