# NVIDIA Quantum InfiniBand

<mark style="color:blue;">**NVIDIA Quantum InfiniBand**</mark> is a high-performance <mark style="color:yellow;">**networking solution**</mark> designed for AI and high-performance computing (HPC) workloads in data centres.

It enables fast, <mark style="color:yellow;">**low-latency communication**</mark> between servers, storage systems, and NVIDIA GPUs.&#x20;

### <mark style="color:purple;">**InfiniBand Architecture**</mark>

* NVIDIA Quantum InfiniBand is based on the <mark style="color:blue;">**InfiniBand networking standard**</mark>, which provides high bandwidth and low latency.
* It uses a <mark style="color:blue;">**switched fabric topology**</mark>, allowing multiple devices to communicate simultaneously without contention.
* The latest generation, NVIDIA Quantum-2, offers speeds up to <mark style="color:blue;">**400 Gb/s per port**</mark>.

This speed of 400 Gb/s per port is incredibly fast. &#x20;

At 400 Gb/s, you could transfer a 100 GB dataset in just 2 seconds.   In one minute, you could transfer 3 TB of data, which is equivalent to the storage capacity of a high-end consumer desktop computer.

To put this speed into context, let's compare it with some <mark style="color:yellow;">**common networking standards**</mark>:

<mark style="color:purple;">**Gigabit Ethernet (GbE)**</mark>

Gigabit Ethernet offers a maximum speed of 1 Gb/s per port. - NVIDIA Quantum InfiniBand's 400 Gb/s speed is 400 times faster than GbE.

<mark style="color:purple;">**10 Gigabit Ethernet (10GbE)**</mark>

10 Gigabit Ethernet provides speeds up to 10 Gb/s per port. NVIDIA Quantum InfiniBand is 40 times faster than 10GbE.

<mark style="color:purple;">**PCI Express (PCIe) Gen 4**</mark>

PCIe Gen 4 provides a bandwidth of up to 16 GT/s (GigaTransfers per second) per lane, with a x16 link offering a maximum theoretical bandwidth of 32 GB/s. &#x20;

NVIDIA Quantum InfiniBand's 400 Gb/s speed is equivalent to around 50 GB/s, exceeding the bandwidth of a PCIe Gen 4 x16 link.

<figure><img src="https://1839612753-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpV8SlQaC976K9PPsjApL%2Fuploads%2FNRzq4VUlvts58lxIgnC3%2Fimage.png?alt=media&#x26;token=1f8b2269-d2de-4276-96f2-f5d127b9eefe" alt=""><figcaption></figcaption></figure>

### <mark style="color:blue;">**Definition - Switched Fabric Topology**</mark>

<mark style="color:blue;">**Fabric topology**</mark> in networking refers to the layout or structure of interconnected nodes, including switches, servers, and storage devices, within a network. &#x20;

It is designed to support high levels of data transmission and communication efficiency. The term "fabric" comes from the idea of interweaving threads, symbolising the complex and interconnected nature of the network paths.

NVIDIA Quantum InfiniBand uses a <mark style="color:blue;">**switched fabric topology**</mark>, which means it can easily be scaled up by *<mark style="color:yellow;">**adding more switches or nodes**</mark>*.  It also means the network can ensure continued operation even if a component fails, which is critical for mission-critical applications in data centres.

### <mark style="color:purple;">Some worthwhile reading on InfiniBand</mark>

{% file src="<https://1839612753-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpV8SlQaC976K9PPsjApL%2Fuploads%2FC6SehVmAqBopk2cCd2Nb%2FIntroduction%20to%20Infiniband.pdf?alt=media&token=0769c8d8-b58f-4d42-8ced-15748e09aa7c>" %}
From NVIDIA
{% endfile %}

### <mark style="color:purple;">Interconnect for GPUs</mark>

NVIDIA Quantum InfiniBand is designed to work with NVIDIA GPUs

It supports [<mark style="color:blue;">**NVIDIA GPUDirect**</mark>](https://training.continuumlabs.ai/infrastructure/networking-and-connectivity/nvidia-gpudirect), a technology that *<mark style="color:yellow;">**allows GPUs to directly access the memory of other GPUs or network adapters**</mark>*, reducing latency and improving performance.

With <mark style="color:blue;">**GPUDirect RDMA**</mark> (Remote Direct Memory Access), <mark style="color:yellow;">**GPUs can bypass the CPU**</mark> and directly access data from remote servers or storage systems over the InfiniBand network.

#### <mark style="color:green;">**In-Network Computing**</mark>

NVIDIA Quantum InfiniBand supports <mark style="color:blue;">**In-Network Computing**</mark>, which offloads certain computations to the network fabric itself.

It includes preconfigured and programmable compute engines, such as [<mark style="color:blue;">**NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARPv3)**</mark>](https://training.continuumlabs.ai/infrastructure/networking-and-connectivity/scalable-hierarchical-aggregation-and-reduction-protocol-sharp), Message Passing Interface (MPI) Tag Matching, and MPI All-to-All.

These engines accelerate collective operations, reduce network traffic, and improve overall application performance.

#### <mark style="color:green;">**Performance Isolation**</mark>

NVIDIA Quantum InfiniBand provides proactive <mark style="color:yellow;">**monitoring and congestion management**</mark> to ensure performance isolation.

It minimises <mark style="color:blue;">**performance jitter**</mark> and guarantees predictable performance for applications, as if they were running on dedicated systems.

This is particularly important in multi-tenant environments where multiple users or applications share the same infrastructure.

<mark style="color:blue;">**Performance jitter**</mark> refers to the *<mark style="color:yellow;">**variability in computational performance or network latency over time**</mark>*. Factors contributing to performance jitter may include fluctuating network traffic, shared system resources, or varying workloads.

#### <mark style="color:green;">**Cloud-Native Supercomputing**</mark>

NVIDIA Quantum InfiniBand, combined with [<mark style="color:blue;">**NVIDIA BlueField Data Processing Units (DPUs)**</mark>](https://training.continuumlabs.ai/infrastructure/data-and-memory/nvidia-bluefield-data-processing-units-dpus), enables cloud-native supercomputing.

It provides bare-metal performance, user management, data protection, and on-demand provisioning of HPC and AI services in a cloud environment.

This allows organisations to leverage the flexibility and scalability of the cloud while maintaining the performance characteristics of dedicated supercomputing systems.

#### <mark style="color:green;">**Adapters and Switches**</mark>

* [**NVIDIA ConnectX-7 InfiniBand adapters**](https://training.continuumlabs.ai/infrastructure/networking-and-connectivity/nvidia-connectx-infiniband-adapters), available in various form factors, provide single or dual network ports at <mark style="color:yellow;">**400 Gb/s**</mark>.
* These adapters include advanced <mark style="color:blue;">**In-Network Computing**</mark> capabilities and programmable engines for data preprocessing and offloading application control paths to the network.
* NVIDIA Quantum-2 switches offer high-density, high-bandwidth switching with up to <mark style="color:yellow;">**64**</mark> 400 Gb/s ports or <mark style="color:yellow;">**128**</mark> 200 Gb/s ports in a compact 1U form factor.

#### <mark style="color:green;">**Cables and Transceivers**</mark>

* NVIDIA Quantum InfiniBand supports a variety of connectivity options, including transceivers[^1], [multi-fibre push-on connectors (MPOs)](#user-content-fn-2)[^2], [active copper cables (ACCs)](#user-content-fn-3)[^3], and [direct attached cables (DACs)](#user-content-fn-4)[^4].
* These options provide flexibility in building network topologies and enable backward compatibility with existing <mark style="color:yellow;">**200 Gb/s**</mark> or <mark style="color:yellow;">**100 Gb/s**</mark> infrastructures.

### <mark style="color:purple;">Summary</mark>

NVIDIA Quantum InfiniBand is a networking solution that offers extreme performance and efficiency for modern data centres.&#x20;

As data centres continue to evolve and adopt GPU-accelerated computing and cloud-native architectures, <mark style="color:blue;">**NVIDIA Quantum InfiniBand**</mark> will play a role in ensuring optimal system performance, scalability, and flexibility.&#x20;

By investing in this technology, organisations can future-proof their data centre infrastructure and unlock new possibilities for innovation and discovery.

### <mark style="color:purple;">Three applications for NVIDIA Quantum InfiniBand</mark>

#### <mark style="color:green;">Real-time, high-resolution video processing in media and entertainment</mark>

NVIDIA Quantum InfiniBand could enable distributed, GPU-accelerated processing of high-resolution video content (e.g., 8K or higher) in real-time.&#x20;

This would allow media and entertainment companies to collaborate on complex video editing, visual effects, and animation projects across multiple locations, with minimal latency and maximum performance.

<mark style="color:green;">**Federated learning for healthcare and medical research**</mark>

Quantum InfiniBand could facilitate secure, high-speed data sharing and model training across multiple healthcare institutions or research centres.&#x20;

This would enable federated learning, where AI models are trained on decentralised data without compromising patient privacy. The low latency and high bandwidth of Quantum InfiniBand would ensure rapid model updates and faster discovery of new medical insights.

<mark style="color:green;">**Real-time, GPU-accelerated intrusion detection and cybersecurity**</mark>

NVIDIA Quantum InfiniBand could power distributed, GPU-accelerated intrusion detection systems (IDS) for large-scale networks.&#x20;

By leveraging GPUs and high-speed, low-latency networking, these systems could analyse massive amounts of network traffic in real-time, detecting and responding to potential security threats with unprecedented speed and accuracy.&#x20;

This would help organisations to better protect their critical assets and data from increasingly sophisticated cyber attacks.

[^1]: Transceivers are devices designed to both transmit and receive data. They convert electrical signals into optical signals for transmission over fibre-optic cables and then back into electrical signals upon receipt. This conversion is vital for long-distance, high-bandwidth data transfers in modern networks.

[^2]: MPOs are high-density fibre optic connectors that can connect multiple fibre optic cables at once. They are primarily used for quick and efficient connection of multiple fibres in a single plug, facilitating rapid deployment of high-performance computing and telecommunications systems.

[^3]: ACCs are copper cables with electronic components built into the connectors. These components boost the signal strength, allowing the cables to support higher data rates over longer distances than standard copper cables, without the cost of fibre optics. They are commonly used in data centres for interconnecting network devices.

[^4]: DACs are high-speed cables with connectors fixed on both ends, used to connect network appliances directly to one another. They are typically used for short distances within racks in a data centre. DACs provide a cost-effective and energy-efficient solution for high-speed data transfers, as they do not require transceivers.