NVLink Switch

NVIDIA NVLink is a high-speed, direct interconnect technology designed to enable fast communication between GPUs within a server or across multiple servers.

By providing CUDA acceleration across different layers, NVLink reduces communication-related network overhead.

NVLink vastly improves scalability for larger multi-GPU systems.

A single NVIDIA Blackwell Tensor Core GPU supports up to 18 NVLink 100 gigabyte-per-second (GB/s) connections for a total bandwidth of 1.8 terabytes per second (TB/s)— 2 times more bandwidth than the previous generation and over 14 times the bandwidth of PCIe Gen5.

Server platforms like the GB200 NVL72 take advantage of this technology to deliver greater scalability for today’s most complex large models.

NVLink in NVIDIA H100 increases inter-GPU communication bandwidth 1.5 times compared to the previous generation, so researchers can use larger, more sophisticated applications to solve more complex problems.

NVLINK addresses the limitations of traditional PCIe switches by providing higher bandwidth and lower latency connections between GPUs.

PCE Switches versus NVLINK

PCIe (Peripheral Component Interconnect Express) switches act as a central hub that allow multiple PCIe devices, such as GPUs, to communicate with each other and with the host system.

Before NVLink technology was introduced (before 2014), GPUs had to be interconnected through a PCIe switch.

PCIe stood as the traditional backbone for GPU interconnectivity in servers. While it offers lower bandwidth compared to NVLink, PCIe's strength lies in its flexibility and broad compatibility. It caters to a diverse range of server architectures, making it a versatile choice for many AI applications, especially where the inter-GPU communication load is moderate.

However, PCIe switches have several limitations that can impact the performance of multi-GPU systems:

Bandwidth limitations

PCIe switches have limited bandwidth compared to direct GPU-to-GPU connections like NVLink.

The total bandwidth available is shared among all the connected devices, which can lead to bottlenecks when multiple GPUs are transferring large amounts of data simultaneously.

Latency

PCIe switches introduce additional latency due to the time required for data to pass through the switch. This latency can accumulate when multiple switches are used in a system, resulting in slower communication between GPUs.

Scalability

As the number of GPUs in a system increases, the number of PCIe switches required also increases, leading to more complex topologies.

This complexity can make it challenging to maintain optimal performance and can introduce additional points of failure.

Power consumption

PCIe switches consume additional power, which can be a concern in large-scale systems with many GPUs. The increased power consumption can lead to higher operating costs and may require more advanced cooling solutions.

Introduction of NVLink

To address these limitations, NVIDIA developed NVLink, which provides higher bandwidth and lower latency connections between GPUs. NVLink enables direct GPU-to-GPU communication, bypassing the need for PCIe switches.

This direct connection allows for faster data transfer and reduces the latency associated with traditional PCIe switches.

In summary, while PCIe switches play a role in connecting PCIe devices, they have limitations in terms of bandwidth, latency, scalability, power consumption, and cost.

NVLink addresses these limitations by providing high-speed, direct GPU-to-GPU connections, enabling faster and more efficient communication between GPUs in multi-GPU systems.

NVLINK - Key Points

Purpose

NVLink enables high-speed, direct communication between GPUs, reducing the overhead associated with traditional networks and allowing GPUs to work together more efficiently.

NVIDIA has introduced significant improvements to its NVLink interconnect technology.

The fifth-generation NVLink can support up to 576 GPUs concurrently, providing high-speed, low-latency communication between GPUs within a server or across a data centre.

Bandwidth

NVLink offers significantly higher bandwidth compared to PCIe. The fourth-generation NVLink provides 112Gbps per lane, which is three times faster than PCIe Gen5.

NVSwitch Chip

The NVSwitch chip is a physical chip that connects multiple GPUs using high-speed NVLink interfaces.

The third-generation NVSwitch chip (NVSwitch3) can interconnect each pair of GPUs at 900 GB/s and includes the SHARP function for aggregating and updating computation results across multiple GPU units.

NVLink Servers

NVLink servers, such as NVIDIA's DGX series or OEM HGX servers, incorporate NVLink and NVSwitch technologies to provide GPU interconnectivity, scalability, and HPC capabilities.

These servers are widely used in scientific computing, AI, big data processing, and data centres.

NVLink Switch

The NVLink Switch is a critical component of this ecosystem that enables the creation of a dedicated NVLink network for GPU-to-GPU communication.

With a bandwidth of 1.8 terabytes per second per GPU, NVLink Switch offers over 14 times the bandwidth of PCIe Gen 5.

This high-speed interconnect is crucial for achieving optimal performance in AI workloads and large-scale GPU deployments.

NVLink Network

The NVLink network is created by connecting multiple NVLink GPU servers using NVSwitch physical switches.

This network is independent of IP Ethernet and dedicated to GPU service, providing high-speed communication bandwidth and efficiency between GPUs.

Development History

NVLink has evolved alongside GPU architecture, progressing from NVLink1 for P100 to NVLink4 for H100, as depicted in the figure.

The key difference among NVLink 1.0, NVLink 2.0, NVLink 3.0, and NVLink 4.0 lies in the connection method, bandwidth, and performance.

Comparison with InfiniBand

NVLink Network is a proprietary technology designed specifically for high-speed direct connections between GPUs, while InfiniBand Network is an open-standard networking technology used in high performance computing clusters and large-scale data centres.

InfiniBand has been the go-to technology for high-performance computing applications, including AI, due to its low latency and high bandwidth.

However, Ethernet has been catching up in terms of speed and is significantly cheaper. Infiniband is relatively difficult to configure, maintain, and scale, and it requires specialised hardware, making expansion more costly than Ethernet.

Ethernet outperforms InfiniBand in terms of performance, cost-effectiveness, and openness. Ethernet's 10% performance advantage can lead to significant cost savings in large-scale AI/ML infrastructures.

NVIDIA, the main vendor for InfiniBand, is slowly migrating to an Ethernet-based framework, which could impact the future support for InfiniBand.

NVLink Network offers higher bandwidth and lower latency between GPUs compared to InfiniBand.

In summary, NVIDIA NVLink is a technology that enables fast, direct communication between GPUs, enhancing performance and enabling efficient parallel processing in HPC and AI applications.