The modern data centre

This is an analysis of an article by the semiconductor genius Doug O'Laughlin:

The Data Center is the New Compute Unit: Nvidia's Vision for System-Level Scaling

The rack itself is similar to a chip

When the author says "the rack itself is similar to a chip," he is drawing an analogy between the structure and function of a computer chip and a data centre rack.

Let's break this down further.

A computer chip, such as a CPU or GPU, is composed of several components:

1. Transistors: The basic building blocks that perform logical operations.

2. Memory: On-chip memory, such as caches or registers, for fast data access.

3. Interconnects: Wires that connect different components within the chip.

4. I/O: Pins or ports that allow the chip to communicate with external components.

Similarly, a data centre rack can be thought of as a larger-scale "chip" with analogous components:

1. Servers: The individual computing units, containing CPUs, GPUs, and memory, analogous to transistors.

2. Storage: High-capacity storage devices, such as SSDs or HDDs, analogous to on-chip memory.

3. Networking: High-speed interconnects, like Ethernet or Infiniband, that connect servers within and between racks, analogous to on-chip interconnects.

4. Power and Cooling: Infrastructure that supplies power and removes heat, analogous to the power delivery and heat dissipation systems of a chip.

Moore's Law, which originally described the trend of doubling the number of transistors on a chip every two years, led to significant improvements in chip performance and power efficiency.

The author suggests that by applying similar principles to data centre racks, we can achieve comparable gains at the rack level.

Just as chip designers optimise the placement and connectivity of transistors, memory, and interconnects to maximise performance and minimize power consumption, data centre architects can optimise the arrangement and interconnection of servers, storage, and networking components within a rack.

This optimisation includes:

Increasing component density: Packing more servers and storage devices into a rack to increase computing power and capacity.

Improving interconnect bandwidth: Using faster and more efficient networking technologies, like NVLink or Infiniband, to reduce communication bottlenecks between components.

Enhancing power and cooling efficiency: Implementing advanced power delivery and cooling systems, such as liquid cooling, to support higher power densities and reduce energy consumption.

By treating the rack as a "giant chip" and applying these optimisation techniques, data centre designers can achieve significant improvements in performance, power efficiency, and scalability, similar to the gains driven by Moore's Law at the chip level.

This approach allows for a new vector of growth and innovation in data centre architecture, beyond the traditional focus on individual server or chip performance.

Overall - the author is suggesting that the principles that have driven the evolution of computer chips can now be applied to the larger scale of data centre racks, opening up new opportunities for performance and efficiency gains in the realm of high-performance computing and artificial intelligence.

Core Thesis

Nvidia CEO Jensen Huang believes that the data centre is the new unit of compute, similar to how the chip was the unit of compute in the past.

Nvidia is applying the principles of Moore's Law, which drove chip-level advancements, to the data centre level.

Miniaturisation

Just as transistors were shrunk to improve performance and power efficiency, Nvidia is focusing on moving components closer together within the data centre to reduce power consumption and increase performance.

System Technology Co-Optimisation (STCO)

To execute the process the System Technology Co-Optimisation (STCO) approach is being used.

This approach seeks to optimise the entire computing system, including chips, interconnects, packaging, and even the rack-level architecture, to achieve better overall performance and efficiency.

This holistic view of system design is becoming increasingly important as the traditional methods of improving performance, such as transistor scaling (Moore's Law), are reaching their limits.

Chip-level optimisation

This involves designing chips that are optimised for specific workloads, such as AI or high-performance computing (HPC).

Techniques include the use of domain-specific architectures (DSAs), the integration of multiple types of cores (e.g., CPUs, GPUs, and AI accelerators), and the use of advanced packaging technologies like 2.5D and 3D integration.

Interconnect optimisation

As the number of components within a system grows, the interconnects between them become a critical factor in determining overall performance.

STCO involves the use of high-bandwidth, low-latency interconnects like Nvidia's NVLink, as well as the optimisation of interconnect topologies to minimise data movement and energy consumption.

Rack-level optimisation

STCO extends beyond individual nodes to the entire rack or even data centre level.

This includes the use of high-speed interconnects between nodes, the optimisation of power delivery and cooling systems, and the use of software-defined infrastructure to dynamically allocate resources based on workload requirements.

Co-design of hardware and software

STCO involves the close collaboration between hardware and software teams to ensure that the system is optimised for the target workloads.

This includes the development of software libraries and frameworks that are optimised for the specific hardware architecture, as well as the use of hardware-software co-design methodologies to explore the design space and make informed trade-offs.

As the demand for high-performance, energy-efficient computing continues to grow, driven by applications like AI, data analytics, and scientific simulations, STCO is expected to become an increasingly important approach to system design.

Companies like Nvidia, AMD, and Intel are already employing STCO techniques in their latest products, and this trend is likely to continue in the future.

Copper Interconnects

The article emphasises the importance of using passive copper interconnects (like NVLink) as much as possible before resorting to more expensive options like optics.

Keeping the interconnects short and efficient is also key to system performance.

Passive copper interconnects within the rack are seen as the most cost-effective and power-efficient solution before relying on optics for networking. So passive copper interconnects should be used as much as possible before resorting to more expensive options like optics.

Copper interconnects can support very high bandwidths over short distances.

For example, Nvidia's NVLink provides up to 900 GB/s of bandwidth between GPUs, which is much higher than what can be achieved with traditional PCIe interconnects. This high bandwidth is essential for data-intensive workloads like AI and HPC.

Scale up as much memory in HBM3 before we consider NVLink and then try to keep as much computing within NVLink before considering scaling to the network

Copper interconnects can provide very low latency communication between components, which is critical for workloads that require frequent communication and synchronisation between nodes. By keeping the interconnects short and using high-speed signalling techniques, latency can be minimised.

Passive copper interconnects are more energy-efficient than active optical interconnects, especially over short distances. This is because passive copper interconnects do not require the same level of signal conditioning and regeneration as optical interconnects, which can consume significant amounts of power.

In the context of rack-scale systems like Nvidia's DGX SuperPOD, the use of passive copper interconnects like NVLink enables higher power density and faster communication between GPUs and other components within the rack.

By keeping the interconnects short and using high-speed signalling techniques, the system can achieve very high bandwidth and low latency communication between nodes.

However, as the size and complexity of rack-scale systems continue to grow, there is a limit to how far copper interconnects can be used effectively.

Beyond a certain distance (typically a few metres), the signal integrity and power efficiency of copper interconnects degrade rapidly. This is where optical interconnects become necessary to maintain high bandwidth and low latency communication between racks and across larger distances.

NVLINK - use of copper

NVLink is a high-bandwidth, low-latency interconnect that allows GPUs to communicate directly with each other, bypassing the traditional PCIe bus.

By using passive copper cables, NVLink can achieve much higher bandwidth and lower latency compared to traditional networking technologies like InfiniBand or Ethernet.

Comparing InfiniBand and Ethernet misses the point because the real advantage lies in the NVLink domain created within a rack using passive copper connections.

The NVLink domain allows GPUs to work together as a single, massive GPU, enabling faster communication and more efficient parallel processing.

By focusing on the NVLink domain and the compute density within a rack will lead to better performance, scalability, and efficiency compared to traditional distributed computing methods that rely on high-speed networking between nodes.

Liquid Cooling

Liquid cooling has emerged as a necessary technology for enabling higher power densities in rack-scale systems, particularly for AI and high-performance computing workloads.

As the power consumption of individual components like CPUs and GPUs continues to rise, traditional air cooling methods are reaching their limits in terms of the amount of heat they can effectively remove from a rack.

Liquid cooling allows for much higher power densities than air cooling, because liquids have a much higher heat capacity and thermal conductivity than air. This means that liquid cooling can remove more heat from a given volume than air cooling, enabling higher power densities in a rack.

The latest liquid cooling technologies, such as cold plate cooling and immersion cooling, allow for direct cooling of high-power components like CPUs and GPUs. By bringing the cooling liquid in direct contact with the heat-generating components, these technologies can provide more effective cooling than traditional air cooling methods.

Many of the latest liquid cooling technologies, such as rack-level liquid cooling and immersion cooling, are designed to be modular and scalable. This means that they can be easily deployed in existing data centre infrastructure, and can be scaled up as the power density and cooling requirements of the system increase.

The shift to liquid cooling is allowing for a doubling of the power density in a rack, with the potential to reach 300+ kW in a single rack in the future.

This is a significant increase over the power densities that are currently achievable with air cooling, which are typically in the range of 20-30 kW per rack.

To put this in perspective, a single rack with 300 kW of power consumption would be equivalent to the power consumption of several hundred high-end gaming PCs. This level of power density is essential for enabling the next generation of AI and HPC systems, which will require even more processing power and memory bandwidth than today's systems.

However, deploying liquid cooling at scale also presents some challenges, such as the need for specialised infrastructure and the potential for leaks and spills.

As a result, the adoption of liquid cooling is likely to be gradual, with early adopters being those who have the most demanding workloads and the greatest need for high power densities.