The energy challenge of powering AI chips

The pace of artificial intelligence (AI) software adoption is one of the fastest adoption curves markets have ever seen. The large language models (LLM) used by ChatGPT and similar AI bots to generate humanlike conversations are just one of many new AI applications that rely on ‘parallel computing’, the term used to describe the massive computational work performed by networks of chips that carry out many calculations or processes simultaneously.

At the core of the AI infrastructure are GPUs (graphic processing units) that excel at the type of specialized, high-performance parallel computing work required by AI. All that processing power also results in higher energy inputs that generate more heat output compared to their CPU (central processing units) counterparts used in PCs. See Figure 1.

High-end GPUs are about four times more power dense than CPUs. This constitutes significant new problems for the data center planning as the originally calculated power supply is now only 25% of what is needed to run modern AI data centers. Even the cutting-edge, hyperscaler data centers used by Amazon, Microsoft and Alphabet for cloud-based computing are still CPU-driven. To illustrate, Nvidia’s currently supplied A100 AI chip has a constant power consumption of roughly 400W per chip, whereas the power intake of its latest microchip, the H100, nearly doubles that – consuming 700W, similar to a microwave. If a full hyperscaler data center with an average of one million servers replaced its current CPU servers with these types of GPUs, the power needed would increase 4-5 times (1500MW) – equivalent to a nuclear power station!

This increase in power density means that these chips also generate significantly more heat. Consequently, the cooling systems also have to become more powerful. Power and cooling changes of these magnitudes will require totally new designs for future AI-driven data centers. This creates a huge demand-supply imbalance on the underlying chip and data center infrastructure. Given the time it takes to build data centers, industry experts project that we are in the first innings of a decade-long modernization of data centers to make them more intelligent.

Structural changes of this magnitude will lead to massive upgrades of not only chips and servers but also the electric infrastructure that supplies them with the power to operate.

The laws of physics dictate that increasing power supply means increasing electric voltage and/or current (Power = voltage x current). Increasing voltage is the more viable option.¹ Accordingly, the industry is working on increasing the voltage, which requires redesigning many standard components that were defined in the early era of computers and servers, when power densities were relatively low (2-3kW per rack). This means new configurations for power cords, power distribution racks and converters as they exceed the current formats (See Figure 3). On a chip level, even bigger challenges exist, as higher voltage and therefore more power for the GPU requires a full redesign of the chip power layout.

Data center cooling is key to ensure high system performance and prevent malfunctioning. Traditional HVAC² solutions that use air conditioning and fans to cool the air in data center server rooms are sufficient for CPUs whose server racks manage power densities between 3-30 kW, but not for GPUs whose power densities easily go beyond 40kW.³ As the latest GPU racks exceed these power levels, additional liquid cooling is (once again) at the forefront. They allow for even higher heat dissipation on a server rack or chip level because liquids have a greater capacity to capture heat per unit volume than air. Some of the biggest challenges for liquid cooling however are 1) the lack of standardized designs and components for such systems, 2) different technology options such as chip or rack cooling, and 3) high costs for pipes and measures to prevent leakage.

The AI revolution will require nothing short of a complete re-engineering of the data center infrastructure from the inside out to accommodate for the much higher energy needs of new AI technology. This will result in a strong boost in demand and investments for data centers on low-power computing applications, energy-efficient HVAC and power management solutions – all of these Big Data energy efficiency solutions are key investment areas for the smart energy strategy.

Revenues for companies supplying energy efficiency solutions to data centers are expected to show strong growth. This has also led to a re-rating in the market with these companies now trading more on the upper end of their historical range on the back of the much stronger growth outlook. Given the strong momentum and underlying structural drivers, we fundamentally really like this part of the smart energy investment universe. We also believe that this area can decouple from a potential recession, as spending on much-needed data centers and related energy efficiency solutions will not be dependent on the economic cycle.