Deploying air-cooled GPUs in today's data centres

Liquid cooling is the future of AI data centres. But for most data centres, air-cooling is what's available now. Here's how air-cooled GPU servers are deployed.

Doing AI today

For large enterprises, educational institutions, or government agencies, Nvidia's H100 and H200 GPUs are the current gold standard for AI.

Both H100 and H200 GPUs are available in SXM (socketed) and PCIe (card) variants. Up to 8 of these are typically installed in a single server chassis.

Considering that each H200 (SXM) uses up to 700W, this means the power requirement adds up quickly. In fact, an Asus GPU server I looked at last week has 6x 3,000W PSUs.

PS: This means a GPU server can max out a 32A PDU all by itself; 6 x 3,000W = 18,000W (~80% loading).

Space things out

According to the 2024 Uptime Institute Survey, the average server rack density is just below 8kW. That's clearly a huge gap between what most data centres offer and the needs of modern GPU servers.

What challenges must data centres overcome to support GPU servers?

a) Power

This is easier to address. Data centre halls are designed with a certain power density in mind. But it isn't that hard to divert power meant for adjacent rows over.

Of course, considering that a rack packed with GPU servers could consume as much power as five to 10 standard racks, that's a lot of racks left unpowered or empty.

b) Cooling

This can be trickier. Traditional hot aisle configuration designed for a lower power capacity is probably not enough to support fully populated GPU racks.

One solution is to install dedicated cooling solutions with adequate cooling capacity. The easier option, though, is to simply spread-out GPU servers across multiple racks.

A typical GPU deployment

From what I've heard, GPU servers produce so much heat that most GPU deployments installed in Singapore and Malaysia have just 1-2 GPU servers (8 to 16 GPUs) in one rack.

This also alleviates the problem of network cabling. Below is the average network cables for a single GPU server:

• InfiniBand: 1x per GPU.
• 10Gb Ethernet: 2x per server.
• BlueField: 1-4 per DPU.

This means that each GPU server have between 10 to 14 networking cables. Put 5 servers in a rack and it adds up really quickly!

Of course, spreading things out is detrimental if you are Meta training Llama 4 using 100,000 GPUs - due to cabling limits.

However, smaller GPU deployments of dozens or even 100s of GPU servers will do just fine.

I had a close-up look at one of the latest Asus GPU servers last week, which I'll share tomorrow. Stay tuned!