The simple reason behind the AI data centre boom
Here's the explanation that's harder to explain than it looks.
Why are we building such massive AI data centres? Here's the simple explanation to a reason that's harder to explain than it looks.
You've read about it: the Stargate project, and how firms like Meta, Google, and Alibaba Cloud are splurging tens of billions on data centres to power AI.
What's more, these data centres are also getting much larger. But why?
Short reason: Scaling law
The chapter on generative AI arguably started with the release of ChatGPT in Nov 2022. Unsurprisingly, everyone wanted in - and the field exploded.
Result: "Let's build larger data centres."
Situation: As "pre-training" improvements taper off due to dwindling high-quality data, new techniques such as reinforcement training proved effective. However, it requires a lot more compute.
Result: "Let's build larger data centres."
Situation: As "pre-training" improvements taper off due to dwindling high-quality data, new techniques such as reinforcement training proved effective. However, it requires a lot more compute.
Result: "Let's build data centre campuses with multiple data centres in each compound."
Rack density now matters
As I wrote a post a week ago, rack density in the data centre has shot through the roof with AI.
Power consumption per rack:
- Estimate for 2018-2022: 5kW
- Average in 2023: 6kW
- Average in 2024: 8kW
- One rack of H200 GPUs: ~50kW
- GB200 NVL72 (2025): ~130kW
- Rubin Ultra NVL576 (2027): 600kW
AI servers consume a lot more power and require a corresponding level of cooling capacity that older data centres might not have.
With the H100 or H200 GPUs, one recommendation is to spread them across racks. I've been told that some deployments have just 1 to 2 GPU servers per rack.
But AI training won't work if latency is impacted with servers that are placed too far apart. And as GPU clusters grow larger, it's become vital to pack them as close as possible.
The AI data centre
The result is the rise of AI data centres, which are facilities designed from the get-go to accommodate 50kW, 130kW or higher workload densities per rack.
We are going into unchartered territory here because - as noted above - the average data centre today is just 8kW per rack.
Check out the details in the feature I wrote for W. Media's Jan 2025 magazine here.
I'll love to hear your thoughts.