AI

100,000-strong GPU clusters are next

LLMs show no sign of plateauing, says OpenAI's CTO Mira Murati.

Paul Mah

Paul Mah

— 1 min read
100,000-strong GPU clusters are next
Photo Credit: Unsplash/Ludde Lorentz

LLMs show no sign of plateauing, says OpenAI's CTO Mira Murati - so we can expect ever-larger AI models.

Mira spoke at a fireside chat at Qualtrics X4 which I attended in May, and recent developments show that we are indeed on the trajectory she painted.

Here are some comments she made that caught my interest:

  1. Expect another leap in performance

Even the researchers at OpenAI were astounded by the new capabilities that GPT-3, then GPT-4 delivered. She expects GPT-5 to offer yet another "step change" in new capabilities.

  1. How you are helping train AI

As observers have long pointed out, OpenAI isn't offering a free version of ChatGPT for purely altruistic reasons. By Mira's admission, inputs from users are used for reinforcement learning, helping make AI more steerable and useful".

  1. 100,000 GPU AI models coming soon

Mira says there is no sign that the scaling paradigm - which posits that AI models will become more powerful with more compute and data - will taper off soon.

  • To add to point 3, we know that LLama 3 was trained on 24,000 GPU clusters and that Meta is currently working to build GPU clusters in the "hundreds of thousands".
  • Elon Musk has separately told investors he wants to deploy a 100,000 H100 GPU data centre for his xAI startup, which recently closed a US$6 billion fundraising round.

The next generation of AI models will be trained on supercomputers with GPUs numbering in the 100,000s - likely leaving the supercomputers of just a few years ago in the dust.

Where will the AI race end? Or should I ask instead: How will the AI race end?

Read my full article here.

Read more