Made in Singapore: The story behind the SEA-LION LLM

"We want to do things correctly and uphold a higher standard with our AI model in Singapore."

Dr Leslie Teo, senior director of AI Products at AI Singapore, said this at a media briefing this afternoon.

First, the context.

An LLM for Southeast Asia

In the briefing, Leslie and team members shared about SEA-LION and pitched it against language models like Meta's Llama, Alibaba's SeaLLM, and OpenAI's GPT-4 Turbo.

And SEA-LION apparently fares well when quizzed in regional languages such as Bahasa Indonesia, Thai, or even Tamil, on local topics.

As it should: The LLM was created in Singapore to outperform other LLMs on Southeast Asian tasks, trained on some 128 billion SEA-centric tokens. (It has 981 billion overall tokens).

There are currently three models:

• SEA-LION 3B
• SEA-LION 7B
• SEA-LION 7B Instruct

And it's put together by a team of just 20 Singaporeans.

Ethical data sourcing

Last week, I wrote about how tech giants such as OpenAI, Meta, and Google have cut corners to harvest enough high-quality data to train their AI models.

In many cases, published books, YouTube videos and online content are used without permission. Read about it here: https://lnkd.in/gP42YFi8

It was clear from the get-go that the AI Singapore team took great pains to source their data ethically.

"We are aware of what others are doing... But because of who we are, we are very conservative. We have to make sure we are doing things correctly."

In fact, Leslie shared how he turned down data brokers who offered to sell high-quality data from dubious or unknown sources.

"We pay a price... our models will not be as good," he conceded.

Too cheap to believe?

When I wrote about Singapore's National Multimodel LLM Programme (NMLP) and the budget of SG$70 million for it, it garnered substantial interest.

Read about it here: https://lnkd.in/gfWReYS8

There was also some incredulity that a "proper" LLM could be trained so cheaply.

I didn't know then, but the NMLP is essentially about creating newer, better versions of SEA-LION. And not all the money will go towards GPUs.

Leslie says the money is enough for the next two SEA-LION models planned.

🔸Leslie says he is not working to pit SEA-LION against state-of-the-art AI LLMs like GPT-4 with its estimated trillion parameters.

🔸Moreover, GPU prices are trending down right now; he's also working with a motivated, mission-driven team that is paid "academic rates".

Ultimately, Leslie sees the development of SEA-LION as a sacred mission to both fill a gap and put Singapore on the global map for AI.

"We believe in the mission. We are mission-driven."