Google Cloud wipes major customer account without warning

Google Cloud abruptly deletes a large customer account, along with all backups. Without warning.

The start of May hasn't been a good one for UniSuper, an Australian pension that manages US$135 billion worth of funds.

It had its entire account wiped out at Google Cloud on May 2. Thankfully, backups with a different cloud provider made restoration possible.

Still, a full restoration of service didn't happen until May 15, almost two full weeks later.

Imagine the CPF Board being down for so long.

So, what happened?

As with all major outages, there were multiple technical updates, PR-glossed statements, as well as updates to affected users.

What was clear:

• Google Cloud CEO signed off on an apology.
• It was the first time this happened.
• It should not happen again.

The root cause?

Miles Ward, the CTO of a cloud solutions provider appears to have the details.

In a nutshell, a manual script used to create the underlying service (Google Cloud VMware Engine) passed a null value when it shouldn't.

• This inadvertently created a one-year "subscription".
• After a year, Google proceeded to shut it down.
• This method was deprecated months ago.
• Other Google services never affected.

Multi-cloud for reliability?

It does sound like the UniSuper outage is a genuine one-of-a-kind bug, though one that shouldn't have happened - why even default to a one-year term?

The unsung hero would probably be the technical decision-maker who decided to host a separate set of backups on another cloud. In this case, it literally saved the day.

Does it mean that we should go multi-cloud? To be clear, multi-cloud deployments are complex beasts that can get very expensive very quickly.

Moreover, there may be no equivalent to some capabilities across the different clouds, necessitating kludgy fixes or keeping away from certain features. So there's no simple solution here.

For now, I just wonder if UniSuper is back on Google Cloud.