3.5 million websites went offline after a fire in a data center

On March 10, 2021, more than 3.5 million websites went offline after a fire destroyed one OVH data center and seriously damaged another. The fire broke out in Strasbourg, where the company had four data centers.

The cloud provider OVH (now OVHcloud) had 27 data centers at the time, 15 of them in Europe.

According to The Register, 115 firefighters took part in putting out the fire, working nonstop for 6 hours. The SBG2 data center was completely destroyed, and SBG1 was also badly damaged. Shortly after the fire broke out, all 4 Strasbourg data centers were shut down to limit the damage.

The company advised its clients on Twitter to activate their disaster recovery procedures, but some were caught unprepared.

For example, Trust Bank in Algeria had all its services offline for 15 hours. Notable clients affected also included the Villarreal football club and Strasbourg airport.

When this article was published, the official cause was not known; the fire was presumed to have started with a fault in the electrical system. The data restoration plan was put into action immediately, but the company estimated at the time that it would take at least 7 days for all services to work again.

In 2021, OVH was one of the largest cloud providers in Europe.

What can companies learn from this event?

However secure cloud services may be, every company needs a solid backup policy and a disaster recovery plan that works well operationally.

Some of the affected companies might not have been able to recover their data in full, and a loss like that can seriously harm a business.

Whether your infrastructure is in the cloud or consists of servers at your office, your company should have an effective backup policy.

A good practice is to have a local backup, which can be restored quickly because it’s on the same network as the servers, plus a “backup of the backup,” kept in cloud storage, in another location.

How do you prepare your company for an incident like the one at OVH?

In short: keep a copy of your data somewhere other than with your main provider, and check regularly that you can restore it. Specifically:

• Apply the 3-2-1 rule: three copies of your data, on two types of media, with one of them in another location. You’ll find details in the article on the 3-2-1 backup policy.
• Don’t keep the backup in the same building as the data. A backup in the same data center as the servers can disappear along with them.
• Read the contract with your cloud provider. Backup isn’t always included in the service. Check where the copies are kept and how long a restore takes.
• Decide how much you can afford to lose. How long the company can go without a system (RTO) and how many hours of data it can lose (RPO). These two figures determine how often you back up and where you keep the backup.
• Test restores. An untested backup is just a hope. Schedule a test restore regularly, for example once a quarter.
• Write the disaster recovery plan: who does what, in what order the systems are restarted and who you notify. For companies covered by NIS2 (in Romania, GEO 155/2024), business continuity, backup management and disaster recovery are among the mandatory measures.

RezolvIT offers Backup as a Service, with backups of your data on your network and in the cloud. If you want to know how quickly you could recover after an incident, ask for a free IT audit or contact us.