Support and Evolution

A backup is worthless until you restore it

In this article

Nori.mx7 min read

Copy 1Office serverCopy 2External driveCopy 3OffsiteRestore testFile: invoices-2026.xlsxCopy used: 3, offsiteOn another machineIt opened in full

Ask any company whether it backs up its data and the answer is almost always “yes.” A distributor has an external drive plugged into the server. A school pays for a cloud service. A clinic has “whatever the vendor set up.” An accounting firm copies the folder on Fridays.

Now ask the other question: when was the last time you restored something from that backup, all of it, to see whether it worked? That answer is almost always silence.

A backup that has never been restored is not a backup. It is an assumption. And the day you need it is the worst day to find out it was wrong.

The case everyone in IT knows

In January 2017, an engineer at GitLab, a company whose business is hosting other companies’ code, accidentally deleted the production database. It happens: that is why backups exist. They went looking for them.

They had several backup and replication mechanisms, and according to the report the company itself published, almost none of them came through. The daily backup had been failing for a while because the tool that produced it was a different version from the database, and the bucket where it should have been stored was empty. The alerts for that error were sent by email, and the receiving server rejected them, so nobody found out. Disk snapshots were not enabled for those servers. The replica had been wiped along with the original. In the end they recovered almost everything from a six-hour-old copy that existed almost by chance, because the engineer had taken it for testing; the alternative was losing almost a full day. The restore took more than 18 hours.

The most useful part of the report is one line. Asked why nobody tested the backups regularly, the answer was: because there was no ownership, so nobody was responsible for testing it.

If that happened to a company whose business is software, it can happen to anyone.

Backing up is not syncing

The most common mix-up is thinking that because files are in OneDrive, Google Drive or Dropbox, they are already backed up.

Syncing copies everything that happens, including the bad things. If someone deletes a folder, the deletion syncs. If a virus encrypts the files on a computer, the encrypted files are uploaded over the good ones. These services have a recycle bin and version history, and they help, but they have time limits and caps, and they were not designed to recover a whole company at once.

The same goes for a mirrored disk in the server or a replicated database: they protect against a disk failing, not against someone deleting or encrypting the data. That is exactly what happened to GitLab’s replica.

A backup is something else: a separate copy, from a point in time, that whatever happens to the original cannot touch.

The 3-2-1 rule

It is the oldest rule in the trade, and the one recommended by the US government’s cybersecurity agency (CISA):

  • 3 copies of important data: the original and two backups.
  • 2 different media, so a single problem cannot take both. A drive and a cloud service, for example.
  • 1 copy offsite.

Today you need to add one thing: one of the copies must not be connected, or must not be deletable or editable for a period of time. CISA puts it plainly: many ransomware variants look for the backups within reach and delete or encrypt them before demanding the ransom. The external drive that lives plugged into the server goes down with the server.

What needs backing up, which is not the obvious part

Almost everyone backs up files. What gets forgotten:

  • The systems’ databases, which are not backed up by copying the folder while the system is running: you have to use the system’s own tool, or you get a copy that will not open.
  • The invoicing or accounting system, which often lives on a desktop computer rather than the server.
  • Email, especially if that is where customer agreements live.
  • The configuration: users, permissions, the order in which everything gets installed. Recovering the data without being able to bring up the system that reads it does not get you far.
  • The passwords and keys to get into all of the above, kept somewhere that does not depend on the server that went down.

Two numbers someone has to decide

Before buying anything, management has to answer two questions, and it is a business decision, not a technical one:

How much data can you afford to lose? If you back up every night and the server fails at six in the evening, you lose the day. For a school on break it may not matter. For a distributor that took a hundred orders that day, it does.

How long can you go without the system? A restore can take minutes or hours, depending on how much there is and where it is kept. Pulling several hundred gigabytes down from the cloud over the office connection can take more than a day.

Those two answers decide how often to back up, where, and how much to spend. Without them, any backup looks good enough until it is not.

How to prove your backup works

You do not need a weekend-long drill. This can be done in a morning:

  1. Pick something that matters: the main system’s database, not a stray file.
  2. Restore it somewhere else, never on top of what is running: a separate computer or a virtual machine.
  3. Open it with the system and check something you can verify: yesterday’s orders, the last student enrolled, the last invoice.
  4. Time it from start to finish, and compare it with how long you said you could go without the system.
  5. Write down what you did, step by step, so that on the day of the emergency it does not depend on who is around.

If any of this fails, you found the problem on a quiet day. That is exactly the point.

Then put it on the calendar. Once a quarter is a good start, plus every time something big changes: a new server, a system upgrade, a different vendor.

Give it an owner

GitLab’s lesson was not technical. They had the tools; what was missing was a person responsible for checking that they worked.

In a mid-sized company that owner can be someone in-house or the vendor that supports you, but it has to be written down: who checks that last night’s backup finished, who runs the restore test, who gets the alert when it fails and what happens if that person is on vacation. If your systems vendor tells you “yes, it gets backed up,” ask for the date of the last tested restore and how long it took. If they do not have it, you know what to ask for first.

It is one of the things a technology audit should check, and it goes hand in hand with keeping what sits underneath up to date, like a server running out of patches.

Where we come in

We provide support and evolution for the systems of companies of every kind in Juárez and El Paso, and backing up and testing the restore is part of the service, not an extra: we put it in writing, with dates and an owner. If you are not sure where you stand, Chira asks the questions and tells you in writing what to check first. And if you would rather talk it through, book a call.

References

  • Postmortem of database outage of January 31, GitLab — the company’s own report: its backup mechanisms, why they failed, the restore that took more than 18 hours and the lack of anyone responsible for testing them.
  • #StopRansomware Guide, CISA — the recommendation to keep encrypted, offline backups and regularly test that they can be restored, and the warning that ransomware looks for and deletes the backups within its reach.
  • Data Backup Options, CISA — the 3-2-1 rule: three copies, two different media and one offsite.

Related service

Support and Evolution

Launching the software is the beginning. Keeping it alive is the work.

See the service

Articles on this service

NOVEMBERSunday12:00 → 1:00Juárez and El PasoScans1:40Pallet A1st pass1:10Pallet B2nd passDowntime−45 min1:50 → 1:05actual: 15 minBy time: B before ANight shift23:00 → 7:00Clock says8 hWorked9 h

Support and Evolution

On Sunday, 1 a.m. happens twice

This Sunday Juárez sets its clocks back an hour. What breaks that night in time clocks, reports and servers, and what to check before it happens.

7 min read

Plant serverYour applicationSQL Server 2016Windows Server 2016Last patchJanuary 12, 20271Upgrade2022 or 2025, app on top2Extended updatesThrough 2030, via Azure Arc3IsolateOff the general network

Support and Evolution

Windows Server 2016 runs out of patches in January

Windows Server 2016 gets its last patch on January 12, 2027. What end of support really means, the three ways out, and why the hard part is the app that runs on it.

8 min read

final_v3system.execonfig.inidata.mdb12 years on the floorThe two-hour testIs the source code there?Is it in version control?Does it build today?Does it run on a copy of the DB?If #1 fails, it is an installation

Support and Evolution

You inherited software nobody documented

It works, nobody knows how, and the person who wrote it is gone. How to tell whether it can be rescued or has to be replaced, and what to do in week one.

7 min read