The Backups Nobody Has Tested

A backup is not a copy of your data. It is a claim that you could produce a working system from a file, and until someone has done that, it is untested.

For a separate people-operations perspective, this independent guide covers separating accountability from responsibility.

The failures below are not exotic. They are what people find the first time they try, usually during an incident, usually at the worst hour.

What actually goes wrong at restore time

The backup was empty, or nearly. A configuration change altered a path, the job kept running, and it has been writing a few kilobytes nightly for eight months. The job succeeded every night. Nobody checks size.

The backup is not consistent. Copying files from a running database without coordinating with it produces a snapshot of files at different moments. It may restore. It may restore and be subtly corrupt. Use the database's own backup mechanism, or a filesystem snapshot the database was told about.

Something was not included. The database is there; the uploaded files are not. Or the database is there and the schema migrations state is not. Or everything is there except the encryption keys, which were on the machine that died.

It cannot be decrypted. The key is in the password manager that authenticates against the system you are restoring.

Nobody knows how. No written procedure, and the person who set it up left. This is common and it is entirely preventable.

It takes far longer than anyone assumed. A terabyte restored over a link that delivers 100 Mbit/s takes roughly a day. That number was never calculated, and the recovery objective on the slide says four hours.

It restores to a version nothing else works with. The dump was taken from a newer engine than the one available now.

The two numbers, and the one that gets forgotten

Recovery point objective — how much data you can afford to lose. Determined by backup frequency. Nightly backups mean up to 24 hours lost.

Recovery time objective — how long you can afford to be down. Determined by how long a restore takes, and this is the one nobody measures.

Neither number is real until you have restored. RPO is checkable from the schedule; RTO is a guess until it has been timed with the actual data volume, over the actual network, with the actual people.

Time it, then multiply. The rehearsed restore is faster than the real one, because during the real one something else is also broken and the person doing it has not slept.

What tested actually means

Not "the job reports success." Not "the file exists." Not "we restored it once in 2023."

A full restore into a clean environment, from the backup alone, following the written procedure, by someone who did not create the backup.

That last clause is the test. If it only works when the person who built it does it, you have a person, not a procedure.

Then verify the result:

• Row counts against expectations
• The most recent records present, with the timestamp you expect given your RPO
• Referential integrity intact
• The application actually starts against it and serves a request
• A known record, chosen in advance, has the right value

Automate the restore test. Weekly, into a scratch environment, with the verification as assertions. A restore test that requires a human to remember is a restore test that stops happening in about two months.

The 3-2-1 rule and where it now falls short

Three copies, two media types, one off-site. Still a reasonable frame, with two additions that modern failures require.

One copy must be immutable or offline. Ransomware encrypts what it can reach, and a backup on a share the compromised host can write to is not a backup. Object-lock retention, write-once storage, or genuinely offline media. A backup an attacker can delete is not a backup.

One copy must be outside the account. A cloud backup in the same account as production shares a failure domain with it — a compromised or closed account takes both. A separate account, separate credentials.

And a third, less often stated: the restore procedure must be available when the systems are down. A runbook in the wiki that the outage took down is a runbook you do not have.

Backups are not replication, and neither is a snapshot

Replication protects against hardware failure. It does not protect against a bad DELETE, which replicates in milliseconds. It also does not protect against corruption, which replicates just as fast.

Snapshots are fast and local. Excellent for quick recovery, useless if the storage system or account is the thing that failed.

Point-in-time recovery is the one that saves you from a bad statement. A base backup plus the write-ahead log lets you restore to the moment before the mistake. If you have it, know the exact command, and know how far back the log retention goes — that window is your real protection against human error, and it is often shorter than people think.

All three are useful and they are not substitutes for each other.

Restoring a single row

Most real recovery is not the whole system. Someone deleted one customer's data, or one table, and everything else is fine.

Restoring the entire database to recover one row is usually unacceptable — it loses everything written since.

Practise the partial restore: restore into a separate instance, extract what you need, insert it back. This is a different procedure from a full restore, it is far more frequently required, and it is almost never documented.

The checklist worth running this quarter

• [ ] Restore into a clean environment, from the backup alone
• [ ] Done by someone who did not build it, following the written procedure
• [ ] Time it, and compare against the stated RTO
• [ ] Check the most recent data present, against the stated RPO
• [ ] Start the application against the restored data and serve a request
• [ ] Verify a specific known record
• [ ] Confirm the encryption key is reachable without the system being restored
• [ ] Confirm one copy cannot be deleted by a compromised production host
• [ ] Confirm one copy is outside the production account
• [ ] Practise recovering a single table, not just everything
• [ ] Confirm the runbook is readable when the primary systems are down

Alert on backup size and age, not just on job success. A job that succeeds while producing nothing is the failure this catches, and it is more common than a job that fails.

The summary

An untested backup is a hypothesis, and the test is a restore by someone else, from the procedure, timed.

RTO is a guess until measured. Measure it with the real data volume.

One copy immutable, one copy outside the account — the two additions that ransomware and account-level failures made necessary.

Practise the single-table restore, because that is the one you will actually need.