Your backup dies with your machine

Nightly dumps, a retention window, a scheduled job. Most teams have this and consider the problem solved. Here are the three ways it is not, roughly in the order they catch people.

One: it lives on the machine it protects

A dump written to local disk on the database host protects you from bad data. It does not protect you from losing the host. Fire, a deleted volume, a compromised machine, a provider incident — the database and its backups go together, because they were always the same failure domain.

So the dump has to leave. That much is obvious. What is less obvious is how.

Two: it is mirrored, not copied

Syncing is the obvious command and the wrong one, because sync propagates deletions.

Which means local retention pruning deletes those same files from the bucket. And anything that wipes the local backup directory — a bad script, a stray recursive delete, ransomware that specifically hunts for backup directories — is replicated straight to your only other copy, promptly and reliably.

Copy only ever uploads. Nothing the syncing container can do removes a remote object. The bucket therefore grows forever, and that is deliberate: expiry belongs in a bucket lifecycle rule, set in the provider’s console, where a compromised host cannot reach it.

The thing that writes your backups should have no ability to delete them.

Three: it has never been restored

An unrestored backup is a hypothesis. The ways it fails are mundane and common: a dump taken with the wrong flags, a truncated upload, a version mismatch, a missing database extension, an encryption key nobody kept.

So every copy is verified — the sync re-reads the remote and compares hashes against local, and a silently truncated upload fails the run rather than sitting there looking fine until the day it matters.

And the restore script targets a scratch database by default. That is the point: you can prove a backup restores without touching live data, which is the only way to find out whether it is real. Rehearse it on an ordinary Tuesday, with nobody shouting.

Three smaller decisions that carry weight

The backup runs in its own container, not as a job inside the application. Backups have to keep working on exactly the days the application does not — a crash loop, a failed migration, a bad deploy. Those are the days last night’s dump matters most, and they are precisely the days an in-process scheduled job is not running.

Old backups are pruned only after a new one is verified. Prune-then-write leaves a window, however brief, in which the old copy is gone and the new one is unconfirmed. It is a small window and it is entirely avoidable.

One backup is taken at startup. Not because a dump is needed at deploy time, but because bad credentials or a broken volume mount should fail in front of someone who is watching, rather than silently at two in the morning.

None of this is clever

There is no interesting engineering in any of it. It is the entire difference between having backups and being able to say that you do.