Backups & Disaster Recovery: RPO vs RTO

RPO and RTO

• RPO is measured backwards from the disaster. It is the most data loss you accept, shown as a length of time.
  • RPO = 24 hours: daily backups are enough.
  • RPO = 5 minutes: you need continuous log shipping (sending the database's change log to another place all the time) or replication.
  • RPO ≈ 0: you need synchronous replication, where a write counts as done only after the copy has it (and you still need backups for logical errors, which means human or software mistakes).
• RTO is measured forwards from the disaster. It is the longest downtime you accept.
  • RTO = 1 day: restoring from backups onto new servers is fine.
  • RTO = 15 minutes: you need a warm standby (a smaller copy that is already running) ready to take over.
  • RTO ≈ 0: you need active-active across regions (all copies serve traffic all the time).

A lower RPO and RTO cost more money and add more complexity. Set them for each system, based on the harm to the business:

Replication is not a backup

You need both. Replication keeps the system running. Backups let you go back to an earlier time.

Types of backups

• Full backup: a complete copy of everything. It is simple to restore, but slow to make and large.
• Incremental backup: only what changed since the last backup (of any kind). It is small and fast to make. But to restore you need the full backup plus every incremental in the chain.
• Differential backup: everything that changed since the last full backup. To restore you need the full backup plus the latest differential. (Difference in one line: incremental looks back to the last backup of any kind, differential looks back to the last full one.)
• Snapshots: copies of a disk or volume at one moment (cloud disk snapshots, file-system snapshots). They are fast. For databases, check that they are consistent, which means the data is not caught half-written. Use the database's own backup tools, or pause writes for a moment.
• Logical backups: exports of the data as SQL or another format (like pg_dump). They are portable and good for small databases or single tables, but slow for large ones.
• Physical backups: copies of the database's own files (like pg_basebackup). They are fast for large databases.

Point-in-time recovery (PITR)

Point-in-time recovery (PITR) means restoring a database to an exact moment. Databases write every change to their write-ahead log first (WAL, post 21). The WAL is a record of all changes, in order. If you keep:

1. a base backup (say, nightly), plus
2. every WAL file since then (continuous archiving, often to object storage, a service for storing files),

then you restore the base backup and replay the WAL (apply the changes again, in order). You can stop at any moment, for example 10:41:59, one second before the bad DELETE ran.

PITR gives a very low RPO for logical mistakes. Managed databases (Amazon RDS, Cloud SQL, Azure Database) offer it with a few clicks. You can only go back as far as the retention window (how long data is kept). The window is set by you, often between 1 and 35 days, and the limits differ between services.

The 3-2-1 rule (and beyond)

A classic rule for safe backups is called 3-2-1:

• 3 copies of your data (the original plus 2 backups),
• on 2 different types of storage or media,
• with 1 copy off-site (in another place, such as another region or another provider).

Modern additions, especially because of ransomware:

• Immutable backups: backups that cannot be changed or deleted for a set time, even by administrators. Example: the object lock feature in S3-compatible storage.
• Isolated or "air-gapped" backups: kept in a separate account with different logins (air gapped originally meant not connected to any network). An attacker who breaks into production then cannot delete the backups too.
• Encrypted backups (scrambled so only key holders can read them), with the keys managed carefully (Part 9). A backup that you cannot decrypt is useless.

Other protective habits

• Soft deletes: mark rows as deleted (with a deleted_at column) instead of removing them at once, so they are easy to undo.
• Delayed replicas: a replica that stays, on purpose, about 1 hour behind. If a bad change happens, you can stop the replica before the change reaches it, and recover quickly.
• Audit logs: a record of who changed what and when, so you know what to restore.
• Safe migrations (changes to the database structure): scripts that are reviewed and tested, a backup taken right before a risky change, and limited direct access to production.

Disaster recovery strategies

AWS describes four common DR strategies. They go from cheapest and slowest to most expensive and fastest. A region is a cloud area, and a standby is a spare copy:

Infrastructure as code means describing your servers and networks in code files. Tools are Terraform, CloudFormation and Pulumi. It makes the cheaper strategies much faster, because you can rebuild whole environments from code and not from memory.

Test your restores

A backup that you have never restored is not a backup. It is only a hope. These are common failures that people find only during a real emergency:

• backups were silently failing for months,
• backups were empty, partial or corrupted,
• the restore takes 12 hours, far beyond the RTO,
• nobody knows the steps, or the only person who does is not available,
• the encryption keys or logins needed to restore are missing,
• the backup is fine, but the app cannot start because other pieces are missing (secrets, DNS, settings).

Good habits:

• automatic restore tests. For example, restore last night's backup to a test environment every day and run checks on it,
• regular DR drills: really switch to the DR region, or rebuild from backups, and measure how long it takes,
• monitor backups: send an alert for failed or missing backups, and for backups that are smaller than expected,
• written runbooks (step-by-step guides), updated after every drill.