Skip to content

docs: add a backup recovery drill and fix the database restore steps - #260

Merged
rfay merged 1 commit into
mainfrom
20261117_rfay_backup_restore_drill
Nov 17, 2026
Merged

rfay merged 1 commit into
mainfrom
20261117_rfay_backup_restore_drill

Conversation

@rfay

@rfay rfay commented Nov 17, 2026

Copy link
Copy Markdown
Member

Short Summary (TL;DR)

The operations guide now explains how to recover the Coder database backup on another machine, using only Docker and the secrets in 1Password, with a new scripts/coder-backup-verify.sh. It also fixes the restore steps for a lost server, which could not work as written.

The Issue

The only documented check of the backup, coder-restore-test, runs on the server itself. After the server is lost, nothing showed that the backup could be recovered with only the secrets stored elsewhere.

The "Restore after losing the server" steps had three errors:

  • Step 1 followed the setup through Step 5, which creates the coder database and user, but step 3 creates them again.
  • Step 3 said to create the coder user but gave no command for it.
  • sudo -u postgres pg_restore /root/restore/... fails, because the postgres user can't read /root.

How This PR Solves The Issue

  • New "Practice a recovery" section. It writes the backup secrets from 1Password to a private env file and runs scripts/coder-backup-verify.sh. It also says what to check in the output. It recommends a drill once a quarter and after any change to the backup setup.
  • scripts/coder-backup-verify.sh downloads the latest backup with duplicity in an ubuntu:24.04 container. It loads the dump into a postgres:16 container and lists users, templates, workspaces and the newest audit log entry. The restored files contain secrets, so it deletes them on exit.
  • The restore steps now skip "Create the Coder database and user" in the setup. They create the user with the password from the backup, pipe the dump to pg_restore, and install /root/coder-pg-password.

Manual Testing Instructions

On any machine with Docker and the 1Password CLI, follow Practice a recovery with your vault and item names. Expect:

Chain end time: <last night's backup>
present  /var/backups/coder/coder.dump
present  /root/coder-pg-password
present  /etc/coder.d/coder.env
== Users
...

The users, templates and workspaces should match coder users list and coder list -a. Afterwards, ls -d "${TMPDIR:-/tmp}"/coder-backup-verify.* should find nothing.

Automated Testing Overview

  • Claude Code ran the documented drill on macOS with OrbStack against the coder.ddev.com backup from 2026-11-17 02:38 UTC. It restored 3 users, 4 templates and 2 workspaces, which match the live server, and it left no files or containers behind.
  • Claude Code ran restore step 3 verbatim in an ubuntu:24.04 container with Ubuntu's PostgreSQL 16 and a test dump. The coder user logged in with a restored password containing quotes, $, a backtick and a space, and the restored table kept coder as owner.
  • Claude Code ran shellcheck on the script and make docs-check, which passed.
  • Nobody has tested a full server restore with steps 4 and 5.

🤖 Generated with Claude Code

## Short Summary (TL;DR)

The operations guide now explains how to recover the Coder database backup on another machine, using only Docker and the secrets in 1Password, with a new `scripts/coder-backup-verify.sh`. It also fixes the restore steps for a lost server, which could not work as written.

## The Issue

- Refs #259, which lists a practice recovery as a follow-up to the 2026-11-16 outage.

The only documented check of the backup, `coder-restore-test`, runs on the server itself. After the server is lost, nothing showed that the backup could be recovered with only the secrets stored elsewhere.

The "Restore after losing the server" steps had three errors:

- Step 1 followed the setup through Step 5, which creates the `coder` database and user, but step 3 creates them again.
- Step 3 said to create the `coder` user but gave no command for it.
- `sudo -u postgres pg_restore /root/restore/...` fails, because the `postgres` user can't read `/root`.

## How This PR Solves The Issue

- New "Practice a recovery" section. It writes the backup secrets from 1Password to a private env file and runs `scripts/coder-backup-verify.sh`. It also says what to check in the output. It recommends a drill once a quarter and after any change to the backup setup.
- `scripts/coder-backup-verify.sh` downloads the latest backup with duplicity in an `ubuntu:24.04` container. It loads the dump into a `postgres:16` container and lists users, templates, workspaces and the newest audit log entry. The restored files contain secrets, so it deletes them on exit.
- The restore steps now skip "Create the Coder database and user" in the setup. They create the user with the password from the backup, pipe the dump to `pg_restore`, and install `/root/coder-pg-password`.

## Manual Testing Instructions

On any machine with Docker and the 1Password CLI, follow [Practice a recovery](https://github.com/ddev/coder-ddev/blob/20261117_rfay_backup_restore_drill/docs/content/self-hosting/operations-guide.md#practice-a-recovery) with your vault and item names. Expect:

```text
Chain end time: <last night's backup>
present  /var/backups/coder/coder.dump
present  /root/coder-pg-password
present  /etc/coder.d/coder.env
== Users
...
```

The users, templates and workspaces should match `coder users list` and `coder list -a`. Afterwards, `ls -d "${TMPDIR:-/tmp}"/coder-backup-verify.*` should find nothing.

## Automated Testing Overview

- Claude Code ran the documented drill on macOS with OrbStack against the coder.ddev.com backup from 2026-11-17 02:38 UTC. It restored 3 users, 4 templates and 2 workspaces, which match the live server, and it left no files or containers behind.
- Claude Code ran restore step 3 verbatim in an `ubuntu:24.04` container with Ubuntu's PostgreSQL 16 and a test dump. The `coder` user logged in with a restored password containing quotes, `$`, a backtick and a space, and the restored table kept `coder` as owner.
- Claude Code ran `shellcheck` on the script and `make docs-check`, which passed.
- Nobody has tested a full server restore with steps 4 and 5.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Previews

The start.coder.ddev.com landing-page preview has been removed. Read the Docs keeps or removes its own docs preview.

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 coder-ddev | 🛠️ Build #35961628 | 📁 Comparing a2080c7 against latest (4e928dc)

  🔍 Preview build  

1 file changed
± self-hosting/operations-guide/index.html

@rfay
rfay merged commit 8e550ed into main Nov 17, 2026
21 checks passed
@rfay
rfay deleted the 20261117_rfay_backup_restore_drill branch November 17, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant