Repository navigation
docs: add a backup recovery drill and fix the database restore steps - #260
Merged
Merged
Conversation
## Short Summary (TL;DR) The operations guide now explains how to recover the Coder database backup on another machine, using only Docker and the secrets in 1Password, with a new `scripts/coder-backup-verify.sh`. It also fixes the restore steps for a lost server, which could not work as written. ## The Issue - Refs #259, which lists a practice recovery as a follow-up to the 2026-11-16 outage. The only documented check of the backup, `coder-restore-test`, runs on the server itself. After the server is lost, nothing showed that the backup could be recovered with only the secrets stored elsewhere. The "Restore after losing the server" steps had three errors: - Step 1 followed the setup through Step 5, which creates the `coder` database and user, but step 3 creates them again. - Step 3 said to create the `coder` user but gave no command for it. - `sudo -u postgres pg_restore /root/restore/...` fails, because the `postgres` user can't read `/root`. ## How This PR Solves The Issue - New "Practice a recovery" section. It writes the backup secrets from 1Password to a private env file and runs `scripts/coder-backup-verify.sh`. It also says what to check in the output. It recommends a drill once a quarter and after any change to the backup setup. - `scripts/coder-backup-verify.sh` downloads the latest backup with duplicity in an `ubuntu:24.04` container. It loads the dump into a `postgres:16` container and lists users, templates, workspaces and the newest audit log entry. The restored files contain secrets, so it deletes them on exit. - The restore steps now skip "Create the Coder database and user" in the setup. They create the user with the password from the backup, pipe the dump to `pg_restore`, and install `/root/coder-pg-password`. ## Manual Testing Instructions On any machine with Docker and the 1Password CLI, follow [Practice a recovery](https://github.com/ddev/coder-ddev/blob/20261117_rfay_backup_restore_drill/docs/content/self-hosting/operations-guide.md#practice-a-recovery) with your vault and item names. Expect: ```text Chain end time: <last night's backup> present /var/backups/coder/coder.dump present /root/coder-pg-password present /etc/coder.d/coder.env == Users ... ``` The users, templates and workspaces should match `coder users list` and `coder list -a`. Afterwards, `ls -d "${TMPDIR:-/tmp}"/coder-backup-verify.*` should find nothing. ## Automated Testing Overview - Claude Code ran the documented drill on macOS with OrbStack against the coder.ddev.com backup from 2026-11-17 02:38 UTC. It restored 3 users, 4 templates and 2 workspaces, which match the live server, and it left no files or containers behind. - Claude Code ran restore step 3 verbatim in an `ubuntu:24.04` container with Ubuntu's PostgreSQL 16 and a test dump. The `coder` user logged in with a restored password containing quotes, `$`, a backtick and a space, and the restored table kept `coder` as owner. - Claude Code ran `shellcheck` on the script and `make docs-check`, which passed. - Nobody has tested a full server restore with steps 4 and 5. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 of 6 tasks
PreviewsThe start.coder.ddev.com landing-page preview has been removed. Read the Docs keeps or removes its own docs preview. |
Documentation build overview
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Summary (TL;DR)
The operations guide now explains how to recover the Coder database backup on another machine, using only Docker and the secrets in 1Password, with a new
scripts/coder-backup-verify.sh. It also fixes the restore steps for a lost server, which could not work as written.The Issue
The only documented check of the backup,
coder-restore-test, runs on the server itself. After the server is lost, nothing showed that the backup could be recovered with only the secrets stored elsewhere.The "Restore after losing the server" steps had three errors:
coderdatabase and user, but step 3 creates them again.coderuser but gave no command for it.sudo -u postgres pg_restore /root/restore/...fails, because thepostgresuser can't read/root.How This PR Solves The Issue
scripts/coder-backup-verify.sh. It also says what to check in the output. It recommends a drill once a quarter and after any change to the backup setup.scripts/coder-backup-verify.shdownloads the latest backup with duplicity in anubuntu:24.04container. It loads the dump into apostgres:16container and lists users, templates, workspaces and the newest audit log entry. The restored files contain secrets, so it deletes them on exit.pg_restore, and install/root/coder-pg-password.Manual Testing Instructions
On any machine with Docker and the 1Password CLI, follow Practice a recovery with your vault and item names. Expect:
The users, templates and workspaces should match
coder users listandcoder list -a. Afterwards,ls -d "${TMPDIR:-/tmp}"/coder-backup-verify.*should find nothing.Automated Testing Overview
ubuntu:24.04container with Ubuntu's PostgreSQL 16 and a test dump. Thecoderuser logged in with a restored password containing quotes,$, a backtick and a space, and the restored table keptcoderas owner.shellcheckon the script andmake docs-check, which passed.🤖 Generated with Claude Code