Skip to content

postgres: add support for restoring backups #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@ jobs:
popd

ansible:
strategy:
matrix:
db_backup:
# Clean install
- ''
# Restore a database backup
- 'testdata/xsnippet-api_20241003-030004.pgc'

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -96,8 +104,28 @@ jobs:

- name: Run the playbook
run: |
read -r -d '' extra_vars << 'EOF' || true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dark magic. 🪄 Apparently this is a shell builtin, and in fish it's different.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I was just struggling to find a different way how to pass that many variables to ansible w/o creating a temporary YAML file :( We could do that instead to make this portable across different shells.

{
"volume_device": "${{ steps.volume-device.outputs.uri }}",
"postgres_users": [
{
"database": "{{ xsnippet_api_user }}",
"username": "{{ xsnippet_api_user }}",
"backup_schedule": "*-*-* 3:00:00",
"backup_restore": "${{ matrix.db_backup }}"
}
]
}
EOF

ansible-playbook \
-vvv \
-e volume_device="${{ steps.volume-device.outputs.uri }}" \
-e "${extra_vars}" \
--inventory inventories/ci \
site.yml

- name: Verify that the database backup has been restored correctly
if: matrix.db_backup != ''
run: |
# Expect at least one full page of results
test "$(curl http://127.0.0.1:8080/v1/snippets | jq length)" == "20"
5 changes: 5 additions & 0 deletions roles/postgres/meta/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ argument_specs:
description: |
The time of when database backups should be triggered. Uses the systemd calendar event expression syntax (see man 7 systemd.time).
If not set, backups will not be created.
backup_restore:
type: str
required: false
description: |
Path to a database backup to be restored.
default: []
description: |
The list of database/username pairs to create.
6 changes: 6 additions & 0 deletions roles/postgres/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@
become: true
become_user: postgres

- name: Restore database backups if necessary
ansible.builtin.include_tasks:
file: restore.yml
with_items: "{{ postgres_users }}"
when: item.backup_restore is defined and item.backup_restore

- name: Install the script for backup rotation
ansible.builtin.copy:
src: 'rotate.py'
Expand Down
35 changes: 35 additions & 0 deletions roles/postgres/tasks/restore.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
- name: Restore database backups if necessary
become: true
become_user: postgres
block:
- name: Check if we need to restore a database backup
community.postgresql.postgresql_query:
db: "{{ item.database }}"
# pg_tables is a system view that is implicitly created in every
# database. The information in is local to that particular database.
query: "SELECT * FROM pg_catalog.pg_tables WHERE tableowner = %s"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the user we have owns multiple databases? Is there any way to check if there are any tables in the database instead of checking owned tables number across the server?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it's grand: pg_catalog.pg_tables is a magical system view that exists in all databases, and all the information is local to a particular database.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and we connect to the database above.

positional_args:
- "{{ item.username }}"
register: existing_tables

- name: Copy and restore a database backup (only if the database is empty)
when: existing_tables.rowcount == 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this to prevent restoration when the database is not empty?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct! Let me add a comment.

block:
- name: Create a temporary backup directory
ansible.builtin.tempfile:
state: directory
suffix: backup
register: backup_tmp_dir

- name: Copy the database backup
ansible.builtin.copy:
src: "{{ item.backup_restore }}"
dest: "{{ [backup_tmp_dir.path, item.backup_restore | basename] | path_join }}"
mode: 'u=rw,g=r,o='

- name: Restore the database backup
community.postgresql.postgresql_db:
name: "{{ item.database }}"
state: "restore"
target: "{{ [backup_tmp_dir.path, item.backup_restore | basename] | path_join }}"
target_opts: "--single-transaction --exit-on-error"
Binary file added testdata/xsnippet-api_20241003-030004.pgc
Binary file not shown.
Loading