Skip to content

River Migrations: DDL and schema version bookkeeping are not atomic when using rivermigrate.Migrate leading to partial migrations #1118

@jpixel745-dev

Description

@jpixel745-dev

We are using River fully programmatically in a Go service using rivermigrate.Migrate Go api as mentioned here: https://riverqueue.com/docs/migrations.

We observed that the river migration process can leave the database in a partially-applied state when the service crashes or restarts in the middle of the migration. Sharing the steps below to reproduce the issue:

  1. River successfully applies all migration files having DDL commands (creates tables / indexes / constraints) (ex.: riverqueue/river/riverdriver/riverdatabasesql/migration/main/001_create_river_migration.up.sql)

  2. Schema versions applied is yet to be recorded in river_migration table and the application crashes or restarts

  3. On the next startup, River retries the same full migration as there are no version entries recorded in river_migration table and the migration fails because the DDL already exists(PostgresSql errors like: relation already exists or index already exists)

This requires manual intervention against River’s internal tables, which is undesirable for application operators.

Expected Behavior:

River migrations should be:

  1. Atomic (DDL + schema version bookkeeping happen together in single transaction)

or

  1. DDL commands are Idempotent (safe to re-run)

This ensures that applications will not require to touch internal River tables manually.

Because of this potential issue, it prevents the service from starting until River’s tables are manually removed. Please suggest on how to proceed in such situations and how others are addressing this potential issue?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions