Skip to content

Re-drive index/constraint creation for split tables on --resume - #49

Merged
teknogeek0 merged 1 commit into
mainfrom
fix/resume-split-table-index-election
Jul 24, 2026
Merged

Re-drive index/constraint creation for split tables on --resume#49
teknogeek0 merged 1 commit into
mainfrom
fix/resume-split-table-index-election

Conversation

@teknogeek0

Copy link
Copy Markdown
Collaborator

Problem

A migration of a large natively-partitioned table (composite PK, many partitions) failed and was resumed. After clone --resume, pg_restore/pgcopydb threw a cascade of errors:

  • relation "<table>_p32_pkey" already exists (leaf ADD CONSTRAINT)
  • cannot attach index "<table>_p32_pkey" ... no constraint exists for index (ATTACH PARTITION)
  • there is no unique constraint matching given keys for referenced table "<table>" (FKs)

Exactly one partition was left as a bare index: pgcopydb built its index in the first run but crashed before promoting it to a PK constraint, and the resume run never re-drove that work.

Root cause

For a table split into multiple parallel COPY parts, index/constraint enqueue is gated on a per-run "who finished the last part" election:

  • copydb_table_parts_are_all_done stores the current getpid() into the SQLite table s_table_parts_done(tableoid, pid).
  • The enqueue gate only fires for the worker whose getpid() equals the stored pid.
  • On resume, the stored pid belongs to a dead prior-run process. No live worker matches, so copydb_add_table_indexes is never called and the table's indexes/constraints are never re-enqueued. Whatever the prior run left half-built stays that way.

The count of done parts comes from the summary table's done_time_epoch, not from s_table_parts_done — so s_table_parts_done / s_table_indexes_done hold only per-run election tokens, not progress. The sibling election s_table_indexes_done (which elects who builds constraints once all indexes are done) has the identical stale-pid flaw.

Only tables with partCount > 1 are affected — single-part tables short-circuit the election.

Fix

Clear both election tables once at the start of a --resume clone, single-threaded and before any table-data worker is forked (in cloneDB, right after schema fetch, so it serves both clone and clone --follow). This forces a fresh election among live workers. Re-enqueue is idempotent (copydb_create_index uses IF NOT EXISTS + doneTime; copydb_create_constraints checks for the constraint already on target). allPartsDone / countPartsDone are recomputed from the summary table, so clearing the election tables loses no progress.

--restart is unaffected (it wipes the whole workdir, so the tables start empty); only --resume reuses the DB and inherits stale pids.

Testing (PostgreSQL 18)

Check Result
make build clean
make tests/unit (incl. 5-split-resume.sh) pass
make tests/pagila pass
citus_indent --check pass
make tests (full suite) pass

No new automated test (a deterministic mid-run crash between index build and PK promotion is not practical to inject in the harness). Verified manually end-to-end on a throwaway Postgres:

  1. Create a source table large enough to split into ≥2 COPY parts, with a PRIMARY KEY.
  2. Run a full clone, SIGKILL after all COPY parts finish but before the PK constraint is built.
  3. clone --resume with the same --dir and flags.
  4. Confirm the run logs the "Resetting stale part/index election state" notice, the PK constraint now exists and is valid on the target, and s_table_parts_done / s_table_indexes_done are empty at the start of the resume run. Before the fix, the constraint enqueue is skipped and the PK stays missing.

For a table split into multiple parallel COPY parts, index and
constraint creation is gated on a per-run "who finished the last part"
election stored in the s_table_parts_done / s_table_indexes_done SQLite
tables (tableoid, pid). The gate only fires for the worker whose
getpid() matches the stored pid.

On --resume the stored pid belongs to a dead prior-run process, so no
live worker matches and the index/constraint enqueue is never
re-driven. A table whose parts all finished but whose PK constraint was
not yet built in the prior run stays with a bare leaf index, and
pg_restore post-data then fails to attach it (relation "..._pkey"
already exists / cannot attach index / no unique constraint matching
given keys).

Clear both election tables once at the start of a --resume clone,
single-threaded and before any table-data worker is forked, so a fresh
election runs among live workers. These tables only hold per-run
election tokens; progress comes from the summary table (done_time_epoch),
so clearing them loses no progress, and re-enqueue is idempotent.
@teknogeek0
teknogeek0 merged commit c186893 into main Jul 24, 2026
96 checks passed
@teknogeek0
teknogeek0 deleted the fix/resume-split-table-index-election branch July 24, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant