Re-drive index/constraint creation for split tables on --resume - #49
Merged
Conversation
For a table split into multiple parallel COPY parts, index and constraint creation is gated on a per-run "who finished the last part" election stored in the s_table_parts_done / s_table_indexes_done SQLite tables (tableoid, pid). The gate only fires for the worker whose getpid() matches the stored pid. On --resume the stored pid belongs to a dead prior-run process, so no live worker matches and the index/constraint enqueue is never re-driven. A table whose parts all finished but whose PK constraint was not yet built in the prior run stays with a bare leaf index, and pg_restore post-data then fails to attach it (relation "..._pkey" already exists / cannot attach index / no unique constraint matching given keys). Clear both election tables once at the start of a --resume clone, single-threaded and before any table-data worker is forked, so a fresh election runs among live workers. These tables only hold per-run election tokens; progress comes from the summary table (done_time_epoch), so clearing them loses no progress, and re-enqueue is idempotent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A migration of a large natively-partitioned table (composite PK, many partitions) failed and was resumed. After
clone --resume, pg_restore/pgcopydb threw a cascade of errors:relation "<table>_p32_pkey" already exists(leafADD CONSTRAINT)cannot attach index "<table>_p32_pkey" ... no constraint exists for index(ATTACH PARTITION)there is no unique constraint matching given keys for referenced table "<table>"(FKs)Exactly one partition was left as a bare index: pgcopydb built its index in the first run but crashed before promoting it to a PK constraint, and the resume run never re-drove that work.
Root cause
For a table split into multiple parallel COPY parts, index/constraint enqueue is gated on a per-run "who finished the last part" election:
copydb_table_parts_are_all_donestores the currentgetpid()into the SQLite tables_table_parts_done(tableoid, pid).getpid()equals the stored pid.copydb_add_table_indexesis never called and the table's indexes/constraints are never re-enqueued. Whatever the prior run left half-built stays that way.The count of done parts comes from the
summarytable'sdone_time_epoch, not froms_table_parts_done— sos_table_parts_done/s_table_indexes_donehold only per-run election tokens, not progress. The sibling elections_table_indexes_done(which elects who builds constraints once all indexes are done) has the identical stale-pid flaw.Only tables with
partCount > 1are affected — single-part tables short-circuit the election.Fix
Clear both election tables once at the start of a
--resumeclone, single-threaded and before any table-data worker is forked (incloneDB, right after schema fetch, so it serves bothcloneandclone --follow). This forces a fresh election among live workers. Re-enqueue is idempotent (copydb_create_indexusesIF NOT EXISTS+doneTime;copydb_create_constraintschecks for the constraint already on target).allPartsDone/countPartsDoneare recomputed from thesummarytable, so clearing the election tables loses no progress.--restartis unaffected (it wipes the whole workdir, so the tables start empty); only--resumereuses the DB and inherits stale pids.Testing (PostgreSQL 18)
make buildmake tests/unit(incl.5-split-resume.sh)make tests/pagilacitus_indent --checkmake tests(full suite)No new automated test (a deterministic mid-run crash between index build and PK promotion is not practical to inject in the harness). Verified manually end-to-end on a throwaway Postgres:
clone, SIGKILL after all COPY parts finish but before the PK constraint is built.clone --resumewith the same--dirand flags.s_table_parts_done/s_table_indexes_doneare empty at the start of the resume run. Before the fix, the constraint enqueue is skipped and the PK stays missing.