Skip to content

Retry post-copy VACUUM ANALYZE on transient target connection loss - #48

Merged
teknogeek0 merged 1 commit into
mainfrom
fix/vacuum-retry-on-connection-loss
Jul 24, 2026
Merged

Retry post-copy VACUUM ANALYZE on transient target connection loss#48
teknogeek0 merged 1 commit into
mainfrom
fix/vacuum-retry-on-connection-loss

Conversation

@teknogeek0

@teknogeek0 teknogeek0 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Problem

During a migration, the post-copy VACUUM ANALYZE step failed on a single table when the target's SSL connection dropped mid-query:

ERROR pgsql.c  Failed to get async query results: SSL error: unexpected eof while reading
ERROR pgsql.c  [TARGET 0] connection pointer is NULL
ERROR pgsql.c  [TARGET 0] SQL query: VACUUM ANALYZE <schema>.<table>
ERROR vacuum.c Failed to vacuum table with oid <oid>

A momentary network blip during the optional, post-copy VACUUM ANALYZE step was treated as a fatal, non-retryable error, and the real cause (SSL EOF) was buried under a misleading "connection pointer is NULL" line. The data copy itself succeeded; only the optimization step tripped, yet the whole worker exited non-zero.

Changes

  • pgsql_stream_log_error: set pgsql->status = PG_CONNECTION_BAD before the teardown pgsql_finish. This is the single teardown path for all three pgsql_fetch_results error branches (invalid socket, select failed, PQconsumeInput == 0), so every mid-query connection loss becomes observable to callers via pgsql->status.
  • pgsql_execute_log_error: guard the libpq message/PID block when connection == NULL. PQerrorMessage(NULL) returns the literal "connection pointer is NULL"; the real error was already logged upstream. The useful SQL query/params context is still logged, with a backend PID of 0. Mirrors the existing guard in log_connection_error.
  • vacuum_analyze_table_by_oid: retry VACUUM ANALYZE up to 3 times on a lost/broken connection. Each attempt uses a fresh connection; the interactive open-retry policy (60s) absorbs the reconnect wait. Only PG_CONNECTION_BAD is retried — genuine SQL errors (lock timeout, undefined table, etc.) leave the status OK and are not retried. The step stays fatal once retries are exhausted.

Testing (PostgreSQL 18)

Check Result
make build clean
make tests/pagila pass — VACUUM ran on all tables, clone succeeded
citus_indent --check pass
make tests (full suite) pass

A deterministic mid-VACUUM SSL EOF is not practical to inject in the test harness, so the retry path was verified by code trace: (a) dst.status is PG_CONNECTION_BAD after an async teardown, (b) a real non-class-08 SQL error leaves status OK and is not retried, and (c) the "connection pointer is NULL" line no longer appears.

A momentary connection loss (e.g. an SSL EOF) during the optional
post-copy VACUUM ANALYZE step was treated as fatal, failing the worker
even though the data copy itself succeeded.

- Mark the connection PG_CONNECTION_BAD on async teardown so callers can
  tell a lost connection from a genuine SQL error via pgsql->status.
- Guard pgsql_execute_log_error against a NULL connection so the
  misleading "connection pointer is NULL" line no longer masks the real
  error; still log the SQL query/params context.
- Retry VACUUM ANALYZE up to 3 times on connection loss, each attempt on
  a fresh connection. Real SQL errors are not retried; the step stays
  fatal once retries are exhausted.
@teknogeek0
teknogeek0 merged commit c06bb0c into main Jul 24, 2026
190 of 192 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant