fix(db): retry the migration connection on transient slot exhaustion

The migration opens its session on the first query (the advisory-lock
acquire). When the deploy database briefly exhausts every non-superuser
connection slot at peak, that connect fails with 53300 ("remaining connection
slots are reserved for roles with the SUPERUSER attribute") and the whole
deploy's migrate step errors out — even when the spike clears within seconds.

Add a bounded connectWithRetry() before acquiring the lock that retries 53300,
the 08xxx connection_exception class, and the driver's transport errors with
backoff (10 attempts, ~90s ceiling). Non-transient errors (auth, bad config)
still fail fast. The migration is a single short-lived session, so waiting out
a transient spike is far safer than failing the deploy.

Author: waleedlatif1

packages/db/scripts/migrate.ts

@@ -71,10 +71,42 @@ const DDL_LOCK_TIMEOUT = '5s'
 const MAX_MIGRATE_ATTEMPTS = 8
 const MIGRATE_RETRY_BACKOFF = { baseMs: 2_000, maxMs: 30_000 } as const
 
+const CONNECT_MAX_ATTEMPTS = 10
+const CONNECT_RETRY_BACKOFF = { baseMs: 1_000, maxMs: 15_000 } as const
+
+/**
+ * Error codes that mean the database was momentarily unreachable rather than
+ * the migration being wrong: chiefly `53300` (too_many_connections — every
+ * non-superuser slot was taken, surfaced as "remaining connection slots are
+ * reserved for roles with the SUPERUSER attribute"), the `08xxx`
+ * connection_exception class, and the postgres-js driver's own transport
+ * codes. These are retried while opening the session; anything else is fatal.
+ */
+const TRANSIENT_CONNECT_CODES = new Set([
+  '53300',
+  '53400',
+  'CONNECT_TIMEOUT',
+  'CONNECTION_CLOSED',
+  'CONNECTION_DESTROYED',
+  'CONNECTION_ENDED',
+  'ECONNREFUSED',
+  'ECONNRESET',
+  'ETIMEDOUT',
+  'EHOSTUNREACH',
+  'ENOTFOUND',
+])
+
+function isTransientConnectError(error: unknown): boolean {
+  const code = getPostgresErrorCode(error)
+  if (!code) return false
+  return TRANSIENT_CONNECT_CODES.has(code) || code.startsWith('08')
+}
+
 /** Backend pid of the lock-holding session; a change means the lock was lost. */
 let lockSessionPid = 0
 
 try {
+  await connectWithRetry()
   await acquireMigrationLock()
   try {
     await runMigrationsWithRetry()
@@ -91,6 +123,31 @@ try {
   await client.end()
 }
 
+/**
+ * Open the migration session before taking the advisory lock, retrying
+ * transient connection failures with bounded backoff. The deploy database can
+ * briefly exhaust every non-superuser connection slot at peak (`53300`); the
+ * migration is a single short-lived session, so waiting out a spike that frees
+ * within seconds is far safer than failing the whole deploy. Non-transient
+ * errors (auth, unknown host config, etc.) still fail fast.
+ */
+async function connectWithRetry(): Promise<void> {
+  for (let attempt = 1; ; attempt++) {
+    try {
+      await client`SELECT 1`
+      return
+    } catch (error) {
+      if (!isTransientConnectError(error) || attempt >= CONNECT_MAX_ATTEMPTS) throw error
+      const delayMs = backoffWithJitter(attempt, null, CONNECT_RETRY_BACKOFF)
+      console.warn(
+        `WARN: database unavailable (${getPostgresErrorCode(error)}); ` +
+          `attempt ${attempt}/${CONNECT_MAX_ATTEMPTS}, retrying in ${Math.round(delayMs)}ms.`
+      )
+      await sleep(delayMs)
+    }
+  }
+}
+
 /**
  * Acquire the cross-process migration lock, failing loudly after the deadline
  * instead of blocking forever behind a wedged runner.