Retry a PAM lease mint the provider could not serialize - #8253
Conversation
CreateFromApprovedRequestAsync enforces its per-cipher singleton by reading a predicate rather than a row, inside a Serializable transaction. PostgreSQL records a predicate lock for that read, so the transaction becomes a candidate for abort whenever any other transaction inserts a lease before it commits -- including one granting access to an unrelated cipher, and including one running at a weaker isolation level, since only the reader has to be Serializable for SSI to find a read-write dependency. The abort surfaced as a raw 40001 out of the repository: a 500 for a caller whose activation was merely unlucky, and an intermittently red CI (observed on #8243). Retry the attempt instead. Each attempt gets a fresh scope, context and transaction, and re-reads the state its guard needs, so the loser reaches the same deterministic outcome the stored procedure's UPDLOCK/HOLDLOCK blocks for. Retries are bounded, and anything that outlives them propagates -- on a path that grants access to Vault Data, an unresolved persistence failure must not be reported as a benign mint outcome. The regression test races two activations of one cipher on separate connections and asserts exactly one mint, one conflict, and one lease. It reproduces the escaping 40001 on every run without the retry.
🤖 Bitwarden Claude Code ReviewOverall Assessment: APPROVE Reviewed the bounded retry added to the EF Code Review Details
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8253 +/- ##
==========================================
- Coverage 68.88% 68.88% -0.01%
==========================================
Files 2410 2410
Lines 104442 104463 +21
Branches 9457 9461 +4
==========================================
+ Hits 71946 71959 +13
- Misses 30115 30120 +5
- Partials 2381 2384 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
IsSerializationFailure was inserted between the duplicate-key summary/remarks block and IsDuplicateKeyException, leaving the new method with two <summary> tags and the older one with none. The compiler does not warn on a repeated <summary>, so the build stayed green.
🎟️ Tracking
No ticket — a CI flake, first triaged on #8243 where it failed the mint on Postgres.
📔 Objective
AccessLeaseRepository.CreateFromApprovedRequestAsync(EF) enforces its per-cipher singleton with a predicate read inside aSerializabletransaction:PostgreSQL records a predicate lock for that read, so the transaction becomes a candidate for abort whenever any other transaction inserts a lease before it commits — including one granting access to an unrelated cipher, and including one at a weaker isolation level, since only the reader has to be
Serializablefor SSI to find a read-write dependency. The abort escaped the repository as a rawNpgsql.PostgresException: 40001, which means a 500 for a caller whose activation was merely unlucky, and an intermittently red CI as soon as anything else seeds a lease concurrently.What this changes
The single attempt is now retried on a provider serialization failure. Each attempt takes a fresh scope, context and transaction and re-reads the state its guard needs, so a losing caller reaches the same deterministic outcome that the SQL Server procedure's
UPDLOCK/HOLDLOCKblocks for. Retries are bounded and anything outliving them propagates — on a path that grants access to Vault Data, an unresolved persistence failure must not be reported as a benign mint outcome.This is the contract PostgreSQL specifies for
SERIALIZABLE: applications at that level must be prepared to retry on serialization failure. The stale comment telling callers to treat the exception as a conflict is updated, since the repository now handles it.Deliberately unchanged:
(CipherId) WHERE Status = Activecan't replace the guard: the invariant is "Active and the window coversnow", and an index predicate can't referencenow.Testing
CreateFromApprovedRequestAsync_ConcurrentSameCipherActivations_OneMintsAndTheOtherConflictsraces two activations of one cipher on separate connections and asserts exactly one mint, one conflict, and one surviving lease. Without the retry it reproduces the escaping40001on 3 of 3 runs; with it, it passes on SqlServer, Postgres and Sqlite.At suite level,
--filter "FullyQualifiedName~IntegrationTest.Pam"on Postgres hit40001in 1 of 6 runs before the change and 0 of 8 after.Not verified locally: MySQL (my local container's credentials are broken, so every MySql case errors on connect) — relying on CI for that provider.
📸 Screenshots
n/a