Refactor mutex handling, continued

[achamayou]

Follow-up to #7744

Status update (2026-08-13)

This issue was automatically closed when #8139 merged because that PR used Closes #8123. It remains open for the final frontend lock-order and TSAN cleanup.

Work item	Status	Tracking
LedgerSecrets lock ordering	Merged	#8125
RpcFrontend consensus/history publication race	Merged	#8127
Node certificate synchronization and NodeClient ownership	Merged	#8135
Backup snapshot task synchronization	Merged	#8139
Remove NodeState::lock from transition_service_to_open()	Merged	#8144
Defer member/user frontend opening from KV global hooks	Not started	This issue
Re-test AFT startup ordering and change it only if still reported	Not started / conditional	This issue
Remove the remaining broad deadlock suppressions	Blocked on the remaining fixes and TSAN evidence	This issue

The snapshot-fetch version - 1 underflow identified during the original review was fixed independently by #7775.

What's left to do

1. Defer member and user frontend opening until after their KV global hooks return. Keep node frontend opening synchronous because boot and in-process node RPC tasks depend on it.
2. Run focused TSAN coverage without the store.h and untyped_map.h deadlock suppressions. Preserve a reproducer/stack pair for each remaining report.
3. If TSAN still reports the AFT startup ordering, introduce passive pre-publication state seeding and post-publication role activation. Do not call become_leader(true) from the constructor, because leader-side effects can run before the store and frontends are wired.
4. Remove the two broad deadlock suppressions once the focused runs are clean, then run the long TSAN configuration and affected startup, governance, recovery, snapshot, and forwarding tests.

Completed work

#8125 - LedgerSecrets lock ordering

Moved encrypted-ledger-secret KV dependency reads before LedgerSecrets::lock in get_latest(), get_latest_and_penultimate(), and get(). The in-memory map remains mutex-protected. A focused test runs with TSAN deadlock detection enabled and without repository-wide deadlock suppressions.

#8127 - frontend consensus/history publication

NodeState now explicitly publishes initialized consensus and history pointers to every frontend. RpcFrontend and EndpointRegistry use synchronized atomic publication, request paths use stable local loads, unpublished consensus is handled safely during redirect resolution, focused concurrent coverage was added, and the obsolete node/rpc/frontend.h race suppression was removed.

#8135 - certificate synchronization and ownership

Both mutable node certificate fields now use a dedicated mutex and copy-returning access. Certificate hooks no longer take the broad NodeState::lock, lock scopes do not span TLS/frontend/RPC work, and asynchronous NodeClient users own immutable certificate snapshots instead of references into mutable NodeState state. Focused ownership and certificate-renewal coverage was added.

#8139 - backup snapshot task locking

backup_snapshot_fetch_task now has a dedicated mutex. Work is enqueued after releasing that mutex, task completion cannot clear a newer replacement task, and the snapshot-evidence global hook no longer takes the broad NodeState::lock for this state.

#8144 - transition_service_to_open() lock ordering

Removed NodeState::lock -> KV locks from transition_service_to_open() after auditing member access. Recovery-secret fields that genuinely require synchronization now use a dedicated recovery_secrets_lock and copy-returning accessors, so the mutex is not held while accessing the KV store.

Remaining implementation details

Defer member/user frontend opening

The endorsed-certificate and service global hooks still open the member/user frontends synchronously. Global hooks can execute while KV map locks are held, while RpcFrontend::open() takes open_lock and initializes handlers. Replace this with an idempotent post-hook task. Keep node frontend opening synchronous.

Revisit AFT startup only if TSAN still requires it

#8060 added store-readiness and node-state gates, removing the strongest correctness reason to move role activation into the AFT constructor. Startup may still expose a NodeState::lock -> AFT state lock ordering. After the other fixes land, test this without the broad suppressions. If it remains, use a two-phase flow:

1. Construct an unpublished AFT object.
2. Seed passive term/index/history/ledger state without externally visible side effects.
3. Publish it to the store and frontends.
4. Activate leader-only behavior after wiring is complete and without holding NodeState::lock.

Suppression exit criteria

main currently retains:

deadlock:*/store.h
deadlock:*/untyped_map.h

Remove each suppression only when focused TSAN coverage is green without it. Finish with the long-test TSAN configuration and normal C++ unit/e2e coverage affected by startup, governance, recovery, snapshot fetching, and forwarding.

Background

The original assessment compared the unmerged draft #7744 with main at commit 0f8233cc on 2026-08-06. #7744 was closed without merging, and its work has been split into the focused PRs above rather than revived wholesale.