Skip to content

fix: TOCTOU race in openbao secret-manager script - #813

Merged
Q-Dub merged 1 commit into
mainfrom
fix-openbao-secret-manage-script-race
Aug 18, 2026
Merged

fix: TOCTOU race in openbao secret-manager script#813
Q-Dub merged 1 commit into
mainfrom
fix-openbao-secret-manage-script-race

Conversation

@lauri-amd

Copy link
Copy Markdown
Contributor

The secret-manager job classified an unreachable/sealed vault the same as "secret does not exist" (both a non-zero bao kv get), so it could overwrite a live random secret if it raced the unseal job. Distinguish those cases explicitly and make writes self-defending instead of trusting the read:

  • kv_status() classifies existence by HTTP status from /v1/secrets/metadata/ (matches what -cas checks) instead of a plain CLI exit code, with connect/max timeouts so a hung vault can't wedge the job
  • random secrets are written with -cas=0, so even a wrong "missing" read gets rejected server-side instead of clobbering
  • apply the same status-based check + CAS guard to the AIWB scoped token mint/rotation, which had the identical blind-overwrite bug
  • guard envsubst substitutions and count/report failures instead of swallowing them under set -e, so the job actually exits non-zero on partial failure
  • offset the cronjob schedule from the unseal job's to shrink the overlap window (defense in depth; -cas=0 is the real guard)
  • give the init Job an activeDeadlineSeconds, since it can now legitimately exit non-zero where it previously always exited 0

The secret-manager job classified an unreachable/sealed vault the same
as "secret does not exist" (both a non-zero `bao kv get`), so it could
overwrite a live random secret if it raced the unseal job. Distinguish
those cases explicitly and make writes self-defending instead of
trusting the read:

- kv_status() classifies existence by HTTP status from
  /v1/secrets/metadata/ (matches what -cas checks) instead of a plain
  CLI exit code, with connect/max timeouts so a hung vault can't wedge
  the job
- random secrets are written with -cas=0, so even a wrong "missing"
  read gets rejected server-side instead of clobbering
- apply the same status-based check + CAS guard to the AIWB scoped
  token mint/rotation, which had the identical blind-overwrite bug
- guard envsubst substitutions and count/report failures instead of
  swallowing them under set -e, so the job actually exits non-zero on
  partial failure
- offset the cronjob schedule from the unseal job's to shrink the
  overlap window (defense in depth; -cas=0 is the real guard)
- give the init Job an activeDeadlineSeconds, since it can now
  legitimately exit non-zero where it previously always exited 0
@lauri-amd
lauri-amd marked this pull request as ready for review August 18, 2026 12:34
@lauri-amd
lauri-amd requested a review from a team as a code owner August 18, 2026 12:34

@Q-Dub Q-Dub left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validated via test plan in Jira: https://amd.atlassian.net/browse/EAI-8139

@Q-Dub
Q-Dub merged commit e2ef4ec into main Aug 18, 2026
14 checks passed
@Q-Dub
Q-Dub deleted the fix-openbao-secret-manage-script-race branch August 18, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants