Skip to content

fix(audit): a feature flag change is saved with its audit entries or not at all - #2079

Open
amarkanday wants to merge 2 commits into
mainfrom
fix/1042-flag-audit-strict
Open

amarkanday wants to merge 2 commits into
mainfrom
fix/1042-flag-audit-strict

Conversation

@amarkanday

Copy link
Copy Markdown
Contributor

What changed

A change to a feature flag is now saved together with its audit-log entry, and for create, update and delete with its compliance event, or not at all. The routes are create (and its hidden twin without the slash), update, delete, activate, deactivate, unarchive, toggle, enable, disable and bulk toggle. When an entry cannot be written, nothing is saved and the route answers 500 with a fixed sentence and the request ID, such as Could not toggle the feature flag (request ID: …).; it used to keep the change and answer 2xx with no entry. Sending the same request again once the problem is fixed saves it, with exactly one entry.

Bulk toggle saves all of its changes or none: when they cannot be saved it answers 500, where it used to answer 200 and report flags as changed that were not saved. Refusals of single flags (not found, no permission, archived) still answer 200 with a result per flag.

Closes #1042

  • AuditService.audit_with_change(db, action=..., entity_type=..., entity_id=...) flushes the caller's change first, so the change's own failure keeps its own error and answer (a key taken by another flag is still 409). It then writes the entries. Any failure there is raised as AuditWriteFailed, a plain Exception (never a ValueError, so never a 400), from None, so the database's text is in no traceback. It never commits, logs nothing, and leaves audit_write_failures_total and the audit-write alarm alone: a refused change has no missing entry.
  • The five FeatureFlagService writers take record=, called before their commit; with None (the tests that call the service directly) they behave as before. The routes pass a closure that writes both records inside audit_with_change. A new flag gets its id before the commit.
  • A refusal logs the route's <operation> failed (AuditWriteFailed) at ERROR (for example Feature flag toggle failed (AuditWriteFailed)); a create or an update also logs the service's existing Error creating feature flag / Error updating feature flag line with the cause's type.
  • The compliance event's content is unchanged. The first commit adds a test, green on main, that pins all five values of the update event as main writes them (including the two status spellings, see A feature flag update's compliance event shows a status change that did not happen #2066).
  • Unchanged on purpose: rollout schedule and stage changes (a row in the durability table says they keep today's behaviour), safety rollbacks (a superuser's rollback to 0% still turns a flag off when its entry cannot be written), experiments and every other route.

OpenAPI: the snapshots change only in the descriptions of toggle, enable, disable, delete and bulk toggle, and of the Compliance tag, which now says "creates, updates and deletes" instead of "every change" (a flag's status change never wrote a compliance event). stability.md has a #1042 row for the stable operations, unarchive included.

Docs: audit-logging.md (bulk toggle; the action types' introduction; the durability table: one row for flags, one for rollout schedule changes, one for every other route; the safety rollback; what a refusal logs and which alarm counts it), compliance.md (flags leave the warning-only sentence), stability.md. Every new or edited sentence is registered. The docs floor lists POST /api/v1/feature-flags/{flag_id}/disable, which the stability row names in full.

Tests changed

Inverted, deliberately, each with its setup kept and the opposite assertion:

  • test_audit_route_events.py::test_bulk_toggle_keeps_the_change_when_the_audit_write_fails → test_bulk_toggle_keeps_nothing_when_the_audit_write_fails
  • test_flag_status_audit_write.py::test_toggle_answers_200_when_the_audit_write_fails → test_toggle_answers_500_when_the_audit_write_fails; its resend, once the problem is fixed, flips the flag once.
  • test_flag_status_audit_write.py::test_bulk_toggle_error_lines_carry_no_values expects 500. Its helper _assert_no_values_logged is relaxed on purpose for the route loggers only: a route's ERROR record may carry a traceback (unexpected_failure logs one), and the formatted traceback must hold none of the values. The audit writer's records still carry no traceback at all.
  • test_change_audit_saved.py::test_failed_audit_write_keeps_the_answer_and_the_change keeps its experiment cases; the flag cases are test_failed_flag_audit_write_keeps_neither.
  • test_change_audit_saved.py::test_failed_audit_write_logs_the_error_type_only keeps its experiment cases; the flag cases are test_failed_flag_audit_write_logs_the_error_type_only.
  • test_feature_flag_toggle.py::test_toggle_audit_logging_error → test_toggle_answers_500_when_audit_logging_fails.

An existing integration test counts the toggle's transactions; the toggle now commits once per request instead of twice, so its pinned count moves from 48 to 24.

test_flag_key_conflict.py's three replacement writers take and forward record=.

Every patch or mock of log_action, record_after_commit, log_toggle_operation, log_bulk_toggle or AuditLogService under backend/tests/unit and backend/tests/integration:

  • unit/api/test_feature_flag_toggle.py (seven tests replaced log_action, which these routes no longer call): rewritten to replace AuditService.record and assert it was called (and with reason=None, and the signed-in actor). Planted: toggle back to log_action after its commit → test_toggle_feature_flag_inactive_to_active (Expected 'commit' to have been called once. Called 2 times.), test_toggle_feature_flag_superuser_access, test_toggle_answers_500_when_audit_logging_fails (assert 200 == 500) fail; a reason sent where none was given → test_toggle_without_reason (assert 'planted' is None) fails.
  • unit/services/test_audit_log_rules.py::test_a_flag_the_caller_may_not_change_is_refused_and_left_as_it_was (replaced record_after_commit, no longer called): rewritten to replace AuditService.record. Planted: an entry for a refused flag → Expected 'record' to not have been called. Called 1 times.
  • unit/api/test_bulk_toggle_endpoints.py::test_bulk_toggle_enable_returns_200: unchanged. Its replacement of log_toggle_operation was already inert on main (bulk toggle has not called it since Audit log records only feature-flag status changes #221); it still tests a 200 on the success path with a working audit write.
  • unit/api/test_compliance_api.py::TestFeatureFlagAuditGeneration: unchanged. Its patches sit in a block that runs nothing; it still tests that the route module names the three compliance actions.
  • integration/api/test_change_audit_saved.py: the two inversions above; the experiment halves are unchanged.
  • unit/core/test_optional_modules.py, unit/services/test_compliance_trail_writers.py (the compliance read route), unit/services/test_audit_service*.py, unit/services/test_audit_write_failed_line.py, integration/api/test_audit_route_events.py::test_writer_log_lines_carry_no_values, integration/services/test_audit_service_sites.py, integration/api/test_flag_status_audit_write.py::test_log_*: unchanged; none reaches a changed route (they call the writers directly or read compliance events).

How it was verified

Locally, on PostgreSQL 14 with the venv's requirements:

  • backend/tests/unit: 25371 passed, 170 skipped.
  • backend/tests/smoke: all passed except test_sdk_release_wiring.py's two pep_440 cases, which fail the same way on main in this environment (No module named 'packaging' in the step's subprocess).
  • The integration files this PR adds or edits, three runs each in a different random order: 160 passed each time. The 85 integration files that touch flag routes or FeatureFlagService: all passed.
  • A copy of the tree with modules/ removed and no .git: backend/tests/smoke and the changed integration files pass (the same two environment failures).
  • ruff check and ruff format --check over backend/ modules/ scripts/; scripts/doc_claims.py check (problems: 0); mkdocs build --strict; scripts/doc_examples.py --check.
  • make openapi: only the two snapshots change, in the six descriptions named above; the dashboard fixture does not. Comparing main's snapshots with these after deleting only string-valued description keys, all three files are equal; changing FeatureFlagRead.properties.description to an integer type makes that comparison report a difference.

Seen failing on main (373744b), passing here:

  • test_flag_change_audit_strict.py::test_a_refused_entry_saves_nothing_and_answers_500, all 11 routes (main keeps the change: create assert 201 == 500, delete assert 204 == 500, the others assert 200 == 500), and test_a_refused_compliance_event_saves_nothing_and_answers_500, all 4.
  • test_flag_change_audit_strict.py::test_bulk_toggle_answers_500_when_its_commit_fails (@pytest.mark.regression): on main 200 with "succeeded":1 and the flag still inactive.
  • Every inverted test above.

Each gate was broken on purpose and seen to fail, then restored:

  • Delete writing its entry after the commit (record_after_commit) → the delete case (assert 204 == 500) and the smoke pin's banned-writer and record= checks fail.
  • Activate called without record= → the activate case (assert 200 == 500) and the pin's record= check fail.
  • Update writing its compliance event after the commit → the compliance case for update (assert 200 == 500) and the pin's inside-the-block check fail.
  • AuditWriteFailed raised from its cause → the route's traceback carries the refused row's values: the toggle case of test_a_refused_entry_saves_nothing_and_answers_500, test_toggle_answers_500_when_the_audit_write_fails and test_bulk_toggle_error_lines_carry_no_values fail, and the unit test sees __cause__.
  • The update's compliance snapshot taken before the change → test_flag_update_record_carries_its_five_values fails ({'name': 'au471 flag'} != {'name': 'renamed'}).
  • AuditWriteFailed based on ValueError → the unit test fails; audit_with_change catching too little → create answers 400 with the error's text and its test fails.
  • audit_with_change counting a refusal as a lost entry → all 11 cases of test_a_refused_entry_saves_nothing_and_answers_500 fail (assert (0, 0, 1.0) == (0, 0, 0)); the existing alarm-line unit test stays green, so that integration test is the gate here.
  • A record_after_commit call in unarchive, an entry written outside the block, an await inside a block, a sixth service writer call → each fails its smoke check.
  • No flush before the guarded block → test_create_race_is_409 and test_put_key_race_is_409 answer 500, and toggle, enable, disable and unarchive's own failures are logged as AuditWriteFailed; with bulk toggle's own flush removed too, bulk's is as well.
  • Bulk toggle swallowing a failed commit → test_bulk_toggle_answers_500_when_its_commit_fails fails (200 with "succeeded":1).
  • A commit inside create's block → the create case (the flag is saved) and the pin fail; the service's error line moved to WARNING → the create case's exact ERROR lines fail.
  • The old Compliance tag text → test_the_compliance_tag_says_what_writes_an_event fails; the old toggle description → both snapshot tests fail.
  • A docs sentence edited without its registry, and a registry citing a test that does not exist → doc_claims.py check reports problems.

NOT VERIFIED

  • CI's PostgreSQL image and its four integration shards; Aurora.
  • The core build CI runs (scripts/core_build.sh); the local copy only deleted modules/ and .git.
  • What the dashboard shows when a flag change answers 500.
  • The fuzz run, which happens when this goes into the OpenAPI train.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn

amarkanday and others added 2 commits November 19, 2026 16:34
…rries

A characterisation test, added before any code change for #1042 and green
on main: a PUT that renames an active flag writes one compliance record
whose old and new values hold exactly the key, name, status, rollout
percentage and value type, in the formats main stores today. Until now the
update record was compared on the key and the name only.

The status pin keeps main's two formats (`FeatureFlagStatus.ACTIVE` before,
`active` after); #2066 tracks that difference. Moving the record into the
change's transaction must leave this test green unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn
Signed-off-by: amarkanday <a.markanday@gmail.com>
…not at all

The routes that change a feature flag (create and its hidden twin without
the slash, update, delete, activate, deactivate, unarchive, toggle, enable,
disable) and bulk toggle now write their audit_logs entry, and for create,
update and delete their compliance event, in the transaction that saves the
change. When an entry cannot be written, nothing is saved and the route
answers 500 with its fixed sentence and the request ID; it used to keep the
change and answer 2xx with no entry. Sending the same request again once the
problem is fixed saves it with exactly one entry. Bulk toggle saves all of
its changes or none: when they cannot be saved it answers 500, where it used
to answer 200 and report flags as changed that were not saved (#1042).

- AuditService.audit_with_change flushes the caller's change first (its own
  failure keeps its own error and answer: a key conflict is still 409), then
  writes the entries; any failure there is raised as AuditWriteFailed, from
  None, so the database's text is in no traceback. It never commits, logs
  nothing and leaves audit_write_failures_total alone: a refused change has
  no missing entry. AuditWriteFailed is a plain Exception, never a
  ValueError, so no route answers it with 400.
- The five FeatureFlagService writers take record=, called before their
  commit; with None they behave as before. The routes pass a closure that
  writes both records inside audit_with_change. The new flag gets its id
  before the commit, so a refused create names it.
- The refusal logs the route's "<operation> failed (AuditWriteFailed)" at
  ERROR; a create or update also logs the service's existing line.
- The compliance event keeps its content; a new test pins all five values
  of the update event as main writes them.
- The OpenAPI document changes only in the descriptions of toggle, enable,
  disable, delete and bulk toggle, and of the Compliance tag, which now says
  creates, updates and deletes rather than every change.
- Docs: audit-logging.md (bulk toggle, the action types' introduction, the
  durability table with a row for flags, one for rollout schedule changes,
  which keep today's behaviour, and one for every other route, the safety
  rollback that still turns a flag off, and what a refusal logs),
  compliance.md (flags leave the warning-only sentence) and a #1042 row in
  stability.md, each registered. The docs floor lists the disable route,
  which the stability row names in full.

Tests inverted deliberately, each with the same setup and the opposite
assertion:
- test_audit_route_events.py::test_bulk_toggle_keeps_the_change_when_the_audit_write_fails
  is test_bulk_toggle_keeps_nothing_when_the_audit_write_fails
- test_flag_status_audit_write.py::test_toggle_answers_200_when_the_audit_write_fails
  is test_toggle_answers_500_when_the_audit_write_fails (the resend after
  the problem is fixed flips the flag once)
- test_flag_status_audit_write.py::test_bulk_toggle_error_lines_carry_no_values
  expects 500; its helper now allows the route's traceback and checks its
  text instead (the audit writer's lines still carry none)
- test_change_audit_saved.py::test_failed_audit_write_keeps_the_answer_and_the_change
  keeps the experiment cases; the flag cases are
  test_failed_flag_audit_write_keeps_neither
- test_change_audit_saved.py::test_failed_audit_write_logs_the_error_type_only
  keeps the experiment cases; the flag cases are
  test_failed_flag_audit_write_logs_the_error_type_only
- test_feature_flag_toggle.py::test_toggle_audit_logging_error is
  test_toggle_answers_500_when_audit_logging_fails

An existing integration test counts the toggle's transactions; the toggle now commits once per request instead of twice, so its pinned count moves from 48 to 24.

The other unit tests that replaced AuditService.log_action on the toggle,
enable and disable routes, and the bulk refusal test that replaced
record_after_commit, now replace AuditService.record, the call those routes
make. test_flag_key_conflict.py's replacement writers take and forward
record=.

New: test_flag_change_audit_strict.py (every route with each audit table
refusing writes, then the same request again; the change's own failure keeps
its own error type; waiting for a row another change holds saves nothing; a
bulk commit that fails answers 500; a rollout schedule change still keeps its
change), test_flag_audit_strict_sites.py (smoke: the writers that commit on
their own are not called from these routes, every entry is written inside
audit_with_change, nothing there commits or awaits, and the five service
writers are called five times, each with record=), test_audit_with_change.py
(unit) and a test that the Compliance tag says what writes an event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn
Signed-off-by: amarkanday <a.markanday@gmail.com>
@amarkanday amarkanday added the bug Something isn't working label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature-flag changes can be saved without a compliance audit entry; state the audit log's durability per resource

1 participant