Repository navigation
fix(audit): a feature flag change is saved with its audit entries or not at all - #2079
Open
amarkanday wants to merge 2 commits into
Open
amarkanday wants to merge 2 commits into
amarkanday wants to merge 2 commits into
Conversation
…rries A characterisation test, added before any code change for #1042 and green on main: a PUT that renames an active flag writes one compliance record whose old and new values hold exactly the key, name, status, rollout percentage and value type, in the formats main stores today. Until now the update record was compared on the key and the name only. The status pin keeps main's two formats (`FeatureFlagStatus.ACTIVE` before, `active` after); #2066 tracks that difference. Moving the record into the change's transaction must leave this test green unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn Signed-off-by: amarkanday <a.markanday@gmail.com>
…not at all The routes that change a feature flag (create and its hidden twin without the slash, update, delete, activate, deactivate, unarchive, toggle, enable, disable) and bulk toggle now write their audit_logs entry, and for create, update and delete their compliance event, in the transaction that saves the change. When an entry cannot be written, nothing is saved and the route answers 500 with its fixed sentence and the request ID; it used to keep the change and answer 2xx with no entry. Sending the same request again once the problem is fixed saves it with exactly one entry. Bulk toggle saves all of its changes or none: when they cannot be saved it answers 500, where it used to answer 200 and report flags as changed that were not saved (#1042). - AuditService.audit_with_change flushes the caller's change first (its own failure keeps its own error and answer: a key conflict is still 409), then writes the entries; any failure there is raised as AuditWriteFailed, from None, so the database's text is in no traceback. It never commits, logs nothing and leaves audit_write_failures_total alone: a refused change has no missing entry. AuditWriteFailed is a plain Exception, never a ValueError, so no route answers it with 400. - The five FeatureFlagService writers take record=, called before their commit; with None they behave as before. The routes pass a closure that writes both records inside audit_with_change. The new flag gets its id before the commit, so a refused create names it. - The refusal logs the route's "<operation> failed (AuditWriteFailed)" at ERROR; a create or update also logs the service's existing line. - The compliance event keeps its content; a new test pins all five values of the update event as main writes them. - The OpenAPI document changes only in the descriptions of toggle, enable, disable, delete and bulk toggle, and of the Compliance tag, which now says creates, updates and deletes rather than every change. - Docs: audit-logging.md (bulk toggle, the action types' introduction, the durability table with a row for flags, one for rollout schedule changes, which keep today's behaviour, and one for every other route, the safety rollback that still turns a flag off, and what a refusal logs), compliance.md (flags leave the warning-only sentence) and a #1042 row in stability.md, each registered. The docs floor lists the disable route, which the stability row names in full. Tests inverted deliberately, each with the same setup and the opposite assertion: - test_audit_route_events.py::test_bulk_toggle_keeps_the_change_when_the_audit_write_fails is test_bulk_toggle_keeps_nothing_when_the_audit_write_fails - test_flag_status_audit_write.py::test_toggle_answers_200_when_the_audit_write_fails is test_toggle_answers_500_when_the_audit_write_fails (the resend after the problem is fixed flips the flag once) - test_flag_status_audit_write.py::test_bulk_toggle_error_lines_carry_no_values expects 500; its helper now allows the route's traceback and checks its text instead (the audit writer's lines still carry none) - test_change_audit_saved.py::test_failed_audit_write_keeps_the_answer_and_the_change keeps the experiment cases; the flag cases are test_failed_flag_audit_write_keeps_neither - test_change_audit_saved.py::test_failed_audit_write_logs_the_error_type_only keeps the experiment cases; the flag cases are test_failed_flag_audit_write_logs_the_error_type_only - test_feature_flag_toggle.py::test_toggle_audit_logging_error is test_toggle_answers_500_when_audit_logging_fails An existing integration test counts the toggle's transactions; the toggle now commits once per request instead of twice, so its pinned count moves from 48 to 24. The other unit tests that replaced AuditService.log_action on the toggle, enable and disable routes, and the bulk refusal test that replaced record_after_commit, now replace AuditService.record, the call those routes make. test_flag_key_conflict.py's replacement writers take and forward record=. New: test_flag_change_audit_strict.py (every route with each audit table refusing writes, then the same request again; the change's own failure keeps its own error type; waiting for a row another change holds saves nothing; a bulk commit that fails answers 500; a rollout schedule change still keeps its change), test_flag_audit_strict_sites.py (smoke: the writers that commit on their own are not called from these routes, every entry is written inside audit_with_change, nothing there commits or awaits, and the five service writers are called five times, each with record=), test_audit_with_change.py (unit) and a test that the Compliance tag says what writes an event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn Signed-off-by: amarkanday <a.markanday@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
A change to a feature flag is now saved together with its audit-log entry, and for create, update and delete with its compliance event, or not at all. The routes are create (and its hidden twin without the slash), update, delete, activate, deactivate, unarchive, toggle, enable, disable and bulk toggle. When an entry cannot be written, nothing is saved and the route answers
500with a fixed sentence and the request ID, such asCould not toggle the feature flag (request ID: …).; it used to keep the change and answer2xxwith no entry. Sending the same request again once the problem is fixed saves it, with exactly one entry.Bulk toggle saves all of its changes or none: when they cannot be saved it answers
500, where it used to answer200and report flags as changed that were not saved. Refusals of single flags (not found, no permission, archived) still answer200with a result per flag.Closes #1042
AuditService.audit_with_change(db, action=..., entity_type=..., entity_id=...)flushes the caller's change first, so the change's own failure keeps its own error and answer (a key taken by another flag is still409). It then writes the entries. Any failure there is raised asAuditWriteFailed, a plainException(never aValueError, so never a400), fromNone, so the database's text is in no traceback. It never commits, logs nothing, and leavesaudit_write_failures_totaland the audit-write alarm alone: a refused change has no missing entry.FeatureFlagServicewriters takerecord=, called before their commit; withNone(the tests that call the service directly) they behave as before. The routes pass a closure that writes both records insideaudit_with_change. A new flag gets its id before the commit.<operation> failed (AuditWriteFailed)at ERROR (for exampleFeature flag toggle failed (AuditWriteFailed)); a create or an update also logs the service's existingError creating feature flag/Error updating feature flagline with the cause's type.OpenAPI: the snapshots change only in the descriptions of toggle, enable, disable, delete and bulk toggle, and of the Compliance tag, which now says "creates, updates and deletes" instead of "every change" (a flag's status change never wrote a compliance event).
stability.mdhas a #1042 row for the stable operations, unarchive included.Docs:
audit-logging.md(bulk toggle; the action types' introduction; the durability table: one row for flags, one for rollout schedule changes, one for every other route; the safety rollback; what a refusal logs and which alarm counts it),compliance.md(flags leave the warning-only sentence),stability.md. Every new or edited sentence is registered. The docs floor listsPOST /api/v1/feature-flags/{flag_id}/disable, which the stability row names in full.Tests changed
Inverted, deliberately, each with its setup kept and the opposite assertion:
test_audit_route_events.py::test_bulk_toggle_keeps_the_change_when_the_audit_write_fails→test_bulk_toggle_keeps_nothing_when_the_audit_write_failstest_flag_status_audit_write.py::test_toggle_answers_200_when_the_audit_write_fails→test_toggle_answers_500_when_the_audit_write_fails; its resend, once the problem is fixed, flips the flag once.test_flag_status_audit_write.py::test_bulk_toggle_error_lines_carry_no_valuesexpects500. Its helper_assert_no_values_loggedis relaxed on purpose for the route loggers only: a route's ERROR record may carry a traceback (unexpected_failurelogs one), and the formatted traceback must hold none of the values. The audit writer's records still carry no traceback at all.test_change_audit_saved.py::test_failed_audit_write_keeps_the_answer_and_the_changekeeps its experiment cases; the flag cases aretest_failed_flag_audit_write_keeps_neither.test_change_audit_saved.py::test_failed_audit_write_logs_the_error_type_onlykeeps its experiment cases; the flag cases aretest_failed_flag_audit_write_logs_the_error_type_only.test_feature_flag_toggle.py::test_toggle_audit_logging_error→test_toggle_answers_500_when_audit_logging_fails.An existing integration test counts the toggle's transactions; the toggle now commits once per request instead of twice, so its pinned count moves from 48 to 24.
test_flag_key_conflict.py's three replacement writers take and forwardrecord=.Every patch or mock of
log_action,record_after_commit,log_toggle_operation,log_bulk_toggleorAuditLogServiceunderbackend/tests/unitandbackend/tests/integration:unit/api/test_feature_flag_toggle.py(seven tests replacedlog_action, which these routes no longer call): rewritten to replaceAuditService.recordand assert it was called (and withreason=None, and the signed-in actor). Planted: toggle back tolog_actionafter its commit →test_toggle_feature_flag_inactive_to_active(Expected 'commit' to have been called once. Called 2 times.),test_toggle_feature_flag_superuser_access,test_toggle_answers_500_when_audit_logging_fails(assert 200 == 500) fail; a reason sent where none was given →test_toggle_without_reason(assert 'planted' is None) fails.unit/services/test_audit_log_rules.py::test_a_flag_the_caller_may_not_change_is_refused_and_left_as_it_was(replacedrecord_after_commit, no longer called): rewritten to replaceAuditService.record. Planted: an entry for a refused flag →Expected 'record' to not have been called. Called 1 times.unit/api/test_bulk_toggle_endpoints.py::test_bulk_toggle_enable_returns_200: unchanged. Its replacement oflog_toggle_operationwas already inert on main (bulk toggle has not called it since Audit log records only feature-flag status changes #221); it still tests a200on the success path with a working audit write.unit/api/test_compliance_api.py::TestFeatureFlagAuditGeneration: unchanged. Its patches sit in a block that runs nothing; it still tests that the route module names the three compliance actions.integration/api/test_change_audit_saved.py: the two inversions above; the experiment halves are unchanged.unit/core/test_optional_modules.py,unit/services/test_compliance_trail_writers.py(the compliance read route),unit/services/test_audit_service*.py,unit/services/test_audit_write_failed_line.py,integration/api/test_audit_route_events.py::test_writer_log_lines_carry_no_values,integration/services/test_audit_service_sites.py,integration/api/test_flag_status_audit_write.py::test_log_*: unchanged; none reaches a changed route (they call the writers directly or read compliance events).How it was verified
Locally, on PostgreSQL 14 with the venv's requirements:
backend/tests/unit: 25371 passed, 170 skipped.backend/tests/smoke: all passed excepttest_sdk_release_wiring.py's two pep_440 cases, which fail the same way on main in this environment (No module named 'packaging'in the step's subprocess).FeatureFlagService: all passed.modules/removed and no.git:backend/tests/smokeand the changed integration files pass (the same two environment failures).ruff checkandruff format --checkoverbackend/ modules/ scripts/;scripts/doc_claims.py check(problems: 0);mkdocs build --strict;scripts/doc_examples.py --check.make openapi: only the two snapshots change, in the six descriptions named above; the dashboard fixture does not. Comparing main's snapshots with these after deleting only string-valueddescriptionkeys, all three files are equal; changingFeatureFlagRead.properties.descriptionto an integer type makes that comparison report a difference.Seen failing on main (373744b), passing here:
test_flag_change_audit_strict.py::test_a_refused_entry_saves_nothing_and_answers_500, all 11 routes (main keeps the change: createassert 201 == 500, deleteassert 204 == 500, the othersassert 200 == 500), andtest_a_refused_compliance_event_saves_nothing_and_answers_500, all 4.test_flag_change_audit_strict.py::test_bulk_toggle_answers_500_when_its_commit_fails(@pytest.mark.regression): on main200with"succeeded":1and the flag still inactive.Each gate was broken on purpose and seen to fail, then restored:
record_after_commit) → the delete case (assert 204 == 500) and the smoke pin's banned-writer andrecord=checks fail.record=→ the activate case (assert 200 == 500) and the pin'srecord=check fail.assert 200 == 500) and the pin's inside-the-block check fail.AuditWriteFailedraisedfromits cause → the route's traceback carries the refused row's values: the toggle case oftest_a_refused_entry_saves_nothing_and_answers_500,test_toggle_answers_500_when_the_audit_write_failsandtest_bulk_toggle_error_lines_carry_no_valuesfail, and the unit test sees__cause__.test_flag_update_record_carries_its_five_valuesfails ({'name': 'au471 flag'} != {'name': 'renamed'}).AuditWriteFailedbased onValueError→ the unit test fails;audit_with_changecatching too little → create answers400with the error's text and its test fails.audit_with_changecounting a refusal as a lost entry → all 11 cases oftest_a_refused_entry_saves_nothing_and_answers_500fail (assert (0, 0, 1.0) == (0, 0, 0)); the existing alarm-line unit test stays green, so that integration test is the gate here.record_after_commitcall in unarchive, an entry written outside the block, anawaitinside a block, a sixth service writer call → each fails its smoke check.test_create_race_is_409andtest_put_key_race_is_409answer500, and toggle, enable, disable and unarchive's own failures are logged asAuditWriteFailed; with bulk toggle's own flush removed too, bulk's is as well.test_bulk_toggle_answers_500_when_its_commit_failsfails (200with"succeeded":1).test_the_compliance_tag_says_what_writes_an_eventfails; the old toggle description → both snapshot tests fail.doc_claims.py checkreports problems.NOT VERIFIED
scripts/core_build.sh); the local copy only deletedmodules/and.git.500.🤖 Generated with Claude Code
https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn