Repository navigation
fix(bandit): read converting users from PostgreSQL first, not DynamoDB conversion events - #2076
Open
amarkanday wants to merge 1 commit into
Open
amarkanday wants to merge 1 commit into
amarkanday wants to merge 1 commit into
Conversation
…B conversion events The DynamoDB conversion counter adds one per conversion event (and the increment API adds any amount), so a user who converts three times counts three there, while /results and the PostgreSQL fallback count converting users. The bandit scheduler preferred DynamoDB whenever its pulls and successes were at least PostgreSQL's for every variant, and an event-inflated success count passes that check, so the bandit could weight arms on events. PostgreSQL is now the first source: when any variant has an assignment there, its pulls and converting users are returned whole and DynamoDB is not read. DynamoDB is used only when PostgreSQL has no assignments or cannot be read, with successes capped at pulls. Mixing DynamoDB pulls with PostgreSQL successes was rejected: the two describe different populations. The completeness check is deleted. No writer, schema, table or infrastructure change. CLAUDE.md and the three docs pages that state the source order are updated with their claim units. Not changed: process_conversion_event (still unwired) and the increment API still count events, and /realtime-counters reports events per assignment. Resolves #581 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kqtSDC7sd5ue3m6LuwdXV Signed-off-by: amarkanday <a.markanday@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #581
What
The bandit scheduler could weigh arms on conversion events instead of converting users.
DynamoDBCounterService'sconversionscounter is incremented once per event (process_conversion_event, and the increment/bulk API with any amount);/resultsand the PostgreSQL fallback (#576) count converting users. The scheduler chose DynamoDB when its pulls AND successes were each at least PostgreSQL's (#430), and an event-inflated success count satisfies that.Not changed:
process_conversion_event(still unwired) and the increment API still count events, and/realtime-countersreports events divided by assignments; the bandit uses those counts only when PostgreSQL has no assignments or cannot be read.SPEC
Choice. Fix it in the scheduler, the one place the bandit reads, not in a writer:
process_conversion_eventhas no production caller, the live writers ofconversionsare the generic increment/bulk endpoints (arbitrary amounts), and a per-user write needs a new item shape in the counters table. Mixing sources (DynamoDB's pulls with PostgreSQL's successes) would count two populations: an arm with DynamoDB-only pulls would get 0 successes and weight 0, and a DynamoDB lead in one arm would halve its rate and flip the ranking. So one source is used, whole.Contract (
backend/app/core/bandit_scheduler.py,get_variant_stats_from_counters).BanditState, then zero priors.event_matching.count_converting_users) are returned whole. DynamoDB is not read.successes = min(successes, pulls), failures = pulls - successes. These successes are still events; nothing user-based exists to use._at_least_as_completeis deleted: fix(bandit): a partial DynamoDB count no longer replaces the complete PostgreSQL count #430's property (a partial DynamoDB count never replaces PostgreSQL's) holds because PostgreSQL is consulted first./realtime-countersAPI and its numbers: unchanged. No infrastructure, no migration, no new table or index.docs/api/multi-armed-bandit.md,docs/getting-started/architecture.md,docs/architecture/technical-guide.md) and their claim units state this order.Where it runs. The bandit scheduler tick (every
BANDIT_UPDATE_INTERVAL_MINUTES) andPOSTtrigger-update, core app.Failure modes. (1) An experiment with no PostgreSQL assignment rows but DynamoDB counters (counters written through the increment API): the bandit weighs DynamoDB's event-based successes, capped at pulls. The first
/tracking/assignof a bandit experiment writes a PostgreSQL row, which ends this. (2) Real-time counters are no longer preferred while PostgreSQL has data, so the bandit reads PostgreSQL's assignment rows at tick time; those are committed at assignment, so there is no staleness premise to lose. (3) PostgreSQL has pulls for only some variants: those pulls are used as they are (an arm with no assignment row has the prior), not filled in from DynamoDB.Verification. Below; every new test is a regression test.
Verification
Run from a clean clone, DB-free, no AWS:
pytest backend/tests/unit/services/test_bandit_scheduler.py modules/backend/tests/unit/services/test_bandit_scheduler_counters.py: 56 passed.origin/main: 9 failed. Red on main:test_postgres_wins_when_dynamodb_equal_or_larger_on_everything,test_postgres_wins_when_dynamodb_is_ahead_of_postgres, the test of an arm whose pulls are only in DynamoDB,test_uneven_dynamodb_lead_does_not_flip_the_ranking,test_user_converting_three_times_counts_once,test_mixed_users_one_converting_twice,test_event_inflated_dynamodb_successes_never_win,test_dynamodb_successes_capped_at_pulls_without_postgres_data, modulestest_postgres_stats_win_over_the_real_dynamodb_provider.test_postgres_pulls_win_even_when_dynamodb_successes_are_lower, which passes on main (main's completeness check includes successes) and fails only under that plant.python scripts/doc_claims.py check(problems: 0),mkdocs build --strict,ruff checkandruff format --check: clean.NOT VERIFIED
get_experiment_counters, as the existing ones do.lint-importsis not installed in the local venv; the change adds no import, so the core/modules boundary is unchanged. CI runs it.🤖 Generated with Claude Code
https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn