Skip to content

fix(bandit): read converting users from PostgreSQL first, not DynamoDB conversion events - #2076

Open
amarkanday wants to merge 1 commit into
mainfrom
fix/581-bandit-converting-users
Open

amarkanday wants to merge 1 commit into
mainfrom
fix/581-bandit-converting-users

Conversation

@amarkanday

Copy link
Copy Markdown
Contributor

Resolves #581

What

The bandit scheduler could weigh arms on conversion events instead of converting users. DynamoDBCounterService's conversions counter is incremented once per event (process_conversion_event, and the increment/bulk API with any amount); /results and the PostgreSQL fallback (#576) count converting users. The scheduler chose DynamoDB when its pulls AND successes were each at least PostgreSQL's (#430), and an event-inflated success count satisfies that.

Not changed: process_conversion_event (still unwired) and the increment API still count events, and /realtime-counters reports events divided by assignments; the bandit uses those counts only when PostgreSQL has no assignments or cannot be read.

SPEC

Choice. Fix it in the scheduler, the one place the bandit reads, not in a writer: process_conversion_event has no production caller, the live writers of conversions are the generic increment/bulk endpoints (arbitrary amounts), and a per-user write needs a new item shape in the counters table. Mixing sources (DynamoDB's pulls with PostgreSQL's successes) would count two populations: an arm with DynamoDB-only pulls would get 0 successes and weight 0, and a DynamoDB lead in one arm would halve its rate and flip the ranking. So one source is used, whole.

Contract (backend/app/core/bandit_scheduler.py, get_variant_stats_from_counters).

  • Order: PostgreSQL, then DynamoDB, then BanditState, then zero priors.
  • If any variant has a pull in PostgreSQL, PostgreSQL's pulls and converting users (event_matching.count_converting_users) are returned whole. DynamoDB is not read.
  • Otherwise (no PostgreSQL assignments, or PostgreSQL raised) DynamoDB's counts are used with successes = min(successes, pulls), failures = pulls - successes. These successes are still events; nothing user-based exists to use.
  • _at_least_as_complete is deleted: fix(bandit): a partial DynamoDB count no longer replaces the complete PostgreSQL count #430's property (a partial DynamoDB count never replaces PostgreSQL's) holds because PostgreSQL is consulted first.
  • DynamoDB writers, schema, /realtime-counters API and its numbers: unchanged. No infrastructure, no migration, no new table or index.
  • CLAUDE.md, the module docstring and three docs pages (docs/api/multi-armed-bandit.md, docs/getting-started/architecture.md, docs/architecture/technical-guide.md) and their claim units state this order.

Where it runs. The bandit scheduler tick (every BANDIT_UPDATE_INTERVAL_MINUTES) and POST trigger-update, core app.

Failure modes. (1) An experiment with no PostgreSQL assignment rows but DynamoDB counters (counters written through the increment API): the bandit weighs DynamoDB's event-based successes, capped at pulls. The first /tracking/assign of a bandit experiment writes a PostgreSQL row, which ends this. (2) Real-time counters are no longer preferred while PostgreSQL has data, so the bandit reads PostgreSQL's assignment rows at tick time; those are committed at assignment, so there is no staleness premise to lose. (3) PostgreSQL has pulls for only some variants: those pulls are used as they are (an arm with no assignment row has the prior), not filled in from DynamoDB.

Verification. Below; every new test is a regression test.

Verification

Run from a clean clone, DB-free, no AWS:

  • pytest backend/tests/unit/services/test_bandit_scheduler.py modules/backend/tests/unit/services/test_bandit_scheduler_counters.py: 56 passed.
  • Scheduler reverted to origin/main: 9 failed. Red on main: test_postgres_wins_when_dynamodb_equal_or_larger_on_everything, test_postgres_wins_when_dynamodb_is_ahead_of_postgres, the test of an arm whose pulls are only in DynamoDB, test_uneven_dynamodb_lead_does_not_flip_the_ranking, test_user_converting_three_times_counts_once, test_mixed_users_one_converting_twice, test_event_inflated_dynamodb_successes_never_win, test_dynamodb_successes_capped_at_pulls_without_postgres_data, modules test_postgres_stats_win_over_the_real_dynamodb_provider.
  • Plant, mixed sources (DynamoDB pulls with PostgreSQL successes): 5 failed, including test_postgres_pulls_win_even_when_dynamodb_successes_are_lower, which passes on main (main's completeness check includes successes) and fails only under that plant.
  • Plant, DynamoDB preferred when it is at least PostgreSQL's on pulls and successes: 7 failed. Plant, DynamoDB read before PostgreSQL: 11 failed. Plant, no cap on DynamoDB's successes: 1 failed.
  • Bandit/counter/docs-gate tests, python scripts/doc_claims.py check (problems: 0), mkdocs build --strict, ruff check and ruff format --check: clean.

NOT VERIFIED

  • Live DynamoDB: tests use fakes of get_experiment_counters, as the existing ones do.
  • lint-imports is not installed in the local venv; the change adds no import, so the core/modules boundary is unchanged. CI runs it.
  • PostgreSQL-backed integration tests (CI).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn

…B conversion events

The DynamoDB conversion counter adds one per conversion event (and the
increment API adds any amount), so a user who converts three times counts
three there, while /results and the PostgreSQL fallback count converting
users. The bandit scheduler preferred DynamoDB whenever its pulls and
successes were at least PostgreSQL's for every variant, and an
event-inflated success count passes that check, so the bandit could weight
arms on events.

PostgreSQL is now the first source: when any variant has an assignment
there, its pulls and converting users are returned whole and DynamoDB is not
read. DynamoDB is used only when PostgreSQL has no assignments or cannot be
read, with successes capped at pulls. Mixing DynamoDB pulls with PostgreSQL
successes was rejected: the two describe different populations. The
completeness check is deleted. No writer, schema, table or infrastructure
change. CLAUDE.md and the three docs pages that state the source order are
updated with their claim units.

Not changed: process_conversion_event (still unwired) and the increment API
still count events, and /realtime-counters reports events per assignment.

Resolves #581

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kqtSDC7sd5ue3m6LuwdXV
Signed-off-by: amarkanday <a.markanday@gmail.com>
@amarkanday amarkanday added the bug Something isn't working label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The DynamoDB conversion counter counts conversion events, not converting users

1 participant