Repository navigation
fix(results): the heterogeneous-effects banner no longer fires when the effect is the same in every segment (engine 1.12.0) - #1981
Merged
Conversation
…he effect is the same in every segment (engine 1.12.0) A breakdown's has_heterogeneous_effects is now true only when a test that the first treatment's difference from the control, in percentage points, is the same in every tested segment rejects at 1 minus the confidence level the results were computed at (#219's interaction_test, on the segments past a floor of 10 expected converters and non-converters per arm). The converters-only chi-squared test at a fixed 0.05, the coefficient-of-variation and opposite-sign heuristic and the silent except are deleted. detect_hte's alpha is required, and _compute_dimensional_breakdown's base_alpha has no default. On the reported case (4 segments of 90 users per arm, the same lift in each) the flag goes from 1216 to 108 of 2000 seeded tables at alpha .05. Engine 1.12.0; the OpenAPI snapshots change in the engine_version defaults and the field's description. Part of #1969. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ Signed-off-by: amarkanday <a.markanday@gmail.com>
…oors Review nit on #1981. interaction_test said to call it only on a table floor_reason accepts, and InteractionFitError that the fit cannot raise on a table that passes "the floor". detect_hte now calls it on the segments hte_tested_segments keeps, a lower floor, so both lines read as broken by that call. Each now names both floors: floor_reason for #219's pair analysis, hte_tested_segments for the breakdown's heterogeneity flag, with a pointer to the module docstring and the simulations behind the breakdown's floor. Docstrings only; the code is unchanged. Part of #1969. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ Signed-off-by: amarkanday <a.markanday@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
the heterogeneous-effects banner no longer fires when the effect is the same in every segment. It now needs a test of equal percentage-point differences to reject at the results' significance level; in the reported case (4 segments of 90 users per arm, the same lift in each) it fired on about 60% of experiments, and now on about 5%.
has_heterogeneous_effectsis true less often. Statistics engine 1.12.0.Part of #1969. Built to the reviewed plan and its binding conditions, PEU3-1..14, EMU3-1..8, EMCF-1 and EMCF-2, each listed below with its evidence.
What changed
DimensionalAnalysisService.detect_hte(segment_results, *, alpha)now runs Interaction analysis reports an interaction from user counts; novelty is never computed #219'sinteraction_analysis.interaction_test: the additive model (one baseline per segment, one common difference) fitted by binomial maximum likelihood, and Pearson's X² against it on J - 1 df. It tests the first treatment against the control on the segments that pass the floor, and returnsbool(p < alpha).alphais keyword-only and required. A value that is not finite and strictly between 0 and 1 raisesValueError.chi2_contingencyat a fixed 0.05 is deleted, along with the coefficient-of-variation / opposite-sign heuristic and theexcept Exception: logger.debug.InteractionFitErrorpropagates. The whole/results?breakdown=request and the whole/balanced-groupsrequest (estimate included) then answer 500 throughunexpected_failure.hte_tested_segments, withHTE_MIN_EXPECTED_PER_CELL = 10shared with the tests' replica):n * conv >= 10 * usersandn * (users - conv) >= 10 * users. The comparison is made in integers.results.py:detect_hte(segment_results, alpha=base_alpha)._compute_dimensional_breakdown'sbase_alphahas no default.has_heterogeneous_effects: PEU3-8's text.make openapiregenerated both snapshots.docs/api/dimensional-analysis.mdHTE Detection is PEU3-9's text, and its five claims are bound in PEU3-9's order.docs/api/stability.mdgains the 1.12.0 row.interaction_analysis.pygains a docstring paragraph (no code change; Interaction analysis reports an interaction from user counts; novelty is never computed #219's golden tests do not move).ENGINE_VERSIONmoves, and 1.12.0 is appended toRELEASED_ENGINE_VERSIONS,FROZEN_DIGESTSandFROZEN_VERSIONS.ENGINE_FINGERPRINTS["1.12.0"]is a new hash. The fingerprint's call gainsalpha=0.05, and its dataset gains the walk data (breakdown_walk)."1.12.0".backend/tests/unit/stats_validation/test_golden_hte_false_positive_rate.py(18 tests) andbackend/tests/unit/services/test_hte_detection.py(33).test_collection_countgains the new module.**kwargsthrough.Before and after (RandomState(1969), 2,000 tables per shape, control drawn before treatment)
unknownsegment is below the floor; the other three give X² 2.97, df 2, p 0.227.Each condition and its evidence
PEU3-1, as replaced by EMCF-1 (V8): RandomState(1969), control drawn first.
test_an_ordinary_null_is_flagged_at_most_alphachecks each of the four ordinary shapes forflagged <= 133,tested >= 1900and the exact pins:test_a_null_in_the_liberal_corner_stays_below_its_boundchecks the two corner shapes forflagged <= 147,tested == 2000and the exact pins: 22 x 50 at 50% flags 121, 21 x 50 at 30% -> 38% flags 139.test_a_shape_below_the_floor_is_never_tested: 21 x 50 at 10% -> 18% and 4 x 90 at 2% -> 4% are tested 0 times.Plant: floor 5 in the constant (plant table).
PEU3-2 (+ EMCF-1's 21 x 50 sentence): re-measured below.
PEU3-3:
test_the_replica_agrees_with_detect_hte.compute_segment_resultsand the realdetect_hteat .01, .05 and .10.interaction_testmemoised per table inside the test.hte_tested_segments.PEU3-4: a plant inside
detect_htenever moves the replica's pins. The plant table shows the cross-check, V4 and V16 catching those plants. The pins catch the shared-constant floor plants, which move the replica too.PEU3-5 (floor):
hte_tested_segments).test_a_segment_at_the_floor_is_tested(True; p 4.4e-5) andtest_a_segment_one_conversion_below_the_floor_is_not_tested(False; p 1.0), both through_compute_dimensional_breakdown.test_the_floor_is_computed_in_integerspins an exact boundary where77 * (100 / 770)is 9.999999999999998 in floating point.PEU3-6: R2 is the cross-check (
@pytest.mark.regression). On v0.47.0 it fails by TypeError onalpha=. The rule-level evidence is 315 of 600 (below).PEU3-7:
InteractionFitErrorpropagates.test_a_failed_fit_fails_the_whole_breakdownmonkeypatchesinteraction_analysis.interaction_testto raise it, and_compute_dimensional_breakdownraises. The plant that catches it and returns False fails that test. The x > n change is not made (declined by the EM).PEU3-8: the schema description is PEU3-8's text. Both snapshots equal it character for character (checked by script against the review file).
PEU3-9:
dimensional-analysis.mdHTE Detection is PEU3-9's text, character for character. Its five claims are bound in order:test_the_first_treatment_by_name_is_the_one_tested.test_the_breakdown_tests_at_the_alpha_it_is_given,test_the_alpha_fixtures_p_valuesandtest_the_replica_agrees_with_detect_hte.test_too_little_to_test_is_false_without_a_warningandtest_a_segment_one_conversion_below_the_floor_is_not_tested.test_each_route_hands_its_confidence_level_down_as_the_alpha.V13's grep over
docs/(Markdown, JSON, TOML) finds 0 hits each for "varies widely", "positive in one segment", "chi-squared test detects" and "Cochran's Q". On main the first is in the page and the third in both snapshots. Plant: editing the doc alone givesdoc_claims.py check2 problems andtest_the_registries_holdfails.PEU3-10: the warning is PEU3-10's text, character for character. Its
'<dimension>'is the requested dimension.PEU3-11: this body opens with it, after "fix(results): ".
PEU3-12:
_compute_dimensional_breakdown(..., base_alpha: float, ...)has no default. Both routes andtest_breakdown_reads.pypass it.PEU3-13:
test_dimensional_analysis_api.py's copy of the warning is the new text. V4's True cases asserthte_warning ==the sentence with the dimension, written out in the test.PEU3-14: the IRREVERSIBLE section below.
EMU3-1: this list.
EMU3-2: the title is exactly the EMU3-2 line; the body opens with PEU3-11 after "fix(results): ".
EMU3-3, as replaced by EMCF-2 (V14):
test_the_first_treatment_by_name_is_the_one_testedruns through_compute_dimensional_breakdownand the_Readersharness.00000000-…-0001, name "zeta"; T2 = idffffffff-…-0003, name "alpha". The pairs are fixed, and the route listscontrol, alpha, zeta.EMU3-4: the regression proof and the plant table, below.
EMU3-5:
interaction_analysis.pydocstring names the liberal corner and quotes only 108 (V1), 121 and 139 (V8).EMU3-6:
stability.mdgains a 1.12.0 row in the 1.11.0 row's form.GET /results/{id}withbreakdown, and/balanced-groups. No other operation serves the breakdown:_compute_dimensional_breakdownhas only these two callers, and/experiments/{id}/resultssetsbreakdown=None.has_heterogeneous_effectsis true less often, andhte_warninghas new text. No answer cached before the upgrade is served.engine_versiondefaults and the field's description, in both snapshots. Every other result is unchanged.exempt:baseline.EMU3-7 (at merge):
make openapiwas run on it.make openapi.EMU3-8: the review request is the coordinator's.
EMCF-1 / EMCF-2: as PEU3-1 and EMU3-3 above.
Review round 1 nit, fixed in c504fe0:
interaction_test's precondition andInteractionFitError's docstring said "the floor", meaningfloor_reason, whichdetect_hte's call did not meet. They now name both callers' floors:floor_reasonfor Interaction analysis reports an interaction from user counts; novelty is never computed #219's pair analysis,hte_tested_segmentsfor the breakdown. Docstrings only, and the AST is unchanged. All tests that import the module pass: 165 passed, 2 skipped (the two SLSQP cases already skipped).Regression proof (EMU3-4, PEU3-6)
git archivecopy of this head withdimensional_analysis_service.py,results.pyandstats_engine.pyrestored from v0.47.0 (cmpagainstgit show v0.47.0:<path>: identical).pytest --collect-only -q -m regressionon the two new files: 8 tests.assert True is False), 4:test_the_walk_data_shows_no_heterogeneity(R1);test_equal_differences_on_an_unequal_split_are_not_flagged(R4 = V9);test_a_segment_without_control_users_is_not_tested(R3);test_the_breakdown_tests_at_the_alpha_it_is_given[152-0.01-False](R5).TypeError: DimensionalAnalysisService.detect_hte() got an unexpected keyword argument 'alpha', 4:test_the_replica_agrees_with_detect_hte[rw1_null|unequal_split|moderate_heterogeneity|floor_binding](R2, PEU3-6).detect_hte, throughcompute_segment_results, on V1's 2,000 draws gives 1216. V2's pin (RW1_NULL_0_47_0_PIN, the test's frozen copy of the 0.47.0 rule) is 1216.Plant table (EMU3-4)
Each plant was made in its own
git archivecopy of this head (no.git). The gates run were:test_dimensional_analysis_service.py,test_engine_fingerprint.py,test_engine_versions_frozen.pyandtest_dimensional_analysis_api.py;The unplanted baseline in the same harness (with the snapshot and collection-count tests): 120 passed.
Cross-check counts are disagreements out of 600 decisions for each shape: RW-1 / unequal / V7 / floor-binding.
detect_htedetect_htedetect_htedetect_htedetect_htedetect_hteonlydetect_hteonlyresults.pyadjusted_alpharesults.pyresults.pyalpha/resultspassesbase_alpha=0.05results.py[0.05] != [0.01]/balanced-groupspassesbase_alpha=0.05results.py[0.01, 0.05] != [0.01, 0.10]detect_htedetect_hteinteraction_testcalled with one or no segmentnumpy.bool_detect_htetype is bool) ×4; V10 zero-variance; TestDetectHTEis True/False,isinstance(bool)InteractionFitErrorcaught, False returneddetect_htestats_engine.pytest_a_frozen_versions_pins_are_unchanged[1.11.0]test_core_api_matches_stable_snapshot,test_full_api_matches_full_snapshot, balanced-groups schema testtest_no_gate_here_can_skipdoc_claims.py checkproblems 2;test_the_registries_holdis_control(S1)nan < alphais False, so the answer stayed right by accident.interaction_testis never called there.Re-measure of PEU3-2's calibration (EMU3-5)
The replica is the one the cross-check holds to the real path. 20,000 tables, seed 2026; each figure is the share flagged among the tables tested.
Local runs (venv311: numpy 1.26.4, scipy 1.17.1; no database, no Redis)
The whole-directory runs were made on this commit before its last amend. The amend changed only two test files: the cross-check's failure message, and V10's check that the test is not run. Both files and stats_validation were re-run after it.
backend/tests/unit/stats_validation: 130 passed, 0 skipped (check_junit_skips.pyclean). The new module has 18 tests and takes 38 s locally, 30.5 s of it the cross-check.test_collection_countpasses.backend/tests/unit/services, the whole directory:3638 passed.
3 skipped, all already skipped before this change: two SLSQP cases in
test_interaction_analysis.pyand a benchmark.61 errors, every one a PostgreSQL connection at setup (
POSTGRES_PORT=1on purpose), intest_audit_service*.pyandtest_experiment_scheduling.py. This PR touches neither; CI runs them.Among them:
test_hte_detection.pytest_dimensional_analysis_service.pytest_breakdown_reads.pytest_engine_fingerprint.pytest_engine_versions_frozen.pytest_sufficient_stats_fingerprint.pytest_lower_is_better_fingerprint.pytest_results_correction_pins.pytest_revenue_per_user_routing.pytest_value_metrics_per_user.pytest_binomial_metric_result_characterisation.pytest_dimensional_analysis_api.py: 12 passed.backend/tests/smoke/test_openapi_snapshot.py: 13 passed.test_doc_claims.py: 172 passed, andpython scripts/doc_claims.py checkreports problems 0.test_docs_only_gate.py: 46 passed.test_openapi_dump_and_stability.py: 6 passed.A core-shaped copy (
git archive, no.git,modules/deleted): stats_validation plus the changed breakdown, engine and API tests give 239 passed, anddoc_claims.py checkreports problems 0.ruff checkandruff format --checkoverbackend/ modules/ scripts/are clean.check_dco.py,leak_guard.py --rangeandgitleaks gitare clean on the final commit.OpenAPI
make openapichanges, in each ofopenapi-v1.full.jsonandopenapi-v1.stable.json:engine_versiondefaults, 1.11.0 -> 1.12.0;has_heterogeneous_effects, which is PEU3-8's text, checked character for character against the gate.Nothing else changes.
frontend/src/tests/fixtures/openapi.jsondoes not change.IRREVERSIBLE
Nothing. Undoing it after release is a new engine-changing PR (1.13.0), per the rule in test_engine_versions_frozen.py ("A revert of an engine-changing PR bumps again").
NOT VERIFIED
**kwargs;"1.12.0";InteractionFitErroris tested at_compute_dimensional_breakdown. That both route handlers turn it into a 500 is read from source (except Exception->unexpected_failure), not run.ENGINE_VERSION, and the cache-key test passes.lint-imports,reuse lint, eslint/tsc, hadolint, actionlint andmkdocs build --strict. The one new import is a core service importing a core service.has_heterogeneous_effects && hte_warning.🤖 Generated with Claude Code
https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ