Skip to content

fix(results): the heterogeneous-effects banner no longer fires when the effect is the same in every segment (engine 1.12.0) - #1981

Merged
amarkanday merged 4 commits into
mainfrom
fix/1969-hte-test
Nov 15, 2026
Merged

amarkanday merged 4 commits into
mainfrom
fix/1969-hte-test

Conversation

@amarkanday

@amarkanday amarkanday commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

the heterogeneous-effects banner no longer fires when the effect is the same in every segment. It now needs a test of equal percentage-point differences to reject at the results' significance level; in the reported case (4 segments of 90 users per arm, the same lift in each) it fired on about 60% of experiments, and now on about 5%. has_heterogeneous_effects is true less often. Statistics engine 1.12.0.

Part of #1969. Built to the reviewed plan and its binding conditions, PEU3-1..14, EMU3-1..8, EMCF-1 and EMCF-2, each listed below with its evidence.

What changed

  • DimensionalAnalysisService.detect_hte(segment_results, *, alpha) now runs Interaction analysis reports an interaction from user counts; novelty is never computed #219's interaction_analysis.interaction_test: the additive model (one baseline per segment, one common difference) fitted by binomial maximum likelihood, and Pearson's X² against it on J - 1 df. It tests the first treatment against the control on the segments that pass the floor, and returns bool(p < alpha).
    • alpha is keyword-only and required. A value that is not finite and strictly between 0 and 1 raises ValueError.
    • The converters-only chi2_contingency at a fixed 0.05 is deleted, along with the coefficient-of-variation / opposite-sign heuristic and the except Exception: logger.debug.
    • InteractionFitError propagates. The whole /results?breakdown= request and the whole /balanced-groups request (estimate included) then answer 500 through unexpected_failure.
  • Floor (hte_tested_segments, with HTE_MIN_EXPECTED_PER_CELL = 10 shared with the tests' replica):
    • The candidates are the segments with at least one user in both arms, and the pooled rate is taken over them.
    • A segment is tested when, in both arms, n * conv >= 10 * users and n * (users - conv) >= 10 * users. The comparison is made in integers.
    • With fewer than two tested segments the answer is False.
  • The treatment is the first non-control variant of the first segment. The route lists the control first and then the variants by name, so it is the non-control variant whose name sorts first. It is looked up by id in every segment.
  • results.py:
    • detect_hte(segment_results, alpha=base_alpha).
    • _compute_dimensional_breakdown's base_alpha has no default.
    • The warning is PEU3-10's text.
  • Schema description of has_heterogeneous_effects: PEU3-8's text. make openapi regenerated both snapshots.
  • Docs:
  • Engine 1.12.0 by the bump template:
    • ENGINE_VERSION moves, and 1.12.0 is appended to RELEASED_ENGINE_VERSIONS, FROZEN_DIGESTS and FROZEN_VERSIONS.
    • ENGINE_FINGERPRINTS["1.12.0"] is a new hash. The fingerprint's call gains alpha=0.05, and its dataset gains the walk data (breakdown_walk).
    • The other six pin tables copy their 1.11.0 entry. No frozen entry is edited.
    • The integration literal moves to "1.12.0".
  • Tests:
    • New: backend/tests/unit/stats_validation/test_golden_hte_false_positive_rate.py (18 tests) and backend/tests/unit/services/test_hte_detection.py (33).
    • test_collection_count gains the new module.
    • The integration spy passes **kwargs through.

Before and after (RandomState(1969), 2,000 tables per shape, control drawn before treatment)

shape 0.47.0 rule now, alpha .05
RW-1 null (4 x 90/arm, 10% -> 18% everywhere) 1216 108 (24 at .01, 219 at .10)
A/A 4 x 1000/arm at 10% 1940 120
unequal split (600/200 alternating), 10% -> 15% everywhere 2000 124
strong heterogeneity (4 x 90, 30/30/5/5% vs 10%) 2000 1967
moderate heterogeneity (4 x 500, 18/18/10/10% vs 10%) 1983 1835 (theory 0.921, band 1806-1879)
  • The RW-1 walk data goes from flagged to not flagged. The unknown segment is below the floor; the other three give X² 2.97, df 2, p 0.227.
  • The engine fingerprint's own breakdown goes from flagged (by lifts of both signs) to not flagged (p 0.061).

Each condition and its evidence

  • PEU3-1, as replaced by EMCF-1 (V8): RandomState(1969), control drawn first.

    • test_an_ordinary_null_is_flagged_at_most_alpha checks each of the four ordinary shapes for flagged <= 133, tested >= 1900 and the exact pins:

      shape flagged tested
      tiny 300/90/30/10 117 1988
      500/500/5/2 99 2000
      4 x 500 at 2% -> 4% 108 2000
      unequal 600/200 124 2000
    • test_a_null_in_the_liberal_corner_stays_below_its_bound checks the two corner shapes for flagged <= 147, tested == 2000 and the exact pins: 22 x 50 at 50% flags 121, 21 x 50 at 30% -> 38% flags 139.

    • test_a_shape_below_the_floor_is_never_tested: 21 x 50 at 10% -> 18% and 4 x 90 at 2% -> 4% are tested 0 times.

    • Plant: floor 5 in the constant (plant table).

  • PEU3-2 (+ EMCF-1's 21 x 50 sentence): re-measured below.

  • PEU3-3: test_the_replica_agrees_with_detect_hte.

    • It sends the first 200 tables of RW-1's null, the unequal null, V7 and the floor-binding null (300/90/60/40, 10% -> 18%) through compute_segment_results and the real detect_hte at .01, .05 and .10.
    • Result: 2,400 decisions, 0 disagreements, 30.5 s locally (7.2-8.0 s per shape) with interaction_test memoised per table inside the test.
    • The replica calls the production hte_tested_segments.
    • All seven plants fail it (plant table).
  • PEU3-4: a plant inside detect_hte never moves the replica's pins. The plant table shows the cross-check, V4 and V16 catching those plants. The pins catch the shared-constant floor plants, which move the replica too.

  • PEU3-5 (floor):

    • The floor is computed in integers (hte_tested_segments).
    • V16 is test_a_segment_at_the_floor_is_tested (True; p 4.4e-5) and test_a_segment_one_conversion_below_the_floor_is_not_tested (False; p 1.0), both through _compute_dimensional_breakdown.
    • test_the_floor_is_computed_in_integers pins an exact boundary where 77 * (100 / 770) is 9.999999999999998 in floating point.
    • Floor 5 and floor 15 are planted in both places (plant table).
  • PEU3-6: R2 is the cross-check (@pytest.mark.regression). On v0.47.0 it fails by TypeError on alpha=. The rule-level evidence is 315 of 600 (below).

  • PEU3-7: InteractionFitError propagates. test_a_failed_fit_fails_the_whole_breakdown monkeypatches interaction_analysis.interaction_test to raise it, and _compute_dimensional_breakdown raises. The plant that catches it and returns False fails that test. The x > n change is not made (declined by the EM).

  • PEU3-8: the schema description is PEU3-8's text. Both snapshots equal it character for character (checked by script against the review file).

  • PEU3-9: dimensional-analysis.md HTE Detection is PEU3-9's text, character for character. Its five claims are bound in order:

    1. V14: test_the_first_treatment_by_name_is_the_one_tested.
    2. V4 + the cross-check: test_the_breakdown_tests_at_the_alpha_it_is_given, test_the_alpha_fixtures_p_values and test_the_replica_agrees_with_detect_hte.
    3. V16: both floor tests.
    4. V10 + V16's second fixture: test_too_little_to_test_is_false_without_a_warning and test_a_segment_one_conversion_below_the_floor_is_not_tested.
    5. V5: test_each_route_hands_its_confidence_level_down_as_the_alpha.

    V13's grep over docs/ (Markdown, JSON, TOML) finds 0 hits each for "varies widely", "positive in one segment", "chi-squared test detects" and "Cochran's Q". On main the first is in the page and the third in both snapshots. Plant: editing the doc alone gives doc_claims.py check 2 problems and test_the_registries_hold fails.

  • PEU3-10: the warning is PEU3-10's text, character for character. Its '<dimension>' is the requested dimension.

  • PEU3-11: this body opens with it, after "fix(results): ".

  • PEU3-12: _compute_dimensional_breakdown(..., base_alpha: float, ...) has no default. Both routes and test_breakdown_reads.py pass it.

  • PEU3-13: test_dimensional_analysis_api.py's copy of the warning is the new text. V4's True cases assert hte_warning == the sentence with the dimension, written out in the test.

  • PEU3-14: the IRREVERSIBLE section below.

  • EMU3-1: this list.

  • EMU3-2: the title is exactly the EMU3-2 line; the body opens with PEU3-11 after "fix(results): ".

  • EMU3-3, as replaced by EMCF-2 (V14): test_the_first_treatment_by_name_is_the_one_tested runs through _compute_dimensional_breakdown and the _Readers harness.

    • Setup: control; T1 = id 00000000-…-0001, name "zeta"; T2 = id ffffffff-…-0003, name "alpha". The pairs are fixed, and the route lists control, alpha, zeta.
    • Case A (only T1's difference varies): False. Case B (the varying counts moved to T2, T1 homogeneous): True.
    • Plant (the treatment picked by smallest id): Case A returns True and Case B returns False; both fail (plant table). This is the run EMCF-2 asked for.
  • EMU3-4: the regression proof and the plant table, below.

  • EMU3-5:

    • No rate appears in the docs, the schema, the warning or the claims.
    • The interaction_analysis.py docstring names the liberal corner and quotes only 108 (V1), 121 and 139 (V8).
    • The PR's re-measure of PEU3-2 is below.
  • EMU3-6: stability.md gains a 1.12.0 row in the 1.11.0 row's form.

    • Operations: GET /results/{id} with breakdown, and /balanced-groups. No other operation serves the breakdown: _compute_dimensional_breakdown has only these two callers, and /experiments/{id}/results sets breakdown=None.
    • What changes: has_heterogeneous_effects is true less often, and hte_warning has new text. No answer cached before the upgrade is served.
    • The OpenAPI document changes only in the engine_version defaults and the field's description, in both snapshots. Every other result is unchanged.
    • Its claim binds to V1, V4, V11 (fingerprint, frozen), V12 (both snapshot tests) and the cache-key test. There is no exempt:baseline.
  • EMU3-7 (at merge):

    • This head is based on current main (517080c, a docs-only roadmap commit on top of v0.47.0), and make openapi was run on it.
    • No open PR touches the engine. The release PR chore(main): release 0.48.0 #1979 (0.47.1) touches both snapshots' version string, so whichever merges second re-runs make openapi.
    • The ADMIT-0480 line is the coordinator's.
  • EMU3-8: the review request is the coordinator's.

  • EMCF-1 / EMCF-2: as PEU3-1 and EMU3-3 above.

  • Review round 1 nit, fixed in c504fe0: interaction_test's precondition and InteractionFitError's docstring said "the floor", meaning floor_reason, which detect_hte's call did not meet. They now name both callers' floors: floor_reason for Interaction analysis reports an interaction from user counts; novelty is never computed #219's pair analysis, hte_tested_segments for the breakdown. Docstrings only, and the AST is unchanged. All tests that import the module pass: 165 passed, 2 skipped (the two SLSQP cases already skipped).

Regression proof (EMU3-4, PEU3-6)

  • Setup: a git archive copy of this head with dimensional_analysis_service.py, results.py and stats_engine.py restored from v0.47.0 (cmp against git show v0.47.0:<path>: identical).
  • Collected: pytest --collect-only -q -m regression on the two new files: 8 tests.
  • Run: 8 failed, 0 passed.
    • By assertion (assert True is False), 4:
      • test_the_walk_data_shows_no_heterogeneity (R1);
      • test_equal_differences_on_an_unequal_split_are_not_flagged (R4 = V9);
      • test_a_segment_without_control_users_is_not_tested (R3);
      • test_the_breakdown_tests_at_the_alpha_it_is_given[152-0.01-False] (R5).
    • By TypeError: DimensionalAnalysisService.detect_hte() got an unexpected keyword argument 'alpha', 4: test_the_replica_agrees_with_detect_hte[rw1_null|unequal_split|moderate_heterogeneity|floor_binding] (R2, PEU3-6).
  • The real 0.47.0 rule, same copy: the real 0.47.0 detect_hte, through compute_segment_results, on V1's 2,000 draws gives 1216. V2's pin (RW1_NULL_0_47_0_PIN, the test's frozen copy of the 0.47.0 rule) is 1216.
  • Rule-level evidence (PEU3-6): under the new signature, the 0.47.0 body disagrees with the replica on 315 of 600 decisions (the first 200 RW-1 tables at .01, .05 and .10).

Plant table (EMU3-4)

Each plant was made in its own git archive copy of this head (no .git). The gates run were:

  • the two new modules;
  • test_dimensional_analysis_service.py, test_engine_fingerprint.py, test_engine_versions_frozen.py and test_dimensional_analysis_api.py;
  • plus the gate the row names.

The unplanted baseline in the same harness (with the snapshot and collection-count tests): 120 passed.

Cross-check counts are disagreements out of 600 decisions for each shape: RW-1 / unequal / V7 / floor-binding.

plant where gate that fired count
alpha fixed at 0.05 detect_hte cross-check ×4; V4 (152 at .01, 144 at .10) 16 / 16 / 33 / 26
OR the 0.47.0 rule detect_hte cross-check ×4; R1, R3, R4, R5; V4 144 at .05; V16 below floor; V10 one-empty-control; PEU3-7 test; fingerprint 315 / 568 / 66 / 349
OR the count chi-square detect_hte cross-check ×4; R4 (V9); V10 one-empty-control 10 / 568 / 3 / 11
df = K detect_hte cross-check ×4; V4 144 at .10 13 / 12 / 38 / 33
alpha/K detect_hte cross-check ×4; V4 152 at .05, 144 at .10 22 / 21 / 72 / 35
floor 5 detect_hte only cross-check (RW-1, floor-binding); V16 second fixture (True) 1 / 0 / 0 / 39
floor 15 detect_hte only cross-check (RW-1, floor-binding); V16 first fixture (False); V10 zero-variance (False) 30 / 0 / 0 / 53
floor 5 shared constant exact pins: V1 .05 110≠108, .10 222≠219; V6 1993≠1967; V8 tiny tested 2000≠1988; 21x50 at 10->18 tested 2000≠0. V16 second fixture (True); integer-floor test (cross-check agrees: replica and real path share it)
floor 15 shared constant exact pins and bands: V1 0 / 2 / 4 of 2000 (below every band); V6 18 < 1900; V8 tiny 26 and 4x500 1007 tested < 1900. V16 first fixture (False); integer-floor test; V10 zero-variance (as above)
call site passes 0.05 results.py V4: 152 at .01 True, 144 at .10 False 2 tests
call site passes adjusted_alpha results.py V4: 152 at .05 False (.0125 < p .0214), 144 at .10 False 2 tests
call site omits alpha results.py every real-path test: TypeError, missing required keyword-only alpha 12 tests
/results passes base_alpha=0.05 results.py V5: [0.05] != [0.01] 1 test
/balanced-groups passes base_alpha=0.05 results.py V5: [0.01, 0.05] != [0.01, 0.10] 1 test
treatment picked by smallest id detect_hte V14: Case A returns True, Case B returns False (EMCF-2) 2 tests
no fewer-than-two guard detect_hte V10: interaction_test called with one or no segment 4 tests
returns numpy.bool_ detect_hte cross-check (type is bool) ×4; V10 zero-variance; TestDetectHTE is True/False, isinstance(bool) 9 tests
InteractionFitError caught, False returned detect_hte PEU3-7 test: DID NOT RAISE 1 test
fix without the bump (ENGINE_VERSION 1.11.0) stats_engine.py fingerprint ("output changed under ENGINE_VERSION 1.11.0"); running engine != last frozen 2 tests
1.11.0 fingerprint edited pin table test_a_frozen_versions_pins_are_unchanged[1.11.0] 1 test
snapshots not regenerated OpenAPI test_core_api_matches_stable_snapshot, test_full_api_matches_full_snapshot, balanced-groups schema test 3 tests
a skip marker on the new module gate module test_no_gate_here_can_skip 1 test
doc edited without its claim docs doc_claims.py check problems 2; test_the_registries_hold 1 test
harness without is_control (S1) gate module V2 structure check; cross-check ×4 30 / 32 / 528 / 53
  • One plant fired nothing on the first run: removing the fewer-than-two guard. With fewer than two tested segments the test has no degrees of freedom, scipy returns a nan p-value, and nan < alpha is False, so the answer stayed right by accident.
    • The V10 test now also requires that interaction_test is never called there.
    • The second run above shows it firing (4 tests).

Re-measure of PEU3-2's calibration (EMU3-5)

The replica is the one the cross-check holds to the real path. 20,000 tables, seed 2026; each figure is the share flagged among the tables tested.

shape tested alpha .05 alpha .10
RW-1 null 99% 4.94% (z -0.4) 10.08%
A/A 4 x 1000 at 10% 100% 5.13% (z +0.9) 10.07%
unequal 600/200 alternating, 10% -> 15% 100% 5.16% (z +1.0) 9.95%
4 x 500 at 2% -> 4% 100% 4.92% (z -0.5) 10.11%
90/10 split (4 x 900/100), A/A at 10% 50% 4.67% (z -1.5) 9.71%
95/5 split (4 x 950/50), A/A at 20% 50% 4.98% (z -0.1) 10.08%
22 x 50/arm at 50% 100% 5.67% (z +4.4) 11.02% (z +4.8)
22 x 30/arm at 50% 100% 5.90% (z +5.8) 11.33% (z +6.3)
21 x 50/arm at 30% -> 38% 100% 5.50% (z +3.2) 10.66% (z +3.1)
  • Power at 4 x 90/arm, +16 points in two of four segments: 75.2% of all tables at .05 (75.8% of those tested). PEU3-2 says 74%.
  • PEU3-2 did not give the shapes for its 95/5 and 90/10 rows. The two above are mine, both about half untested, which matches its note.
  • Every other figure matches PEU3-2 and EMCF-1's addition: RW-1 4.9%; A/A 5.1-5.2%; unequal 5.0-5.2%; 4 x 500 at 2% -> 4% 4.9%; 22 x 50 at 50% 5.7% / 11.0%, z +4.4; 22 x 30 at 50% 5.9% / 11.3%; 21 x 50 at 30% -> 38% 5.5%, z +3.2.

Local runs (venv311: numpy 1.26.4, scipy 1.17.1; no database, no Redis)

The whole-directory runs were made on this commit before its last amend. The amend changed only two test files: the cross-check's failure message, and V10's check that the test is not run. Both files and stats_validation were re-run after it.

  • backend/tests/unit/stats_validation: 130 passed, 0 skipped (check_junit_skips.py clean). The new module has 18 tests and takes 38 s locally, 30.5 s of it the cross-check. test_collection_count passes.

  • backend/tests/unit/services, the whole directory:

    • 3638 passed.

    • 3 skipped, all already skipped before this change: two SLSQP cases in test_interaction_analysis.py and a benchmark.

    • 61 errors, every one a PostgreSQL connection at setup (POSTGRES_PORT=1 on purpose), in test_audit_service*.py and test_experiment_scheduling.py. This PR touches neither; CI runs them.

    • Among them:

      test file passed
      test_hte_detection.py 33
      test_dimensional_analysis_service.py 23
      test_breakdown_reads.py 10
      test_engine_fingerprint.py 2
      test_engine_versions_frozen.py 17
      test_sufficient_stats_fingerprint.py 12
      test_lower_is_better_fingerprint.py 21
      test_results_correction_pins.py 22
      test_revenue_per_user_routing.py 4
      test_value_metrics_per_user.py 20
      test_binomial_metric_result_characterisation.py 7
  • test_dimensional_analysis_api.py: 12 passed.

  • backend/tests/smoke/test_openapi_snapshot.py: 13 passed.

  • test_doc_claims.py: 172 passed, and python scripts/doc_claims.py check reports problems 0.

  • test_docs_only_gate.py: 46 passed. test_openapi_dump_and_stability.py: 6 passed.

  • A core-shaped copy (git archive, no .git, modules/ deleted): stats_validation plus the changed breakdown, engine and API tests give 239 passed, and doc_claims.py check reports problems 0.

  • ruff check and ruff format --check over backend/ modules/ scripts/ are clean.

  • check_dco.py, leak_guard.py --range and gitleaks git are clean on the final commit.

OpenAPI

make openapi changes, in each of openapi-v1.full.json and openapi-v1.stable.json:

  • the four engine_version defaults, 1.11.0 -> 1.12.0;
  • the description of has_heterogeneous_effects, which is PEU3-8's text, checked character for character against the gate.

Nothing else changes. frontend/src/tests/fixtures/openapi.json does not change.

IRREVERSIBLE

Nothing. Undoing it after release is a new engine-changing PR (1.13.0), per the rule in test_engine_versions_frozen.py ("A revert of an engine-changing PR bumps again").

NOT VERIFIED

  • PostgreSQL and integration tests: CI is the first run of:
    • the integration spy's **kwargs;
    • the balanced-groups literal "1.12.0";
    • the 61 database-backed unit tests above.
  • The route's 500: the propagation of InteractionFitError is tested at _compute_dimensional_breakdown. That both route handlers turn it into a 500 is read from source (except Exception -> unexpected_failure), not run.
  • The Redis cache miss after the bump is not run against Redis. The key carries ENGINE_VERSION, and the cache-key test passes.
  • Not run locally: lint-imports, reuse lint, eslint/tsc, hadolint, actionlint and mkdocs build --strict. The one new import is a core service importing a core service.
  • The new fingerprint was computed with numpy 1.26.4 and scipy 1.17.1, the pinned versions. CI is its first run on Linux; outputs are rounded to 10 significant digits.
  • CI time of the new module was measured only locally, and is not asserted.
  • PEU3-2's shapes: its 95/5 and 90/10 rows are re-measured on shapes of my choosing (above).
  • The dashboard was not run. No frontend file changes; the banner still renders on has_heterogeneous_effects && hte_warning.

🤖 Generated with Claude Code

https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ

amarkanday and others added 4 commits November 14, 2026 23:03
…he effect is the same in every segment (engine 1.12.0)

A breakdown's has_heterogeneous_effects is now true only when a test that
the first treatment's difference from the control, in percentage points,
is the same in every tested segment rejects at 1 minus the confidence
level the results were computed at (#219's interaction_test, on the
segments past a floor of 10 expected converters and non-converters per
arm). The converters-only chi-squared test at a fixed 0.05, the
coefficient-of-variation and opposite-sign heuristic and the silent
except are deleted. detect_hte's alpha is required, and
_compute_dimensional_breakdown's base_alpha has no default.

On the reported case (4 segments of 90 users per arm, the same lift in
each) the flag goes from 1216 to 108 of 2000 seeded tables at alpha .05.
Engine 1.12.0; the OpenAPI snapshots change in the engine_version
defaults and the field's description.

Part of #1969.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ
Signed-off-by: amarkanday <a.markanday@gmail.com>
…oors

Review nit on #1981. interaction_test said to call it only on a table
floor_reason accepts, and InteractionFitError that the fit cannot raise
on a table that passes "the floor". detect_hte now calls it on the
segments hte_tested_segments keeps, a lower floor, so both lines read as
broken by that call. Each now names both floors: floor_reason for #219's
pair analysis, hte_tested_segments for the breakdown's heterogeneity
flag, with a pointer to the module docstring and the simulations behind
the breakdown's floor. Docstrings only; the code is unchanged.

Part of #1969.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ
Signed-off-by: amarkanday <a.markanday@gmail.com>
@amarkanday
amarkanday merged commit b5e9248 into main Nov 15, 2026
70 checks passed
@amarkanday
amarkanday deleted the fix/1969-hte-test branch November 15, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant