Skip to content

chore(main): release 0.48.0 - #1979

Open
github-actions[bot] wants to merge 1 commit into
mainfrom
release-please--branches--main--components--experimently-platform
Open

github-actions[bot] wants to merge 1 commit into
mainfrom
release-please--branches--main--components--experimently-platform

Conversation

@github-actions

@github-actions github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

🤖 I have created a release beep boop

0.48.0 (2026-11-21)

⚠ BREAKING CHANGES

  • auth: POST /api/v1/users/me/password answers 200 with a new access token (the login response) for an account that signs in with its password alone. It answers 204, as before, for an account with two-step sign-in on or required for its role. Every token issued before the change stops working, including the one that sent it. Use the returned token, or sign in again after a 204.
  • api: The six /api/v1/wizard operations (POST and GET /api/v1/wizard/drafts, GET /api/v1/wizard/drafts/{draft_id}, PUT /api/v1/wizard/drafts/{draft_id}/step, POST /api/v1/wizard/drafts/{draft_id}/submit and POST /api/v1/wizard/validate), deprecated since 0.15.0, are removed and answer 404. Create an experiment with POST /api/v1/experiments/, as the dashboard's guided setup does, and size it with GET /api/v1/experiments/analysis/sample-size.

Features

  • api: remove the deprecated wizard endpoints (#537) (#2082) (4f611b5)
  • auth: ask local accounts for a second step at sign-in (#1992) (ddf5395)
  • auth: changing a password signs out the user's other sessions (#501) (#2083) (37c90b6)
  • auth: require two-step sign-in for chosen roles (#2010) (373744b)
  • auth: setting up two-step sign-in asks for your current password (#2008) (8471781)
  • bandit: a bandit's weight history in the API (beta) (#2000) (5023032)
  • dashboard: a bandit's weight history on the experiment page (#2016) (fb67dac)
  • dashboard: an admin sets a bandit's weights by hand (#2002) (8d0e2d8)
  • dashboard: create a bandit experiment from the new-experiment form (#1999) (c5c9acc)
  • dashboard: create a split URL experiment from the new-experiment form (full profile) (#2077) (487520c)
  • dashboard: CUPED-adjusted results on the results page (beta) (#1998) (4c53619)
  • dashboard: list the API keys that stop working when you deactivate a user (#2078) (1e0410d)
  • dashboard: results for one saved segment's members (#2006) (21c3935)
  • dashboard: the interactions page, with the scan and the pair test (beta) (#2003) (10d5a19)
  • dashboard: the other confirm panels use the in-page confirm component (#2073) (7105479)
  • dashboard: two-step sign-in on the account page and at sign-in (#1995) (8e95b2a)
  • db: add the identity_aliases table (schema only) (#2080) (723cc97)
  • dotnet-sdk: track sends an event id, so a retried event is stored once (#1987) (9f93c63)
  • infra: the CDK app can require two-step sign-in for chosen roles (#2060) (caaafcf)
  • results: results for one saved segment's members (beta route) (#1997) (0b7e3cc)
  • sdk: a browser script applies visual changes before the page shows (#1545) (#2088) (38fed2d)
  • sdk: the React and React Native SDKs send an event id, so a retried event is stored once (#1985) (9baa90c)
  • site: controlvar.com gets a sitemap, a canonical address and a descriptive title (#2050) (537a391)
  • tracking: resolve identity links on assign and events (no route writes one yet) (#2085) (7d257f2)
  • warehouse: enable Amazon Redshift (live run wl-redshift-20261118T011715Z-6b77b53f) (#2053) (b9887a6)

Bug Fixes

  • api: leave post-stratification out of the API document until it is available (#963) (#2081) (86721d7)
  • audit: a feature flag change is saved with its audit entries or not at all (#2079) (f09a300)
  • audit: a rollout schedule change is saved with its audit entry or not at all (#2099) (01703b8)
  • audit: saving the safety settings writes an audit entry (#1982) (ac14a22)
  • auth: Cognito calls time out after a few seconds (#2072) (da66e16)
  • bandit: read converting users from PostgreSQL first, not DynamoDB conversion events (#2076) (ddd479d)
  • dashboard: a late rollback result no longer shows in another flag's dialog (#1993) (f1c6a15)
  • dashboard: every sign-in page applies the same check to the next path (#2052) (108710f)
  • dashboard: no sideways scroll on a phone, safety-check units, and four wording fixes (#1988) (7d84469)
  • db: database errors are logged without the statement's parameter values (#2036) (d0fdf35)
  • deps: postcss-selector-parser 6.1.5 in the dashboard, including under @tailwindcss/typography (#1984) (3859b1e)
  • deps: postcss-selector-parser 6.1.5 in the StreamPulse demo (#1983) (9cc7db1)
  • infra: cdk destroy of the Fargate stack works in one run whatever target group is live (#2071) (dae7332)
  • infra: remove the CloudWatch dashboards that graph metrics nothing publishes (#2005) (0e7e4ce)
  • notifications: email sends time out instead of waiting for ever (#2034) (27151c6)
  • results: the heterogeneous-effects banner no longer fires when the effect is the same in every segment (engine 1.12.0) (#1981) (b5e9248)
  • sdk: the Android SDK sends the user's attributes when it evaluates a flag (#2074) (9448fb3)
  • sdk: the Edge SDK sends the user's attributes when it evaluates a flag (#2043) (15291cf)
  • sdk: the Elixir SDK sends the user's attributes when it evaluates a flag (#2044) (df64f75)
  • sdk: the Flutter SDK sends the user's attributes when it evaluates a flag (#2062) (b54fc0a)
  • sdk: the iOS SDK sends the user's attributes when it evaluates a flag (#2045) (10dcca7)
  • sdk: the PHP SDK sends the user's attributes when it evaluates a flag (#2037) (f280816)
  • sdk: the Ruby SDK sends the user's attributes when it evaluates a flag (#2035) (5370cca)
  • site: the homepage and docs hub say which features are beta (#2049) (1bf9613)
  • wizard: limits on a draft's fields and on how many drafts one user keeps (#2033) (c894353)
  • workspaces: a workspace slug must match the whole pattern (#2031) (4e8e902)

Documentation

  • claims: the bandit page's three jest targets name the whole it.each title (#2059) (74666ab)
  • link previews, logo and favicon on the docs site (#2075) (82b813e)
  • mark CUPED, sequential testing and the SSO routes as beta (#2032) (ba87d64)
  • page descriptions and robots.txt for search (#2061) (e09dfac)
  • roadmap: name Slack accurately, and point contributors to where to start (#2089) (a20452e)
  • sdk: a key-less track() does not report entries a 200 answer refuses (#1994) (34f0fc3)
  • the capabilities page lists what shipped through 0.47.0 (#1980) (31a8c19)
  • the limitations page names the SDKs whose error reports reach safety rollback (#2051) (4f82358)
  • the roadmap drops five finished items, notes 0.47.0 first steps (#1977) (517080c)

This PR was generated with Release Please. See documentation.

@github-actions
github-actions Bot requested a review from amarkanday as a code owner November 15, 2026 06:25
@github-actions
github-actions Bot force-pushed the release-please--branches--main--components--experimently-platform branch from 42617a4 to a385401 Compare November 15, 2026 09:04
amarkanday added a commit that referenced this pull request Nov 15, 2026
…he effect is the same in every segment (engine 1.12.0) (#1981)

the heterogeneous-effects banner no longer fires when the effect is the
same in every segment. It now needs a test of equal percentage-point
differences to reject at the results' significance level; in the
reported case (4 segments of 90 users per arm, the same lift in each) it
fired on about 60% of experiments, and now on about 5%.
`has_heterogeneous_effects` is true less often. Statistics engine
1.12.0.

Part of #1969. Built to the reviewed plan and its binding conditions,
PEU3-1..14, EMU3-1..8, EMCF-1 and EMCF-2, each listed below with its
evidence.

## What changed

- **`DimensionalAnalysisService.detect_hte(segment_results, *, alpha)`**
now runs #219's `interaction_analysis.interaction_test`: the additive
model (one baseline per segment, one common difference) fitted by
binomial maximum likelihood, and Pearson's X² against it on J - 1 df. It
tests the first treatment against the control on the segments that pass
the floor, and returns `bool(p < alpha)`.
- `alpha` is keyword-only and required. A value that is not finite and
strictly between 0 and 1 raises `ValueError`.
- The converters-only `chi2_contingency` at a fixed 0.05 is deleted,
along with the coefficient-of-variation / opposite-sign heuristic and
the `except Exception: logger.debug`.
- `InteractionFitError` propagates. The whole `/results?breakdown=`
request and the whole `/balanced-groups` request (estimate included)
then answer 500 through `unexpected_failure`.
- **Floor** (`hte_tested_segments`, with `HTE_MIN_EXPECTED_PER_CELL =
10` shared with the tests' replica):
- The candidates are the segments with at least one user in both arms,
and the pooled rate is taken over them.
- A segment is tested when, in both arms, `n * conv >= 10 * users` and
`n * (users - conv) >= 10 * users`. The comparison is made in integers.
  - With fewer than two tested segments the answer is False.
- **The treatment** is the first non-control variant of the first
segment. The route lists the control first and then the variants by
name, so it is the non-control variant whose name sorts first. It is
looked up by id in every segment.
- **`results.py`**:
  - `detect_hte(segment_results, alpha=base_alpha)`.
  - `_compute_dimensional_breakdown`'s `base_alpha` has no default.
  - The warning is PEU3-10's text.
- **Schema description** of `has_heterogeneous_effects`: PEU3-8's text.
`make openapi` regenerated both snapshots.
- **Docs**:
- `docs/api/dimensional-analysis.md` HTE Detection is PEU3-9's text, and
its five claims are bound in PEU3-9's order.
  - `docs/api/stability.md` gains the 1.12.0 row.
- `interaction_analysis.py` gains a docstring paragraph (no code change;
#219's golden tests do not move).
- **Engine 1.12.0** by the bump template:
- `ENGINE_VERSION` moves, and 1.12.0 is appended to
`RELEASED_ENGINE_VERSIONS`, `FROZEN_DIGESTS` and `FROZEN_VERSIONS`.
- `ENGINE_FINGERPRINTS["1.12.0"]` is a new hash. The fingerprint's call
gains `alpha=0.05`, and its dataset gains the walk data
(`breakdown_walk`).
- The other six pin tables copy their 1.11.0 entry. No frozen entry is
edited.
  - The integration literal moves to `"1.12.0"`.
- **Tests**:
- New:
`backend/tests/unit/stats_validation/test_golden_hte_false_positive_rate.py`
(18 tests) and `backend/tests/unit/services/test_hte_detection.py` (33).
  - `test_collection_count` gains the new module.
  - The integration spy passes `**kwargs` through.

## Before and after (RandomState(1969), 2,000 tables per shape, control
drawn before treatment)

| shape | 0.47.0 rule | now, alpha .05 |
|---|---|---|
| RW-1 null (4 x 90/arm, 10% -> 18% everywhere) | 1216 | 108 (24 at .01,
219 at .10) |
| A/A 4 x 1000/arm at 10% | 1940 | 120 |
| unequal split (600/200 alternating), 10% -> 15% everywhere | 2000 |
124 |
| strong heterogeneity (4 x 90, 30/30/5/5% vs 10%) | 2000 | 1967 |
| moderate heterogeneity (4 x 500, 18/18/10/10% vs 10%) | 1983 | 1835
(theory 0.921, band 1806-1879) |

- The RW-1 walk data goes from flagged to not flagged. The `unknown`
segment is below the floor; the other three give X² 2.97, df 2, p 0.227.
- The engine fingerprint's own breakdown goes from flagged (by lifts of
both signs) to not flagged (p 0.061).

## Each condition and its evidence

- **PEU3-1, as replaced by EMCF-1 (V8):** RandomState(1969), control
drawn first.
- `test_an_ordinary_null_is_flagged_at_most_alpha` checks each of the
four ordinary shapes for `flagged <= 133`, `tested >= 1900` and the
exact pins:

    | shape | flagged | tested |
    |---|---|---|
    | tiny 300/90/30/10 | 117 | 1988 |
    | 500/500/5/2 | 99 | 2000 |
    | 4 x 500 at 2% -> 4% | 108 | 2000 |
    | unequal 600/200 | 124 | 2000 |

- `test_a_null_in_the_liberal_corner_stays_below_its_bound` checks the
two corner shapes for `flagged <= 147`, `tested == 2000` and the exact
pins: 22 x 50 at 50% flags **121**, 21 x 50 at 30% -> 38% flags **139**.
- `test_a_shape_below_the_floor_is_never_tested`: 21 x 50 at 10% -> 18%
and 4 x 90 at 2% -> 4% are tested 0 times.
  - Plant: floor 5 in the constant (plant table).
- **PEU3-2 (+ EMCF-1's 21 x 50 sentence):** re-measured below.
- **PEU3-3:** `test_the_replica_agrees_with_detect_hte`.
- It sends the first 200 tables of RW-1's null, the unequal null, V7 and
the floor-binding null (300/90/60/40, 10% -> 18%) through
`compute_segment_results` and the real `detect_hte` at .01, .05 and .10.
- Result: 2,400 decisions, **0 disagreements**, 30.5 s locally (7.2-8.0
s per shape) with `interaction_test` memoised per table inside the test.
  - The replica calls the production `hte_tested_segments`.
  - All seven plants fail it (plant table).
- **PEU3-4:** a plant inside `detect_hte` never moves the replica's
pins. The plant table shows the cross-check, V4 and V16 catching those
plants. The pins catch the shared-constant floor plants, which move the
replica too.
- **PEU3-5 (floor):**
  - The floor is computed in integers (`hte_tested_segments`).
- V16 is `test_a_segment_at_the_floor_is_tested` (True; p 4.4e-5) and
`test_a_segment_one_conversion_below_the_floor_is_not_tested` (False; p
1.0), both through `_compute_dimensional_breakdown`.
- `test_the_floor_is_computed_in_integers` pins an exact boundary where
`77 * (100 / 770)` is 9.999999999999998 in floating point.
  - Floor 5 and floor 15 are planted in both places (plant table).
- **PEU3-6:** R2 is the cross-check (`@pytest.mark.regression`). On
v0.47.0 it fails by TypeError on `alpha=`. The rule-level evidence is
315 of 600 (below).
- **PEU3-7:** `InteractionFitError` propagates.
`test_a_failed_fit_fails_the_whole_breakdown` monkeypatches
`interaction_analysis.interaction_test` to raise it, and
`_compute_dimensional_breakdown` raises. The plant that catches it and
returns False fails that test. The x > n change is not made (declined by
the EM).
- **PEU3-8:** the schema description is PEU3-8's text. Both snapshots
equal it character for character (checked by script against the review
file).
- **PEU3-9:** `dimensional-analysis.md` HTE Detection is PEU3-9's text,
character for character. Its five claims are bound in order:
  1. V14: `test_the_first_treatment_by_name_is_the_one_tested`.
2. V4 + the cross-check:
`test_the_breakdown_tests_at_the_alpha_it_is_given`,
`test_the_alpha_fixtures_p_values` and
`test_the_replica_agrees_with_detect_hte`.
  3. V16: both floor tests.
4. V10 + V16's second fixture:
`test_too_little_to_test_is_false_without_a_warning` and
`test_a_segment_one_conversion_below_the_floor_is_not_tested`.
  5. V5: `test_each_route_hands_its_confidence_level_down_as_the_alpha`.

V13's grep over `docs/` (Markdown, JSON, TOML) finds 0 hits each for
"varies widely", "positive in one segment", "chi-squared test detects"
and "Cochran's Q". On main the first is in the page and the third in
both snapshots. Plant: editing the doc alone gives `doc_claims.py check`
2 problems and `test_the_registries_hold` fails.
- **PEU3-10:** the warning is PEU3-10's text, character for character.
Its `'<dimension>'` is the requested dimension.
- **PEU3-11:** this body opens with it, after "fix(results): ".
- **PEU3-12:** `_compute_dimensional_breakdown(..., base_alpha: float,
...)` has no default. Both routes and `test_breakdown_reads.py` pass it.
- **PEU3-13:** `test_dimensional_analysis_api.py`'s copy of the warning
is the new text. V4's True cases assert `hte_warning ==` the sentence
with the dimension, written out in the test.
- **PEU3-14:** the IRREVERSIBLE section below.
- **EMU3-1:** this list.
- **EMU3-2:** the title is exactly the EMU3-2 line; the body opens with
PEU3-11 after "fix(results): ".
- **EMU3-3, as replaced by EMCF-2 (V14):**
`test_the_first_treatment_by_name_is_the_one_tested` runs through
`_compute_dimensional_breakdown` and the `_Readers` harness.
- Setup: control; T1 = id `00000000-…-0001`, name "zeta"; T2 = id
`ffffffff-…-0003`, name "alpha". The pairs are fixed, and the route
lists `control, alpha, zeta`.
- Case A (only T1's difference varies): False. Case B (the varying
counts moved to T2, T1 homogeneous): True.
- Plant (the treatment picked by smallest id): **Case A returns True and
Case B returns False; both fail** (plant table). This is the run EMCF-2
asked for.
- **EMU3-4:** the regression proof and the plant table, below.
- **EMU3-5:**
  - No rate appears in the docs, the schema, the warning or the claims.
- The `interaction_analysis.py` docstring names the liberal corner and
quotes only 108 (V1), 121 and 139 (V8).
  - The PR's re-measure of PEU3-2 is below.
- **EMU3-6:** `stability.md` gains a 1.12.0 row in the 1.11.0 row's
form.
- Operations: `GET /results/{id}` with `breakdown`, and
`/balanced-groups`. No other operation serves the breakdown:
`_compute_dimensional_breakdown` has only these two callers, and
`/experiments/{id}/results` sets `breakdown=None`.
- What changes: `has_heterogeneous_effects` is true less often, and
`hte_warning` has new text. No answer cached before the upgrade is
served.
- The OpenAPI document changes only in the `engine_version` defaults and
the field's description, in both snapshots. Every other result is
unchanged.
- Its claim binds to V1, V4, V11 (fingerprint, frozen), V12 (both
snapshot tests) and the cache-key test. There is no `exempt:baseline`.
- **EMU3-7 (at merge):**
- This head is based on current main (517080c, a docs-only roadmap
commit on top of v0.47.0), and `make openapi` was run on it.
- No open PR touches the engine. The release PR #1979 (0.47.1) touches
both snapshots' version string, so whichever merges second re-runs `make
openapi`.
  - The ADMIT-0480 line is the coordinator's.
- **EMU3-8:** the review request is the coordinator's.
- **EMCF-1 / EMCF-2:** as PEU3-1 and EMU3-3 above.
- **Review round 1 nit, fixed in c504fe0:** `interaction_test`'s
precondition and `InteractionFitError`'s docstring said "the floor",
meaning `floor_reason`, which `detect_hte`'s call did not meet. They now
name both callers' floors: `floor_reason` for #219's pair analysis,
`hte_tested_segments` for the breakdown. Docstrings only, and the AST is
unchanged. All tests that import the module pass: 165 passed, 2 skipped
(the two SLSQP cases already skipped).

## Regression proof (EMU3-4, PEU3-6)

- **Setup:** a `git archive` copy of this head with
`dimensional_analysis_service.py`, `results.py` and `stats_engine.py`
restored from v0.47.0 (`cmp` against `git show v0.47.0:<path>`:
identical).
- **Collected:** `pytest --collect-only -q -m regression` on the two new
files: 8 tests.
- **Run:** 8 failed, 0 passed.
  - By assertion (`assert True is False`), 4:
    - `test_the_walk_data_shows_no_heterogeneity` (R1);
- `test_equal_differences_on_an_unequal_split_are_not_flagged` (R4 =
V9);
    - `test_a_segment_without_control_users_is_not_tested` (R3);
- `test_the_breakdown_tests_at_the_alpha_it_is_given[152-0.01-False]`
(R5).
- By `TypeError: DimensionalAnalysisService.detect_hte() got an
unexpected keyword argument 'alpha'`, 4:
`test_the_replica_agrees_with_detect_hte[rw1_null|unequal_split|moderate_heterogeneity|floor_binding]`
(R2, PEU3-6).
- **The real 0.47.0 rule, same copy:** the real 0.47.0 `detect_hte`,
through `compute_segment_results`, on V1's 2,000 draws gives **1216**.
V2's pin (`RW1_NULL_0_47_0_PIN`, the test's frozen copy of the 0.47.0
rule) is **1216**.
- **Rule-level evidence (PEU3-6):** under the new signature, the 0.47.0
body disagrees with the replica on **315 of 600** decisions (the first
200 RW-1 tables at .01, .05 and .10).

## Plant table (EMU3-4)

Each plant was made in its own `git archive` copy of this head (no
`.git`). The gates run were:
- the two new modules;
- `test_dimensional_analysis_service.py`, `test_engine_fingerprint.py`,
`test_engine_versions_frozen.py` and `test_dimensional_analysis_api.py`;
- plus the gate the row names.

The unplanted baseline in the same harness (with the snapshot and
collection-count tests): 120 passed.

Cross-check counts are disagreements out of 600 decisions for each
shape: RW-1 / unequal / V7 / floor-binding.

| plant | where | gate that fired | count |
|---|---|---|---|
| alpha fixed at 0.05 | `detect_hte` | cross-check ×4; V4 (152 at .01,
144 at .10) | 16 / 16 / 33 / 26 |
| OR the 0.47.0 rule | `detect_hte` | cross-check ×4; R1, R3, R4, R5; V4
144 at .05; V16 below floor; V10 one-empty-control; PEU3-7 test;
fingerprint | 315 / 568 / 66 / 349 |
| OR the count chi-square | `detect_hte` | cross-check ×4; R4 (V9); V10
one-empty-control | 10 / 568 / 3 / 11 |
| df = K | `detect_hte` | cross-check ×4; V4 144 at .10 | 13 / 12 / 38 /
33 |
| alpha/K | `detect_hte` | cross-check ×4; V4 152 at .05, 144 at .10 |
22 / 21 / 72 / 35 |
| floor 5 | `detect_hte` only | cross-check (RW-1, floor-binding); V16
second fixture (True) | 1 / 0 / 0 / 39 |
| floor 15 | `detect_hte` only | cross-check (RW-1, floor-binding); V16
first fixture (False); V10 zero-variance (False) | 30 / 0 / 0 / 53 |
| floor 5 | shared constant | exact pins: V1 .05 110≠108, .10 222≠219;
V6 1993≠1967; V8 tiny tested 2000≠1988; 21x50 at 10->18 tested 2000≠0.
V16 second fixture (True); integer-floor test | (cross-check agrees:
replica and real path share it) |
| floor 15 | shared constant | exact pins and bands: V1 0 / 2 / 4 of
2000 (below every band); V6 18 < 1900; V8 tiny 26 and 4x500 1007 tested
< 1900. V16 first fixture (False); integer-floor test; V10 zero-variance
| (as above) |
| call site passes 0.05 | `results.py` | V4: 152 at .01 True, 144 at .10
False | 2 tests |
| call site passes `adjusted_alpha` | `results.py` | V4: 152 at .05
False (.0125 < p .0214), 144 at .10 False | 2 tests |
| call site omits alpha | `results.py` | every real-path test:
TypeError, missing required keyword-only `alpha` | 12 tests |
| `/results` passes `base_alpha=0.05` | `results.py` | V5: `[0.05] !=
[0.01]` | 1 test |
| `/balanced-groups` passes `base_alpha=0.05` | `results.py` | V5:
`[0.01, 0.05] != [0.01, 0.10]` | 1 test |
| treatment picked by smallest id | `detect_hte` | V14: Case A returns
True, Case B returns False (EMCF-2) | 2 tests |
| no fewer-than-two guard | `detect_hte` | V10: `interaction_test`
called with one or no segment | 4 tests |
| returns `numpy.bool_` | `detect_hte` | cross-check (`type is bool`)
×4; V10 zero-variance; TestDetectHTE `is True/False`, `isinstance(bool)`
| 9 tests |
| `InteractionFitError` caught, False returned | `detect_hte` | PEU3-7
test: DID NOT RAISE | 1 test |
| fix without the bump (ENGINE_VERSION 1.11.0) | `stats_engine.py` |
fingerprint ("output changed under ENGINE_VERSION 1.11.0"); running
engine != last frozen | 2 tests |
| 1.11.0 fingerprint edited | pin table |
`test_a_frozen_versions_pins_are_unchanged[1.11.0]` | 1 test |
| snapshots not regenerated | OpenAPI |
`test_core_api_matches_stable_snapshot`,
`test_full_api_matches_full_snapshot`, balanced-groups schema test | 3
tests |
| a skip marker on the new module | gate module |
`test_no_gate_here_can_skip` | 1 test |
| doc edited without its claim | docs | `doc_claims.py check` problems
2; `test_the_registries_hold` | 1 test |
| harness without `is_control` (S1) | gate module | V2 structure check;
cross-check ×4 | 30 / 32 / 528 / 53 |

- One plant fired nothing on the first run: removing the fewer-than-two
guard. With fewer than two tested segments the test has no degrees of
freedom, scipy returns a nan p-value, and `nan < alpha` is False, so the
answer stayed right by accident.
- The V10 test now also requires that `interaction_test` is never called
there.
  - The second run above shows it firing (4 tests).

## Re-measure of PEU3-2's calibration (EMU3-5)

The replica is the one the cross-check holds to the real path. 20,000
tables, seed 2026; each figure is the share flagged among the tables
tested.

| shape | tested | alpha .05 | alpha .10 |
|---|---|---|---|
| RW-1 null | 99% | 4.94% (z -0.4) | 10.08% |
| A/A 4 x 1000 at 10% | 100% | 5.13% (z +0.9) | 10.07% |
| unequal 600/200 alternating, 10% -> 15% | 100% | 5.16% (z +1.0) |
9.95% |
| 4 x 500 at 2% -> 4% | 100% | 4.92% (z -0.5) | 10.11% |
| 90/10 split (4 x 900/100), A/A at 10% | 50% | 4.67% (z -1.5) | 9.71% |
| 95/5 split (4 x 950/50), A/A at 20% | 50% | 4.98% (z -0.1) | 10.08% |
| 22 x 50/arm at 50% | 100% | 5.67% (z +4.4) | 11.02% (z +4.8) |
| 22 x 30/arm at 50% | 100% | 5.90% (z +5.8) | 11.33% (z +6.3) |
| 21 x 50/arm at 30% -> 38% | 100% | 5.50% (z +3.2) | 10.66% (z +3.1) |

- Power at 4 x 90/arm, +16 points in two of four segments: 75.2% of all
tables at .05 (75.8% of those tested). PEU3-2 says 74%.
- PEU3-2 did not give the shapes for its 95/5 and 90/10 rows. The two
above are mine, both about half untested, which matches its note.
- Every other figure matches PEU3-2 and EMCF-1's addition: RW-1 4.9%;
A/A 5.1-5.2%; unequal 5.0-5.2%; 4 x 500 at 2% -> 4% 4.9%; 22 x 50 at 50%
5.7% / 11.0%, z +4.4; 22 x 30 at 50% 5.9% / 11.3%; 21 x 50 at 30% -> 38%
5.5%, z +3.2.

## Local runs (venv311: numpy 1.26.4, scipy 1.17.1; no database, no
Redis)

The whole-directory runs were made on this commit before its last amend.
The amend changed only two test files: the cross-check's failure
message, and V10's check that the test is not run. Both files and
stats_validation were re-run after it.

- `backend/tests/unit/stats_validation`: 130 passed, 0 skipped
(`check_junit_skips.py` clean). The new module has 18 tests and takes 38
s locally, 30.5 s of it the cross-check. `test_collection_count` passes.
- `backend/tests/unit/services`, the whole directory:
  - 3638 passed.
- 3 skipped, all already skipped before this change: two SLSQP cases in
`test_interaction_analysis.py` and a benchmark.
- 61 errors, every one a PostgreSQL connection at setup
(`POSTGRES_PORT=1` on purpose), in `test_audit_service*.py` and
`test_experiment_scheduling.py`. This PR touches neither; CI runs them.
  - Among them:

    | test file | passed |
    |---|---|
    | `test_hte_detection.py` | 33 |
    | `test_dimensional_analysis_service.py` | 23 |
    | `test_breakdown_reads.py` | 10 |
    | `test_engine_fingerprint.py` | 2 |
    | `test_engine_versions_frozen.py` | 17 |
    | `test_sufficient_stats_fingerprint.py` | 12 |
    | `test_lower_is_better_fingerprint.py` | 21 |
    | `test_results_correction_pins.py` | 22 |
    | `test_revenue_per_user_routing.py` | 4 |
    | `test_value_metrics_per_user.py` | 20 |
    | `test_binomial_metric_result_characterisation.py` | 7 |

- `test_dimensional_analysis_api.py`: 12 passed.
- `backend/tests/smoke/test_openapi_snapshot.py`: 13 passed.
- `test_doc_claims.py`: 172 passed, and `python scripts/doc_claims.py
check` reports problems 0.
- `test_docs_only_gate.py`: 46 passed.
`test_openapi_dump_and_stability.py`: 6 passed.
- A core-shaped copy (`git archive`, no `.git`, `modules/` deleted):
stats_validation plus the changed breakdown, engine and API tests give
239 passed, and `doc_claims.py check` reports problems 0.
- `ruff check` and `ruff format --check` over `backend/ modules/
scripts/` are clean.
- `check_dco.py`, `leak_guard.py --range` and `gitleaks git` are clean
on the final commit.

## OpenAPI

`make openapi` changes, in each of `openapi-v1.full.json` and
`openapi-v1.stable.json`:
- the four `engine_version` defaults, 1.11.0 -> 1.12.0;
- the description of `has_heterogeneous_effects`, which is PEU3-8's
text, checked character for character against the gate.

Nothing else changes. `frontend/src/tests/fixtures/openapi.json` does
not change.

## IRREVERSIBLE

Nothing. Undoing it after release is a new engine-changing PR (1.13.0),
per the rule in test_engine_versions_frozen.py ("A revert of an
engine-changing PR bumps again").

## NOT VERIFIED

- **PostgreSQL and integration tests:** CI is the first run of:
  - the integration spy's `**kwargs`;
  - the balanced-groups literal `"1.12.0"`;
  - the 61 database-backed unit tests above.
- **The route's 500:** the propagation of `InteractionFitError` is
tested at `_compute_dimensional_breakdown`. That both route handlers
turn it into a 500 is read from source (`except Exception` ->
`unexpected_failure`), not run.
- **The Redis cache miss after the bump** is not run against Redis. The
key carries `ENGINE_VERSION`, and the cache-key test passes.
- **Not run locally:** `lint-imports`, `reuse lint`, eslint/tsc,
hadolint, actionlint and `mkdocs build --strict`. The one new import is
a core service importing a core service.
- **The new fingerprint** was computed with numpy 1.26.4 and scipy
1.17.1, the pinned versions. CI is its first run on Linux; outputs are
rounded to 10 significant digits.
- **CI time** of the new module was measured only locally, and is not
asserted.
- **PEU3-2's shapes:** its 95/5 and 90/10 rows are re-measured on shapes
of my choosing (above).
- **The dashboard** was not run. No frontend file changes; the banner
still renders on `has_heterogeneous_effects && hte_warning`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011kqtSCWnWY2zHdCgKyTKiJ

---------

Signed-off-by: amarkanday <a.markanday@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions
github-actions Bot force-pushed the release-please--branches--main--components--experimently-platform branch 3 times, most recently from 6ea5e2d to c2f7c2d Compare November 15, 2026 17:36
@github-actions github-actions Bot changed the title chore(main): release 0.47.1 chore(main): release 0.48.0 Nov 15, 2026
@github-actions
github-actions Bot force-pushed the release-please--branches--main--components--experimently-platform branch 21 times, most recently from 2a7ad12 to bdff4a5 Compare November 18, 2026 12:02
@github-actions
github-actions Bot force-pushed the release-please--branches--main--components--experimently-platform branch 28 times, most recently from fe71017 to 42a6aa8 Compare November 21, 2026 00:07
@github-actions
github-actions Bot force-pushed the release-please--branches--main--components--experimently-platform branch from 42a6aa8 to 04dd337 Compare November 21, 2026 01:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants