Skip to content

fix(api): leave post-stratification out of the API document until it is available (#963) - #2081

Open
amarkanday wants to merge 2 commits into
mainfrom
fix/963-unpublish-post-stratification
Open

amarkanday wants to merge 2 commits into
mainfrom
fix/963-unpublish-post-stratification

Conversation

@amarkanday

Copy link
Copy Markdown
Contributor

Part of #963.

What and why

POST /api/v1/results/{experiment_id}/post-stratification answers 501 for every existing experiment, yet it is published as an operation. It is now left out of the API document with include_in_schema=False, until it is computed from an experiment's own data. The route is otherwise unchanged: 501 for an existing experiment, 404 for an unknown one, 422 for a malformed body, 401 and 403 as before. It stays beta, so there is no ! and no BREAKING CHANGE.

Changed:

  • backend/app/api/v1/endpoints/post_stratification.py: the keyword argument, one comment line and the module docstring line. Nothing else; git diff origin/main -- backend/app/services backend/app/schemas is empty.
  • make openapi: both snapshots and the dashboard fixture lose the path (below).
  • test_the_post_stratification_route_is_not_in_the_published_documents (regression) replaces the test that pinned the route as deprecated. It checks both snapshots and the live core document: the path and both components are absent, and POST /api/v1/results/{experiment_id}/fdr-correction is present as the positive control.
  • backend/tests/unit/schemas/test_post_stratification_schema.py: the request's documented defaults (metric_col is "metric_value", alpha is 0.05) and bounds (alpha 0.0 and 1.0 are refused), which the snapshot used to hold. The two claim units cite it.
  • Fuzz lists: the "Deprecated and unimplemented" exclusion is deleted (the loader refuses an exclusion for an operation that is not in the document); POST_STRAT leaves EXCLUDED (the constant stays); test_deprecated_operations_stay_fuzzed is now generic (every deprecated operation in the full snapshot is selected in the superuser and viewer passes; it asserts the set is not empty); comments and docstrings reworded; unreached.toml is untouched.
  • tests/acceptance/docs/floor.toml: the route is taken off the floor.
  • docs/api/stability.md and docs/statistics/post-stratification.md say the route is not in the API document, with their claim units.

Snapshot diff (structural, against origin/main)

== docs/api/openapi-v1.stable.json
 top-level keys: same = True | keys changed in value: []
 paths removed: ['/api/v1/results/{experiment_id}/post-stratification'] added: []
 operations removed: [('/api/v1/results/{experiment_id}/post-stratification', 'post')] added: []
 operations changed: []
 components/schemas removed: ['PostStratResultResponse', 'PostStratificationRequest'] added: [] changed: []
 components/(other sections) removed: [] added: [] changed: []
 tags removed: [] added: [] tags equal: True
== docs/api/openapi-v1.full.json
 (identical to the above)
== frontend/src/tests/fixtures/openapi.json
 top-level keys: same = True | keys changed in value: []
 paths removed: ['/api/v1/results/{experiment_id}/post-stratification'] added: []
 operations removed: [('/api/v1/results/{experiment_id}/post-stratification', 'post')] added: []
 operations changed: []
 tags removed: [] added: [] tags equal: True

The regression test, three results

  • (a) On unmodified main: fails, AssertionError: openapi-v1.stable.json / assert '/api/v1/results/{experiment_id}/post-stratification' not in {...}.
  • (b) With include_in_schema=False dropped (snapshots left as committed): fails, AssertionError: the live core document.
  • (c) On this branch: 1 passed.

The loader's refusal with the exclusion entry left in

On this tree, before the entry was deleted:

fuzz_check.ListError: exclusions.toml entry 26: POST /api/v1/results/{experiment_id}/post-stratification is not an operation of the API document

Verification

  • Schema defaults test: planted default="metric_value" -> default="metric": AssertionError: assert 'metric' == 'metric_value' (1 failed, 3 passed); restored, 4 passed.
  • Generic fuzz test: planted a deprecated-operation filter that selects nothing: AssertionError: no deprecated operation to protect; planted an exclusion of POST /api/v1/wizard/validate (viewer): assert [('POST /api/...e', 'viewer')] == []. Restored.
  • pytest backend/tests/unit/scripts/test_fuzz_check.py backend/tests/unit/docs/test_docs_floor.py: 176 passed.
  • python3 -I scripts/doc_claims.py check: exit 0, problems: 0. test_docs_only_gate.py: 47 passed.
  • ruff check and ruff format --check over backend/ modules/ scripts/: exit 0.
  • backend/tests/smoke (full tree): all pass except the known test_version_sources (15, this venv cannot build the sdist) and one test_wiring error (no database). Same set in a git archive HEAD copy with modules/ removed and no .git: 1045 passed.
  • backend/tests/unit: 25142 passed, 0 failed; the 226 errors needed a database and all pass against a scratch PostgreSQL 14 (the files that errored: 1299 and 1139 passed).
  • backend/tests/integration/api/test_post_strat_api.py against a scratch PostgreSQL 14: 20 passed (the 501, 404 and 422 answers).
  • git diff origin/main -- backend/app/services backend/app/schemas is empty.

NOT VERIFIED

  • The fuzz run (its effect on unreached.toml), the docs journeys, core-build and mkdocs in CI.
  • Third-party callers of the route are not visible from this repository.

🤖 Generated with Claude Code

https://claude.ai/code/session_011kqtSDC7sd5ue3m6LuwdXV

amarkanday and others added 2 commits November 18, 2026 21:59
…is available (#963)

POST /api/v1/results/{experiment_id}/post-stratification answers 501 for
every existing experiment, so it is published as an operation that never
succeeds. It is now left out of the API document: the decorator carries
include_in_schema=False, and the route is otherwise unchanged. It still
answers 501 for an existing experiment, 404 for an unknown one, 422 for a
malformed body, and 401 or 403 as before.

The two snapshots and the dashboard fixture lose the path, and the two
components only it used (PostStratificationRequest and
PostStratResultResponse). Nothing else in them changes.

Around that:
- the regression test that pinned the route as deprecated now pins it as
  absent from both snapshots and from the live document, with
  fdr-correction on the same router as the positive control;
- the request's two documented defaults (metric_col, alpha) are held by a
  unit test on the schema instead of by the snapshot;
- the fuzz exclusion for the route is deleted, as the loader refuses an
  exclusion for an operation that is not in the document, and
  test_deprecated_operations_stay_fuzzed now covers every deprecated
  operation in the full snapshot;
- the route is taken off the docs floor, and stability.md and
  post-stratification.md say it is not in the API document.

The route stays beta; no BREAKING CHANGE. Part of #963.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kqtSDC7sd5ue3m6LuwdXV
Signed-off-by: amarkanday <a.markanday@gmail.com>
Merges main at 4c45fcd.

No conflicts; `make openapi` regenerated the snapshots and they differ from
main only by this branch's change.

Gates run on the merged tree: the branch's own test files, backend/tests/smoke,
backend/tests/unit, scripts/doc_claims.py check (0 problems), ruff check and
ruff format --check over backend/ modules/ scripts/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NpxPyav9uBxNuiXsDvD8qn
Signed-off-by: amarkanday <a.markanday@gmail.com>
@amarkanday amarkanday added the bug Something isn't working label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant