Skip to content

chore(AIC-3500): commit an API report for every public module - #195

Merged
jeffdupont merged 4 commits into
mainfrom
jdupont/AIC-3500/api-report
Oct 22, 2026
Merged

jeffdupont merged 4 commits into
mainfrom
jdupont/AIC-3500/api-report

Conversation

@jeffdupont

@jeffdupont jeffdupont commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Why

Nothing in CI notices when a public function's signature changes: a new required parameter, a removed parameter or a changed return type all pass today. That's a breaking change for callers, and after 1.0 it has to be deliberate.

What

  • A committed API report for every public module, at packages/<package>/api-report/<module>.txt. That's every package root (10 reports, including opencode-agents), plus each module of launchdarkly_ai_server.experimental. The umbrella launchdarkly_ai_python re-exports the core package, so the core report covers it.
  • Each report lists every name in __all__ with its full signature, one line per name. A class's constructor and public members are indented below it. Tags show async, classmethod, property and decorators. A re-export from another distribution is recorded as name = target [re-export].
  • scripts/api_report.py generates the reports with griffe (griffelib, now an explicit dev dependency; it was already in the lockfile through openai-agents).
  • tests/test_api_report.py fails while any report is out of date, so make test and CI catch a signature change whose report wasn't regenerated. It also checks the generator against a fixture package: functions, async, dataclasses, properties, classmethods, private members, external re-exports, a missing __all__, and an undefined name in __all__.
  • make api-report writes the reports; make api-report-check checks them.
  • .github/CODEOWNERS gives packages/*/api-report/ to @launchdarkly/team-ai-agents, so a public API change needs their approval of the report diff.
  • AGENTS.md documents the workflow.

Tested

  • uv run pytest: 2556 passed, 14 skipped. ruff check and ruff format --check are clean.
  • A signature change fails the check. Adding a required strict: bool to inspect_config turns make api-report-check red, with the exact one-line diff and "run make api-report and commit the result". Reverting it turns it green again.

Not in this PR

These come in follow-up PRs:

  • detecting breaking changes against the base branch and the last release, declared with ! in the PR title
  • the usage-ID table as a data file
  • the README experimental-features block
  • the dir() ⊆ __all__ sweep for every module

🤖 Generated with Claude Code


Note

Overview
Introduces committed API reports as the public contract for every package module (and experimental submodules): each __all__ name and signature lives at packages/<package>/api-report/<module>.txt, generated by scripts/api_report.py (griffe) via make api-report, with make api-report-check and tests/test_api_report.py failing when reports are stale.

Adds griffelib as a dev dependency, CODEOWNERS review on packages/*/api-report/ for @launchdarkly/team-ai-agents, and AGENTS.md workflow docs. Initial report snapshots are committed for all handler packages plus launchdarkly_ai_server (including experimental skills/evaluations); the umbrella launchdarkly_ai_python package is skipped because it re-exports the core module.

Reviewed by Cursor Bugbot for commit cd40590. Bugbot is set up for automated code reviews on this repo. Configure here.

@jeffdupont
jeffdupont marked this pull request as ready for review October 20, 2026 22:24
@jeffdupont
jeffdupont requested a review from a team October 20, 2026 22:24

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread scripts/api_report.py
Comment thread scripts/api_report.py
Each public module (every package root, and each module of the core
package's experimental subpackage) now has a committed report at
packages/<package>/api-report/<module>.txt, generated by
scripts/api_report.py with griffe. A report lists each name in __all__ with
its full signature, and a class's public members indented below it.

tests/test_api_report.py fails while any report is out of date, so `make
test` and CI catch a public name or signature change whose report wasn't
regenerated. It also checks the generator against a fixture package.
`make api-report` writes the reports, `make api-report-check` checks them,
and .github/CODEOWNERS gives them to team-ai-agents. griffelib is now an
explicit dev dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeffdupont
jeffdupont force-pushed the jdupont/AIC-3500/api-report branch from 13e38fb to 741f773 Compare October 20, 2026 22:33
… reports

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0d155e6. Configure here.

Comment thread scripts/api_report.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	pyproject.toml
@jeffdupont
jeffdupont merged commit e174e84 into main Oct 22, 2026
7 checks passed
@jeffdupont
jeffdupont deleted the jdupont/AIC-3500/api-report branch October 22, 2026 22:56
jeffdupont added a commit that referenced this pull request Oct 22, 2026
…ic API (#196)

**Stacked on #195** (the API reports).

## Why

The API reports make every public change visible, but they don't stop a
breaking change from shipping in a minor or patch release. After 1.0 a
breaking change has to be deliberate: declared, and released as a major
version.

## What

`scripts/api_breaking.py` uses griffe to compare the names in each
public module's `__all__` with another git ref. The new **API
compatibility** workflow runs it on every PR, and re-runs it when the
title or labels change.

- **On a PR (`pr`):** compares with the PR's base. A breaking change
fails unless one of these is true:
- the PR title declares it with `!` (`feat!: ...`, `fix(client)!: ...`),
which makes release-please cut a major version (a minor one before
1.0.0);
- an owner applied the **`api-compatible`** label because they judge the
flagged change compatible.
- **On the release PR (`release`):** for each package whose version
changes in `.release-please-manifest.json`, compares the code with that
package's last release tag (`<component>-<version>`). It fails on a
breaking change unless the new version is a major one (a minor one
before 1.0.0), or an owner applied `api-compatible` to the release PR. A
package with no release tag yet is reported and skipped.
- **What counts as breaking:**
- from griffe: removed names, removed or moved parameters, new required
parameters, changed defaults, changed attribute values;
- added by the script: names dropped from `__all__`, and changed return
types, which griffe doesn't flag.
- **Experimental modules are skipped,** because an experimental feature
may change in a minor release.

## Tested

- **`tests/test_api_breaking.py` (23 tests)** uses a throwaway git repo
with a tagged release. It covers:
  - each breaking kind, and additive changes that pass;
  - experimental modules being skipped;
  - the title and label rules;
  - release versions before and after 1.0, including 0.9.0 → 1.0.0;
  - a package with no release tag;
- `api-compatible` passing an undeclared breaking release, and another
label not passing it.

  The fixture isolates git from the developer's global config.
- **Full suite:** `uv run pytest` gives 2579 passed, 14 skipped. `ruff
check` and `ruff format --check` are clean, and `actionlint` passes on
the workflow.
- **Against real history:**
  - `pr` against `main`: no breaking changes, so it passes.
- `pr` against `launchdarkly-ai-server-0.3.0`: 23 breaking changes.
That's 14 names that moved to `experimental`, plus `capture_content`'s
default changing from `False` to `None` in every provider factory,
`create_handler` and `ProviderHandler`. It exits 1 with a plain title,
and 0 with `!` or the label.

## The open release PR #166

Running `release` against #166 flags **6 provider packages**:
- claude-agents, langchain-agents, openai-agents: 0.2.5 → 0.2.6
- claude-messages, langchain-messages, openai-messages: 0.2.4 → 0.2.5

Each changed `capture_content`'s default from `False` to `None` (#158)
since its last release, but is getting only a patch bump. `client`
(0.3.0 → 0.4.0) passes.

If `None` means "use the default from `configure()`", callers see no
difference. So the **`api-compatible` label now also passes the release
check** (06050f1): an owner can judge a release PR compatible, the same
way as any other PR. Verified on #166: it exits 1 without the label and
0 with it. The label now exists in this repo.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds **CI enforcement** so undeclared public API breaks cannot ship in
patch/minor releases.
> 
> A new **API compatibility** GitHub Actions workflow runs on every PR
(including title/label edits). For normal PRs it runs
`scripts.api_breaking pr`, comparing stable `__all__` exports against
the merge base via griffe (plus extra checks for `__all__` drops and
return-type changes). CI fails unless the title uses conventional-commit
`!` or an owner applies **`api-compatible`**. On **release-please**
branches it runs `release`, comparing each bumped package to its last
release tag and requiring a major (or pre-1.0 minor) bump when breaking
changes slipped through.
> 
> `api_report.resolve` is refactored to take a `ModulesCollection` so
the breaking checker can reuse the same resolution logic.
**`tests/test_api_breaking.py`** covers detection rules, title/label
gates, semver alignment, experimental skips, and removed packages.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a61cf9e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants