Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ jobs:
'^mypy\.ini$' \
'^bin/lint\.sh$' \
'^bin/typecheck\.sh$' \
'^bin/ci_.*baseline\.json$'
'^bin/ci/ci_.*baseline\.json$'

# GFQL core + tests.
emit gfql \
Expand Down Expand Up @@ -863,7 +863,7 @@ jobs:
- name: Pyright ratchet
run: |
source pygraphistry/bin/activate
./bin/ci_pyright_guard.py
./bin/ci/ci_pyright_guard.py

cypher-frontend-strict-typing:
name: cypher-frontend-strict-typing (py3.12)
Expand Down Expand Up @@ -983,7 +983,7 @@ jobs:

- name: Audit cuDF test gates and the CI coverage they lack
run: |
python bin/ci_gpu_gate_audit.py
python bin/ci/ci_gpu_gate_audit.py

cypher-frontend-surface-guard:
name: cypher-frontend-surface-guard
Expand All @@ -1000,7 +1000,7 @@ jobs:

- name: Cypher surface growth guard
run: |
python bin/ci_cypher_surface_guard.py
python bin/ci/ci_cypher_surface_guard.py

test-networkx-scipy-policy:
name: test-networkx-scipy-policy (${{ matrix.label }}, py${{ matrix.python-version }})
Expand Down Expand Up @@ -1248,7 +1248,7 @@ jobs:
graphistry/tests/test_gfql_remote_metadata.py \
graphistry/tests/test_gfql_remote_persistence.py \
tests/gfql/ref
python bin/coverage_audit.py \
python bin/ci/coverage_audit.py \
--profile gfql \
--skip-tests \
--engine-label ci-pandas-py3.12 \
Expand Down Expand Up @@ -1362,7 +1362,7 @@ jobs:
build/changed-line-coverage/artifacts/core/coverage.core-py3.14 \
build/changed-line-coverage/artifacts/gfql/coverage.gfql-py3.12 \
build/changed-line-coverage/artifacts/polars/coverage.polars-py3.12
python bin/changed_line_coverage.py \
python bin/ci/changed_line_coverage.py \
--data-file build/changed-line-coverage/combined.coverage \
--base-ref "$BASE_SHA" \
--head-ref "$HEAD_SHA" \
Expand Down Expand Up @@ -1749,7 +1749,7 @@ jobs:
if: ${{ matrix.python-version == '3.12' }}
run: |
source pygraphistry/bin/activate
python bin/coverage_audit.py \
python bin/ci/coverage_audit.py \
--profile gfql-polars \
--skip-tests \
--engine-label ci-polars-py3.12 \
Expand Down Expand Up @@ -1859,7 +1859,7 @@ jobs:
- name: Derive HF cache key
id: hf-cache-key
run: |
key_output="$(python bin/ci-hf-cache-key.py requirements/test-ai-py${{ matrix.python-version }}.lock)"
key_output="$(python bin/ci/ci-hf-cache-key.py requirements/test-ai-py${{ matrix.python-version }}.lock)"
printf '%s\n' "$key_output"
printf '%s\n' "$key_output" >> "$GITHUB_OUTPUT"

Expand Down
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
### Infrastructure

- **CI: the gfql change filter now includes the chain engine** (`compute/chain*.py`, `hop.py`, `gfql_fast_paths.py`, `filter_by_dict.py`, `ast.py`, `predicates/`, and the chain/hop test files), so tck-gfql, the Cypher-frontend gates and the gfql benchmark lane run on a change to the chain engine; they were skipped on #2055.
- **CI: pyright now runs as a ratcheted gate (`python-pyright`, py3.12)**, catching conditionally-unbound locals, unresolved names and no-effect statements that ruff and mypy miss. `bin/pyright.sh` pins pyright 1.1.414; `bin/ci_pyright_guard.py` holds findings to per-file counts in `bin/ci_pyright_baseline.json`, so existing debt is grandfathered and new or moved code is not. Only five rules gate — those decided by a file's own control flow, names and syntax; rules that read third-party stubs vary by up to 5.5x across environments and are reported but never gated. Unparseable files and a collapse in analysis scope also fail the gate, so widening a `pyrightconfig.json` exclude cannot quietly disable it. See DEVELOP.md "Pyright ratchet". Builds on #1123 by @JithinBathula; closes #1075.
- **CI tooling moves into `bin/ci/`**: `bin/` had grown to 34 flat files. The 13 CI-only scripts and baselines (the four `ci_*_guard.py` with their baselines, `ci_docs_only_check.py`, `ci_gpu_gate_audit.py`, `ci-hf-cache-key.py`, `changed_line_coverage.py`, `coverage_audit.py`) now live under `bin/ci/`, leaving 21 developer-facing entry points at the top level. Pure relocation: no behaviour, flags or output changed, and `git mv` keeps history. The `test-*.sh` runners deliberately stay put — they are referenced 32 times across 15 files including `conftest.py` and committed coverage baselines, which is fixture churn for a cosmetic gain.
- **CI: pyright now runs as a ratcheted gate (`python-pyright`, py3.12)**, catching conditionally-unbound locals, unresolved names and no-effect statements that ruff and mypy miss. `bin/pyright.sh` pins pyright 1.1.414; `bin/ci/ci_pyright_guard.py` holds findings to per-file counts in `bin/ci/ci_pyright_baseline.json`, so existing debt is grandfathered and new or moved code is not. Only five rules gate — those decided by a file's own control flow, names and syntax; rules that read third-party stubs vary by up to 5.5x across environments and are reported but never gated. Unparseable files and a collapse in analysis scope also fail the gate, so widening a `pyrightconfig.json` exclude cannot quietly disable it. See DEVELOP.md "Pyright ratchet". Builds on #1123 by @JithinBathula; closes #1075.
- **CI: `test-docs` runs on docs-only pull requests (#2018)**: the job needed `python-lint-types`, which a docs-only change skips, and GitHub skips a job whose prerequisite was skipped. The gate now accepts skipped prerequisites and refuses only failed or cancelled ones, so documentation changes are built and tested before merge.
### Tests

Expand Down
44 changes: 22 additions & 22 deletions DEVELOP.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,21 +118,21 @@ CI includes `cypher-frontend-surface-guard`, which enforces bounded growth for:

Guard implementation + baseline:

- Script: `bin/ci_cypher_surface_guard.py`
- Baseline: `bin/ci_cypher_surface_guard_baseline.json`
- Script: `bin/ci/ci_cypher_surface_guard.py`
- Baseline: `bin/ci/ci_cypher_surface_guard_baseline.json`

If growth is intentional, regenerate baseline in your branch and include explicit PR rationale:

```bash
python bin/ci_cypher_surface_guard.py --write-baseline
python bin/ci/ci_cypher_surface_guard.py --write-baseline
```

Then commit both code changes and baseline update together.

### Type Hygiene Guard

`bin/lint.sh` (run by the `python-lint-types` matrix on py3.8-3.14) runs
`bin/ci_type_hygiene_guard.py`, a stdlib-only AST check over `graphistry/`
`bin/ci/ci_type_hygiene_guard.py`, a stdlib-only AST check over `graphistry/`
(tests excluded, matching `mypy.ini`). It exists to catch the defect classes
that keep coming back in code review, so a reviewer does not have to.

Expand All @@ -147,15 +147,15 @@ that keep coming back in code review, so a reviewer does not have to.
| `vocab-str-param` | a closed-vocabulary parameter (`table`, `kind`, `direction`, `how`, `mode`, `engine`) annotated as plain `str` |

Enforcement is a **per-file count ratchet** against
`bin/ci_type_hygiene_baseline.json`: a file may not gain findings, and a file
`bin/ci/ci_type_hygiene_baseline.json`: a file may not gain findings, and a file
absent from the baseline must have zero. Existing debt is grandfathered, new and
moved code is not.

```bash
./bin/ci_type_hygiene_guard.py # what CI runs
./bin/ci_type_hygiene_guard.py --report # totals per check
./bin/ci_type_hygiene_guard.py --list plottable-setattr
./bin/ci_type_hygiene_guard.py --strict # show files that improved; time to retighten
./bin/ci/ci_type_hygiene_guard.py # what CI runs
./bin/ci/ci_type_hygiene_guard.py --report # totals per check
./bin/ci/ci_type_hygiene_guard.py --list plottable-setattr
./bin/ci/ci_type_hygiene_guard.py --strict # show files that improved; time to retighten
```

When a finding is genuinely correct, annotate that line and say why:
Expand All @@ -170,8 +170,8 @@ the baseline update together.

### Pyright Ratchet

The `python-pyright` CI job (py3.12) runs `bin/ci_pyright_guard.py`, which invokes `bin/pyright.sh`
and holds the result to a per-file count ratchet against `bin/ci_pyright_baseline.json`. It catches
The `python-pyright` CI job (py3.12) runs `bin/ci/ci_pyright_guard.py`, which invokes `bin/pyright.sh`
and holds the result to a per-file count ratchet against `bin/ci/ci_pyright_baseline.json`. It catches
what ruff and mypy do not: locals bound on only some paths, names that resolve nowhere, statements
with no effect.

Expand All @@ -192,10 +192,10 @@ an installed `pyright` only when it matches, else fetches the pin via `uvx`/`npx
`PYRIGHT_VERSION` and regenerate the baseline in the same commit.

```bash
./bin/ci_pyright_guard.py # what CI runs
./bin/ci_pyright_guard.py --report # gated and ungated totals, always exit 0
./bin/ci_pyright_guard.py --list reportPossiblyUnboundVariable
./bin/ci_pyright_guard.py --strict # show files that improved; time to retighten
./bin/ci/ci_pyright_guard.py # what CI runs
./bin/ci/ci_pyright_guard.py --report # gated and ungated totals, always exit 0
./bin/ci/ci_pyright_guard.py --list reportPossiblyUnboundVariable
./bin/ci/ci_pyright_guard.py --strict # show files that improved; time to retighten
./bin/pyright.sh graphistry/compute # the raw tool, narrowed
```

Expand All @@ -213,7 +213,7 @@ is baselined rather than excluded so that fixing it shows up as slack under `--s
### Comment Density Guard

`bin/lint.sh` (the same `python-lint-types` matrix lane as the type-hygiene
guard) runs `bin/ci_comment_density_guard.py`, a stdlib-only `tokenize` + `ast`
guard) runs `bin/ci/ci_comment_density_guard.py`, a stdlib-only `tokenize` + `ast`
check over `graphistry/`. It enforces the "Encoding: names, tests, and
structure — not prose" rules in `agents/skills/review/SKILL.md`, which were the
last rule class on that stack still enforced only by human review.
Expand All @@ -236,13 +236,13 @@ performance vocabulary (they also name correctness concepts), and a comment that
points at `pyg-bench` is a pointer to the measurement rather than a claim.

Enforcement is a **per-file count ratchet** against
`bin/ci_comment_density_baseline.json`, exactly like the type-hygiene guard.
`bin/ci/ci_comment_density_baseline.json`, exactly like the type-hygiene guard.

```bash
./bin/ci_comment_density_guard.py # what CI runs
./bin/ci_comment_density_guard.py --report # totals per check
./bin/ci_comment_density_guard.py --list comment-block
./bin/ci_comment_density_guard.py --strict # show files that improved; time to retighten
./bin/ci/ci_comment_density_guard.py # what CI runs
./bin/ci/ci_comment_density_guard.py --report # totals per check
./bin/ci/ci_comment_density_guard.py --list comment-block
./bin/ci/ci_comment_density_guard.py --strict # show files that improved; time to retighten
```

The fix is almost never a suppression: extract a helper whose NAME states the
Expand Down Expand Up @@ -299,7 +299,7 @@ a cuDF-gated test can contradict the CPU contract, or rot outright, and stay gre
on master indefinitely. Treat a GPU claim in a PR as unprotected until a GPU lane
exists: re-run it yourself rather than trusting the last receipt.

`bin/ci_gpu_gate_audit.py` (lane `gpu-gate-audit`) keeps the size of that gap
`bin/ci/ci_gpu_gate_audit.py` (lane `gpu-gate-audit`) keeps the size of that gap
visible: it counts the cuDF gates, requires each to be attributable (a `reason=`
naming `TEST_CUDF`, so `pytest -rs` names what was not run rather than reporting a
bare `s`) and to actually read the flag from the environment, and cross-checks this note against
Expand Down
2 changes: 1 addition & 1 deletion agents/skills/review/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ Delete from a diff: narration of the next block (extract a helper named for the
why-this-fix or issue-number rationale (the pin's test name carries it; an issue ref may stay as a
trailing tag), any perf/complexity/benchmark claim (measurement belongs in pyg-bench), and
restatements of the signature. A keep must state a constraint that no name and no test can express
— *defensible* is not the bar, and a doubtful keep deletes. Guard: `bin/ci_comment_density_guard.py`.
— *defensible* is not the bar, and a doubtful keep deletes. Guard: `bin/ci/ci_comment_density_guard.py`.

Typing, same rule: `Any` over a known domain gets the real alias; a new `# type: ignore` or
`hygiene-ok` gets restructured (both are for grandfathered debt only); `cast()` to satisfy the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from typing import Any, Dict, List, Mapping, Optional, Sequence, Set


REPO_ROOT = Path(__file__).resolve().parent.parent
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
ZERO_SHA = "0000000000000000000000000000000000000000"
HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"_comment": "Per-file ratchet for bin/ci_comment_density_guard.py. Counts may shrink, never grow; a file absent here must have zero findings. Regenerate with `./bin/ci_comment_density_guard.py --update-baseline` and explain the delta in the PR description.",
"_comment": "Per-file ratchet for bin/ci/ci_comment_density_guard.py. Counts may shrink, never grow; a file absent here must have zero findings. Regenerate with `./bin/ci/ci_comment_density_guard.py --update-baseline` and explain the delta in the PR description.",
"checks": {
"comment-block": {
"graphistry/Engine.py": 17,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

Meaning belongs in a **name**, a **test/pin**, or the **structure** of the code.
Prose is the last resort. Every other rule on that stack already has a machine
gate (`bin/ci_type_hygiene_guard.py`, `bin/ci_cypher_surface_guard.py`, the
gate (`bin/ci/ci_type_hygiene_guard.py`, `bin/ci/ci_cypher_surface_guard.py`, the
per-file coverage floors); comment discipline was the only one left to human
review, which is why it is the only one that kept reaching the owner.

Expand Down Expand Up @@ -37,11 +37,11 @@
file may not gain findings, and a file absent from the baseline must have zero.
Existing debt is grandfathered; new and moved code is held to the rule.

./bin/ci_comment_density_guard.py # check (this is what CI runs)
./bin/ci_comment_density_guard.py --report # totals per check, always exit 0
./bin/ci_comment_density_guard.py --list CHECK # every current finding for CHECK
./bin/ci_comment_density_guard.py --update-baseline
./bin/ci_comment_density_guard.py --strict # also fail when the baseline has
./bin/ci/ci_comment_density_guard.py # check (this is what CI runs)
./bin/ci/ci_comment_density_guard.py --report # totals per check, always exit 0
./bin/ci/ci_comment_density_guard.py --list CHECK # every current finding for CHECK
./bin/ci/ci_comment_density_guard.py --update-baseline
./bin/ci/ci_comment_density_guard.py --strict # also fail when the baseline has
# gone stale-loose (time to tighten)

Escape hatch: put `# guard-ok: <check> -- <reason>` on the reported line, or on
Expand All @@ -62,9 +62,9 @@
from dataclasses import dataclass
from typing import Dict, List, Optional, Sequence, Tuple

REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
SCAN_ROOT = os.path.join(REPO_ROOT, "graphistry")
DEFAULT_BASELINE = os.path.join(REPO_ROOT, "bin", "ci_comment_density_baseline.json")
DEFAULT_BASELINE = os.path.join(REPO_ROOT, "bin", "ci", "ci_comment_density_baseline.json")

EXCLUDE_DIRS = ("__pycache__",)
EXCLUDE_FILES = ("graph_vector_pb2.py", "_version.py", "versioneer.py")
Expand Down Expand Up @@ -382,9 +382,9 @@ def load_baseline(path: str) -> Counts:
def write_baseline(path: str, counts: Counts) -> None:
payload = {
"_comment": (
"Per-file ratchet for bin/ci_comment_density_guard.py. Counts may shrink, never "
"Per-file ratchet for bin/ci/ci_comment_density_guard.py. Counts may shrink, never "
"grow; a file absent here must have zero findings. Regenerate with "
"`./bin/ci_comment_density_guard.py --update-baseline` and explain the delta in "
"`./bin/ci/ci_comment_density_guard.py --update-baseline` and explain the delta in "
"the PR description."
),
"checks": dict((check, dict(sorted(counts[check].items()))) for check in CHECKS),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@
from typing import Dict


REPO_ROOT = Path(__file__).resolve().parent.parent
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
LOWERING_PATH = REPO_ROOT / "graphistry/compute/gfql/cypher/lowering.py"
DEFAULT_BASELINE_PATH = REPO_ROOT / "bin/ci_cypher_surface_guard_baseline.json"
DEFAULT_BASELINE_PATH = REPO_ROOT / "bin/ci/ci_cypher_surface_guard_baseline.json"
TARGET_CLASSES = (
"CompiledCypherQuery",
"CompiledGraphBinding",
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion bin/ci_gpu_gate_audit.py → bin/ci/ci_gpu_gate_audit.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from pathlib import Path
from typing import List, Optional, Tuple

REPO = Path(__file__).resolve().parent.parent
REPO = Path(__file__).resolve().parent.parent.parent
TESTS = REPO / "graphistry" / "tests"
WORKFLOWS = REPO / ".github" / "workflows"
DEVELOP = REPO / "DEVELOP.md"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"_comment": "Per-file ratchet for bin/ci_pyright_guard.py, built with pyright 1.1.414. Counts may shrink, never grow; a file absent here must have zero findings. Only the rules listed are gated -- the rest move with the installed dependencies. Regenerate with `./bin/ci_pyright_guard.py --update-baseline` and explain the delta in the PR description. files_analyzed records the scope this was built over; a run that sees materially less is a collapsed gate, not an improvement.",
"_comment": "Per-file ratchet for bin/ci/ci_pyright_guard.py, built with pyright 1.1.414. Counts may shrink, never grow; a file absent here must have zero findings. Only the rules listed are gated -- the rest move with the installed dependencies. Regenerate with `./bin/ci/ci_pyright_guard.py --update-baseline` and explain the delta in the PR description. files_analyzed records the scope this was built over; a run that sees materially less is a collapsed gate, not an improvement.",
"pyright_version": "1.1.414",
"files_analyzed": 351,
"rules": {
Expand Down
Loading
Loading