Repository navigation
feat: add graphify verify for basic dependency invariant checks - #4549
rajarshidattapy wants to merge 3 commits into
Conversation
Adds an OSS, local implementation of the "formal verification" feature currently described as Cloud-only in the README: a graphify verify CLI command that checks a graph.json against a small rules file of forbidden dependency patterns (glob over source_file), reporting violations and exiting 1 for CI use.
Adds coverage for malformed/missing rules and graph files, dangling edges, nodes missing source_file, alternate edges/links key, self-loops, glob-matching behavior, and CLI argument handling (missing/dangling flags, default graph path, exit codes, --json output).
|
Thanks for the pull request, @rajarshidattapy. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic. |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; formal verification proved 0 of 1 changed function(s) (1 not verified).
Formal verification. PR-changed functions: 0/1 verified (0 proven, 0 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).
Not verified on this run: dispatch\_command (vacuous: never exercised).
Graphify review — findings
Adds a graphify verify --rules rules.json command that checks the built graph for forbidden dependencies and exits 1 on any violation, so it works as a CI gate. verify_graph matches each edge's source and target source_file against glob rules, while skipping edges with unknown nodes or missing files. A null source_file is stringified rather than skipped, so it can produce false violations, and load_rules coerces non-string from/to values with str() rather than rejecting them. The README documents the rules format and the command, and new tests cover rule loading, matching, and the JSON/text reports.
Worth a look
- Null source_file is treated as literal 'None' and can create false violations —
graphify/verify.py:67· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Rule schema silently coerces non-string glob patterns —
graphify/verify.py:39· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 427 functions depend on the 151 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
dispatch_command()— 5 callers, 138 callees
Verification — 427 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 382 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
433 of 433 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/pi.test.ts— full-run-safetytests/test_affected_cli.py— impact, full-run-safetytests/test_affected_defines.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_affected_overrides.py— full-run-safetytests/test_agents_platform.py— impact, full-run-safetytests/test_alias_folding.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_backend_opt_in.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_benchmark_real_query.py— full-run-safetytests/test_blade_extractor.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_located_semantic_identity.py— full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_c_include_resolution.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_cache_stale_import_target.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_cargo_missing_manifest.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safetytests/test_charmap_encoding.py— full-run-safetytests/test_chunking.py— full-run-safetytests/test_cjs_module_extension.py— full-run-safetytests/test_claude_cli_backend.py— full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— impact, full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cloud_cta.py— impact, full-run-safetytests/test_cluster.py— full-run-safetytests/test_cluster_ambiguous_scale.py— full-run-safetytests/test_cluster_exclude_hubs.py— full-run-safety- … and 383 more
non-code file(s) changed (
README.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
README.md) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
docs/superpowers/plans/2026-05-04-incremental-updates-dedup.md§ Task 4: Incremental updates — semantic cache + manifest in __main__.py (lines 603-911): references changed symbols_run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 40 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
· 1 more finding(s) on lines outside this diff (see the check run).
Summary
graphify verify --rules rules.json [--graph graph.json] [--json]CLI command: loads a rules file of forbidden-dependency glob patterns (from/tomatched against each edge's nodesource_file) and checks them against a builtgraph.json, reporting any edge that actually crosses a forbidden boundary.1on violations (usable as a CI gate),0when clean.graphify/verify.pymodule:load_rules,verify_graph,format_report,format_json. No new dependencies.Test plan
pytest tests/test_verify.py- 34 tests covering:rules, non-dict/invalid entries, empty rules)source_file, non-dict nodes/edges,edgesvslinkskey, self-loops, multi-rule matches, glob/subtree matching)--jsonoutput)ruff checkandmypyclean on changed filespytest tests/run: 778 passed, 22 skipped, 1 pre-existing unrelated failure (test_code_only_update_all_hosts.py::...[devin], a Windows path-quoting issue confirmed present before this change too)