Skip to content

feat: add graphify verify for basic dependency invariant checks - #4549

Open
rajarshidattapy wants to merge 3 commits into
Graphify-Labs:v8from
rajarshidattapy:feat/verify-dependency-invariants
Open

rajarshidattapy wants to merge 3 commits into
Graphify-Labs:v8from
rajarshidattapy:feat/verify-dependency-invariants

Conversation

@rajarshidattapy

Copy link
Copy Markdown
Contributor

Summary

  • The README currently describes "Formal verification: check architectural and dependency invariants against the live graph" as a Graphify Cloud-only feature. This adds a small, local OSS version of it.
  • New graphify verify --rules rules.json [--graph graph.json] [--json] CLI command: loads a rules file of forbidden-dependency glob patterns (from/to matched against each edge's node source_file) and checks them against a built graph.json, reporting any edge that actually crosses a forbidden boundary.
  • Exits 1 on violations (usable as a CI gate), 0 when clean.
  • New graphify/verify.py module: load_rules, verify_graph, format_report, format_json. No new dependencies.

Test plan

  • pytest tests/test_verify.py - 34 tests covering:
    • rules-file parsing (missing file, malformed JSON, missing/non-list rules, non-dict/invalid entries, empty rules)
    • graph parsing (missing/malformed graph, empty graph, dangling edges, nodes missing source_file, non-dict nodes/edges, edges vs links key, self-loops, multi-rule matches, glob/subtree matching)
    • CLI behavior (no args, missing/dangling flags, unknown options, default graph path, exit codes, --json output)
  • ruff check and mypy clean on changed files
  • Full pytest tests/ run: 778 passed, 22 skipped, 1 pre-existing unrelated failure (test_code_only_update_all_hosts.py::...[devin], a Windows path-quoting issue confirmed present before this change too)

Adds an OSS, local implementation of the "formal verification" feature
currently described as Cloud-only in the README: a graphify verify CLI
command that checks a graph.json against a small rules file of forbidden
dependency patterns (glob over source_file), reporting violations and
exiting 1 for CI use.
Adds coverage for malformed/missing rules and graph files, dangling
edges, nodes missing source_file, alternate edges/links key, self-loops,
glob-matching behavior, and CLI argument handling (missing/dangling
flags, default graph path, exit codes, --json output).
@github-actions

Copy link
Copy Markdown

Thanks for the pull request, @rajarshidattapy. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

@github-actions github-actions Bot added the contributor Pull request from a returning community contributor label Oct 18, 2026

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; formal verification proved 0 of 1 changed function(s) (1 not verified).

Formal verification. PR-changed functions: 0/1 verified (0 proven, 0 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).

Not verified on this run: dispatch\_command (vacuous: never exercised).


Graphify review — findings

Adds a graphify verify --rules rules.json command that checks the built graph for forbidden dependencies and exits 1 on any violation, so it works as a CI gate. verify_graph matches each edge's source and target source_file against glob rules, while skipping edges with unknown nodes or missing files. A null source_file is stringified rather than skipped, so it can produce false violations, and load_rules coerces non-string from/to values with str() rather than rejecting them. The README documents the rules format and the command, and new tests cover rule loading, matching, and the JSON/text reports.

Worth a look

  • Null source_file is treated as literal 'None' and can create false violations — graphify/verify.py:67 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Rule schema silently coerces non-string glob patterns — graphify/verify.py:39 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 427 functions depend on the 151 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: dispatch_command() — 5 callers, 138 callees

Verification — 427 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 382 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

433 of 433 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/pi.test.ts — full-run-safety
  • tests/test_affected_cli.py — impact, full-run-safety
  • tests/test_affected_defines.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_affected_overrides.py — full-run-safety
  • tests/test_agents_platform.py — impact, full-run-safety
  • tests/test_alias_folding.py — full-run-safety
  • tests/test_analyze.py — full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — full-run-safety
  • tests/test_astro_import_ids.py — full-run-safety
  • tests/test_atomic_canvas_export.py — full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_backend_opt_in.py — full-run-safety
  • tests/test_benchmark.py — full-run-safety
  • tests/test_benchmark_raw_graph.py — full-run-safety
  • tests/test_benchmark_real_query.py — full-run-safety
  • tests/test_blade_extractor.py — full-run-safety
  • tests/test_build.py — full-run-safety
  • tests/test_build_located_semantic_identity.py — full-run-safety
  • tests/test_build_merge_dedup_scope.py — full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — full-run-safety
  • tests/test_build_merge_shrink_guard.py — full-run-safety
  • tests/test_builtin_global_type_refs.py — full-run-safety
  • tests/test_c_include_resolution.py — full-run-safety
  • tests/test_cache.py — full-run-safety
  • tests/test_cache_stale_import_target.py — full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — full-run-safety
  • tests/test_cargo_missing_manifest.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — full-run-safety
  • tests/test_case_sensitive_resolution.py — full-run-safety
  • tests/test_charmap_encoding.py — full-run-safety
  • tests/test_chunking.py — full-run-safety
  • tests/test_cjs_module_extension.py — full-run-safety
  • tests/test_claude_cli_backend.py — full-run-safety
  • tests/test_claude_md.py — full-run-safety
  • tests/test_cli_broken_pipe.py — full-run-safety
  • tests/test_cli_export.py — impact, full-run-safety
  • tests/test_cli_help.py — full-run-safety
  • tests/test_cloud_cta.py — impact, full-run-safety
  • tests/test_cluster.py — full-run-safety
  • tests/test_cluster_ambiguous_scale.py — full-run-safety
  • tests/test_cluster_exclude_hubs.py — full-run-safety
  • … and 383 more

non-code file(s) changed (README.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (README.md) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

Formal verification

Could not verify: Could not verify dispatch\_command.

The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 40 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)

· 1 more finding(s) on lines outside this diff (see the check run).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor Pull request from a returning community contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant