index_status returns ~26k tokens on a 11.8k-file repo (~8k on a 42-node repo) — payload dominated by deliberately-ignored files, with no way to trim

[artaommahe]

Version

codebase-memory-mcp 0.9.1-rc.1

Platform

macOS (Apple Silicon)

Install channel

GitHub release archive / install.sh / install.ps1

Binary variant

standard

What happened, and what did you expect?

index_status enumerates every parse_partial file and up to 500 not_indexed files inline, with no way to trim or omit them. The result is an enormous response for what is usually a "how big is the graph, is it ready?" question — and the overwhelming majority of the payload is files the indexer deliberately ignores by design (ignored-suffix assets, gitignored files), not gaps anyone can act on.

On a private Angular monorepo the MCP index_status response is 105,771 chars (~26k tokens):

block	chars	content
parse_partial	54,876	all 402 files enumerated
not_indexed	53,829	500 files listed, 494 of them .svg/.png icon assets
everything else	~1,000	project, nodes, edges, status, root_path

That is ~26k tokens of context for ~1k tokens of signal, on the tool an agent naturally calls first to orient itself. It also sits oddly beside the RC's headline "~99% fewer output tokens" — the tree/TOON format applies to the graph query tools, but index_status (and check_index_coverage) still emit raw JSON with full inline listings.

check_index_coverage --scopes . has the same shape: on the same repo it returns ~15k tokens where the first 200 entries are almost entirely ignored-suffix SVG assets, burying the handful of real parse_partial source files.

--verbose only adds a git-context block; there is no inverse.

Suggestions (any one would fix the practical problem):

• Default to counts + a few examples, and require an opt-in flag (or check_index_coverage) for the full listing — the parse_partial block already has a truncated field and a note pointing at the logfile, so the mechanism exists.
• Never inline not_indexed entries whose reason is by-design (gitignore, ignored-suffix, excluded subtree) — a per-reason count is strictly more useful than 494 SVG paths.
• Accept a limit / include parameter so callers can ask for what they need.

Reproduction

Public repro repo: https://github.com/artaommahe/codebase-memory-mcp-rc-repros — see Case C. It contains 600 .svg files under assets/icons/, correctly classified as ignored-suffix.

git clone https://github.com/artaommahe/codebase-memory-mcp-rc-repros
cd codebase-memory-mcp-rc-repros
codebase-memory-mcp cli index_repository --repo-path "$PWD" --mode full --name cbm-rc-repros

codebase-memory-mcp cli index_status --project cbm-rc-repros | wc -c

Observed:

31931

~8k tokens of status for a graph of 42 nodes / 54 edges. 500 of the 600 not_indexed entries are enumerated, and all 500 are .svg assets; the not_indexed block alone is 33,719 chars of JSON.

Project scale (if relevant)

The 105,771-char figure is from a private Angular monorepo: 11,788 files, 63,840 nodes / 120,805 edges, --mode full. The linked repro is self-contained dummy code.

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.

---

Posted by Claude Code on behalf of @artaommahe

[artaommahe]

Three corrections to the body, one disclosure I owe you, and a pointer to evidence in your own source that makes the case better than I did.

Correction 1: "dominated by deliberately-ignored files" is wrong for the large repo

I wrote that "the overwhelming majority of the payload is files the indexer deliberately ignores by design". On the 11.8k-file repo it's roughly an even split, and the ignored half is the smaller one:

block	wire bytes	share	content
parse_partial	53,264	50.4%	402 files — genuine coverage gaps, not by-design
not_indexed	51,802	49.0%	500 listed, 494 of them .svg/.png
everything else	~700	0.6%	project, nodes, edges, status, root_path

The by-design framing only holds on the small repro repo, which has parse_partial: 0. On a real repo half the payload is real signal — just far more of it than any caller asked for. That doesn't change the ask (counts always, entries opt-in), but the issue as written is contradicted by its own table and I'd rather fix that than have you notice it.

Correction 2: my per-block figures were measured differently from my totals

The original body lists not_indexed as 33,719 chars for the repro repo — larger than the 31,931-byte total in the same issue, which is obviously impossible. Cause: I measured sub-blocks with Python's default json.dumps separators and totals as wire bytes. Corrected, consistently in wire bytes:

block	as filed	actual
repro not_indexed	33,719	31,716 (99.2% of the 31,970-byte payload)
frontend parse_partial	54,876	53,264
frontend not_indexed	53,829	51,802

Correction 3: the check_index_coverage figure was 4× too high

I said "~15k tokens". At the documented default scope_limit, check_index_coverage --scopes . is 26,271 chars ≈ 6.6k tokens. My 15k came from a larger page size. It also does have scope_limit / scope_offset, so "the same shape" overstates the comparison — its real gap is narrower: no way to filter out by-design entries.

Disclosure: there is a partial workaround I didn't mention

A .cbmignore containing assets/icons/ collapses the repro's index_status from 31,970 → 489 bytes, because a directory skip is one row instead of 500 file rows. My literal claim (no parameter trims it) still holds, but anyone who finds this unaided would reasonably read the issue as user error, so it belongs in the issue.

It's a workaround, not a fix, for two reasons: it destroys the coverage record (not_indexed count drops from 600 to 0 — you can no longer tell those files exist), and it cannot help parse_partial at all, which is the larger half on a real repo. You can't .cbmignore 240 Angular templates without dropping them from the graph.

You already diagnosed this, in src/mcp/mcp.c

This is the strongest argument in the issue and it isn't mine — it's the comment above INDEX_SKIPPED_FILE_CAP:

/* In-response coverage lists are EXAMPLES, not the record: the full uncapped lists live in the per-run logfile … Five examples orient the agent; anything more duplicates the logfile into every index response (53 KB observed on a large repo). */
enum { INDEX_SKIPPED_FILE_CAP = 5 };

Same problem, same magnitude ("53 KB observed on a large repo"), already fixed for index_repository with a 5-example cap. index_status uses COVERAGE_FILE_CAP = 500 instead. So the ask reduces to: apply the reasoning you already applied to index_repository. Note also that the comment points callers at "index_status (scope_limit)" — that parameter doesn't exist on index_status.

#1066 ("list_projects should paginate and avoid eager per-project details") is the same request for a neighbouring tool and reads as precedent; #1334 is a sibling RC token-safety issue.

One more bug found while measuring

index_status's not_indexed.files_count reports 2000 on the large repo, but check_index_coverage metadata for the same project says ignored_files_stored: 2000, ignored_files_total: 6410. So the summary count is itself silently clamped to the storage cap — the one number a caller would trust to avoid reading the list is wrong by 3.2×.

---

Posted by Claude Code on behalf of @artaommahe