A concurrency test asserts on wall-clock time and fails under coverage instrumentation

[borisno2]

The flake

packages/rag/src/runtime/batch.test.ts › ProcessingQueue › should process items concurrently with concurrency > 1 asserts on wall-clock elapsed time:

AssertionError: expected 79 to be less than 50

It blocked a merge on PR #1318, whose diff is documentation, a changeset and package.json — no runtime source at all.

The evidence that it is a flake, and what actually causes it

In one CI run (34156198755) the identical test ran four times:

job	copy	result
test	src	pass
test	dist	pass
test:coverage	src	pass
test:coverage	dist	fail, 79ms vs a 50ms bound

Three passes and one failure, differing only by which copy ran and whether coverage instrumentation was on. Coverage instrumentation is the cause — it slows execution enough to breach a 50ms bound.

Two things make it worse. The suite runs twice, from src and from dist (#1311), doubling CPU contention on the runner. And a shared runner's load is not controlled at all.

Why the assertion is wrong, not just tight

The test wants to prove concurrency — that a queue with concurrency greater than one overlaps its work. Elapsed time is a proxy for that, and a proxy that a slower machine, an instrumented run, or a noisy neighbour can falsify without the behaviour changing at all.

Widening the bound would only move the threshold. Concurrency should be observed directly: record the number of in-flight tasks and assert the maximum exceeds one, or assert on the interleaving of start and finish events. That holds on any machine at any speed, and it fails when concurrency genuinely breaks — which the timing version does not reliably do in either direction.

What to do

• Rewrite this assertion to observe overlap rather than elapsed time.
• Check the rest of the file, and the repo, for other wall-clock assertions with the same weakness.
• Fixing Four small cleanups from the spec 8 QA pass: duplicate RAG test runs, db:push in the scaffolder template, a wrong README script, and a broken Ollama demo #1311 first would halve the contention, but it does not make a timing assertion sound.

Related

The same shape as the PGlite-absence flake fixed in #1290: a test racing a fixed budget on a shared runner. That one was given a realistic budget because its cost was genuinely bounded; this one should stop measuring time altogether, because time is not what it is testing.

Context

• fix(rag): make the RAG docs' own examples compile, and load the chatbot config #1318 — where it blocked a merge.
• Four small cleanups from the spec 8 QA pass: duplicate RAG test runs, db:push in the scaffolder template, a wrong README script, and a broken Ollama demo #1311 — the duplicate src/dist runs that double the load.
• Flaky: the PGlite-absence test races the default 5s timeout #1290 — the earlier timing flake.

---

Generated by Claude Code