Repository navigation
Conversation
Adds an advisory workflow that mutates only the lines a PR changes. It tests each mutant against its own package's tests, then retests the survivors against the datafusion core tests and sqllogictest, and lists mutants that no test caught in the job summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TAeJdSfRAQtKMjuAtTDnuH
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TAeJdSfRAQtKMjuAtTDnuH
|
Thank you for working on this! I'd like to spend some time experimenting with it locally before deciding whether we should integrate it into our CI infrastructure. Here are my concerns: I now think addressing mutation testing failures can introduce significant maintenance overhead. Here's a concrete example I just ran into: TL;DR: An AI-assisted review found that removing two lines from the execution code wouldn't cause any existing tests to fail, which is a typical mutation testing finding. However, addressing such findings isn't necessarily straightforward. It could encourage adding tests that overfit specific mutations rather than testing meaningful behavior. Such tests tend to be verbose and hard to maintain, which is already a common weakness of AI-generated code. I think a guideline like 'only try with e2e test, don't overfit mutation test violations with verbose UT' should work, but it can be challenging to enforce it 🤔 |
I agree about the maintenance overhead, totally, but I also think somehow.... that is the point. I have been thinking about it, and I am not sure there is a way to contain the defect rate in presence of AI generated code that doesn't involve increasing overhead and friction. This is a non-blocking check for now, so merging it would only help contributors write better tests, if they want. If they don't, they can ignore it :) |
Important
Fork CI setup: For pull requests from a fork to
main, enable workflows in your fork's Actions tab before you push.See the fork CI setup guide.
Which issue does this PR close?
Rationale for this change
In #26865, @2010YOUY01 noted that the earlier attempt (#14590) failed because a full cargo-mutants run over the workspace took too long. They offered to review CI that only mutates code a PR changes. Running it on a PR's own code also addresses @alamb's concern that someone has to watch and fix the results: the author who wrote the code sees the missed mutants while the change is still in review.
What changes are included in this PR?
.github/workflows/mutants.yml: an advisorypull_requestworkflow. It is not in.asf.yamlrequired checks, and it only runs when.rsfiles change. It runscargo mutants --in-diff, which mutates only the lines the PR changes..cargo/mutants.toml: reuses the existingciprofile.docs/source/contributor-guide/testing.md: a "Mutation testing" section with the commands to run the check locally.Why the workflow runs two passes
A mutant is caught when some test fails with it in place. It is missed when all tests still pass. Only missed mutants are reported.
By default, cargo-mutants runs only the tests of the crate the mutant is in. Much of DataFusion is tested from elsewhere. Optimizer rules, for example, are mostly covered by
datafusion/core/testsand.sltfiles. A mutant can therefore be missed by its own crate's tests but caught by sqllogictest.Running every mutant against core and sqllogictest would fix that, but each mutant then needs a rebuild of
datafusionanddatafusion-sqllogictestand a full run of their tests: 2.5–6 minutes per mutant, against seconds for a crate's own tests. So the workflow uses the cheap run as a filter:--iterate, againstdatafusionanddatafusion-sqllogictest. Skipped if pass 1 missed nothing.Mutants still missed after pass 2 are listed in the job summary, and the job fails.
mutants.outis uploaded as an artifact.For #26716 (23 mutants), pass 1 took 1.5 minutes and left 3 missed mutants. Pass 2 caught 1 of them. Running all 23 against core and sqllogictest would take about 1.5 hours. Pass 1 alone would have reported one false miss.
A test in a third crate (not the mutated crate,
datafusionor sqllogictest) is still not run, so the check can report a mutant as missed when that test covers it.Settings needed to avoid false passes
--baseline skipand no test data, every core and sqllogictest test failed. cargo-mutants then counted every mutant as caught, so the check passed without testing anything.--timeout 1800. In cargo-mutants 27.1.0, the pass 2 baseline runs the mutated crate's tests instead of the--test-packageones, and derives a 20 s timeout from them. Every mutant then timed out instead of being reported as missed, and the check passed.What is the testing strategy for this PR?
I ran the workflow on two PRs inside my fork. Both are based on the commit before #26716.
In the #26716 run, pass 1 tested 23 mutants: 16 caught, 4 unviable, 3 missed. Pass 2 caught one more. No unit, integration or
.slttest fails for these two:Cost on
ubuntu-latest:A PR whose mutants are all caught in pass 1 skips pass 2.
actionlint, shellcheck, the license header check, prettier and typos pass locally.
Are there any user-facing changes?
No. This adds a non-required CI check and a contributor guide section.
Open questions
Rustworkflow instead runs fork PRs on the fork. Is the extra runner time acceptable for an advisory check, or should this follow the fork-CI pattern?This PR was written with AI assistance (Claude Code). I have read the whole change and the fork runs.
🤖 Generated with Claude Code
https://claude.ai/code/session_01TAeJdSfRAQtKMjuAtTDnuH