feat(forge): add support for mutation tests

[simon-something]

This is a draft PR for visibility/ request for design and implementation feedback.

As discussed in #478 , this is an implementation of mutation tests for foundry unit tests (I'd suggest closing the other PR which uses Gambit)

Motivation

While coverage explains which lines are executed when running a test suite, mutation tests add another level of security by assessing which lines are tested. To do so, mutations (ie random or constrained changes to the codebase) are introduced, the codebase is then recompiled (only keeping the successful compilation outcome) and the test run against this "mutated codebase". Ideally, the tests should fail against any mutant (ie not a single part of the code can be changed without getting caught by at least one test).

Solution

This implementation started from the existing Certora's Gambit, but diverged (but kept some of the mutation naming). It leverages Solar to create an ast, find mutation and conduct them, in an efficient manner.

The flow is as follow:

• triggered by --mutate flag of forge test
• if a list of contracts is passed after mutate, these are the targets mutated
• if no contract is provided, list all contracts in src/
• for now, mutate isn't excluding test filters (still not sure if it shouldn't be the case tho)
• run all tests, using cache (see below) - if one test fails, interrupt here
• do the following for each contract to mutate:
  • lex and parse it as an ast (using Solar)
  • visit this ast, for each expression, generate each possible mutation (eg x = 4 will be mutated as x = 0 and x = -4)
  • for each mutant:
    • create a temp dir
    • copy the cache, out, test and src (excluding the target contract)
    • emit the solidity code of the mutant in src
    • (try to) compile it, using the cache and out (this uses foundry-compiler, to be "compiler-agnostic" if/when Solar is used, even tho it should then be refactored to avoid writing solidity on the disk/starts from the mutated ast instead)
    • if successfully compiling, run the tests and get the outcome
• collect and display all outcomes, in terms of invalid, dead, surviving mutant <-- current status (for now, it's just some print)

The mutations are stored in a mutator module, to allow easily adding/modifying them (even if it is kept as temporary design, this will help designing them imo).

Beside the (many) todo's in the code, some things to work on are:
tests+++, doc, squash, etc + test shouldn't be copied in the temp folder (update filter instead? I tried unsuccessfully before), add a way to resume a mutation campaign, implement non-duplicate detection (trivial cases at least:compare bytecode hash?) - see the original issue, review other mutation strategies (vertigo and univeralmutator?) or poke the one not implemented by Gambit (https://github.com/Certora/gambit/blob/bf7ab3c91c47a10dcf272380b6406f0404f3b5d1/src/mutation.rs#L323 for instance)

PR Checklist

• Added Tests
• Added Documentation
• Breaking changes

[simon-something]

context: these are the mutators used by Gambit, in theory (in practice, some are not implemented, or not defined - see https://github.com/Certora/gambit/blob/bf7ab3c91c47a10dcf272380b6406f0404f3b5d1/src/mutation.rs#L323 for instance)

[grandizzy]

Hey @simon-something can you make the branch editable for code owners, so can rebase and other changes while reviewing? Thank you!

[simon-something]

Hey @simon-something can you make the branch editable for code owners, so can rebase and other changes while reviewing? Thank you!

Hi @grandizzy ! Sure, only issue is the fork is in the Wonderland github org, and if I'm correct, github doesn't allow external codeowner for org-owned repo (ie I don't have the usual tickbox in this PR) -> is it ok if I invite all codeowners to the repo instead? Alternative is re-opening a PR from my personal account

Sorry for the github-trouble!

[grandizzy]

Hey @simon-something can you make the branch editable for code owners, so can rebase and other changes while reviewing? Thank you!

Hi @grandizzy ! Sure, only issue is the fork is in the Wonderland github org, and if I'm correct, github doesn't allow external codeowner for org-owned repo (ie I don't have the usual tickbox in this PR) -> is it ok if I invite all codeowners to the repo instead? Alternative is re-opening a PR from my personal account

Sorry for the github-trouble!

no worries, opening from your personal account would be better if that works for you too. thanks!

[simon-something]

Superseded by #10193 (for codeowner access)