Mutation Schemata

[rakaramos]

Mutation Schemata

• Authors: Rafael Machado, Zev Eisenberg
• Review Manager: TBD
• Status: Awaiting implementation or Awaiting review
• Vision: if applicable Vision Name
• Roadmap: if applicable Roadmap Name)
• Bug: if applicable apple/swift#NNNNN
• Implementation: apple/swift#NNNNN or apple/swift-evolution-staging#NNNNN

Introduction

Mutation testing introduces bugs into a copy of your code and then runs your test suite to see how good it is at testing your code. However, it can be slow to rebuild the project every time a bug is introduced.

With mutation schemata, all bugs are introduced at the same time, but with flags that allow them to be enabled and disabled at run-time via environment variables. This allows for rapid re-runs of your tests with different bugs enabled. The more potential bugs muter can add to your project, the more time mutation schemata will save you.

Motivation

Mutation schemata is a well-known technique to improve the mutation testing performance.
Although it may sound weird at first, because this is an established term that is being used for years, we are going along with it.
The idea behind it is to encode all of the operators at once (known as program schemata) into a simple compiled binary (known as metaprogram) and use the runtime to enable/disable operators.

Proposed solution

For Swift projects (iOS/macOS) it is possible to have the build system (xcodebuild or swift) running tests without recompiling the app, enabling us to have the metaprogram.

Detailed design

Environment variables

Environment variables are user-definable values that can affect how a running process behaves and are part of the environment in which a process runs.
In the iOS/macOS world, we have an abstraction for that, the ProcessInfo's environment dictionary.
By using it, we can have an implementation that can change, just by manipulating the variables, for instance:
Say we have a Remove Side Effects operator that needs to be applied on a mutation test round:

Original source code:

func foo() {
  bar()
}

Mutated source code:

func foo() {
}

Using ProcessInfo, the mutated source code would look like this:

func foo() {
  ProcessInfo.processInfo.environment["operator_unique_id"] {
    bar()
  }
}

Now, we can turn the operator on and off at will.
For the operator_unique_id, we use the original source code line, column, and the utf8 offset values plus the operator id. This gives us the uniqueness we are looking for.

Schemata mappings

TODO:
Describe the steps and the new data structures like MutationSchema, [CodeBlockItemListSyntax: MutationSchemata] (a mapping), and the SchemataMutationMapping class.

Source compatibility

• limitation to @resultBuilder due to the return statement that the if statement ProcessInfo.processInfo.environment introduces to the source code

ABI compatibility

None?

Implications on adoption

• Requires to ignore SwiftUI code or any other code that uses the @resultBuilders (muter:enable, muter:disable may be helpful)

Future directions

• Partial execution with schemata plan to have it run in parallel or other machines

Alternatives considered

• Nothing

Acknowledgments