Skip to content

Commit b78f3dd

Browse files
AlexDenisovAlexDenisov
AlexDenisov
authored and
AlexDenisov
committed
Update HelloWorld tutorial to use IR Frontend
1 parent 620ff3b commit b78f3dd

File tree

2 files changed

+114
-177
lines changed

2 files changed

+114
-177
lines changed

Diff for: ‎docs/tutorials/HelloWorld.rst

+110-176
Original file line numberDiff line numberDiff line change
@@ -9,29 +9,35 @@ mutants and generate mutation testing reports.
99
**TL;DR version**: if you want to run a single copy and paste example, scroll
1010
down to ``Killing mutants again, all killed`` below.
1111

12+
__Note: Clang 9 or newer is required!__
13+
1214
----
1315

1416
Step 1: Checking version
1517
------------------------
1618

17-
The tutorial assumes that you have `installed <Installation.html>`_ Mull on your system and
18-
have the `mull-cxx` executable available:
19+
Mull comes in a form of a compiler plugin and therefore tied to specific versions
20+
of Clang and LLVM. As a consequence of that, tools and plugins have a suffix with
21+
the actual Clang/LLVM version.
22+
23+
This tutorial assumes that you are using Clang 12 and that you have
24+
`installed <Installation.html>`_ Mull on your system and have the ``mull-runner-12``
25+
executable available:
1926

2027
.. code-block:: bash
2128
22-
$ mull-cxx -version
29+
$ mull-runner-12 -version
2330
Mull: LLVM-based mutation testing
2431
https://github.com/mull-project/mull
25-
Version: 0.8.0
26-
Commit: f94f38ed
27-
Date: 04 Jan 2021
28-
LLVM: 9.0.0
32+
Version: 0.15.0
33+
Commit: a4be349e
34+
Date: 18 Jan 2022
35+
LLVM: 12.0.1
2936
30-
Step 2: Enabling Bitcode
31-
------------------------
37+
Step 2: Enabling compiler plugin
38+
--------------------------------
3239

33-
The most important thing that Mull needs to know is the path to your program
34-
which must be a valid C or C++ executable. Let's create a C program:
40+
Let's create a C++ program:
3541

3642
.. code-block:: c
3743
@@ -43,61 +49,53 @@ and compile it:
4349

4450
.. code-block:: bash
4551
46-
$ clang main.cpp -o hello-world
52+
$ clang-12 main.cpp -o hello-world
4753
48-
We can already try running ``mull-cxx`` and see what happens:
54+
We can already try using ``mull-runner`` and see what happens:
4955

5056
.. code-block:: text
5157
52-
$ mull-cxx hello-world
53-
[info] Extracting bitcode from executable (threads: 1)
54-
[warning] No bitcode: x86_64
55-
[################################] 1/1. Finished in 3ms
56-
[info] Sanity check run (threads: 1)
57-
[################################] 1/1. Finished in 409ms
58-
[info] Gathering functions under test (threads: 1)
59-
[################################] 1/1. Finished in 0ms
58+
$ mull-runner-12 ./hello-world
59+
[info] Warm up run (threads: 1)
60+
[################################] 1/1. Finished in 5ms
61+
[info] Baseline run (threads: 1)
62+
[################################] 1/1. Finished in 4ms
6063
[info] No mutants found. Mutation score: infinitely high
61-
[info] Total execution time: 413ms
64+
[info] Total execution time: 10ms
65+
66+
Notice the ``No mutants found`` message! Now, Mull is ready to work with the executable
67+
but there are no mutants: we haven't compiled the program with the compiler plugin that embeds
68+
mutants into our executable.
6269

63-
Notice the ``No bitcode: x86_64`` warning! Now Mull is already trying to work
64-
with our executable but there is still one important detail that is missing: we
65-
haven't compiled the program with a special option that embeds LLVM bitcode
66-
into our executable.
70+
Let's fix that!
71+
To pass the plugin to Clang, you need to add a few compiler flags.
6772

68-
Mull works on a level of LLVM Bitcode relying on debug information to show
69-
results, therefore you should build your project with ``-fembed-bitcode`` and
70-
``-g`` flags enabled.
73+
.. code-block:: text
74+
75+
$ clang-12 -fexperimental-new-pass-manager \
76+
-fpass-plugin=/usr/local/lib/mull-ir-frontend-12 \
77+
-g -grecord-command-line \
78+
main.cpp -o hello-world
79+
[warning] Mull cannot find config (mull.yml). Using some defaults.
80+
81+
Notice the warning: Mull needs a config.
82+
However, in this tutorial we can ignore the warning and rely on the defaults.
7183

72-
Let's try again:
84+
You can learn more about the config `here <TODO>`_.
85+
86+
Let's run ``mull-runner`` again:
7387

7488
.. code-block:: text
7589
76-
$ clang -fembed-bitcode -g main.cpp -o hello-world
77-
$ mull-cxx hello-world
78-
[info] Extracting bitcode from executable (threads: 1)
79-
[################################] 1/1. Finished in 5ms
80-
[info] Loading bitcode files (threads: 1)
81-
[################################] 1/1. Finished in 11ms
82-
[info] Sanity check run (threads: 1)
83-
[################################] 1/1. Finished in 336ms
84-
[info] Gathering functions under test (threads: 1)
85-
[################################] 1/1. Finished in 1ms
86-
[info] Applying function filter: no debug info (threads: 1)
87-
[################################] 1/1. Finished in 10ms
88-
[info] Applying function filter: file path (threads: 1)
89-
[################################] 1/1. Finished in 10ms
90-
[info] Instruction selection (threads: 1)
91-
[################################] 1/1. Finished in 13ms
92-
[info] Searching mutants across functions (threads: 1)
93-
[################################] 1/1. Finished in 11ms
90+
$ mull-runner-12 ./hello-world
91+
[info] Warm up run (threads: 1)
92+
[################################] 1/1. Finished in 4ms
93+
[info] Baseline run (threads: 1)
94+
[################################] 1/1. Finished in 6ms
9495
[info] No mutants found. Mutation score: infinitely high
95-
[info] Total execution time: 400ms
96+
[info] Total execution time: 12ms
9697
97-
The ``No bitcode: x86_64`` warning has gone and now we can focus on another
98-
important part of the output: ``No mutants found. Mutation score: infinitely
99-
high``. We have our executable but we don't have any code so there is nothing
100-
Mull could work on.
98+
Still no mutants, but this time it is because we don't have any code Mull can mutate.
10199

102100
Step 3: Killing mutants, one survived
103101
-------------------------------------
@@ -114,13 +112,13 @@ Let's add some code:
114112
}
115113
116114
int main() {
117-
int test1 = valid_age(25) == true;
115+
bool test1 = valid_age(25) == true;
118116
if (!test1) {
119117
/// test failed
120118
return 1;
121119
}
122120
123-
int test2 = valid_age(20) == false;
121+
bool test2 = valid_age(20) == false;
124122
if (!test2) {
125123
/// test failed
126124
return 1;
@@ -130,74 +128,41 @@ Let's add some code:
130128
return 0;
131129
}
132130
133-
We compile this new code using the bitcode flags and run the Mull again. This
134-
time we also want to add additional flag ``-ide-reporter-show-killed`` which
131+
We re-compile this new code using the plugin and run the Mull again. This
132+
time we also want to add an additional flag ``-ide-reporter-show-killed`` which
135133
tells Mull to print killed mutations. Normally we are not interested in seeing
136-
killed mutations in console output but in this tutorial we want to be more
134+
killed mutants in console output but in this tutorial we want to be more
137135
verbose.
138136

139137
.. code-block:: text
140138
141-
$ clang -fembed-bitcode -g main.cpp -o hello-world
142-
$ mull-cxx -ide-reporter-show-killed hello-world
143-
[info] Extracting bitcode from executable (threads: 1)
144-
[################################] 1/1. Finished in 6ms
145-
[info] Loading bitcode files (threads: 1)
146-
[################################] 1/1. Finished in 11ms
147-
[info] Sanity check run (threads: 1)
148-
[################################] 1/1. Finished in 341ms
149-
[info] Gathering functions under test (threads: 1)
150-
[################################] 1/1. Finished in 0ms
151-
[info] Applying function filter: no debug info (threads: 3)
152-
[################################] 3/3. Finished in 0ms
153-
[info] Applying function filter: file path (threads: 2)
154-
[################################] 2/2. Finished in 0ms
155-
[info] Instruction selection (threads: 2)
156-
[################################] 2/2. Finished in 11ms
157-
[info] Searching mutants across functions (threads: 2)
158-
[################################] 2/2. Finished in 10ms
159-
[info] Applying filter: no debug info (threads: 6)
160-
[################################] 6/6. Finished in 1ms
161-
[info] Applying filter: file path (threads: 6)
162-
[################################] 6/6. Finished in 0ms
163-
[info] Applying filter: junk (threads: 6)
164-
[################################] 6/6. Finished in 11ms
165-
[info] Prepare mutations (threads: 1)
166-
[################################] 1/1. Finished in 0ms
167-
[info] Cloning functions for mutation (threads: 1)
168-
[################################] 1/1. Finished in 11ms
169-
[info] Removing original functions (threads: 1)
170-
[################################] 1/1. Finished in 10ms
171-
[info] Redirect mutated functions (threads: 1)
172-
[################################] 1/1. Finished in 10ms
173-
[info] Applying mutations (threads: 1)
174-
[################################] 4/4. Finished in 12ms
175-
[info] Compiling original code (threads: 1)
176-
[################################] 1/1. Finished in 11ms
177-
[info] Link mutated program (threads: 1)
178-
[################################] 1/1. Finished in 109ms
139+
$ clang-12 -fexperimental-new-pass-manager \
140+
-fpass-plugin=/usr/local/lib/mull-ir-frontend-12 \
141+
-g -grecord-command-line \
142+
main.cpp -o hello-world
143+
$ mull-runner-12 -ide-reporter-show-killed hello-world
179144
[info] Warm up run (threads: 1)
180-
[################################] 1/1. Finished in 360ms
145+
[################################] 1/1. Finished in 151ms
181146
[info] Baseline run (threads: 1)
182-
[################################] 1/1. Finished in 18ms
147+
[################################] 1/1. Finished in 3ms
183148
[info] Running mutants (threads: 4)
184-
[################################] 4/4. Finished in 63ms
149+
[################################] 4/4. Finished in 10ms
185150
[info] Killed mutants (3/4):
186-
/tmp/sc-PzmaCNIRu/main.cpp:2:15: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
187-
if (age >= 21) {
188-
^
189-
/tmp/sc-PzmaCNIRu/main.cpp:9:33: warning: Killed: Replaced == with != [cxx_eq_to_ne]
190-
int test1 = valid_age(25) == true;
191-
^
192-
/tmp/sc-PzmaCNIRu/main.cpp:15:33: warning: Killed: Replaced == with != [cxx_eq_to_ne]
193-
int test2 = valid_age(20) == false;
194-
^
151+
/tmp/sc-tTV8a84lL/main.cpp:2:11: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
152+
if (age >= 21) {
153+
^
154+
/tmp/sc-tTV8a84lL/main.cpp:9:30: warning: Killed: Replaced == with != [cxx_eq_to_ne]
155+
bool test1 = valid_age(25) == true;
156+
^
157+
/tmp/sc-tTV8a84lL/main.cpp:15:30: warning: Killed: Replaced == with != [cxx_eq_to_ne]
158+
bool test2 = valid_age(20) == false;
159+
^
195160
[info] Survived mutants (1/4):
196-
/tmp/sc-PzmaCNIRu/main.cpp:2:15: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
197-
if (age >= 21) {
198-
^
161+
/tmp/sc-tTV8a84lL/main.cpp:2:11: warning: Survived: Replaced >= with > [cxx_ge_to_gt]
162+
if (age >= 21) {
163+
^
199164
[info] Mutation score: 75%
200-
[info] Total execution time: 996ms
165+
[info] Total execution time: 167ms
201166
202167
What we are seeing now is four mutations: three mutations are ``Killed``, another
203168
one is ``Survived``. If we take a closer look at the code and the contents
@@ -247,84 +212,53 @@ The code:
247212
248213
.. code-block:: text
249214
250-
$ clang -fembed-bitcode -g main.cpp -o hello-world
251-
$ mull-cxx -ide-reporter-show-killed hello-world
252-
[info] Extracting bitcode from executable (threads: 1)
253-
[################################] 1/1. Finished in 4ms
254-
[info] Loading bitcode files (threads: 1)
255-
[################################] 1/1. Finished in 11ms
256-
[info] Sanity check run (threads: 1)
257-
[################################] 1/1. Finished in 7ms
258-
[info] Gathering functions under test (threads: 1)
259-
[################################] 1/1. Finished in 0ms
260-
[info] Applying function filter: no debug info (threads: 3)
261-
[################################] 3/3. Finished in 0ms
262-
[info] Applying function filter: file path (threads: 2)
263-
[################################] 2/2. Finished in 0ms
264-
[info] Instruction selection (threads: 2)
265-
[################################] 2/2. Finished in 12ms
266-
[info] Searching mutants across functions (threads: 2)
267-
[################################] 2/2. Finished in 10ms
268-
[info] Applying filter: no debug info (threads: 5)
269-
[################################] 5/5. Finished in 0ms
270-
[info] Applying filter: file path (threads: 5)
271-
[################################] 5/5. Finished in 1ms
272-
[info] Applying filter: junk (threads: 5)
273-
[################################] 5/5. Finished in 12ms
274-
[info] Prepare mutations (threads: 1)
275-
[################################] 1/1. Finished in 0ms
276-
[info] Cloning functions for mutation (threads: 1)
277-
[################################] 1/1. Finished in 10ms
278-
[info] Removing original functions (threads: 1)
279-
[################################] 1/1. Finished in 11ms
280-
[info] Redirect mutated functions (threads: 1)
281-
[################################] 1/1. Finished in 10ms
282-
[info] Applying mutations (threads: 1)
283-
[################################] 5/5. Finished in 0ms
284-
[info] Compiling original code (threads: 1)
285-
[################################] 1/1. Finished in 11ms
286-
[info] Link mutated program (threads: 1)
287-
[################################] 1/1. Finished in 62ms
215+
$ clang-12 -fexperimental-new-pass-manager \
216+
-fpass-plugin=/usr/local/lib/mull-ir-frontend-12 \
217+
-g -grecord-command-line \
218+
main.cpp -o hello-world
219+
$ mull-runner-12 -ide-reporter-show-killed hello-world
288220
[info] Warm up run (threads: 1)
289-
[################################] 1/1. Finished in 311ms
221+
[################################] 1/1. Finished in 469ms
290222
[info] Baseline run (threads: 1)
291-
[################################] 1/1. Finished in 19ms
223+
[################################] 1/1. Finished in 4ms
292224
[info] Running mutants (threads: 5)
293-
[################################] 5/5. Finished in 63ms
225+
[################################] 5/5. Finished in 12ms
294226
[info] Killed mutants (5/5):
295-
/tmp/sc-PzmaCNIRu/main.cpp:2:15: warning: Killed: Replaced >= with > [cxx_ge_to_gt]
296-
if (age >= 21) {
297-
^
298-
/tmp/sc-PzmaCNIRu/main.cpp:2:15: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
299-
if (age >= 21) {
300-
^
301-
/tmp/sc-PzmaCNIRu/main.cpp:9:34: warning: Killed: Replaced == with != [cxx_eq_to_ne]
302-
bool test1 = valid_age(25) == true;
303-
^
304-
/tmp/sc-PzmaCNIRu/main.cpp:15:34: warning: Killed: Replaced == with != [cxx_eq_to_ne]
305-
bool test2 = valid_age(20) == false;
306-
^
307-
/tmp/sc-PzmaCNIRu/main.cpp:21:34: warning: Killed: Replaced == with != [cxx_eq_to_ne]
308-
bool test3 = valid_age(21) == true;
309-
^
227+
/tmp/sc-tTV8a84lL/main.cpp:2:11: warning: Killed: Replaced >= with > [cxx_ge_to_gt]
228+
if (age >= 21) {
229+
^
230+
/tmp/sc-tTV8a84lL/main.cpp:2:11: warning: Killed: Replaced >= with < [cxx_ge_to_lt]
231+
if (age >= 21) {
232+
^
233+
/tmp/sc-tTV8a84lL/main.cpp:9:30: warning: Killed: Replaced == with != [cxx_eq_to_ne]
234+
bool test1 = valid_age(25) == true;
235+
^
236+
/tmp/sc-tTV8a84lL/main.cpp:15:30: warning: Killed: Replaced == with != [cxx_eq_to_ne]
237+
bool test2 = valid_age(20) == false;
238+
^
239+
/tmp/sc-tTV8a84lL/main.cpp:21:30: warning: Killed: Replaced == with != [cxx_eq_to_ne]
240+
bool test3 = valid_age(21) == true;
241+
^
310242
[info] All mutations have been killed
311243
[info] Mutation score: 100%
312-
[info] Total execution time: 554ms
244+
[info] Total execution time: 487ms
313245
314246
In this last run, we see that all mutants were killed since we covered with tests
315247
all cases around the ``<=``.
316248

317249
Summary
318250
-------
319251

320-
This is a short summary of what we have learned in the tutorial.
321-
Your code has to be compiled with ``-fembed-bitcode -g`` compile flags:
252+
As a summary, all you need to enable Mull is to add a few compiler flags to the
253+
build system and then run ``mull-runner`` against the resulting executable.
254+
Just to recap:
322255

323-
- Mull expects embedded bitcode files to be present in a binary executable
324-
(ensured by ``-fembed-bitcode``).
256+
.. code-block:: text
325257
326-
- Mull needs debug information to be included by the compiler (enabled by
327-
``-g``). Mull uses this information to find mutations in bitcode and source
328-
code.
258+
$ clang-12 -fexperimental-new-pass-manager \
259+
-fpass-plugin=/usr/local/lib/mull-ir-frontend-12 \
260+
-g -grecord-command-line \
261+
main.cpp -o hello-world
262+
$ mull-runner-12 hello-world
329263
330264
The next step is to learn about `Compilation Database and Junk Mutations <CompilationDatabaseAndJunk.html>`_

Diff for: ‎lib/Driver.cpp

+4-1
Original file line numberDiff line numberDiff line change
@@ -394,7 +394,6 @@ void mull::mutateBitcode(llvm::Module &module) {
394394
if (configPath.empty()) {
395395
diagnostics.warning("Mull cannot find config (mull.yml). Using some defaults.");
396396
} else {
397-
diagnostics.info("Using configuration "s + configPath);
398397
configuration = Configuration::loadFromDisk(diagnostics, configPath);
399398
}
400399
configuration.parallelization.normalize();
@@ -409,6 +408,10 @@ void mull::mutateBitcode(llvm::Module &module) {
409408
diagnostics.makeSilent();
410409
}
411410

411+
if (!configPath.empty()) {
412+
diagnostics.info("Using configuration "s + configPath);
413+
}
414+
412415
std::vector<std::unique_ptr<mull::Filter>> filterStorage;
413416
mull::Filters filters;
414417

0 commit comments

Comments
 (0)