During development Graphite2 is regularly checked against its test suite of over a 100 test cases. This happens automatically on every check-in to the default branch thanks to our continuous build server for Windows and Linux. Prior to each major release it’s tested using an automated fuzzing system, which checks for robustness in the face of corrupted font files. This set of fuzz tests uses valgrind to check for rogue memory accesses and runs several thousand tests on each of the 4 major test fonts and takes considerably longer to run. We also have a growing suite of fuzz test regressions culled from logs generated by the above fuzz test system, which can be run with a make command. These are intended to be run before bug fix release and other point releases.
The standard test suite, the same one run on every checkin to the graphite2 project repository, can be run with a simple:
make test
make fuzztest
These should be run before point and bug fix releases.
full-fuzz-test.sh script [fuzztest options]
This script exercises graphite over 4 scripts Myanmar, Devangari, extended Latin
and Arabic using 4 fonts and text in the Myanmar, Nepalese, Yoroba and Arabic
languages. It uses the fuzzcomparerender script to fuzz every byte of each
font with a random value and enforce generous enough runtime and memory resource
limits to detect infinite loops or memory leaks. A successfull run of this
script will produce four empty log files. Passing --valgrind to the script
this is passed down to the fuzztest program which will run the test program
with valgrind, this increases the runtime of script considerably. Normally the
script can run all four tests within 24 hours, fully loading our 4 core
hyperthreaded Xeon/Core i7 system. Using the valgrind option this takes
approximately a week on the same system.
tests/fuzzcomparerender <font name> <text name> [-r] [fuzztest options]
The font name must be the basename of a font file under tests/fonts and the
text name must be the basename of a text file from tests/texts. The -r
option if present is passed to comparerenderer and tells it the text is
right-to-left. Any further options are passed to the fuzztest program, this is
typically one of --valgrind or --input, or less frequently --passes,
--jobs or --status. See tests/fuzztest --help for more information. This
script runs fuzztest so that it corrupts every byte of the required TrueType
and Graphite specific tables with a random value, but excludes OpenType and AAT
related tables. It also imposes a runtime limit of 10 seconds (most test should
compete in a fraction of a second) and a memory limit 200MiB, again a normal run
should only use a tiny fraction of that. If the comparerenderer test
segfaults, exceeds those limits or returns an error value it is logged to a file
named on the following pattern: fuzzfont-<font name>-<text name>.log, which is
written to the script’s current directory.
tests/fuzztest --font=font [options] -- <test harness>
A multiprocess fuzz test framework that generates corrupt fonts to run a user
supplied test harness against. This will check each byte in every table of a
TTF or subset of it’s tables if specified, by overwriting with a random or user
specified value. Using the --input option it can also re-run tests using
previous output logs as input, it will ignore any line that doesn’t match the
format of a fuzz line generated by the program, so such input fuzzes can be well
annotated. It is this facility that is used to drive the make fuzztest
regression check.
By default this will try to ensure there is always one test harness running on
each OS reported core at all times, the --jobs option can be used to limit
this if need to limit the load. Unless told otherwise the program will display a
status line inidcating the percentage of the tests run so far, the rate of
testing and an estimated time and date for completion. Once the status has
updated 4-5 times the estimate usually settles down to a frequently accurate
estimate.
Fuzz regression test files are simply copies of the log generated by the
fuzzcomparerender script. Once you have a log that generates a test case, you
can edit it to produce a more targeted set of fuzz lines (a log can generate
several different bugs). You should try to produce a minimal set of fuzz lines
for that particular test case, however the more fuzz lines that casue the same
bug the better.
The test case should be placed as follows:
tests/fuzz-tests/<font name>/<text name>/<bug description>.fuzz
where:
- font name
-
The name of a font, minus the .ttf extension from tests/fonts.
- text name
-
The name of a text file, minus .txt extension from tests/texts.
- bug description
-
A short description of the bug, this cannot contain any ":" characters or other characters forbidden by Windows or Unix filename schemes. For
class::memberreferences use an_e.g.class_member.
We now build a number of Clang libFuzzer test targets in
<src>/tests/fuzz-tests whenever fuzzer is one of the sanitizers found in
GRAPHITE2_SANITIZERS. There are a series of seed inputs in the
<src>/tests/fuzz-tests/libfuzzer-corpus directory each one coresponding to
one of the existing unit tests pairing a font with text and parameters. These
are binary files which consist of:
| Length | Input parameter |
|---|---|
<variable> |
TrueType file of font |
utint16 * N |
Feature values (N = number of feature ids in the font) |
uint32_t |
Text Script ID |
uint8_t |
UTF encoding form uint size (1,2 or 4) see |
uint32_t |
Text directionality see |
uint_8_t[128] |
Text in the UTF encoding form specified |
Run as follows:
cd <src>/build cp ../tests/fuzz-tests/libfuzz-corpus corpus tests/fuzz-tests/gr-fuzzer-font corpus
To rerun a test just do:
tests/fuzz-tests/gr-fuzz-font crash-test-case-produced-by-fuzzer
See the libFuzzer documentation for more advanced options standard to all libFuzzer produced fuzzing targets.