Code coverage analysis¶

undo coverage extracts line coverage information from an Undo recording and presents it in various ways.

Warning

undo coverage is experimental. It may be withdrawn, or changed in incompatible ways, in a future release. It is not listed by undo --help and has no man page.

Added in version 10.2.

The key advantage compared to other approaches is that there is no need to maintain a separate coverage-enabled build; instead coverage can be derived from tests run on the standard production build of your application.

There are some caveats and limitations:

Only line coverage is supported.
The recorded program must have been built with debug information. See Debugging Information for how it is found, and --symbols for naming the files that hold it.
The count of times a line has been executed is always reported as 0 or 1.
undo coverage works by analyzing each machine instruction that was executed, and assigning it to one or more source lines. When run on highly optimized code, the results may be inaccurate, for instance if a block of code is completely optimized away because the compiler can determine that it would have no effect.

Usage synopsis¶

undo coverage [GENERAL-OPTIONS ...] COMMAND [COMMAND-OPTIONS ...]

Recordings may be created by the live-record tool, as described in Using the LiveRecorder tool; by the LiveRecorder API, as described in Using the LiveRecorder API; or by the usave command in UDB.

For example, to record a test suite and turn the recordings into an HTML report:

$ live-record --recording-dir recordings ./run-tests
$ undo coverage lcov -j 8 -o coverage.info recordings
$ genhtml -o coverage-html coverage.info

genhtml generates reports in HTML format from coverage data in LCOV format. It is part of the LCOV package, which can be installed with dnf install lcov on Fedora or Red Hat, or apt install lcov on Ubuntu.

General options¶

These options apply to undo coverage itself, so they come before the command.

--help, -h¶: Print usage information.

--version, -v¶: Print the version.

The remaining general options run extra UDB commands before coverage analysis is performed. They behave the same as the corresponding UDB options.

--command FILE, -x FILE¶: See the udb --command option.

--eval-command COMMAND, -ex COMMAND¶: See the udb --eval-command option.

--init-command FILE, -ix FILE¶: See the udb --init-command option.

--init-eval-command COMMAND, -iex COMMAND¶: See the udb --init-eval-command option.

Options for all commands¶

--symbols SYMBOLS¶

Load symbols from SYMBOLS, which is either the name of a file containing symbols, or a shell wildcard pattern matching such files. The option can be given more than once.

For example, the following command loads every file under the .debug directory with the extension .debug, including those in subdirectories.

$ undo coverage lcov --symbols '.debug/**/*.debug' recordings

Arguments passed to this option may require quoting to avoid expansion by the shell.

undo coverage lcov¶

undo coverage lcov [OPTIONS ...] RECORDING|DIRECTORY

undo coverage lcov writes coverage information as an LCOV-compatible tracefile, suitable for processing with genhtml.

The argument names either a single recording, which is used whatever it is called, or a directory to search, in which case the files named *.undo in it are treated as recordings.

The tracefile covers the source files reached by the recordings, provided they are present on the machine running undo coverage. It excludes files under /usr/src/debug, where debuginfo packages install the sources of system components. Each file left out is named on standard error.

A source file that has moved since the recording was made can still be found by giving UDB the usual command for it with undo coverage --init-eval-command:

$ undo coverage -iex 'set substitute-path /build/src /home/user/src' lcov recordings

The tracefile then names the file where it was found, rather than where it was built.

--exclude EXCLUDE¶: Ignore recording files whose path matches EXCLUDE. Unlike lcov’s option of the same name, this matches the recordings to process rather than the source files they cover. EXCLUDE is a shell wildcard pattern matched against the whole path, in which * also matches / and ** has no special meaning.

--include INCLUDE¶: Only process recording files whose path matches INCLUDE, which is interpreted as for --exclude.

--jobs JOBS, -j JOBS¶

Process up to JOBS recordings at a time, using one UDB session for each recording. The default is 1, and the number of sessions is capped at the number of CPUs.

Each udb session needs a license, so JOBS should be no larger than the number of concurrent sessions your license allows.

--no-recursion¶: Do not descend into subdirectories when searching for recording files.

--output-file FILE, -o FILE¶: Write the tracefile to FILE. Without this option, or with - as FILE, the tracefile is written to standard output.

--test-name TEST_NAME, -t TEST_NAME¶: Store TEST_NAME as the test name in the tracefile.

undo coverage report¶

undo coverage report [--symbols SYMBOLS] RESULT_FILE

undo coverage report produces overall information about a test run: a total coverage metric, a grouping of the failing tests by their coverage patterns, the source lines covered by every failing test and by no passing test, and the tests that contributed no coverage of their own.

RESULT_FILE must be a file in JSON format containing a single object whose keys are the names of the tests. Each value is an object with two keys: recording, the name of a recording of that test, and success, which is true if the test passed and false if it failed. For example:

{
    "passing_test": {
        "recording": "passing_test.undo",
        "success": true
    },
    "failing_test": {
        "recording": "failing_test.undo",
        "success": false
    }
}

For example, running two tests over a program whose negated function is reached only by the failing one:

Lines present:
/home/user/prog/prog.c: 5-7,10-12,15-19,21-23
...
Lines covered:
/home/user/prog/prog.c: 5-7,10-12,15-19,21-23
...
Line coverage = 1177/8340 (14.112709832134293%)

These source lines were covered by all failing tests but no passing tests:
/home/user/prog/prog.c: 10-12,21-22

Failure clusters:

The lists of present and covered lines name every source file the tests reached, including the C library’s own, so most of their output is elided above.