- Generated by
1.15.0
|
ZeroErr
|
Build an optional instrumented configuration to find which production code each top-level TEST_CASE executes. Normal builds have no LLVM runtime dependency.
Keep existing TEST_CASEs. Configure once using Clang and -DZEROERR_ENABLE_COVERAGE=ON, then run:
This runs the tests, collects profiles, analyzes them, and prints coverage percentages, the ten slowest tests, and identical-coverage candidate groups. Each run gets a new zeroerr-coverage/run-<id> directory under the working directory, containing report.json, report.summary.txt, and raw profiles. No manual LLVM or Python commands are needed. Tests that fail still produce a report and return failure; infrastructure/report failures return 2. --list-test-cases only lists.
CMake finds Python 3.9+ and matching LLVM tools at configuration time. The default scope is the consuming project's source root, excluding executed test source files, zeroerr framework sources/headers, and third_party/_deps. If production code shares a file with tests, explicitly select that file. Explicit includes replace automatic exclusions; additional excludes always apply:
For a reusable CMake target in an application project:
Run cmake --build build --target my_tests-coverage to build and report in one step. The helper also accepts INCLUDE "regex", EXCLUDE "regex" ..., and ARGS "--testcase=parser.*" .... For zeroerr's own suite the target is unittest-coverage; it explicitly includes zeroerr itself as the production code.
Try the complete standalone example from the repository root:
The example instruments a separate library, covers all outcomes of sign, and identifies two positive-input tests with identical measured coverage.
For a manually built/single-header executable, place scripts/coverage-report.py beside it as zeroerr-coverage.py (or set ZEROERR_COVERAGE_SCRIPT to its path). Python 3 and matching llvm-cov/llvm-profdata must be on PATH; set ZEROERR_COVERAGE_PYTHON if Python has a different name. The default source root in this mode is the current directory. A standalone launcher is also available: python coverage-report.py --run ./my_tests --source-root . -- --testcase='parser.*'. Pass --llvm-cov/--llvm-profdata to that launcher for versioned tool names. The helper is an external dependency and must be deployed with a relocated binary; its CMake-configured source path is used while the checkout is available.
Use llvm-profdata and llvm-cov from the compiler's toolchain. Versioned paths can be passed with --llvm-profdata /path/to/llvm-profdata-21 and --llvm-cov /path/to/llvm-cov-21. The Python analyzer needs Python 3.9+ and accepts LLVM export schema versions 2 and 3, with branch information present. The backend has been exercised with Clang/LLVM 21 on Linux; other platforms need validation.
For application projects linking the zeroerr CMake target, coverage compile/link flags propagate to the test target. Apply zeroerr_instrument_coverage(my_library) to each separately compiled production library that should be measured. Previously compiled, uninstrumented libraries cannot be measured. The helper must be called after adding zeroerr's subdirectory. Instrumented builds use atomic profile updates so worker threads do not lose increments; all workers must finish before the test returns. This backend targets a single executable with statically linked code; separately instrumented shared-library runtimes are not collected.
For the single-header distribution, define ZEROERR_ENABLE_COVERAGE in the implementation translation unit, compile measured code with -fprofile-instr-generate -fcoverage-mapping -fprofile-update=atomic, and link with -fprofile-instr-generate. Coverage mode requires C++17 (std::filesystem).
--coverage-output requires a new directory for every run; existing directories are rejected without changing their contents. Existing test filters still apply. Listing tests does not create output. Skipped tests have no profile. Failed tests and caught exceptions have profiles and retain their test result. Report generation is a separate step, so it can be run even when the test process returns failure.
The runner resets LLVM counters before each top-level test and snapshots them using the LLVM profile buffer API after the test. Nested SUB_CASEs, fixture construction and destruction are included in their parent's profile. Initialization before test execution is excluded. Framework code executed inside the boundary may be present; use a production-only source scope to exclude it. No per-subcase attribution is claimed. Do not run tests concurrently in one process or leave background work running across boundaries. Forked children and subprocess coverage are not collected.
The directory contains manifest.jsonl (name, source location, elapsed seconds, result, profile filename) and one .profraw per executed test. A completion marker distinguishes complete runs from interrupted/crashed runs. Collection I/O failures return exit code 2. Uninstrumented builds reject the coverage option with exit code
In offline mode, --include is a required regex over normalized forward-slash source paths. Repeat --exclude REGEX to narrow the scope. Exclude test implementations, assertion helpers, and third-party code when comparing production coverage. An empty matched scope, incomplete manifest, LLVM diagnostics, or missing branch data is an error.
The JSON report contains source feature dictionaries and compact per-test feature IDs for three independent metrics:
Each metric has covered/total counts and per-test unique coverage counts, relative to the tests collected in this run. Denominators come from selected coverage mappings in the executable, including mapped code not hit by any test; code absent from the binary is not counted. Lines, conditions, MC/DC and paths are explicitly marked as uncollected. The report also preserves instrumented test execution time (excluding profile serialization), which is not a normal-build performance benchmark.
identical_coverage_candidates groups successful tests whose feature sets match across all three metrics. Empty coverage is not grouped. Unique counts consider all collected tests, including failing ones. Matching coverage does not establish equivalent inputs, assertions, or defect detection; no tests are deleted or merged. Grouping uses coverage signatures instead of an all-pairs comparison.
Analyze with the exact executable used for collection and keep profiles with that build. The report records the analyzed executable's SHA-256, but cannot prove it is the original executable if it was replaced before analysis. Feature IDs are local to a report, not persistent IDs across builds. Archive the binary alongside profiles if analysis will be delayed. Raw profiles require space proportional to test count times instrumented binary profile size; analysis currently invokes LLVM tools per test. Use this as an opt-in diagnostic run, not on every edit.
1.15.0