Skip to content

Test coverage

Aggregate line and branch coverage of jzap's own production code, measured with JaCoCo:

./gradlew coverage

coverageGate fails the build if it slips below a recorded floor, a little under what the suite achieves today. The report lands in build/reports/coverage/html.

Where it stands

before this work now target
line 71.33% 93.61% 95%
branch 57.29% 86.08% —

"Before" is the first honest measurement, not the first number JaCoCo printed. That one was 61.16%, and most of the difference was not jzap:

  • Fixtures are not the product. fixtures/ is the code jzap mutates — the input to the tool. It contributed 715 lines, 680 of them generated by the bench fixture, and it executes only inside the analysis JVMs jzap forks, so it read as 0% and moved the headline by fifteen points while saying nothing about whether jzap is tested.
  • Two things cannot be measured from the test JVM at all, and are excluded rather than counted as zero, which would put a false floor under every figure:
  • io.github.huyz0.jzap.minion exists only to run inside a forked JVM. MinionIntegrationTest drives a live one over the wire, so it is tested; it cannot be observed from here. Attaching a second JaCoCo agent to each minion would change the thing under test — the minion asserts it is dependency-free, and its class-redefinition path is exactly what another bytecode-rewriting agent interferes with.
  • JzapAgent and OverrideTransformer need a real Instrumentation, which only a JVM started with -javaagent has. The rest of io.github.huyz0.jzap.agent is ordinary static state and pure functions, and is measured: the module's own tests run with the shipped agent jar attached, so ClassOverrides is tested against real class retransformation.

Measurement is aggregated across modules on purpose. The tests that exercise most of jzap-core live in jzap-e2e, because what they assert is a whole analysis of a real fixture; per-module figures would report a fraction of what is actually covered.

What the remaining gap is made of

244 lines across 57 classes, almost all of it one to six lines at a time. Three groups:

  1. Fault-injection paths. catch blocks for a minion that dies mid-conversation, a schemata class that will not build, bytecode that cannot be generated for a key discovery returned, an interrupted analysis. Each is a handful of lines, and reaching them means making a component fail on demand — which needs a test-only seam in production code. The judgement here is that a seam whose only purpose is to be triggered by a test is worse than an untested catch block, so these are left.

  2. Paths needing an environment the build does not have. The 60-second connect timeout when a minion never calls back, the daemon's 30-minute idle exit, a second JDK for the javaHome branch, and a classpath packaged as jars rather than directories for the other half of RuntimeJars.locate.

  3. Defensive branches that are unreachable by construction. default -> arms over closed opcode sets, and null checks on values the callers cannot produce. Worth keeping and not worth contorting a test to reach.

Closing the last 1.4% would mean the first group, and the cost is production complexity rather than test effort. If it is wanted, the cheapest honest route is a package-private failure hook on MinionProcess that MutantExecutor's tests can arm — one seam, used by several tests, rather than one per path.

What raising it found

Writing these tests turned up seven real defects, which is the argument for the exercise over and above the number:

  • The daemon dropped most of its options. run --daemon forwarded a hand-written list that had fallen behind: --dry-run performed a real analysis, --include analysed everything, and --fail-on-survivors turned a failing build green. Rebuilt from picocli's parse result.
  • An unknown --reporters name exited 1, the same code as "score below threshold", after running every mutant. Now a usage error, before the work starts.
  • Scope.isClassGranularity() was serialised as a classGranularity field that ModelIo then warned about as unknown — jzap complaining about its own output.
  • Minion.requireHarness's message could never be delivered: it threw from a command handler, which killed the process, so the controller saw a truncated response instead.
  • GitScope's "no git repository found — use a patch file instead" was unreachable; JGit threw its own One of setGitDir or setWorkTree must be called first.
  • An integer overflow in the worker-count rule, reachable from a corrupted cache, which would have produced a negative thread-pool size.
  • ParallelExecutionTest had become vacuous: the adaptive worker cap collapsed its 8-thread request to one worker, so it asserted that verdicts agree across thread counts while every run was single-threaded.

Fixtures added for coverage that could not otherwise be reached

  • fixtures/parallel-java — deliberately slow tests, so the work justifies more than one analysis JVM and the concurrent execution path is actually exercised.
  • fixtures/blocking-java — a mutant that blocks rather than loops. The runaway-loop guard counts back edges, so blocking trips nothing; this is the one case the wall-clock backstop has to catch on its own, and it is the only place a jzap verdict depends on the clock.