beyond-test-coverage

beyond-test-coverage

A Claude Code skill that hardens test quality, not coverage — packaged with the cross-language benchmark that built and validated it. The skill (test-quality) audits, hardens, or generates unit tests against a multi-axis quality scorecard. Everything else in this repo is the experiment that derived that scorecard and proved the approach beats human-written suites across nine libraries in three languages. Install it →

Are LLM-generated tests better than human-written tests?

We need to put numbers to it. Code coverage is a vanity metric. 100%-covered suites are routinely brittle, over-mocked, and noisy; coverage tells you a line ran, not that a test would catch the regression that breaks it.

beyond-test-coverage measures the thing coverage can't: test quality. Point it at a repository and it regenerates the test suite from the source through a read → build → evaluate loop, optimizing for the qualities that make a suite actually load-bearing:

• Anti-fragility — tests that survive a benign refactor. No asserting on error-message substrings, no recomputing the expected value with the same code under test, no reaching into private internals, no ||-joined "passes if any of these" assertions.
• Rigor — assert against known-good fixed values (precomputed vectors, inline snapshots), cover the boundaries, and exercise real behavior end-to-end rather than re-deriving it.
• Mocking discipline — zero hand-rolled mocks of the unit under test. Drive the real thing through framework primitives (supertest, httptest, fake timers) or real objects; mock only true external boundaries.
• Reuse & efficiency — fold repetition into table-driven / parametrized cases instead of copy-paste; fewer lines per test.
• Correctness — every test passes, and every non-obvious assumption is verified against the real library, not guessed.
• Coverage as a floor, not a target — never regress it, never chase it.

An evaluate function (the test-quality scorer) measures each of these axes on the generated suite and on the original, and reports a win/loss/tie per axis. If the suite hasn't beaten the baseline — or isn't green — the loop reads, rebuilds, and re-evaluates. Through that loop the suite improves not just in coverage but in quality: the result is a test suite that is stronger, more robust, and less noisy than the surface it replaced.

Does it actually work?

To find out, we took a set of widely-used open-source libraries with exceptional, maintainer-written test suites, measured their baseline on every axis, deleted every test, and regenerated the suite from the source under increasing iteration budgets (a single pass, up to iterate-until-it-stops-improving) — scoring each generated suite against the human original. Every round's findings fed back into the prompts and the scorer, hardening the tool with each pass.

Nine libraries, three languages, the strongest test suites we could find:

Results

The headline is the gap between chasing coverage and chasing quality. When the loop was pointed at coverage, the regenerated suites beat the human baseline on only 2 of 9 runs — they hit coverage parity and stopped. Re-aimed at the quality scorecard, they won every run:

Regenerated suites that beat the human-written test suite
──────────────────────────────────────────────────────────
Coverage-driven   (Python)   ██░░░░░░░░   2 / 9     22%
Quality-driven    (Python)   ██████████   9 / 9    100%
Quality-driven    (JS / TS)  ██████████   9 / 9    100%
Quality-driven    (Go)       ██████████   9 / 9    100%

Across 27 quality-driven runs (9 libraries × 3 iteration budgets), every one beat its human-written baseline on the auto-countable axes — 9/9 on Python, 18/18 across JS/TS and Go. The wins are consistent and structural: the generated suites are dramatically more LOC-efficient than every baseline, carry zero substring-match assertions and no hand mocks of the code under test, cut private-symbol access sharply (the coverage-driven arms' 185 total fell to 59 under quality-driven across the Python matrix — though not to zero), and are more parametrized than most baselines.

Three honest caveats the numbers also surfaced:

• A single pass already wins the quality axes for every library — but it doesn't ship clean: every JS/Go oneshot suite carried 1–4 self-authored failing tests, and all three Kotlin/Swift oneshot reds failed to compile. The iteration budget is what makes the suite green and widens the margin. Quality and shippability are different milestones.
• Coverage held or rose in 25 of 27 runs. The two exceptions are both Python oneshot arms (httpx −7.0 pp line, requests −3.2 pp); their iterated arms recovered the floor and then some.
• The axis the generated suites most often lose is fixed-vector count, because we score it as an absolute count that scales with suite size and competes with LOC-efficiency. (Re-shaping it into a per-test ratio is the next change — see CHANGELOG.md.) On Python, a few arms also lost an individual axis outright — one iter20 arm regressed private-access (A.2 14 v 10), three arms lost parametrize-ratio — while still winning their overall tally.

Explore the numbers yourself:

• docs/cross-language-results.html — interactive dashboard for the cross-language experiment (JS/TS + Go): per-axis win/tie/loss across all 18 arms, the one-shot-vs-iterate split, and the full per-suite matrix.
• docs/python-results.html — the Python dashboard, including the decomposition of why coverage-driven scored 2/9 and quality-driven scored 9/9.
• docs/kotlin-swift-results.html — the Kotlin/Swift matrix dashboard: six libraries × three policies, all 18 arms vs their maintainer-written baselines.
• FINDINGS.md — the full running analysis.

Static previews live in examples/ (GitHub renders the HTML dashboards as source, so open them locally for the interactive charts):

The quality scorecard

The criteria above are a multi-axis rubric; the scorer auto-counts the mechanical ones and leaves the judgement ones to review:

scripts/...score.py auto-counts A.1/A.2/A.4/A.5, B.1, C.1/C.2, D.1/D.2 across Python, JS/TS, Go, Kotlin (kotlin.test/JUnit5/Kotest), and Swift (XCTest/Swift Testing/Quick) and prints the head-to-head Win/Loss/Tie tally. The rest are review-time judgement calls.

Use it on your own suite

The reusable artifacts are two bundled Claude Code skills:

• test-quality — the scorer, the anti-fragility contract, and the read→build→evaluate loop definition. This is the product: point it at a suite to audit, harden, or generate tests against the quality scorecard.
• results-dashboard — renders a scorecard JSON into the same interactive HTML readouts you see in docs/.

A clone is self-contained, so both skills are auto-discovered when you open the repo in Claude Code. You can also install them into any project in one step via the bundled .claude-plugin/marketplace.json:

/plugin marketplace add <owner>/<this-repo>
/plugin install test-quality-tools@beyond-test-coverage

Or score any suite directly, without the skill harness:

python .claude/skills/test-quality/scripts/score.py \
    --tests path/to/tests --baseline path/to/old_tests --lang python|js|go|kotlin|swift

The Python profile is the empirically-validated one (it drove the experiment above). js, go, kotlin, and swift apply the same rubric with heuristic, framework-appropriate regexes; the Kotlin and Swift profiles were calibrated against six real, well-tested suites — kotlinx.serialization, kotlinx-datetime, kotlin-result, swift-argument-parser, swift-collections, and SwiftyJSON — with every calibration finding pinned as a named regression in tests/test_score.py.

Beyond calibration, the full Kotlin/Swift generation matrix was executed the same way as the nine-library experiment — six repos × three iteration policies (oneshot / iter2 / iter20) = 18 arms, each deleting the human suite, regenerating from source against the scorecard, building/running with the real toolchain (JDK 17 + Gradle; Apple Swift 6.1.2), and scoring vs the human baseline. All 18 arms beat their baseline on the countable axes; 15 are green — the three reds are all oneshot (one pass, no repair) and fail to compile, the same "one pass wins quality but ships failures; iteration makes it green" pattern the Python/JS runs showed. Every W/L/T was re-scored independently of the generating agent, and every green count re-run from the real toolchain. The two largest repos (kotlinx.serialization, swift-collections) are scoped to a coherent core module, stated explicitly. See the interactive dashboard, results-kotlin-swift-scorecard.md and reports/kotlin-swift-generation.md. (These profiles are heuristic, not the empirically-validated Python set, so read the axis counts as toolchain-verified trends confirmed by reading, not validated ground truth.)

Repository layout

.
├── .claude/skills/               # two bundled Claude Code skills:
│   ├── test-quality/             #   scorer + rubric + read→build→evaluate loop
│   └── results-dashboard/        #   scorecard JSON → interactive HTML readout
├── .claude-plugin/               # marketplace.json — one-command install of both
├── docs/                         # interactive results dashboards
├── prompts/                      # the prompt sets that drive generation
├── scripts/                      # setup, scoring, and aggregation harness
├── configs/                      # per-repo coverage config
├── reports/ · runs/              # preserved reports & worktree inventories
├── results-*.{json,md}           # the scored results behind the dashboards
├── FINDINGS.md                   # running analysis (what we learned)
└── CHANGELOG.md                  # how the rubric & prompts evolved per stage

The nine libraries under test are each cloned locally into their own git-ignored directory; they are not redistributed here and remain under their own licenses. Generated suites live on preserved branches inside those clones.

How the experiment is organized

The work proceeded in three stages. Each froze a prompt set, ran it against the libraries, and fed its findings into the next:

• Coverage-driven control (Python). Prompts aimed at coverage %. Beat the human baseline on only 2/9 — the suites hit coverage parity and stopped. This is the control that motivated everything else.
• Quality-driven (Python). Same repos, prompts re-aimed at the quality scorecard. 9/9. An ablation arm held the model fixed at Opus 4.7 and swapped only the prompts, isolating the effect: the prompt redesign did the heavy lifting; the model upgrade was a marginal top-up (FINDINGS §10).
• Cross-language (JS/TS + Go). New libraries, same approach. 18/18 — the result generalizes beyond Python.

The frozen prompt sets live under prompts/quality/ and prompts/cross-language/; the scored artifacts are results-{coverage,quality,ablation,cross-language}-*.{json,md}. CHANGELOG.md records the rubric/prompt delta per stage and FINDINGS.md the evidence; proposed rubric changes queue under CHANGELOG [Unreleased].

License

MIT © Michael Rollins. The third-party repositories under test are not included in this repo (each is cloned locally and git-ignored) and remain under their own respective licenses.