> ## Documentation Index
> Fetch the complete documentation index at: https://docs.whyfile.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 0030 panel evaluation harness

# ADR-0030: The panel evaluation harness

**Status:** Accepted
**Date:** 2026-07-14

## Context

ADR-0026 (C1) defined the panel roster and the pure `converge` function; ADR-0027 (C2) built the
executor that runs the roster over the real graph; ADR-0028 (C3) added convergence, the trust-ledger
write, and the merge gate. The court now runs end to end — but it is **trusted, not measured**.
Nothing quantifies whether the five refuter lenses actually catch bad reconstructed intent or how
often they mis-fire, and the hand-picked `3/2/1` convergence policy (ADR-0026) has no evidence behind
it. Epic #9 named this as deferred — *"automated panel scoring + an evaluation harness (measure the
panel once it exists)"* — and the precondition (a running panel) is now met (issue #20).

Measuring the panel means running the *real* lenses, not inspecting their prompts: ADR-0022 already
established that a reviewer handed only a node's own fields will rubber-stamp it, so an honest eval
has to execute the query tools over a real graph the same way C2 does. That pulls in a tension with
the rest of this codebase, where every test fakes the LLM: the harness needs a live half (measure the
real court) and a deterministic half (the scoring and sweep math, in CI).

A hostile 5-lens panel reviewed the spec for this work before code was written (ADR-0022's own
practice applied to itself), running the query tools and the real `select`/`converge` code rather
than judging the spec on the page. It caught two blockers that reshaped the design:

* **Circular sweep.** The original spec derived each fixture's expected node-level outcome by applying
  the `3/2/1` policy to its refutation count — so the policy sweep was scored against an answer key
  the incumbent policy itself produced, guaranteeing the sweep would "validate" `3/2/1`. The fix: node
  labels are a **policy-free, design-time judgment** about the intent's trustworthiness, never a
  function of any threshold, and the ranking carries **no tie-break toward the incumbent**.
* **Toothless relational lenses.** The original seed graph held only reconstructed nodes, so
  `contradiction-supersession` (needs a higher-provenance node to contradict) and `duplication-merge`
  (needs a twin) had nothing to bite — their recall was structurally forced to zero. The fix: the
  seed set includes an **authored companion** and a deliberate **near-twin** so the relational lenses
  judge real, non-empty tool output.

## Decision

**Ship a panel evaluation harness as pure machinery plus one live orchestrator, run on demand from a
maintainer script.** `src/whyfile/panel_eval.py` holds the seeded fixtures, the graph builder, the
confusion-matrix scorer, and the policy sweep — all pure and unit-tested with a fake `complete`
callable — plus `run_eval`, which reuses `audit.run_audit`'s injectable `complete` seam to run the
real lenses over the seed graph. The entrypoint is `scripts/panel_eval.py` (beside `dogfood.py` and
`ensure_self_graph.py`), **not** a `whyfile` subcommand: this is maintainer measurement tooling, and a
shipped subcommand pointing at fixtures baked into the module would still be the wrong surface for an
end user's installed CLI. It makes live LLM calls, so it never runs in CI.

**Measure per-lens quality as raw counts, roll up by macro and worst-lens, never by blending.** The
scorer reports `tp/fp/tn/fn/errored` per lens (positive class = `refuted`) with their denominators;
derived precision/recall/fp-rate/fn-rate are `null` on a zero denominator. An errored pair is excluded
from precision/recall and counted separately (ADR-0027: a failed call is never a silent survive), with
an `effective_recall` that does count an error on a known-bad case as a miss. There is **no single
volume-blended precision/recall** — that would violate the epic's design law 3 (metrics never blend
evidence classes) and let one dead lens hide behind four healthy ones. Instead the roll-up is a macro
average plus an explicit `worst_lens` (minimum recall), because a **false negative — a missed defect —
is the guardrail error** for a refuter court (a promoted bad node has no human gate; only the advisory
band does).

**Sweep convergence policies and recommend only on a strict win.** `sweep_policies` scores a bounded
grid of `structural/sharpen/note` candidates (bounded by the greatest refutation count actually
observed — a policy that can never dispute any fixture is not a candidate) against the policy-free node
labels, using the *actual* panel findings. The ranking is a total deterministic order with no
incumbent tie-break; `recommended` is set only when the top policy strictly out-scores the second,
otherwise the harness reports the tie honestly (`tied_top`) and says the fixture set does not
discriminate. On the current seed set every `structural=3` policy scores best but ties on the
`sharpen`/`note` split — so the harness reports "structural=3 is pinned, sub-tiers under-determined"
rather than manufacturing a false-precise `3/2/1` recommendation. That expected non-discrimination is
itself the evidence the sweep is not rigged toward the default. A human reads this and edits the
Whyfile `convergence` block (ADR-0029); the harness never writes it.

## Assumptions

* We assume the seed corpus stays small and its cases stay *blatantly* bad or *clearly* good, so a
  competent model classifies them from real tool output and the eval measures the lens **prompt**, not
  fixture subtlety (review by 2026-10-01, or when borderline fixtures are added — issue #20 §11).
* We assume the five-lens roster (ADR-0026) stays the source of which lenses exist; the harness has no
  roster logic of its own, only labels keyed to `panel.LENSES` and a self-consistency test that fails
  loud if a label's lens stops selecting its node (review by 2026-10-01, or whenever the roster
  changes shape).
* We assume single-shot per (node, lens) is enough for a directional signal; run-to-run LLM variance
  is documented, not measured (expires: if a maintainer needs confidence intervals or a regression
  gate, `--repeat N` and a variance model are the next step — issue #20 §11).

## Alternatives considered

1. **Deterministic-only harness (recorded verdicts, no live calls).** Rejected: it exercises the
   scoring math but measures nothing about the real LLM, which is the entire point of issue #20 and
   contradicts ADR-0022's execution-over-inspection rule.
2. **A shipped `whyfile panel-eval` subcommand.** Rejected: this is maintainer measurement tooling,
   and the repo already homes that in `scripts/`; a subcommand also invites confusion with `audit`
   (which runs the panel over *your* repo, not the seed graph).
3. **A markdown renderer / CI regression gate.** Rejected for v1: markdown here would feed a PR comment
   or a merge gate, both non-goals; and LLM non-determinism makes a score-diff gate flaky. JSON plus a
   plain terminal summary is the whole surface.
4. **Measure-only, no policy sweep.** Rejected: too thin for the tuning intent — a measured court
   should give a team *some* directional input on the threshold that governs it.
5. **Auto-write the tuned policy into the Whyfile.** Rejected outright: the tool choosing its own trust
   thresholds is the panel acting as a certifier over a human, exactly what ADR-0021 forbids.

## Consequences

* The panel's per-lens accuracy is finally a queryable artifact instead of an article of faith, and a
  team can see *which* lens is weak (worst-lens recall) rather than a blended headline that hides it.
* The sweep gives honest, directional input to convergence tuning (ADR-0029) without ever certifying a
  policy — and its built-in non-discrimination reporting means it cannot launder the incumbent default
  into false evidence for itself.
* Because the machinery is pure and the fixtures ship inside the package, the whole harness is
  unit-tested in CI with no live backend or network — the same property ADR-0026/0027/0028 established
  for `converge`, the executor, and the gate.
* The seed corpus is small and synthetic, so the relational and retrieval lenses judge shallow tool
  output; per-lens numbers are a smoke test on blatant cases, not a production accuracy estimate. A
  richer corpus is the deferred path to deeper measurement.
* This ADR is itself authored intent the ingest pass reconstructs into the self-graph (the dogfood
  loop, ADR-0022), so the harness's own "why" becomes something `whyfile` can `explain`/`why` about
  itself.