Evaluations

Three checked-in evaluations live in evals/, each with its harnesses, scoring code, and committed results. They answer three different questions, and the answers are not the same. Harness value is measured per task and model, not assumed — it can be a rescue, a compliance floor, or nothing at all.

1. Does a harness rescue a weak model? (article-extractor) #

The article-extractor benchmark evaluates 32 live URLs over three epochs (96 runs per arm). Raw and harness arms use the same prompt and fetch tool; the difference is the harness scaffolding.

The Haiku harness gained 61.5 percentage points over raw Haiku and cut cost per successful result by 12×. Read the rest of the table before generalising from that: Haiku's is the only delta that survives a paired test over the 32 URLs (p < 0.0001). The three local models move by 10 points or less, which is noise at this sample size, and Gemma is slightly worse harnessed. Scaffolding rescues a model that cannot hold the output contract. It does not improve one that already can — and it did not beat raw Sonnet, which wins outright on both success and cost.

Hallucination is the more robust signal, because the effect sizes are large relative to the sample. Every output is checked verbatim against a re-fetch of the live page:

2. Does it still earn its place on frontier models? (article-digest) #

article-digest runs an output-heavy task — a 120–200 word original summary plus five verbatim quotes and a verbatim outline — on Claude Opus 5 and Sonnet 5. Both arms go through hiveloom with the same prompt, tool, guardrails, and loop policy; the raw arms only drop the validators and loop.require_verification, so the measured delta is validators plus retry-with-feedback, and nothing else.

Neither model fabricated anything. What the raw arms lost was the contract: one run emitted invalid JSON, another a quote outside the required length. The harness is not buying accuracy from a frontier model — it is buying the tail of contract compliance, at roughly unchanged cost per success.

3. What do frontier models still get wrong? (page-audit) #

page-audit targets what remains: exhaustiveness past a truncated tool view, aggregation, and date arithmetic. The fetch tool clips its digest, and half the pages have more headings than the digest shows, so no complete answer is reachable from the tool alone. The metric that matters is not success but whether a wrong answer arrives labelled.

Raw arms confidently returned truncated heading lists and off-by-one day counts. The harnessed arms either recovered on retry or exited verify_failed — they never returned a wrong audit as a success. That is the property a downstream automation can actually build on.

Prompt caching compounds it #

Prompt caching is on by default for the claude provider. In a live measurement on a 7k-token harness prompt (Haiku 4.5, two-turn run), the first run wrote the prefix to cache and every later run inside the cache TTL read it back at a tenth of the input price — $0.0019 per warm run vs $0.0100 cold, an 81% reduction. A harness runs the same prompt shape every time, which is exactly the workload prompt caching rewards.

Sample sizes, scoring code, and caveats are in evals/README.md.