Evaluations

The checked-in evaluations live in evals/, each with its harnesses, scoring code, and committed results. They address task execution and, in 1.3.0, autonomous improvement. These are historical measurements on specific code revisions, models, and datasets, not a fresh benchmark of the 1.3.0 checkout. Harness value is measured per task and model, not assumed — it can be a rescue, a compliance floor, or nothing at all.

1. Does a harness rescue a weak model? (article-extractor) #

The article-extractor benchmark used hiveloom 0.3.1 on 32 live URLs over three epochs (96 runs per arm). Repeated epochs are not 96 independent samples. Raw and harness arms use the same prompt and fetch tool; the difference is the harness scaffolding.

The Haiku harness gained 61.5 percentage points over raw Haiku and cut cost per successful result by 12×. Read the rest of the table before generalising from that: Haiku's is the only delta that survives a paired test over the 32 URLs (p < 0.0001). The local-model deltas are not statistically established over the 32 URLs; that does not establish either no effect or harm. In this experiment, raw Sonnet achieved higher success and lower cost per success than harnessed Haiku. Measure the tradeoff on your own task and model.

Hallucination is the more robust signal, because the effect sizes are large relative to the sample. Every output is checked verbatim against a re-fetch of the live page:

2. Does it still earn its place on frontier models? (article-digest) #

article-digest runs an output-heavy task — a 120–200 word original summary plus five verbatim quotes and a verbatim outline — on Claude Opus 5 and Sonnet 5. Both arms go through hiveloom with the same prompt, tool, guardrails, and loop policy; the raw arms only drop the validators and loop.require_verification, so the measured delta is validators plus retry-with-feedback. There were only five runs per arm: the 100% versus 80% difference is one run per model, not established general superiority. The prompt requested 120–200 summary words; the scoring contract accepted 100–260, with explicit quote/outline tolerances documented in the results.

Neither model fabricated anything. What the raw arms lost was the contract: one run emitted invalid JSON, another a quote outside the required length. The checks recovered these observed contract failures. This small sample cannot quantify the reliability gain on future inputs.

3. What do frontier models still get wrong? (page-audit) #

page-audit targets what remains: exhaustiveness past a truncated tool view, aggregation, and date arithmetic. The fetch tool clips its digest, and half the pages have more headings than the digest shows, so no complete answer is reachable from the tool alone. The metric that matters is not success but whether a wrong answer arrives labelled.

Raw arms confidently returned truncated heading lists and off-by-one day counts. The harnessed arms either recovered on retry or exited verify_failed — no wrong audit was labelled successful in these six cases per arm. This illustrates explicit failure reporting; it is not a guarantee for checks or inputs outside this benchmark.

Prompt caching compounds it #

Prompt caching is on by default for the claude provider. In a live measurement on a 7k-token harness prompt (Haiku 4.5, two-turn run), the first run wrote the prefix to cache and every later run inside the cache TTL read it back at a tenth of the input price — $0.0019 per warm run vs $0.0100 cold, an 81% reduction. A harness runs the same prompt shape every time, which is exactly the workload prompt caching rewards.

Sample sizes, scoring code, and caveats are in evals/README.md.

Autonomous improvement: research-bakeoff #

The research-bakeoff results compare research with evolve --experiment using the same live director model and scripted executors. Each run used four planted harnesses, two arms, and three repeats; fixes were scored on 40 fresh cases unseen by either arm.

Both comparisons recorded zero false promotions. Neither met the pre-registered 1.5× defect-fix target: ratios were 1.12× and 1.05×. Research also took longer. The later product criterion—useful unattended improvement rather than replacing deliberate evolution—was a disclosed post-hoc change; it does not erase those results.

Later autonomy runs (eight programs each) tested the engine after additional fixes:

These runs used a live gpt-5-mini director, fresh case sets, and different engine revisions as fixes were made. They are evidence about those controlled tasks, not a clean causal estimate of the current engine's general advantage. The concepts-mode live check also exposed an invented rate table in a drafted contract: contract approval is a meaningful part of the workflow.

Read Research programs to run your own charter. Choose scoring that matches real acceptance, keep sealed cases independent, and inspect unmeasured criteria and confirmation strength before promotion.