# Evaluations

Three checked-in evaluations live in
[`evals/`](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/README.md),
each with its harnesses, scoring code, and committed results. They answer three
different questions, and **the answers are not the same**. Harness value is
measured per task and model, not assumed — it can be a rescue, a compliance
floor, or nothing at all.

![Task success by model, raw versus the same model inside a hiveloom harness](/assets/01-task-success.png)

## 1. Does a harness rescue a weak model? (article-extractor)

The
[article-extractor benchmark](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/article-extractor/RESULTS.md)
evaluates 32 live URLs over three epochs (96 runs per arm). Raw and harness arms
use the same prompt and fetch tool; the difference is the harness scaffolding.

| Model / arm | Task success | Hallucination | Cost per success | p50 latency |
|---|---:|---:|---:|---:|
| Claude Haiku 4.5, raw | 3% | 96% | $0.2212 | 5.8s |
| Claude Haiku 4.5 + hiveloom | **65%** | **11%** | **$0.0186** | 10.3s |
| Claude Sonnet 5, raw baseline | 100% | 0% | $0.0073 | 6.4s |
| Qwen 3 4B, raw / harness | 58% / **69%** | 19% / 16% | local | 3.9s / 6.3s |
| Qwen 3.6 35B, raw / harness | 75% / **84%** | 16% / **0%** | local | 13.5s / 16.4s |
| Gemma 4 12B, raw / harness | **92%** / 90% | 1% / **0%** | local | 26.2s / 9.3s |

The Haiku harness gained 61.5 percentage points over raw Haiku and cut cost per
successful result by 12×. Read the rest of the table before generalising from
that: **Haiku's is the only delta that survives a paired test over the 32 URLs**
(p < 0.0001). The three local models move by 10 points or less, which is noise
at this sample size, and Gemma is slightly *worse* harnessed. Scaffolding
rescues a model that cannot hold the output contract. It does not improve one
that already can — and it did not beat raw Sonnet, which wins outright on both
success and cost.

Hallucination is the more robust signal, because the effect sizes are large
relative to the sample. Every output is checked verbatim against a re-fetch of
the live page:

![Hallucination rate, raw versus harnessed](/assets/02-hallucination.png)

![Cost per successful extraction](/assets/03-cost-per-success.png)

## 2. Does it still earn its place on frontier models? (article-digest)

[article-digest](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/article-digest/RESULTS.md)
runs an output-heavy task — a 120–200 word original summary plus five verbatim
quotes and a verbatim outline — on Claude Opus 5 and Sonnet 5. Both arms go
through hiveloom with the same prompt, tool, guardrails, and loop policy; the
raw arms only drop the validators and `loop.require_verification`, so the
measured delta is **validators plus retry-with-feedback, and nothing else**.

| Arm | Success | Hallucinated quotes | Cost per success |
|---|---:|---:|---:|
| Opus 5 + hiveloom | **100%** | 0% | $0.0383 |
| Opus 5, raw | 80% | 0% | $0.0320 |
| Sonnet 5 + hiveloom | **100%** | 0% | $0.0142 |
| Sonnet 5, raw | 80% | 0% | $0.0141 |

Neither model fabricated anything. What the raw arms lost was the *contract*:
one run emitted invalid JSON, another a quote outside the required length. The
harness is not buying accuracy from a frontier model — it is buying the tail of
contract compliance, at roughly unchanged cost per success.

## 3. What do frontier models still get wrong? (page-audit)

[page-audit](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/page-audit/RESULTS.md)
targets what remains: exhaustiveness past a truncated tool view, aggregation,
and date arithmetic. The fetch tool clips its digest, and half the pages have
more headings than the digest shows, so no complete answer is reachable from the
tool alone. The metric that matters is not success but whether a wrong answer
arrives **labelled**.

| Arm | Silently wrong | Flagged (`verify_failed`) |
|---|---:|---:|
| Opus 5 + hiveloom | **0/6** | 1/6 |
| Opus 5, raw | 5/6 | 0/6 |
| Sonnet 5 + hiveloom | **0/6** | 1/6 |
| Sonnet 5, raw | 3/6 | 0/6 |

Raw arms confidently returned truncated heading lists and off-by-one day counts.
The harnessed arms either recovered on retry or exited `verify_failed` — they
never returned a wrong audit as a success. That is the property a downstream
automation can actually build on.

## Prompt caching compounds it

Prompt caching is on by default for the `claude` provider. In a live measurement
on a 7k-token harness prompt (Haiku 4.5, two-turn run), the first run wrote the
prefix to cache and every later run inside the cache TTL read it back at a tenth
of the input price — **$0.0019 per warm run vs $0.0100 cold, an 81% reduction**.
A harness runs the same prompt shape every time, which is exactly the workload
prompt caching rewards.

![Prompt caching, cold versus warm run cost](/assets/04-prompt-caching.png)

Sample sizes, scoring code, and caveats are in
[`evals/README.md`](https://github.com/FrancescoMrn/hiveloom/blob/main/evals/README.md).