Research programs

A research program runs a bounded improvement loop on a harness. You define what better means, which spec fields may change, which tools may execute, and how much the program may spend. A director model investigates and proposes; the engine runs the measurements and decides what to keep. The default output is one proposal for the live harness, with evidence attached.

Use this when you have a repeatable task and want to explore changes across multiple rounds. For one change you want to inspect yourself, use evolve --propose. For a simpler measured keep-or-revert loop, use evolve --experiment.

Start with an existing eval #

The eval must live in the harness folder and target that harness (harness: .). Its dataset and scoring policy determine what the program can learn. Levers must already be permitted by evolution.mutable and must not be frozen. For harnesses with evolution.objectives, the engine supplies objective expectations to the evolution gate: the hypothesis's metric target takes precedence, then charter goal metrics, then all configured objectives. Each expected direction comes from the declared objective.

hiveloom eval validate ./my-harness/eval.yaml --json
hiveloom evolve ./my-harness --research --program trial-1 \
  --budget 1 --rounds 4 --json

The second command makes paid model calls. It uses research.yaml in the harness folder if present, otherwise builds a charter from the harness and eval.yaml. Automatic levers are the permitted subset of system_prompt, loop.max_turns, and memory.entries. Without an explicit director model, the automatic charter uses the generation/evolution default described in Models and providers. A supplied charter keeps its own director unless you override it.

Unclassified code, shell, network, or MCP tools cause initialization to fail. Classify known effects explicitly, for example --tool http_get=replay or --tool lookup_record=allow for a reviewed local read-only tool. Replayed calls need matching results from the base harness's existing journals.

To inspect a charter before any model call, save a separate research.yaml:

goal: Increase correct answers without reducing the success rate.
eval: eval.yaml
holdout: 0.25
levers:
  - system_prompt
  - loop.max_turns
guard:
  success_rate: no_regression
  cost_per_run_increase: 0.2
budget:
  usd: 1.0
  rounds: 4
  wall_clock_minutes: 240
  split: {exploration: 0.25, experiments: 0.60, confirmation: 0.15}
stop:
  no_progress_rounds: 2
execution:
  tools: {}                 # classify this harness's effectful tools here
  mcp: {}                   # classify declared servers here
  max_cost_per_run: 0.05
models:
  director: claude/claude-opus-5-5
experiments_per_round: 2

This is a separate program document; it does not replace harness.yaml. Then initialize, inspect, and run:

hiveloom research init ./my-harness --name trial-1 \
  --charter ./my-harness/research.yaml --json
hiveloom research status ./my-harness --name trial-1 --json
hiveloom research run ./my-harness --name trial-1 --json

init validates, loads the eval, freezes the split, and copies the base harness as candidate c0; it makes no model call. Referenced local code keeps the normal trust gate. At least two working cases must remain after holdout.

What happens during a program #

charter → baseline → survey → hypotheses → experiments → interpretation
                       ↑                                      │
                       └──────────────────────────────────────┘
stop → sealed confirmation → report → queued proposal → explicit application

The director runs inside a packaged harness with tools to read working-split evidence, register hypotheses, design experiments, and record findings. Its brief includes failing examples, the signal map, cost and sample-size planning, previous verdicts, and findings from earlier programs on this harness. It has no shell or network tool and cannot directly edit the live harness or score its own candidate.

The engine applies permitted changes to candidate copies through the evolution gate. It pairs candidate and incumbent runs by eval case, inspecting results at approximately one third, two thirds, and the full planned size. Harm or futility can stop an experiment early; benefit is decided at the planned size. Multiple eligible winners can be combined, rebased, and measured again before entering the incumbent. Refuted ideas and already-tested candidate versions are refused rather than silently repeated.

Candidate runs use a program-scoped harness identity. They do not enter the live harness's production fitness bucket. Delegation is disabled in candidates because peer tools would execute outside the research tool policy; the live harness keeps its configuration.

Tool effects during research #

Research executes a harness repeatedly, so tool policy matters as much as the model budget. Classification can only tighten the harness's capabilities.

Read-only builtins default to allow; supported write tools default to sandbox. Code, network, shell, and MCP tools need an explicit choice. MCP servers accept a mode or per-tool mapping in execution.mcp; they cannot use sandbox. Normal MCP discovery still happens when the registry is built, including for denied or replayed calls. Classification does not turn Python hooks or server startup into isolated code execution.

When tools is a lever, a candidate cannot introduce unclassified effects or change an effectful declaration's hosts, code, or command. It can change a description, drop a tool, or add an allowed read-only builtin. Egress, confinement, guardrails, MCP declarations, and the executor model remain protected by the existing frozen-field rules.

Read experiment verdicts and confirmation separately #

These are working-set decisions. If changes were kept, the engine attempts one confirmation of the base against the final incumbent on sealed cases. It records that access in the ledger and gives the proposal a separate evidence strength:

Small samples can leave a useful-looking change uncertain. A working-set goal or ceiling is not proof of generalization. The report shows the evidence you need to judge that distinction.

Budget, stopping, and resume #

The total USD budget splits into exploration (director calls), experiments (baseline and candidate runs), and reserved confirmation. Defaults are 25% / 60% / 15%. Concepts mode adds a data pool. Shares must sum to one. The director can move limited amounts among non-confirmation pools; it cannot borrow the confirmation reserve.

Costs are recorded in the ledger, and executor calls can have an additional execution.max_cost_per_run cap. Budgeting depends on reported billing or configured token-price estimates; it is not a provider account spending limit. Time limits are checked between units and rounds, rather than cancelling every in-flight request at the deadline.

Stop conditions include working-set goal thresholds, round/time/budget limits, no progress, a director stop, an operator stop, and ceiling. A ceiling means the remaining evidence is predominantly content failures and no addressable mechanism is left under the charter's levers, after an unproductive round. The recommendation may be a stronger executor, wider capabilities, or a narrower task. It describes the current evidence and allowed search space.

hiveloom research run ./my-harness --name trial-1 --until unit --json
hiveloom research run ./my-harness --name trial-1 --until round --json
hiveloom research stop ./my-harness --name trial-1 --reason 'Review the current result' --json
hiveloom research run ./my-harness --name trial-1 --json
hiveloom research report ./my-harness --name trial-1 --json

stop requests a stop; the running unit finishes. The next units confirm and report, so a CLI program needs another run if no process is advancing it. Only one process advances a program at once; contention returns runtime exit code 4. Status and report make no model calls.

State and a hash-chained ledger live at <harness>/.hiveloom/research/<name>/, alongside candidate copies, director journals, replay data, and the final report. Interrupted evals resume from their manifests. Use the same research run --name or evolve --research --program to continue; an existing program keeps its original charter, so charter overrides require a new program.

Review and promote #

Kept changes are merged three-way against the live harness and queued as a trigger=research proposal. Nonconflicting live edits are preserved. Conflicts produce a rejected proposal that names the path. A program with no kept change need not produce a proposal.

hiveloom proposals list ./my-harness --json
hiveloom proposals show ./my-harness <proposal_id> --json
hiveloom proposals apply ./my-harness <proposal_id> --yes --json

For unattended application, explicitly opt in when starting or resuming:

hiveloom evolve ./my-harness --research --yes --json

Program status and reports show the proposal's current status, including application or rejection after the program finished.

This applies only a pending promotion whose strength is confirmed or supported. Other evidence remains for review. Applying does not authorize frozen-field changes or code regeneration.

Start from concepts when you have no eval #

A concepts charter uses concepts and optional seeds instead of eval. Concepts can be prose or a file in the harness folder. Seed JSONL rows can carry input, expected, and notes. Provide the same levers and execution policy as an eval charter, plus models and a data budget:

goal: Quotes include a price and the JSON fields billing expects.
concepts: concepts.md
seeds: seeds.jsonl
levers: [system_prompt, loop.max_turns]
budget:
  usd: 1.0
  split: {data: 0.2, exploration: 0.25, experiments: 0.4, confirmation: 0.15}
models:
  director: claude/claude-opus-5-5
  examiner: claude/claude-sonnet-5       # must differ from the director
  judges: [claude/claude-sonnet-5]
human: {questions: 12, batch: 4}
trust: {kappa: 0.6, agreement: 0.85, min_anchors: 8}

The director drafts criteria and working cases. Checks may be deterministic (json_field, json_present, contains, not_contains, exact, regex) or use a judge rubric. Framing consumes model budget, but experiments cannot start until you approve the contract and sample cases:

hiveloom research contract ./my-harness --name trial-1 --json
hiveloom research approve ./my-harness --name trial-1 --json
hiveloom research run ./my-harness --name trial-1 --json
hiveloom research questions ./my-harness --name trial-1 --json
hiveloom research answer <question_id> pass --dir ./my-harness --name trial-1 --json

Use --contract edited.yaml with approve to supply an edited contract. A separate examiner creates sealed cases without seeing the working cases; near-copies are removed from the working set.

Deterministic criteria are measured immediately. Judged criteria count only when unanimous judge labels satisfy your agreement and κ thresholds on enough human anchors. The engine asks for labels, prioritizes disagreements, and audits trust during rounds within the question budget. A pass covers only criteria currently measured. Review any unmeasured criteria: they are not evidence that the output satisfies the complete concept. Confirmation strength is reduced when criteria were never measured, and the report flags a large working-versus-sealed gap as possible overfitting.

Workbench and current limits #

The workbench's Research tab has the charter form, background start/step/ run/stop controls, eval progress, per-case links into Trace, contract approval, question cards, budgets, reports, and the ledger. It resumes interrupted jobs when its server restarts. Improve → Evolve autonomously opens this workflow; the copilot can relay it in conversation. See Workbench.

Dataset growth during a program, research candidates in the Versions graph, and an optional critic/coaching phase are not implemented. The director changes permitted spec values, not arbitrary tool or validator code. The existing research benchmarks are small controlled experiments; they establish neither universal improvement nor superiority to a person using evolve --experiment.