# Research programs A research program runs a bounded improvement loop on a harness. You define what better means, which spec fields may change, which tools may execute, and how much the program may spend. A director model investigates and proposes; the engine runs the measurements and decides what to keep. The default output is one proposal for the live harness, with evidence attached. Use this when you have a repeatable task and want to explore changes across multiple rounds. For one change you want to inspect yourself, use [`evolve --propose`](signal-driven-evolution.md). For a simpler measured keep-or-revert loop, use `evolve --experiment`. ## Start with an existing eval The eval must live in the harness folder and target that harness (`harness: .`). Its dataset and scoring policy determine what the program can learn. Levers must already be permitted by `evolution.mutable` and must not be frozen. For harnesses with `evolution.objectives`, the engine supplies objective expectations to the evolution gate: the hypothesis's metric target takes precedence, then charter goal metrics, then all configured objectives. Each expected direction comes from the declared objective. ```bash hiveloom eval validate ./my-harness/eval.yaml --json hiveloom evolve ./my-harness --research --program trial-1 \ --budget 1 --rounds 4 --json ``` The second command makes paid model calls. It uses `research.yaml` in the harness folder if present, otherwise builds a charter from the harness and `eval.yaml`. Automatic levers are the permitted subset of `system_prompt`, `loop.max_turns`, and `memory.entries`. Without an explicit director model, the automatic charter uses the generation/evolution default described in [Models and providers](models.md). A supplied charter keeps its own director unless you override it. Unclassified code, shell, network, or MCP tools cause initialization to fail. Classify known effects explicitly, for example `--tool http_get=replay` or `--tool lookup_record=allow` for a reviewed local read-only tool. Replayed calls need matching results from the base harness's existing journals. To inspect a charter before any model call, save a separate `research.yaml`: ```yaml goal: Increase correct answers without reducing the success rate. eval: eval.yaml holdout: 0.25 levers: - system_prompt - loop.max_turns guard: success_rate: no_regression cost_per_run_increase: 0.2 budget: usd: 1.0 rounds: 4 wall_clock_minutes: 240 split: {exploration: 0.25, experiments: 0.60, confirmation: 0.15} stop: no_progress_rounds: 2 execution: tools: {} # classify this harness's effectful tools here mcp: {} # classify declared servers here max_cost_per_run: 0.05 models: director: claude/claude-opus-5-5 experiments_per_round: 2 ``` This is a separate program document; it does not replace `harness.yaml`. Then initialize, inspect, and run: ```bash hiveloom research init ./my-harness --name trial-1 \ --charter ./my-harness/research.yaml --json hiveloom research status ./my-harness --name trial-1 --json hiveloom research run ./my-harness --name trial-1 --json ``` `init` validates, loads the eval, freezes the split, and copies the base harness as candidate `c0`; it makes no model call. Referenced local code keeps the normal trust gate. At least two working cases must remain after holdout. ## What happens during a program ```text charter → baseline → survey → hypotheses → experiments → interpretation ↑ │ └──────────────────────────────────────┘ stop → sealed confirmation → report → queued proposal → explicit application ``` The director runs inside a packaged harness with tools to read working-split evidence, register hypotheses, design experiments, and record findings. Its brief includes failing examples, the signal map, cost and sample-size planning, previous verdicts, and findings from earlier programs on this harness. It has no shell or network tool and cannot directly edit the live harness or score its own candidate. The engine applies permitted changes to candidate copies through the evolution gate. It pairs candidate and incumbent runs by eval case, inspecting results at approximately one third, two thirds, and the full planned size. Harm or futility can stop an experiment early; benefit is decided at the planned size. Multiple eligible winners can be combined, rebased, and measured again before entering the incumbent. Refuted ideas and already-tested candidate versions are refused rather than silently repeated. Candidate runs use a program-scoped harness identity. They do not enter the live harness's production fitness bucket. Delegation is disabled in candidates because peer tools would execute outside the research tool policy; the live harness keeps its configuration. ## Tool effects during research Research executes a harness repeatedly, so tool policy matters as much as the model budget. Classification can only tighten the harness's capabilities. | Mode | Effect | |---|---| | `allow` | The declared tool executes normally, including any real external effects. | | `sandbox` | Supported writes occur in the candidate's workspace. Supported builtins are `file_write`, `notes`, and `propose_memory`; shell requires no-network confinement. | | `replay` | Returns a recorded result matching the tool name and arguments. A missing match is a tool error, with no fallback to live execution. | | `deny` | Removes the tool from the run. | Read-only builtins default to `allow`; supported write tools default to `sandbox`. Code, network, shell, and MCP tools need an explicit choice. MCP servers accept a mode or per-tool mapping in `execution.mcp`; they cannot use `sandbox`. Normal MCP discovery still happens when the registry is built, including for denied or replayed calls. Classification does not turn Python hooks or server startup into isolated code execution. When `tools` is a lever, a candidate cannot introduce unclassified effects or change an effectful declaration's hosts, code, or command. It can change a description, drop a tool, or add an allowed read-only builtin. Egress, confinement, guardrails, MCP declarations, and the executor model remain protected by the existing frozen-field rules. ## Read experiment verdicts and confirmation separately | Experiment verdict | Meaning | Eligible to keep | |---|---|---| | `confirmed` | The predicted target moved as predicted and the guard held. | Yes | | `improved` | Success rose significantly, although the predicted target did not establish the mechanism. | Yes, ranked below `confirmed` | | `refuted` / `inconclusive` | The experiment did not establish an effect worth keeping. | No | | `futile` | An early look could not reach the target's significance threshold even with a favorable remainder. | No | | `shifted` | A failure mechanism disappeared because runs stopped answering; the failure moved rather than being resolved. | No | | `regressed` | Harm or a guard violation was detected. | No | These are working-set decisions. If changes were kept, the engine attempts one confirmation of the base against the final incumbent on sealed cases. It records that access in the ledger and gives the proposal a separate evidence strength: | Strength | Interpretation | |---|---| | `confirmed` | Improvement is significant on the sealed cases. | | `supported` | Evidence points toward improvement but does not establish significance. | | `provisional` | Confirmation did not establish improvement, or could not be funded. | | `contradicted` | Sealed evidence shows a significant deterioration. | Small samples can leave a useful-looking change uncertain. A working-set goal or ceiling is not proof of generalization. The report shows the evidence you need to judge that distinction. ## Budget, stopping, and resume The total USD budget splits into exploration (director calls), experiments (baseline and candidate runs), and reserved confirmation. Defaults are 25% / 60% / 15%. Concepts mode adds a data pool. Shares must sum to one. The director can move limited amounts among non-confirmation pools; it cannot borrow the confirmation reserve. Costs are recorded in the ledger, and executor calls can have an additional `execution.max_cost_per_run` cap. Budgeting depends on reported billing or configured token-price estimates; it is not a provider account spending limit. Time limits are checked between units and rounds, rather than cancelling every in-flight request at the deadline. Stop conditions include working-set goal thresholds, round/time/budget limits, no progress, a director stop, an operator stop, and `ceiling`. A ceiling means the remaining evidence is predominantly content failures and no addressable mechanism is left under the charter's levers, after an unproductive round. The recommendation may be a stronger executor, wider capabilities, or a narrower task. It describes the current evidence and allowed search space. ```bash hiveloom research run ./my-harness --name trial-1 --until unit --json hiveloom research run ./my-harness --name trial-1 --until round --json hiveloom research stop ./my-harness --name trial-1 --reason 'Review the current result' --json hiveloom research run ./my-harness --name trial-1 --json hiveloom research report ./my-harness --name trial-1 --json ``` `stop` requests a stop; the running unit finishes. The next units confirm and report, so a CLI program needs another `run` if no process is advancing it. Only one process advances a program at once; contention returns runtime exit code 4. Status and report make no model calls. State and a hash-chained ledger live at `/.hiveloom/research//`, alongside candidate copies, director journals, replay data, and the final report. Interrupted evals resume from their manifests. Use the same `research run --name` or `evolve --research --program` to continue; an existing program keeps its original charter, so charter overrides require a new program. ## Review and promote Kept changes are merged three-way against the live harness and queued as a `trigger=research` proposal. Nonconflicting live edits are preserved. Conflicts produce a rejected proposal that names the path. A program with no kept change need not produce a proposal. ```bash hiveloom proposals list ./my-harness --json hiveloom proposals show ./my-harness --json hiveloom proposals apply ./my-harness --yes --json ``` For unattended application, explicitly opt in when starting or resuming: ```bash hiveloom evolve ./my-harness --research --yes --json ``` Program status and reports show the proposal's current status, including application or rejection after the program finished. This applies only a pending promotion whose strength is `confirmed` or `supported`. Other evidence remains for review. Applying does not authorize frozen-field changes or code regeneration. ## Start from concepts when you have no eval A concepts charter uses `concepts` and optional `seeds` instead of `eval`. Concepts can be prose or a file in the harness folder. Seed JSONL rows can carry `input`, `expected`, and `notes`. Provide the same levers and execution policy as an eval charter, plus models and a data budget: ```yaml goal: Quotes include a price and the JSON fields billing expects. concepts: concepts.md seeds: seeds.jsonl levers: [system_prompt, loop.max_turns] budget: usd: 1.0 split: {data: 0.2, exploration: 0.25, experiments: 0.4, confirmation: 0.15} models: director: claude/claude-opus-5-5 examiner: claude/claude-sonnet-5 # must differ from the director judges: [claude/claude-sonnet-5] human: {questions: 12, batch: 4} trust: {kappa: 0.6, agreement: 0.85, min_anchors: 8} ``` The director drafts criteria and working cases. Checks may be deterministic (`json_field`, `json_present`, `contains`, `not_contains`, `exact`, `regex`) or use a judge rubric. Framing consumes model budget, but experiments cannot start until you approve the contract and sample cases: ```bash hiveloom research contract ./my-harness --name trial-1 --json hiveloom research approve ./my-harness --name trial-1 --json hiveloom research run ./my-harness --name trial-1 --json hiveloom research questions ./my-harness --name trial-1 --json hiveloom research answer pass --dir ./my-harness --name trial-1 --json ``` Use `--contract edited.yaml` with `approve` to supply an edited contract. A separate examiner creates sealed cases without seeing the working cases; near-copies are removed from the working set. Deterministic criteria are measured immediately. Judged criteria count only when unanimous judge labels satisfy your agreement and κ thresholds on enough human anchors. The engine asks for labels, prioritizes disagreements, and audits trust during rounds within the question budget. **A pass covers only criteria currently measured.** Review any unmeasured criteria: they are not evidence that the output satisfies the complete concept. Confirmation strength is reduced when criteria were never measured, and the report flags a large working-versus-sealed gap as possible overfitting. ## Workbench and current limits The workbench's **Research** tab has the charter form, background start/step/ run/stop controls, eval progress, per-case links into Trace, contract approval, question cards, budgets, reports, and the ledger. It resumes interrupted jobs when its server restarts. Improve → **Evolve autonomously** opens this workflow; the copilot can relay it in conversation. See [Workbench](workbench.md). Dataset growth during a program, research candidates in the Versions graph, and an optional critic/coaching phase are not implemented. The director changes permitted spec values, not arbitrary tool or validator code. The existing [research benchmarks](evals.md#autonomous-improvement-research-bakeoff) are small controlled experiments; they establish neither universal improvement nor superiority to a person using `evolve --experiment`.