# Testing and evaluating a harness

A model can produce a convincing answer while the harness loses a queued message, executes a tool twice, or records an interrupted operation as completed. Testing the final prose alone misses these failures. Start by checking the machinery with scripted model responses, then evaluate whether real models use that machinery effectively.

Suppose a model requests two writes. An approval hook cancels during preparation of the second call. Did either write execute? Does every call receive a terminal result? Does the next request contain a valid conversation? Those are harness contracts: properties that should hold regardless of how capable the model is.

This page examines tests and evaluation infrastructure in the pinned [Codex](../../snapshots/codex/8c076942cb70/chapter.md), [pi](../../snapshots/pi/482c1381a00d/chapter.md), [Goose](../../snapshots/goose/34a222e3f2b9/chapter.md), and [Sidekick](../../snapshots/sidekick/463d653124e1/chapter.md) snapshots. Sidekick source links are private and require repository access. The tests were inspected, not executed; their assertions describe intended contracts, not a passing-suite or production-reliability claim.

## Three different questions

| Layer | Question | Useful evidence |
|---|---|---|
| Contract tests | Does the harness behave correctly for a controlled sequence? | Captured requests, event ordering, persisted records, counted effects |
| Behavioral evaluations | Can a real model complete a task with this configuration? | Task-specific grading, complete trajectories, repeated trials |
| Comparative benchmarks | Does configuration A improve on B under a defined workload? | Matched tasks and budgets, pinned versions, outcome distributions and costs |

A fake provider can prove that steering reaches the next request. It cannot prove that a model follows that steering. A benchmark can reveal a lower success rate without identifying whether the cause is the prompt, tools, provider adapter, or execution environment. These layers complement each other; neither test count nor one aggregate score measures harness quality by itself.

## What the four codebases demonstrate

### Codex: control the stream and inspect the next request

Codex's test helpers serve scripted server-sent events, capture outgoing requests, and expose request fields such as instructions, tools, and call outputs. A separate streaming server gates individual chunks with explicit signals. Tests can therefore pause at a known boundary, inject an action, and inspect what the harness subsequently sends. This is more precise than hoping a live provider responds slowly enough to reproduce a race. [Request capture][C1] [Scripted sequence][C2] [Gated delivery][C3]

The cancellation tests distinguish admission from completion. One holds the execution gate and cancels before admission, asserting timing reports no execution start. Another allows a handler to finish, pauses its lifecycle notification, then cancels: the successful output and completed lifecycle must survive. Cancellation should not rewrite an already completed action into a fictional failure. [Cancellation contracts][C4]

Tool-search tests also exercise catalog changes: reuse for unchanged handlers, rebuilding for changed registry inputs, and removal from the deferred catalog when exposure becomes direct. This is a useful pattern for testing tool negotiation: assert the actual resulting catalog, not merely that an update method ran. These are catalog-cache tests, not measurements of provider prompt-cache hits. [Catalog tests][C5]

### pi: script the provider while exercising the session

pi's coding-agent test harness registers its faux provider, constructs an actual `AgentSession`, and allows controlled tools, extensions, settings, and session storage. Faux responses can be functions of the supplied context, making it possible to fail a test when the next request lacks expected input. [Session fixture][P1]

The two-write example is a real regression test: the second preflight aborts, neither tool executes, result hooks do not run, and both calls receive matching terminal error events and aborted results. Another test injects a custom message during tool execution and checks ordering in agent state, session entries, emitted events, and model history. A steering test pauses a tool, queues user input, then verifies the next faux response sees it. [Preflight regression][P2] [Message-order regression][P3] [Steering boundary][P4]

“Faux” does not mean every byte is deterministic. Its stream can split content into randomized chunks, and its token/cache accounting is an estimate built from text length and common prefixes. Builders should fix or normalize randomness when asserting exact traces, and avoid treating simulated usage as provider economics. [Synthetic usage and chunks][P5]

pi also has separate real-model documentation evaluations. Its documented protocol pairs runs with and without documentation in isolated containers using the same staged packages. The report code rejects incomplete or unscored pairs and withholds headline pass rates when pairs are blocked. This separates infrastructure failures from low task scores instead of silently dropping inconvenient runs. [Evaluation protocol][P6] [Pair validation][P7] [Headline calculation][P8]

### Goose: reconstruct between state-machine steps

Goose's pipeline tests script a dummy provider and rebuild the pipeline after each applied step. A tool-turn test asserts two inference calls, the call/result exchange, the final reply, and successful continuation after reconstruction. Other assertions check restored model, mode, recipe, tools, and isolation between sessions. [Reconstruction tests][G1]

The fixture exposes a crucial limitation: reconstruction shares the session manager, dummy API, scheduler, and retained extension lease. It tests reconstruction from stored session state within that controlled environment; it does not simulate every consequence of losing the whole process or remote connection. Read the fixture as carefully as the test name. [Reconstruction implementation][G2]

Goose separately includes Harbor benchmark tooling. Its adapter uploads a selected local binary and configuration into a task environment, creates a recipe, and captures streaming JSON output. That makes the executable and extension configuration explicit experimental inputs. The presence of benchmark tooling establishes a measurement mechanism, not a comparable score for these four pinned snapshots. [Benchmark adapter][G3]

### Sidekick: inject failure around durable effects

Sidekick's intent-contract integration tests use the real workflow task body, native loop, and PostgreSQL, while faking the model stream, Redis transport, and selected executors. They count simulated tool effects and inject worker-death exceptions at specific boundaries. These are stronger persistence checks than an in-memory transcript assertion, but are not literal process-kill tests of a deployed cluster. [Fixture boundaries][S1]

One scenario crashes after both a replay-safe search and an unsafe send have executed, before their outcomes settle. On redispatch, assertions expect the search count to reach two, the send count to remain one, and the unsafe result to explain its uncertain outcome. A second scenario fails before execution; its intended record permits the previously unstarted call to run. This tests the distinction between “never started” and “may have completed.” [Crash-window assertions][S2]

Ownership tests separately check refusal when a heartbeat is fresh, takeover when the cluster attempt is stale, and rejection of heartbeats from superseded attempts. These contracts matter in [event-driven execution](event-driven.md), where a valid conversation history alone does not establish which worker may act. [Ownership assertions][S3]

## Build a failure matrix before a leaderboard

The following is a proposed exercise, not a claim that every repository tests every row. Use a scripted provider and a counted, reversible tool; retain requests, events, durable records, and execution counts.

| Injected transition | Assert at the boundary |
|---|---|
| Split a tool's JSON arguments across chunks; stop the stream early | No incomplete call executes; interruption is explicit |
| Queue steering while a tool waits | Input appears at the documented next boundary, once |
| Cancel before admission, then separately after completion | Unstarted effects remain zero; completed outcomes remain truthful |
| Change tools, system context, or model between requests | The outgoing request reflects the intended configuration and valid call/result pairing |
| Crash after an effect but before recording its result | Replay policy governs retry; uncertainty is visible; effects are counted |
| Resume after compaction or fork | The projected context matches the selected history and configuration |

Use explicit barriers to trigger each transition rather than arbitrary sleeps. Then deliberately break one invariant and confirm the test catches it. Recording a trace is useful only if assertions inspect the contract it is supposed to protect. Replaying a recorded provider response tests a known response shape; it cannot establish that the live provider still behaves the same way.

Finally, add a small real-model task set. Grade observable outcomes, such as a passing parser fixture or the correct structured artifact, independently of the model's claim of success. Pin the harness commit, model identifier, effort, prompt, tools, environment, and budgets. Change one factor per comparison, repeat trials, retain failures and timeouts, and report success alongside cost and duration. Keep missing telemetry distinct from zero. Use any discovered protocol failure to create a deterministic regression; use genuine strategy failures to improve the behavioral evaluation set.

Continue with [provider adaptation](provider-adaptation.md) and [retries and failure semantics](retries.md) for the boundaries to test, then [steering and cancellation](steering.md), [persistence and recovery](persistence.md), and [observability](observability.md) for their surrounding contracts.

[C1]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/tests/common/responses.rs#L40-L247
[C2]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/tests/common/responses.rs#L1471-L1508
[C3]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/tests/common/streaming_sse.rs#L13-L176
[C4]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/tools/parallel.rs#L606-L883
[C5]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/tools/handlers/tool_search.rs#L375-L440
[P1]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/test/suite/harness.ts#L111-L250
[P2]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/test/suite/regressions/8935-parallel-preflight-abort.test.ts#L16-L72
[P3]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/test/suite/regressions/8537-custom-message-tool-result-ordering.test.ts#L21-L151
[P4]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/test/suite/agent-session-queue.test.ts#L92-L123
[P5]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/ai/src/providers/faux.ts#L251-L297
[P6]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/evals/README.md#L74-L118
[P7]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/evals/src/report.ts#L249-L281
[P8]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/evals/src/report.ts#L359-L410
[G1]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose/src/agents/state_machine/tests/reconstruction_isolation_lifecycle.rs#L18-L212
[G2]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose/src/agents/state_machine/tests/pipeline.rs#L280-L295
[G3]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/evals/harbor/agent.py#L153-L235
[S1]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/tests/integration/test_turn_contract_intents.py#L1-L16
[S2]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/tests/integration/test_turn_contract_intents.py#L1000-L1089
[S3]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/tests/integration/test_turn_record.py#L424-L530