# Prompt caching: preserve what the next request can reuse

An agent repeatedly sends much of the same context: instructions, tool definitions, and earlier conversation. **Prompt caching** lets a model provider reuse eligible computation from an earlier request. The harness controls the request it constructs; the provider controls whether reuse occurs. A cache key, retention preference, or breakpoint is a request to participate in that mechanism, not proof of a hit.

Keep this separate from application caches such as tool-discovery indexes, stored token counts, credentials, or loaded skill files. Also distinguish provider caching from a transport optimization that sends only new messages: reducing bytes sent over a WebSocket does not by itself prove model computation was reused.

This page compares the pinned request-building paths. It does not assert universal provider invalidation rules, current service guarantees, or measured savings. Sidekick citations require access to its private repository; its native OpenRouter path is separate from any externally wrapped harness.

## A stable prefix is an architectural choice

Imagine an effective prompt containing stable instructions and tools, then conversation history, then newly changing information. Appending a tool result can leave the earlier prefix unchanged. Editing an early instruction or tool declaration changes the request before that history. A prefix-based cache may then reuse less of the request. Actual eligibility also depends on the provider, model, request options, routing, and retention behavior.

![Unchanged context followed by new results versus an early tool-catalog change](../guide/assets/prompt-cache-prefix.svg)

The diagram is conceptual. It does not specify every provider's wire ordering, tokenization, cache hierarchy, or hit boundary. The implementations below demonstrate different ways to preserve reusable input.

## Four approaches in the inspected code

| Harness | Concrete mechanism | Important limit |
| --- | --- | --- |
| Codex | Stable prompt-item identities, session-scoped cache affinity, compatible incremental requests, and gated effort updates in conversation history. | WebSocket continuation compatibility and provider cache hits are different decisions. [Request construction][c-request], [continuation][c-continuation] |
| pi | Structured prompt/tool changes plus provider-specific conversion; compatible adapters can append changes while retaining initial declarations. | Unsupported APIs collapse updates into current leading instructions/tools. [Prompt changes][p-prompt], [transcript conversion][p-transcript] |
| Goose | Sorted tools, less volatile prompt timestamps, provider/model cache semantics, and explicit breakpoint placement on applicable routes. | Sorting avoids accidental differences; a real catalog or prompt change still changes input. [Tool ordering][g-order], [prompt time][g-time], [cache policy][g-cache] |
| Sidekick | Deterministic skill metadata, stable skill dispatcher tools, up to two message breakpoints, and changing “vanishing” context appended after history. | Rebuilding the catalog or tool configuration can still alter the early request. [Catalog][s-catalog], [wire formatting][s-wire], [skill tools][s-tools] |

## What changes when configuration changes?

“Invalidated” is often too vague. Record which input changed and which reuse mechanism stopped applying.

| Change | Potential effect | Harness response or boundary |
| --- | --- | --- |
| Model or provider | A different execution target and request representation; do not assume reuse across models. | Codex rejects incremental continuation when the model field changes. Pi changes model state and records the selection. Neither proves the provider deleted an older cache entry. [Codex check][c-mismatch], [pi selection][p-model] |
| Reasoning effort | May change a top-level setting or become an appended configuration item. | Compatible Codex and pi paths preserve a request-level baseline and express later effort changes separately. Other paths send the changed setting directly. [Codex effort][c-effort], [pi effort][p-effort] |
| System instructions | Replacing early text changes the prefix; appending a supported update can preserve it. | Pi supports both representations, selected by compatibility. Codex rebuilds base-instruction items with identities derived from their content. [Pi conversion][p-transcript], [Codex construction][c-request] |
| Tool catalog | Addition, removal, redefinition, or ordering can change earlier input. | Goose sorts declarations. Pi has gated incremental tool representations, with fallbacks. A stable loader/dispatcher avoids inserting every newly discovered capability into the initial catalog. [Ordering][g-order], [pi tools][p-tools], [Sidekick dispatcher][s-tools] |
| Skill metadata | A changed description, name, version, or visible inventory can change prompt content. | Stable ordering removes incidental differences, not meaningful edits. Sidekick sorts the rendered catalog; Goose sorts discovered entries. [Sidekick catalog][s-render], [Goose skills][g-skills] |
| Skill body or reference loaded on demand | Normally adds content later through a reader, loader, or explicit injection. | Codex can inject selected instructions as turn input; pi reads files; Goose and Sidekick have loader tools. This differs from rewriting earlier metadata. [Loading paths](skills.md). |
| Compaction or history editing | The next request may no longer extend the previous input. | Codex's incremental path detects shortening or mismatch; successful compaction also allows a new effort baseline. Provider reuse of any surviving portion remains a separate question. [Continuation][c-continuation], [effort baseline][c-effort] |

## Codex: request identity, continuation, and effort

Codex derives its normal prompt cache key from the session identity, with explicit overrides and parent-scoped internal-session cases. It also gives rebuilt base-instruction and Responses Lite tool items deterministic identities derived from their visible contents within the thread. These reduce accidental request differences; a changed instruction still produces different content and identity. [Cache identity][c-key], [request construction][c-request].

Its WebSocket continuation path checks that request properties match and that input extends the previous request plus returned output. Model, top-level tools, reasoning, cache key, and other property changes can reject that incremental path. It then needs a non-incremental request; this is not evidence that every provider-side prefix was lost. [Compatibility checks][c-mismatch], [input comparison][c-continuation].

There is a deliberate exception to “effort changes rewrite request configuration.” On the feature/model-supported path, Codex pins request-level effort for the context window and appends trusted configuration updates for changed effort. Successful compaction retires that baseline; unsupported models use the selected request-level effort. This preserves the instruction to use a new effort while avoiding unnecessary changes to the older request representation. It does not guarantee a cache hit. [Effort implementation][c-effort].

## pi: preserve changes in the transcript when the API supports it

Pi compares structured system-prompt sections and produces an update for the next request. An API supporting mid-conversation system messages can retain those updates in place. Otherwise conversion collapses them into the current leading system message, changing the earlier prompt when necessary. A forced system prompt from an extension can also replace the request's leading prompt. Correct instructions take precedence over prefix preservation. [Prompt preparation][p-prompt], [conversion fallback][p-transcript].

Tool updates are similarly conditional. The inspected Anthropic adapter can keep initial request-level tools fixed and encode later additions/removals in messages, including a deferred placeholder from the initial request. This path requires support for both mid-conversation system messages and tool changes, plus a nonempty initial tool list. The OpenAI-style transcript helper anchors additions only when supported and when there are no removals or redefinitions; otherwise it sends the current catalog. These are compatibility-gated implementations, not universal API capabilities. [Anthropic conversion][p-anthropic], [tool fallback][p-tools].

Effort updates can use message/configuration items on supported paths; the OpenAI adapter excludes its configuration-update path for ChatGPT sign-in. Cache retention is also translated per adapter: pi exposes `none`, `short`, and `long`, while emitted fields depend on capabilities and authentication. Treat these as requested policies, not identical lifetimes across providers. [Effort and retention options][p-effort], [request fields][p-request].

Pi additionally contains configurable cache warming: it may replay the prior request with a small output budget when its eligibility and estimated-economics checks permit. It stops when the saved request is no longer current and skips unsupported replay cases. This makes extra provider calls; its cost estimate is not a measured saving for a particular workload. [Warmer eligibility][p-warmer], [refresh and cost decision][p-warmer-cost].

## Goose: deterministic assembly and provider-specific breakpoints

Goose sorts model-visible tools by name and rounds its generated prompt timestamp to an hour. Both avoid changes unrelated to the task; the timestamp can still change at an hour boundary. When caching is enabled, its native Anthropic formatter marks the system prompt, final tool declaration, and last two user messages. Its OpenRouter-style chat helper instead marks eligible content near the tail and an earlier anchor, plus system/tools. Different envelopes require different placement strategies. [Ordering][g-order], [time][g-time], [Anthropic format][g-anthropic], [chat breakpoints][g-cache].

The OpenAI provider can attach the session ID as `prompt_cache_key`. Goose also accepts a configured cache TTL and re-derives it on session resume rather than treating a saved value as permanent session state. These controls do not preserve reuse after arbitrary model, reasoning, or prompt changes. No general mechanism deferring those changes solely to protect caching was established in this review. [OpenAI key][g-key], [retention configuration][g-retention].

## Sidekick: stable catalog, changing tail

Sidekick marks the first non-vanishing system message and last non-vanishing message as cache breakpoints. Its wire formatter consistently uses content arrays, avoiding a representation change just because a message stops being the active breakpoint. Configured vanishing context—such as current contextual information—is appended after history and excluded from breakpoint selection. This is a concrete separation between stable and frequently changing input. [Wire formatting][s-wire], [tail assembly][s-tail].

Its skill catalog is sorted and includes resolved version metadata in the system prompt. Loading instructions or references returns tool-result content; the fixed `Skill`, `ReadSkillReference`, and `UseSkillTool` interface avoids adding each skill's tools to the advertised array. However, editing attachments, descriptions, or resolved versions can change the catalog, and the catalog is rebuilt during context assembly. “Stable” means deterministic for the same inputs, not frozen forever. Native reasoning options are passed to OpenRouter; no equivalent effort-baseline preservation was established here. [Catalog assembly][s-catalog], [catalog rendering][s-render], [skill interface][s-tools], [reasoning parameters][s-reasoning].

## Trace a skill change and design for it

Suppose request A advertises a `review` skill. Request B loads its body through a tool and appends the result. That can preserve A's earlier prefix. Before C, someone edits the advertised description and publishes a new version. If the harness rebuilds that early catalog, C changes before the appended result—even though the visible conversation looks unchanged. A supported appended instruction update offers a different representation; simply retaining the same cache key does not undo the change.

For a custom harness, make request assembly deterministic, place volatile data deliberately, and distinguish catalog edits from on-demand loading. Snapshot or version configuration at a documented boundary; defer optional refreshes only when doing so preserves correct behavior. Never delay revoked permissions or required instruction updates for cache savings.

Measure provider-reported cache reads/writes alongside model, request settings, and a digest of the effective prompt components. Track transport continuation separately. Test a repeated request, appended tool result, reordered catalog, changed schema, changed effort, skill load, skill edit, and compaction. These experiments reveal your actual contracts; code comments about caching do not substitute for observed provider usage.

[c-request]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/client.rs#L955-L1006
[c-continuation]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/client.rs#L1496-L1556
[c-mismatch]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/client.rs#L389-L457
[c-key]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/client.rs#L646-L669
[c-effort]: https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/src/session/reasoning_effort.rs#L1-L160
[p-prompt]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/src/core/agent-session.ts#L1734-L1768
[p-transcript]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/ai/src/utils/transcript.ts#L104-L120
[p-model]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/src/core/agent-session.ts#L2507-L2527
[p-tools]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/ai/src/utils/transcript.ts#L221-L237
[p-anthropic]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/ai/src/api/anthropic-messages.ts#L1172-L1295
[p-effort]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/ai/src/api/openai-responses.ts#L69-L134
[p-request]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/ai/src/api/openai-responses.ts#L331-L407
[p-warmer]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/src/core/cache-warmer.ts#L186-L243
[p-warmer-cost]: https://github.com/earendil-works/pi/blob/482c1381a00d8d00bc81697e4a980c593db6803e/packages/coding-agent/src/core/cache-warmer.ts#L300-L399
[g-order]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose/src/agents/reply_parts.rs#L184-L207
[g-time]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose/src/agents/prompt_manager.rs#L177-L205
[g-cache]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose-provider-types/src/cache_semantics.rs#L1-L110
[g-anthropic]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose-provider-types/src/formats/anthropic.rs#L521-L600
[g-key]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose-providers/src/openai.rs#L328-L341
[g-retention]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose/src/model_config.rs#L145-L170
[g-skills]: https://github.com/aaif-goose/goose/blob/34a222e3f2b95d8b4a43482b53c2cc957ecce26d/crates/goose/src/agents/state_machine/ops_skills.rs#L62-L106
[s-catalog]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/src/sidekick/services/context_builder_service.py#L838-L868
[s-render]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/src/sidekick/services/skill_service.py#L212-L248
[s-wire]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/src/sidekick/utils/wire_format.py#L25-L138
[s-tools]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/src/sidekick/services/context_builder_service.py#L1903-L1918
[s-tail]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/src/sidekick/services/context_builder_service.py#L995-L1038
[s-reasoning]: https://github.com/datex/sidekick/blob/463d653124e1c2cda46a3f14f01f3d6f384326b1/backend/src/sidekick/services/llm_providers/openrouter_native.py#L273-L325