← Jianming Qu

Field Note 01 · November 2026 · 5 min read

97.5% Prefix Cache and Why 100 Plain-Text Assistant Turns Break Tool Calling

Long-horizon multi-agent simulations fail economically long before they fail intellectually. In AI Detective, seven agents spend hours cross-examining each other across investigation rounds, private side-channels, and public debates. By the second meeting, a single seat's prompt reaches 200,000 to 360,000 tokens, and one full game consumes roughly 80 million cumulative input tokens across 600 to 840 player calls.

Without near-total KV prefix caching, running frontier models across seven seats is cost-prohibitive. Over four full-game verification runs, we pushed player prefix cache hit rates from 62.5% to 97.5% — only to uncover a subtle context bug that caused tool-calling rejections to climb even as cache efficiency peaked.

1. Lifting Prefix Cache from 62.5% to 97.5%

Run Player Calls Peak Input Cache Hit Rate Rejection Rate
Run 02 599 359.8K 62.5% 13.2% (79)
Run 03 840 205.5K 97.5% 18.3% (154)
Run 04 812 198.3K 97.4% 21.7% (176)

To hold 97.5% across 800+ asynchronous calls, the prompt layout must behave like an append-only log. Anything mutable — turn counters, private scratchpads, remaining action quotas, and current-step task prompts — cannot touch the system block or historical turns. It must live exclusively in the trailing user message:

[immutable system] -> [static tool schemas (37 tools)] -> [append-only event log] -> [trailing dynamic state board]

Once tool schemas and early context stopped mutating mid-game, roughly 78 million of 80 million cumulative input tokens hit warm cache.

2. The Hidden Failure Mode: 100-Shot Contamination

Even with cache fixed, a troubling trend appeared in the telemetry: player tool rejection rate kept climbing from 13.2% to 18.3% to 21.7%. Nearly half of all rejections were the exact same error — no tool call emitted, with the model writing raw JSON into plain message text instead.

When we ported the same request trace to a second provider with weaker instruction-hierarchy adherence, the symptom exploded. On an identical 138K-token prompt, the model ignored tool_choice: "required" and generated 65,536 tokens of consecutive fabricated event logs until hitting the output token limit.

Inspecting the raw payload exposed the culprit immediately. Out of 615 messages in the trace, 100 messages carried role: "assistant" with tool_calls: 0 — every prior speech and inference from that seat had been serialized back into history as raw log JSON:

role: "assistant" (tool_calls: 0) -> {"id":"E1418","phase":"meeting2","kind":"speech","actor":"R5",...}

To an autoregressive model, 100 consecutive plain-text assistant turns is an overwhelming 100-shot few-shot demonstration: "In this conversation, you never invoke native tools; you simply emit the next raw JSON event object." Stronger models resisted the pull 85% of the time; weaker instruction-hierarchy models followed the 100-shot pattern straight off a cliff.

3. Why Dynamic Tool Pruning Is a Trap

When a model keeps picking auxiliary tools (like note or scratch) and dumping its formal move into text, the intuitive hotfix is to trim the 37-tool array down to only the 1 or 2 tools valid on that step. In isolation, single-turn compliance improves immediately.

In a long-horizon harness, it is financially catastrophic. Because tool definitions sit at the very front of the KV prefix — ahead of all 138K+ conversation tokens — mutating the tool array on every turn invalidates the entire prefix cache on every call. Trading a 97.5% cache hit for single-step tidiness multiplies per-game inference costs by 3× to 4×.

4. Four Architectural Fixes

When a frontier model breaks protocol on turn 400, the prompt wording is rarely guilty. Nine times out of ten, the harness fed it a hundred contradictory examples in the history buffer.