97.5% Prefix Cache and Why 100 Plain-Text Assistant Turns Break Tool Calling
Long-horizon multi-agent simulations fail economically long before they fail intellectually. In AI Detective, seven agents spend hours cross-examining each other across investigation rounds, private side-channels, and public debates. By the second meeting, a single seat's prompt reaches 200,000 to 360,000 tokens, and one full game consumes roughly 80 million cumulative input tokens across 600 to 840 player calls.
Without near-total KV prefix caching, running frontier models across seven seats is cost-prohibitive. Over four full-game verification runs, we pushed player prefix cache hit rates from 62.5% to 97.5% — only to uncover a subtle context bug that caused tool-calling rejections to climb even as cache efficiency peaked.
1. Lifting Prefix Cache from 62.5% to 97.5%
| Run | Player Calls | Peak Input | Cache Hit Rate | Rejection Rate |
|---|---|---|---|---|
| Run 02 | 599 | 359.8K | 62.5% | 13.2% (79) |
| Run 03 | 840 | 205.5K | 97.5% | 18.3% (154) |
| Run 04 | 812 | 198.3K | 97.4% | 21.7% (176) |
To hold 97.5% across 800+ asynchronous calls, the prompt layout must behave like an append-only log. Anything mutable — turn counters, private scratchpads, remaining action quotas, and current-step task prompts — cannot touch the system block or historical turns. It must live exclusively in the trailing user message:
Once tool schemas and early context stopped mutating mid-game, roughly 78 million of 80 million cumulative input tokens hit warm cache.
2. The Hidden Failure Mode: 100-Shot Contamination
Even with cache fixed, a troubling trend appeared in the telemetry: player tool rejection rate kept climbing from 13.2% to 18.3% to 21.7%. Nearly half of all rejections were the exact same error — no tool call emitted, with the model writing raw JSON into plain message text instead.
When we ported the same request trace to a second provider with weaker instruction-hierarchy adherence, the symptom exploded. On an identical 138K-token prompt, the model ignored tool_choice: "required" and generated 65,536 tokens of consecutive fabricated event logs until hitting the output token limit.
Inspecting the raw payload exposed the culprit immediately. Out of 615 messages in the trace, 100 messages carried role: "assistant" with tool_calls: 0 — every prior speech and inference from that seat had been serialized back into history as raw log JSON:
To an autoregressive model, 100 consecutive plain-text assistant turns is an overwhelming 100-shot few-shot demonstration: "In this conversation, you never invoke native tools; you simply emit the next raw JSON event object." Stronger models resisted the pull 85% of the time; weaker instruction-hierarchy models followed the 100-shot pattern straight off a cliff.
3. Why Dynamic Tool Pruning Is a Trap
When a model keeps picking auxiliary tools (like note or scratch) and dumping its formal move into text, the intuitive hotfix is to trim the 37-tool array down to only the 1 or 2 tools valid on that step. In isolation, single-turn compliance improves immediately.
In a long-horizon harness, it is financially catastrophic. Because tool definitions sit at the very front of the KV prefix — ahead of all 138K+ conversation tokens — mutating the tool array on every turn invalidates the entire prefix cache on every call. Trading a 97.5% cache hit for single-step tidiness multiplies per-game inference costs by 3× to 4×.
4. Four Architectural Fixes
-
Evict raw event logs from
role: "assistant". All historical game events, including the agent's own past public actions, belong inside theuserobservation stream. Reserveassistantstrictly for nativetool_callsturns. -
Keep tool schemas static and let auxiliary calls loop. Never prune the global tool array per step. If an agent invokes
noteorrecordswithout submitting a formal action, return the tool result cleanly with a reminder to submit the formal tool on the next hop. -
Make optional expression fields genuinely optional. Marking empty-allowed fields like
say,look_at, andactas schema-required caused dozens of needless retries per game. Defaulting omitted optional strings inside the validator removes the friction without losing structure. -
Clamp per-step
max_tokensas a circuit breaker. Leaving a 65K default output ceiling allows a single format-drift failure to burn ten minutes and 65,000 expensive output tokens. Capping step output at 12K–16K keeps runaway loops cheap and fast to recover.
When a frontier model breaks protocol on turn 400, the prompt wording is rarely guilty. Nine times out of ten, the harness fed it a hundred contradictory examples in the history buffer.