OpenAI Makes Cache Hit Rate a Product Metric for Agents
OpenAI added monitoring and controls for GPT-6 prompt caching. Agent teams now need to manage cache hit rate as a product metric.
Long-running agents resend the same instructions, tool definitions and conversation context. On 22 September, OpenAI expanded prompt caching for GPT-6 with a dashboard, miss diagnostics, explicit breakpoints and prewarming. OpenAI says eligible shared prefixes reused within 30 minutes can receive discounts of up to 90% on cached input tokens.
Context layout now affects an agent's operating economics. A release that changes a tool schema or reorders definitions can break reuse, increasing latency and cost even when task quality stays flat.
Track cache hit rate beside time to first response, inference cost and accepted task completion. Keep stable instructions and tool definitions together, append volatile context later, and investigate misses after every release.
Before scaling a persistent agent, set a cache-regression budget. The discount matters only when it lowers cost per accepted task without creating fragile prompt plumbing.
