OpenAI Prompt Cache Diagnostics Without Guesswork
A practical guide to comparing Responses API requests, interpreting cache-miss reasons, and preventing small request changes from quietly increasing latency and cost.
By Amaresh Adak13 min readfree tools · most run in your browser · no signupfree developer tools/most run right in your browser/no signup needed
A practical guide to comparing Responses API requests, interpreting cache-miss reasons, and preventing small request changes from quietly increasing latency and cost.
By Amaresh Adak13 min readFree toolkit
85+ private dev tools
Everything runs in your browser. Zero tracking, no sign-up.
Browse toolsOpenAI prompt cache diagnostics give engineers something cache dashboards cannot: a reason why one Responses API request failed to reuse the prefix of another. Instead of staring at cached_tokens: 0 and blaming “the cache,” you can compare two response IDs and find whether the model, cache key, tools, output schema, reasoning settings, or input changed.
The feature is useful, but it is deliberately narrow. It explains one comparison at a time, reports only the first classified mismatch, and does not replace usage metrics. This guide turns those constraints into a practical debugging and regression-testing workflow.
Prompt caching reuses model work for an unchanged prefix. OpenAI caches the rendered context, which can include hidden service instructions, tool definitions, developer messages, conversation history, text, images, documents, and supported audio. A match therefore depends on more than the string you call “the prompt.”
Caching is enabled automatically for supported models. For GPT-5.6 and later, a reusable prefix must contain at least 1,024 visible input tokens. The first eligible request writes cache state; later requests can reuse the longest matching eligible prefix. OpenAI’s current prompt-caching guide documents a default minimum lifetime of 30 minutes for GPT-5.6 and later, refreshed by a write or reuse.
Diagnostics add a comparison layer. On a current request, you supply the ID of a recent completed response that should have shared a prefix. The returned prompt_cache_diagnostics object classifies that relationship.
| Signal | Answers | Does not answer |
|---|---|---|
prompt_cache_diagnostics |
Why expected reuse did or did not occur against one baseline | The application-wide hit rate |
usage.input_tokens_details.cached_tokens |
How many input tokens the current request actually reused | Why the remainder missed |
cache_write_tokens |
How many tokens were written to cache | Whether a future request will reach that entry |
| Prompt Caching Dashboard | Aggregate cache performance | Which exact request field changed |
That distinction matters. A diagnostic cache_hit is not a claim that every input token was cached. A request with a 2,000-token reusable prefix and 500 new tokens can be a diagnostic hit while still processing the new suffix normally. Use usage fields for billing and volume; use diagnostics for the explanation.
The comparison requires the Responses API and a supported GPT-5.6-or-later model. Install the current Python SDK, keep an API key in the environment, and use a policy or reference block long enough to cross the model’s 1,024-token cacheability threshold.
python -m pip install --upgrade openai
export OPENAI_API_KEY="your-api-key"
The following example makes a baseline request and then compares a second request against it. The stable cache key is intentional; changing it per request is a wonderfully efficient way to make your cache inefficient.
from pathlib import Path
from openai import OpenAI
client = OpenAI()
policy = Path("support-policy.txt").read_text(encoding="utf-8")
baseline = client.responses.create(
model="gpt-5.6",
instructions=policy,
input="Summarize the refund policy.",
prompt_cache_key="support-policy:v3",
)
current = client.responses.create(
model="gpt-5.6",
instructions=policy,
previous_response_id=baseline.id,
input="Does the policy also apply to sale items?",
prompt_cache_key="support-policy:v3",
prompt_cache_options={
"comparison_response_id": baseline.id,
},
)
diagnostic = current.prompt_cache_diagnostics
usage = current.usage.input_tokens_details
print("diagnostic:", diagnostic.model_dump() if diagnostic else None)
print("cached_tokens:", usage.cached_tokens)
print("cache_write_tokens:", usage.cache_write_tokens)
Two similarly named IDs do different jobs:
previous_response_id continues conversation state. It does not request cache diagnostics.prompt_cache_options.comparison_response_id chooses the diagnostic baseline. It neither imports that response’s conversation nor limits which matching cache entry the current request can reuse.They may point to the same response, as above, but they do not have to. If your application sends complete history in input, omit previous_response_id and keep only the comparison ID.
One subtlety is easy to miss: top-level instructions are not carried forward by previous_response_id. Supply the same instructions again when they are part of the reusable prefix. Otherwise your “continuation” quietly becomes a different rendered request.
The official diagnostics guide defines four top-level outcomes.
type |
Meaning | Next move |
|---|---|---|
cache_hit |
No miss was detected against the comparison | Read cached_tokens to measure actual reuse |
cache_miss |
A classified difference prevented expected prefix reuse | Inspect reason and token estimates, then fix and compare again |
comparison_response_not_found |
The short-lived diagnostic record is missing or expired | Use another recent completed response from the same organization |
unavailable |
The comparison was inconclusive or unsupported | Verify model support; treat the model response itself as valid |
For a miss, cache_missed_tokens estimates how many reusable baseline tokens were not reused. When present, comparison_reusable_tokens is the baseline’s cached_tokens + cache_write_tokens. These are comparison aids, not invoice fields. Actual cache usage and billing come from the current response’s usage object.
Diagnostics surface the first classified reason, not a diff of the entire request. Fixing tools_changed might reveal text_format_changed on the next run. This is less “one report and done” and more peeling a small, expensive onion.
The current API can classify these reasons:
| Reason | Typical production cause | Corrective action |
|---|---|---|
model_changed |
Router, experiment, or fallback chose another model | Keep model selection stable for a cache cohort |
prompt_cache_key_changed |
Key includes a request ID, timestamp, or random value | Use a stable key for requests sharing a prefix |
service_tier_changed |
Actual serving tier differs between requests | Compare returned tiers and stabilize routing policy |
tools_changed |
Tool added, removed, reordered, or its schema changed | Keep the supplied tool catalog and ordering stable |
text_format_changed |
Structured-output schema changed | Version and reuse the same schema when output shape is unchanged |
reasoning_effort_changed |
Request-level reasoning effort changed | Keep the setting consistent within the comparison cohort |
verbosity_changed |
text.verbosity changed |
Stabilize it or deliberately accept the cache break |
context_compacted |
Earlier history was replaced by compacted context | Measure total cost; fewer tokens may still be the better trade |
input_changed |
Timestamp, ID, edited history, or reordered messages appeared early | Move volatile data after the stable prefix and append turns |
Not every miss is a bug. A model fallback may preserve availability, and compaction may cut more ordinary input tokens than it costs in lost reuse. Diagnostics tell you what changed; architecture still decides whether the change was worthwhile.
Tools deserve special attention because their definitions are part of the rendered prefix. Do not remove a tool merely because one turn should not call it. Keep definitions and ordering stable, then use tool_choice: "none" to disable tool use or allowed_tools to limit the callable subset when supported. That keeps authorization explicit without rewriting the catalog.
This pairs naturally with a bounded tool executor. If you are building one, the site’s Programmatic Tool Calling guide covers execution and approval boundaries; cache stability does not make a tool safe to run.
Start by making request construction boring. Boring is a compliment in infrastructure.
Place versioned developer instructions, policy text, few-shot examples, and stable tool definitions before per-request content. Put timestamps, trace IDs, tenant-specific snippets, retrieved documents, and the current user message later.
Bad ordering:
instructions = f"""
Request time: {request_time}
Trace ID: {trace_id}
{stable_policy}
"""
Better ordering:
input_items = [
{"role": "developer", "content": stable_policy},
{
"role": "developer",
"content": f"Request time: {request_time}
Trace ID: {trace_id}",
},
{"role": "user", "content": user_question},
]
Keep observability metadata in your own logs whenever the model does not need it. A trace ID is useful to an engineer; the model rarely becomes wiser after reading it.
For multi-turn work, append messages, tool calls, and results. Editing a prior message, rebuilding history in a different order, or summarizing it changes the prefix from the first changed token onward. If you must compact, establish the compacted state as the new baseline and evaluate the total input cost rather than chasing the old hit rate.
This is also where conversation design affects more than caching: Responses, Conversations, and previous_response_id impose different state and retention choices, so choose the model deliberately.
prompt_cache_key influences routing; it does not pin a request to one machine or guarantee a hit. Use a key that represents the reusable prompt family and version, not an individual HTTP request.
Reasonable:
support-agent:v3:tenant-42:shard-07
Counterproductive:
support-agent:request-01JXYZ...
At higher traffic, deterministic sharding can spread load while keeping each prefix family mapped consistently. Cache entries live on individual machines, and overflow routing can still cause misses. A good key improves the odds; it is not a tiny contractual hostage note to the scheduler.
A unit test cannot prove a future cache hit because placement, lifetime, and capacity are runtime concerns. It can catch the request drift most likely to cause misses.
First, canonicalize the fields your application owns and fingerprint them in CI:
import hashlib
import json
from typing import Any
def cache_contract_fingerprint(request: dict[str, Any]) -> str:
contract = {
"model": request["model"],
"instructions": request.get("instructions"),
"tools": request.get("tools", []),
"text": request.get("text"),
"reasoning": request.get("reasoning"),
"service_tier": request.get("service_tier"),
"prompt_cache_key": request.get("prompt_cache_key"),
}
encoded = json.dumps(
contract,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
Store the expected fingerprint beside a prompt version. A tool reordering, schema edit, or accidental dynamic instruction then becomes a deliberate review event. This fingerprint is an application-side guard, not OpenAI’s cache hash, and it intentionally covers only fields you include.
Second, run a small authenticated integration check in a non-production project after changing a model, prompt, tools, or response schema:
comparison_response_id.Do not make deployment depend on a single live cache hit. A cache can expire or overflow without your request contract being wrong. Gate on deterministic contract checks; use live diagnostics as supporting evidence.
Diagnostics are best used as sampled debugging data. Adding them to every request can create noisy high-cardinality logs even though the diagnostic feature itself has no additional charge or separate rate-limit cost.
For each sampled comparison, log:
type and reason;cached_tokens, cache_write_tokens, and total input tokens;Track ratios by prompt family rather than globally. An agent with a 20,000-token stable tool-and-policy prefix should not be averaged with short classification requests that never meet the minimum cacheable length. The aggregate number will look scientific and answer almost nothing.
Tie cache metrics to economics. The hard spend limits guide covers budget enforcement, while the Fast mode guide explains why cached input can change the value of paying for a lower-latency serving tier.
Four constraints belong in the runbook:
unavailable does not prove a hit or a miss, and diagnostics never fail or block the model response.OpenAI says diagnostics are compatible with Zero Data Retention. The diagnostic record contains configuration metadata, token estimates, and hashes rather than raw prompts or model outputs; it is scoped to the organization and expires after a short period. Supplying a comparison ID does not retrieve or persist the earlier response’s content.
Regional boundaries still matter. Prompt caches are not shared across organizations or regional-processing boundaries, so a US request should not be expected to reuse an EU cache entry. If request geography is policy-driven, keep it explicit as described in the site’s regional processing guide.
The diagnostic feature has no additional charge and does not count separately toward rate limits. The Responses API calls you make for baselines, follow-ups, and retries are billed normally and consume normal rate-limit capacity.
cache_hit mean the full prompt was cached?#No. It means the comparison did not detect a miss for the expected reusable prefix. Read usage.input_tokens_details.cached_tokens to see the current request’s actual reuse; any new suffix still needs processing.
comparison_response_id continue a conversation?#No. It selects a diagnostic baseline only. Use previous_response_id to continue a response chain, or send conversation history yourself. The same response ID can serve both roles when that matches your design.
comparison_response_not_found appear for a valid response ID?#The response object and its diagnostic record have different lifetimes. Diagnostics retain comparison data only briefly, and the baseline must be a recent completed response from the same organization. Choose a newer baseline and rerun the comparison.
Usually not. Diagnostics never change generation and may be inconclusive. Treat cache reuse as a performance and cost optimization, then alert on sustained regressions by prompt family. Fail only if a separate business rule—such as a hard budget boundary—requires it.
OpenAI prompt cache diagnostics replace guesswork with a bounded comparison: choose a recent baseline, inspect the first classified difference, confirm actual reuse in usage fields, and repeat after each repair. The durable fix is rarely a clever cache trick. It is stable request construction—versioned prompts, append-only history, consistent tools and settings, deliberate cache keys, and regression checks that make drift visible before the bill does.
Comments
Sign in to join the discussion.
No comments yet. Be the first to share your thoughts.