Prompt caching for LLM apps: put the stable parts first and watch the usage fields

An agent works in a loop: read the situation, call a tool, read the result, decide again. Every turn sends the model the same long beginning (instructions, tool definitions, earlier messages) plus a little that is new. You pay to process that repeated beginning each time.

Prompt caching lets the provider remember the processed beginning and reuse it. Done right, it lowers both cost and latency. Done carelessly, it quietly never hits. This post explains the mechanics using the Anthropic prompt caching documentation; other providers have similar features with different details, so check yours.

The rule: an exact prefix

Caching works on a prefix of your request. The docs describe the order as tools first, then the system prompt, then messages, up to the block you mark with cache_control. A cache hit needs that whole prefix to match exactly.

That gives one design rule: put what rarely changes at the front and what changes every turn at the back.

flowchart LR
    A[Tool definitions] --> B[System prompt]
    B --> C[Earlier messages]
    C --> D[New user turn]
    A -.stable.-> S((cached))
    B -.stable.-> S
    C -.grows.-> S
    D -.new each time.-> N((not cached))

You can switch caching on with a top-level cache_control field, in which case the breakpoint moves forward as the conversation grows. Or you can place breakpoints on specific blocks yourself, with up to four allowed.

{
  "model": "claude-sonnet-5-5",
  "max_tokens": 1024,
  "cache_control": { "type": "ephemeral" },
  "system": "You are a support agent for ...(long, stable instructions)...",
  "messages": [{ "role": "user", "content": "Where is my order?" }]
}

Lifetimes and prices

Two things to know before you plan around it:

• Time to live. The default is 5 minutes, refreshed each time the cache is read. An optional 1-hour lifetime exists for prompts reused less often than every five minutes but more often than hourly.
• Price shape. Writing to the cache costs a bit more than normal input (the docs list 1.25 times the base price for the 5-minute option and 2 times for the 1-hour option), while reading costs a small fraction of it, 0.1 times or lower depending on the model.

Because a read is so much cheaper than a write, caching pays off when a prefix is reused even a few times. If a prompt is used once, caching only adds the write premium. Prices change, so read the current table before doing any math.

There is also a floor. Prompts shorter than a model's minimum cacheable length are processed without caching, and no error is raised. The docs list minimums that differ by model, from 512 tokens on some current models to 4,096 on older ones. A short system prompt may simply never be cached.

What breaks the cache

This is where teams lose their savings without noticing. Per the docs, changes invalidate the cache from the changed point onward:

Change	What is invalidated
Tool names, descriptions or parameters	Tools, system and messages
System prompt edits, or toggling some features	System and messages
tool_choice changes, adding or removing images	Messages
Cache lifetime lapsing with no read	Everything cached

Common accidental causes are easy to find once you look for them:

• A timestamp or request ID at the top of the system prompt, which makes every request unique.
• Tool lists built from a dictionary or set whose order changes between runs.
• Per-user details placed before the shared instructions instead of after.
• Rewriting or summarizing earlier messages each turn instead of appending to them.

Tip

Treat the front of your prompt like a published file: versioned, deterministic and edited deliberately. If you version your prompts anyway, a cache miss after a release is also a useful signal that something at the front changed.

Measure, do not assume

Every response reports what happened in its usage block:

• cache_creation_input_tokens: tokens written to the cache on this request.
• cache_read_input_tokens: tokens served from the cache.
• input_tokens: tokens after the last breakpoint that were processed normally.

Total input is the sum of those three. If both cache fields are zero, nothing was cached, and the usual reason is a prompt below the minimum length.

A simple habit: log these three numbers for every call and chart the share of reads. On a healthy agent loop, the first turn writes and later turns mostly read. If reads stay near zero, walk back through the list above.

u = response.usage
total = u.input_tokens + u.cache_creation_input_tokens + u.cache_read_input_tokens
hit_rate = u.cache_read_input_tokens / total if total else 0
log.info("llm_call", extra={"cache_hit_rate": round(hit_rate, 2)})

Fitting it into an agent design

A few design choices make caching more effective, and they are good practice anyway:

1. Keep the tool list small and stable. Fewer tools means a smaller, steadier prefix. This lines up with the advice in our post on choosing the simplest design that does the job.
2. Append, do not rewrite. Add new turns to the end of the conversation. If you must compact history, do it rarely and expect one cache write afterwards.
3. Match lifetime to rhythm. A busy loop is fine with the default. A nightly job that reuses a big prompt a few times an hour might justify the longer lifetime.
4. Test with evals. A caching change should not change answers. If you run an eval set, as described in evals for AI agents, run it before and after to confirm the answers match and the cost drops.

A short checklist

• Stable content first, volatile content last.
• Nothing random or time-based near the top.
• Deterministic ordering of tools and examples.
• Prefix long enough to meet the model's minimum.
• Log the three usage fields and track the hit rate.
• Re-read the pricing and minimums page now and then; they change.

Caching is one of the few optimizations that makes an LLM app cheaper and faster without changing what it says. It just needs a prompt that sits still.

Sources

• Anthropic documentation: Prompt caching