Production LLM App Caching Strategy 2026 | Prompt Caching + Semantic Caching to Cut Latency & Cost

Each LLM call is expensive and high-latency. Re-processing the same system prompt and long context in full every time, and generating from scratch for every similar question — leave that unchecked in production and your bill and p95 latency quietly balloon. Caching is the cheapest lever for cutting an LLM app's cost and latency.

But "caching" comes in two kinds with different properties, and the key is using each appropriately.

1. Prompt caching (a provider feature) — reuses the prompt's stable prefix (system instructions, long context) and discounts those input tokens. Exact-match based.
2. Semantic caching (app layer) — returns a past answer to a "paraphrased, similar question" via embedding similarity, skipping the LLM call itself. Threshold-based (= it carries a false-hit risk).

This article applies the production RAG with pgvector know-how to caching, and — after verifying each provider's official docs — designs both in real TypeScript. It's a move that complements reliability & resilience from the latency/cost side of "production AI."

⚠️ The prices and token thresholds here are 2026 official values tied to specific model names (Opus 4.8 / GPT-5 family / Gemini 3.x). They change frequently, so re-verify each source before implementing. Discount rates are model-dependent.

The two kinds of caching

Start with safe prompt caching, then carefully add semantic caching — that's the practical order.

Prompt caching: put static content at the "front"

The effect of prompt caching is decided by the structure of your prompt. The shared principle: put the large unchanging part (system instructions, long context, few-shot examples) at the front, and the changing part (user input) at the back. The more the prefix matches, the better the cache works.

• OpenAI is automatic. No code changes; it works on prompts above a length (1,024 tokens), then hits in 128-token increments. Just put static content at the front to benefit. The cached-read discount is model-dependent (~50% for the GPT-4o generation, up to 90% for the GPT-5 family).
• Anthropic is explicit via cache_control (ephemeral). Up to 4 breakpoints; default TTL 5 minutes (1-hour option). Pricing is cache write 1.25× (5-min) / 2× (1-hour), read 0.1× (relative to base input, as of 2026).
• Google Gemini has both an explicit CachedContent API and, from 2.5 onward, implicit caching on by default (note that keeping caches alive incurs a per-hour storage cost).

With the Vercel AI SDK, the simplest is the AI Gateway's automatic caching.

import { generateText } from "ai";

// 最も簡単：AI Gateway の自動キャッシュ。プロバイダに応じ最適な戦略を適用する
// （Anthropic系は cache_control を注入、OpenAI/Google は暗黙キャッシュを利用）。
const { text } = await generateText({
  model: "anthropic/claude-opus-4-8",
  messages,
  providerOptions: { gateway: { caching: "auto" } },
});

For explicit control, place cacheControl on the stable prefix.

// 明示制御：静的コンテンツを"前"に置き、そこに cacheControl を付ける。
const { usage } = await generateText({
  model: "anthropic/claude-opus-4-8",
  messages: [
    {
      role: "system",
      content: largeStaticContext, // 大きく・変わらない部分（RAG文脈やfew-shot）
      providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
    },
    { role: "user", content: userQuestion }, // 変わる部分は後ろ
  ],
});
// usage.inputTokenDetails.cacheReadTokens でヒットを計測（読み取りは入力の約0.1×課金）

Prompt caching is virtually no-risk (exact match, so no wrong answers). Start here as a rule.

Semantic caching: skip the call itself

Go one step further: "don't call the LLM at all for similar questions." Embed the question, match it against past questions by cosine similarity, and if it exceeds a threshold, return the past answer. Where prompt caching is "exact-prefix reuse," this catches paraphrases — its strength.

Separate the decision logic (threshold, freshness) into a pure function for testability.

// セマンティックキャッシュ：埋め込み類似で「言い換え」まで拾う。判定は純粋関数で分離。
interface CacheHit {
  readonly answer: string;
  readonly similarity: number; // 0..1（コサイン類似）
  readonly ageMs: number;
}

// 純粋・テスト容易：閾値と鮮度(TTL)の両方を満たすヒットだけ採用する（SRP）。
function acceptCacheHit(hit: CacheHit | null, minSimilarity: number, ttlMs: number): boolean {
  return hit !== null && hit.similarity >= minSimilarity && hit.ageMs <= ttlMs;
}

// cache-aside：ヒットなら即返す（LLM呼び出しを省く）、ミスなら生成して保存。
async function cachedGenerate(query: string, scope: string): Promise<string> {
  const embedding = await embed(query);
  // pgvector で最近傍を1件（scope でユーザー/地域を隔離し、他者の応答を返さない）
  const hit = await cache.nearest(embedding, scope);
  if (acceptCacheHit(hit, 0.95, 3_600_000)) return hit!.answer; // 高閾値で偽ヒットを抑制
  const answer = await callLlm(query);
  await cache.put({ query, embedding, answer, scope }); // ミス：保存して次回に備える
  return answer;
}

Semantic caching: risks and controls

The danger of semantic caching is false hits. "What were 2024 sales?" and "What were 2025 sales?" have near-identical embeddings, yet the correct answers differ. A loose threshold mass-produces wrong answers. Four controls:

• Threshold — start high (0.90–0.95) and tune while monitoring the false-hit rate (AWS, too, states "the threshold controls the trade-off between hit rate and answer quality").
• TTL — by data volatility. Real-time (prices/inventory) 5–15 min, static facts 24 hours. Longer TTL raises hit rate but also staleness risk.
• Scope — filter by user, region, product ID, so a personalized response isn't served to someone else.
• Invalidation — invalidate the affected entries content-triggered when source data updates.

And measure hit quality with evaluation. Include "is this cache hit correct?" in offline evaluation and gate on the false-hit rate (LLM evaluation & testing). The go-to OSS is GPTCache, where you can swap the threshold and vector store.

Be equally clear about when NOT to cache. Always needs freshness, heavily personalized, audit requires fresh generation — for these, the cost of false hits/staleness outweighs the benefit, so just call every time.

Observability: measure the effect and the side effects

Caching is only safe once you measure it. Record hit rate, tokens/cost saved, and p95 latency, plus the false-hit rate (from evaluation). Metrics guide tuning — "hit rate is high but false hits rose = threshold too loose." For the foundation, see production observability with OpenTelemetry. Note that caching is a different layer of optimization from model/infra economics (self-host vs API); for that, see generative-AI cost design.

Adoption checklist

• Prompt caching (the safe first move) — static content at the front, OpenAI automatic, Anthropic cacheControl (or Gateway caching:'auto') enabled?
• Structure — the large unchanging part at the front, the variable part at the back?
• Semantic caching (carefully) — starting from a high similarity threshold (0.90+)?
• TTL — set by data volatility, and invalidated on source updates?
• Scope — hits isolated by user/region, not leaking personalization?
• False-hit monitoring — is cache-hit correctness in evaluation and gated?
• Non-cache decision — excluded queries that need freshness / heavy personalization / audit?
• Observability — monitoring hit rate, cost saved, latency, and false-hit rate?

Summary

Caching for LLM apps is, by convention, "a foundation of safe prompt caching, with semantic caching carefully layered on."

• Prompt caching is exact-match and virtually no-risk. Just putting static content at the front lowers input cost (Anthropic reads ≈0.1×, OpenAI automatic).
• Semantic caching is a powerful way to skip whole calls, but false hits are its intrinsic risk. Manage it with a high threshold, TTL, scope, and evaluation.
• Operate by measuring. Monitor not just hit rate but the false-hit rate, and include the decision not to cache in your design.

Fast, cheap, and accurate — you can have all three at once when you design caching correctly.