# Production LLM App Caching Strategy 2026 | Prompt Caching + Semantic Caching to Cut Latency & Cost

> An implementation guide to two kinds of caching that cut latency and cost for generative-AI/LLM apps: provider prompt caching (Anthropic cache_control, OpenAI automatic, Google context caching) and application-layer semantic caching (embedding similarity, pgvector) — with when to use which, false-hit mitigation, TTL design, and real TypeScript code.

- Published: 2026-07-23
- Author: 友田 陽大
- Tags: 生成AI, LLM, キャッシュ, コスト最適化, パフォーマンス, RAG, pgvector, AIエージェント, TypeScript, アーキテクチャ設計
- URL: https://tomodahinata.com/en/blog/llm-application-caching-prompt-semantic-latency-cost-guide
- Category: 本番LLMアプリ実装（信頼できるAI）
- Pillar guide: https://tomodahinata.com/en/blog/production-llm-application-engineering-guide

## Key points

- Each LLM call is expensive and high-latency. Re-processing the same/similar inputs every time is wasteful. Caching cuts it dramatically.
- Use two kinds appropriately. Prompt caching = the provider reuses a stable prefix and discounts the input (Anthropic reads ≈0.1×). Semantic caching = the app layer returns a past answer to a 'paraphrased' question via embedding similarity, skipping the call entirely.
- Prompt caching hits more when static content is at the FRONT. OpenAI is automatic (≥1024 tokens); Anthropic is explicit via cache_control.
- The biggest risk of semantic caching is false hits (semantically similar but different intent). Manage it with a high similarity threshold (start 0.90–0.95), TTL, scope, and quality monitoring via evaluation.
- Set TTL by data volatility. Real-time data (prices/inventory) 5–15 min; static facts ~24 hours.

---

Each LLM call is **expensive and high-latency**. Re-processing the same system prompt and long context in full every time, and generating from scratch for every similar question — leave that unchecked in production and your bill and p95 latency quietly balloon. **Caching is the cheapest lever for cutting an LLM app's cost and latency.**

But "caching" comes in two kinds with different properties, and the key is **using each appropriately.**

1. **Prompt caching (a provider feature)** — reuses the prompt's **stable prefix** (system instructions, long context) and **discounts those input tokens.** Exact-match based.
2. **Semantic caching (app layer)** — returns a past answer to a "paraphrased, similar question" via **embedding similarity**, **skipping the LLM call itself.** Threshold-based (= it carries a false-hit risk).

This article applies the [production RAG with pgvector](/blog/pgvector-postgres-production-rag-hybrid-search) know-how to caching, and — after **verifying each provider's official docs** — designs both in real TypeScript. It's a move that complements [reliability & resilience](/blog/llm-application-reliability-resilience-fallback-retry-guide) from the latency/cost side of "production AI."

> ⚠️ The prices and token thresholds here are **2026** official values tied to specific model names (Opus 4.8 / GPT-5 family / Gemini 3.x). They change frequently, so re-verify each source before implementing. Discount rates are **model-dependent.**

## The two kinds of caching

| | Prompt caching | Semantic caching |
|---|---|---|
| **Layer** | Provider (API feature) | App (DIY / GPTCache, etc.) |
| **Match** | **Exact-match** of the prefix | **Semantic similarity** of embeddings (catches paraphrases) |
| **Effect** | **Discounts** input tokens | **Skips** the LLM call entirely |
| **Risk** | Virtually none (safe) | **False hits** (needs threshold/TTL/eval) |
| **Good for** | Large fixed context (RAG, system instructions, few-shot) | FAQ, canned questions, paraphrase-heavy inquiries |

**Start with safe prompt caching, then carefully add semantic caching** — that's the practical order.

## Prompt caching: put static content at the "front"

The effect of prompt caching is decided by **the structure of your prompt.** The shared principle: **put the large unchanging part (system instructions, long context, few-shot examples) at the front, and the changing part (user input) at the back.** The more the prefix matches, the better the cache works.

- **OpenAI** is **automatic.** No code changes; it works on prompts above a length (**1,024 tokens**), then hits in 128-token increments. Just put static content at the front to benefit. The cached-read discount is **model-dependent** (~50% for the GPT-4o generation, up to 90% for the GPT-5 family).
- **Anthropic** is **explicit** via `cache_control` (ephemeral). Up to 4 breakpoints; default TTL 5 minutes (1-hour option). Pricing is **cache write 1.25× (5-min) / 2× (1-hour), read 0.1×** (relative to base input, as of 2026).
- **Google Gemini** has both an explicit `CachedContent` API and, from 2.5 onward, **implicit caching** on by default (note that keeping caches alive incurs a per-hour storage cost).

With the Vercel AI SDK, the simplest is the AI Gateway's automatic caching.

```ts
import { generateText } from "ai";

// 最も簡単：AI Gateway の自動キャッシュ。プロバイダに応じ最適な戦略を適用する
// （Anthropic系は cache_control を注入、OpenAI/Google は暗黙キャッシュを利用）。
const { text } = await generateText({
  model: "anthropic/claude-opus-4-8",
  messages,
  providerOptions: { gateway: { caching: "auto" } },
});
```

For explicit control, place `cacheControl` on the stable prefix.

```ts
// 明示制御：静的コンテンツを"前"に置き、そこに cacheControl を付ける。
const { usage } = await generateText({
  model: "anthropic/claude-opus-4-8",
  messages: [
    {
      role: "system",
      content: largeStaticContext, // 大きく・変わらない部分（RAG文脈やfew-shot）
      providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
    },
    { role: "user", content: userQuestion }, // 変わる部分は後ろ
  ],
});
// usage.inputTokenDetails.cacheReadTokens でヒットを計測（読み取りは入力の約0.1×課金）
```

Prompt caching is **virtually no-risk** (exact match, so no wrong answers). Start here as a rule.

## Semantic caching: skip the call itself

Go one step further: **"don't call the LLM at all for similar questions."** Embed the question, match it against past questions by **cosine similarity**, and if it exceeds a threshold, return the past answer. Where prompt caching is "exact-prefix reuse," this **catches paraphrases** — its strength.

Separate the decision logic (threshold, freshness) into a **pure function** for testability.

```ts
// セマンティックキャッシュ：埋め込み類似で「言い換え」まで拾う。判定は純粋関数で分離。
interface CacheHit {
  readonly answer: string;
  readonly similarity: number; // 0..1（コサイン類似）
  readonly ageMs: number;
}

// 純粋・テスト容易：閾値と鮮度(TTL)の両方を満たすヒットだけ採用する（SRP）。
function acceptCacheHit(hit: CacheHit | null, minSimilarity: number, ttlMs: number): boolean {
  return hit !== null && hit.similarity >= minSimilarity && hit.ageMs <= ttlMs;
}

// cache-aside：ヒットなら即返す（LLM呼び出しを省く）、ミスなら生成して保存。
async function cachedGenerate(query: string, scope: string): Promise<string> {
  const embedding = await embed(query);
  // pgvector で最近傍を1件（scope でユーザー/地域を隔離し、他者の応答を返さない）
  const hit = await cache.nearest(embedding, scope);
  if (acceptCacheHit(hit, 0.95, 3_600_000)) return hit!.answer; // 高閾値で偽ヒットを抑制
  const answer = await callLlm(query);
  await cache.put({ query, embedding, answer, scope }); // ミス：保存して次回に備える
  return answer;
}
```

## Semantic caching: risks and controls

The danger of semantic caching is **false hits.** "What were 2024 sales?" and "What were 2025 sales?" have near-identical embeddings, yet **the correct answers differ.** A loose threshold mass-produces wrong answers. Four controls:

- **Threshold** — start high (**0.90–0.95**) and tune while monitoring the false-hit rate (AWS, too, states "the threshold controls the trade-off between hit rate and answer quality").
- **TTL** — by data volatility. Real-time (prices/inventory) **5–15 min**, static facts **24 hours**. Longer TTL raises hit rate but also staleness risk.
- **Scope** — filter by user, region, product ID, so a **personalized response isn't served to someone else.**
- **Invalidation** — invalidate the affected entries **content-triggered** when source data updates.

And **measure hit quality with evaluation.** Include "is this cache hit correct?" in offline evaluation and gate on the false-hit rate ([LLM evaluation & testing](/blog/llm-application-evaluation-testing-llm-as-judge-ci-guide)). The go-to OSS is [GPTCache](https://github.com/zilliztech/GPTCache), where you can swap the threshold and vector store.

**Be equally clear about when NOT to cache.** Always needs freshness, heavily personalized, audit requires fresh generation — for these, the cost of false hits/staleness outweighs the benefit, so just call every time.

## Observability: measure the effect and the side effects

Caching is **only safe once you measure it.** Record **hit rate**, tokens/cost saved, and p95 latency, plus the **false-hit rate** (from evaluation). Metrics guide tuning — "hit rate is high but false hits rose = threshold too loose." For the foundation, see [production observability with OpenTelemetry](/blog/opentelemetry-observability-production-tracing-metrics-logs). Note that caching is a different layer of optimization from model/infra economics (self-host vs API); for that, see [generative-AI cost design](/blog/generative-ai-cost-api-vs-self-hosting-decision-guide).

## Adoption checklist

- [ ] **Prompt caching (the safe first move)** — static content at the front, OpenAI automatic, Anthropic cacheControl (or Gateway caching:'auto') enabled?
- [ ] **Structure** — the large unchanging part at the front, the variable part at the back?
- [ ] **Semantic caching (carefully)** — starting from a high similarity threshold (0.90+)?
- [ ] **TTL** — set by data volatility, and invalidated on source updates?
- [ ] **Scope** — hits isolated by user/region, not leaking personalization?
- [ ] **False-hit monitoring** — is cache-hit correctness in evaluation and gated?
- [ ] **Non-cache decision** — excluded queries that need freshness / heavy personalization / audit?
- [ ] **Observability** — monitoring hit rate, cost saved, latency, and false-hit rate?

## Summary

Caching for LLM apps is, by convention, **"a foundation of safe prompt caching, with semantic caching carefully layered on."**

- **Prompt caching** is exact-match and virtually no-risk. Just **putting static content at the front** lowers input cost (Anthropic reads ≈0.1×, OpenAI automatic).
- **Semantic caching** is a powerful way to skip whole calls, but **false hits** are its intrinsic risk. Manage it with a **high threshold, TTL, scope, and evaluation.**
- **Operate by measuring.** Monitor not just hit rate but the **false-hit rate**, and include the decision not to cache in your design.

Fast, cheap, and accurate — you can have all three at once when you design caching correctly.