# Production LLM App Reliability & Resilience Guide 2026 | Multi-Provider Fallback, Retry, Circuit Breaker, Idempotency

> A reliability & resilience design guide for keeping production generative-AI/LLM apps from falling over. Timeouts, retries with exponential backoff + jitter (transient failures only), circuit breakers, multi-provider fallback (AI SDK / Vercel AI Gateway), idempotency, and rate limiting — with primary-source-faithful TypeScript code.

- Published: 2026-07-23
- Author: 友田 陽大
- Tags: 生成AI, LLM, 信頼性, 回復性, 冪等性, AIエージェント, TypeScript, SRE, アーキテクチャ設計, コスト最適化
- URL: https://tomodahinata.com/en/blog/llm-application-reliability-resilience-fallback-retry-guide
- Category: 本番LLMアプリ実装（信頼できるAI）
- Pillar guide: https://tomodahinata.com/en/blog/production-llm-application-engineering-guide

## Key points

- An LLM app is a distributed system depending on a non-deterministic external API — it falls over easily on outages, rate limits, latency spikes, and cost blow-ups. Build resilience with architecture.
- Retry only 'transient failures (429/5xx/timeout), only idempotent calls.' Add full jitter to exponential backoff to prevent the thundering herd.
- The AI SDK retries transient failures by default (maxRetries=2). Do multi-provider fallback via Vercel AI Gateway's providerOptions.gateway.models or yourself. Don't stack retries across layers (retry amplification).
- Use a circuit breaker to temporarily cut off a failing provider, stopping cascading failures and cost waste. Inject the clock to make it unit-testable.
- Protect side-effecting operations with an idempotency key. Structurally prevent double-charges/double-shipments from retries or an agent's repeated calls.

---

"It was perfect in development, then fell over the moment production traffic hit it" — a frequent LLM-app incident. The reason is simple: **an LLM app is a distributed system that depends on a non-deterministic external API.** Provider-side transient failures, rate limits (429), latency spikes, and cost blow-ups. These aren't a question of "if" but "when."

Bottom line up front: the **resilience of a production LLM app is built with architecture, not the model.** Six pillars.

1. **Timeouts** — don't hang on a call that never returns.
2. **Retries (transient only, exponential backoff + jitter)** — absorb transient failures.
3. **Circuit breakers** — stop attacking a failing provider; prevent cascading failures.
4. **Multi-provider fallback** — don't let one vendor's outage stop your service.
5. **Idempotency** — don't double-apply side effects on retries or double calls.
6. **Rate limits / cost caps** — contain abuse and blow-ups.

This article brings the [reliability and idempotency design](/blog/dynamodb-single-table-design-reliability-idempotency-patterns) that kept **production double-charges at zero** on a payments platform into generative AI. Adhering to the **current AI SDK / Vercel AI Gateway API and the primary sources of AWS and Google SRE**, it shows real code. It's the fifth pillar of **trustworthy production AI**, following [security](/blog/llm-application-security-owasp-top-10-prompt-injection-guide), [MCP](/blog/mcp-model-context-protocol-production-server-guide), [evaluation](/blog/llm-application-evaluation-testing-llm-as-judge-ci-guide), and [privacy](/blog/llm-application-data-privacy-compliance-pii-appi-gdpr-guide).

## Design LLM calls on the premise that they fail

First, classify the failures, because the response differs.

| Failure | Examples | Response |
|---|---|---|
| **Transient** | 429 (rate limit), 500/502/503/504, connection drop, timeout | **Retry** (backoff + jitter) → if still failing, **fall back** |
| **Permanent** | 400 (bad request), 401/403 (authz), 422 | **Don't retry** (won't succeed; wasted cost) |
| **Partial degradation** | Response returns but is slow / low quality | **Timeout** + detect via **evaluation** |

The HTTP semantics are clear. **429 (Too Many Requests) is defined by RFC 6585**, and **the Retry-After header and 503 by RFC 9110.** If Retry-After comes back, **prefer it** over your own backoff.

## Timeouts and retries: transient failures only, with jitter

**Two iron rules for retries: "transient failures only" and "idempotent calls only."** Google SRE also says: don't retry permanent errors or malformed requests — they will never succeed.

And **jitter.** If all clients retry at the same interval, the retry spikes synchronize into a **thundering herd** that crushes a provider just as it starts to recover. AWS recommends adding **full jitter** (a uniform random from 0 to backoff) to exponential backoff to level out the spikes.

First, the good news: **the Vercel AI SDK retries transient failures by default** (`maxRetries`, default 2). That covers many cases.

```ts
import { generateText } from "ai";

// AI SDK は一時障害を既定で2回リトライ（maxRetries=2）。timeout は AbortSignal で。
// Vercel AI Gateway なら「モデル・フォールバック」を宣言的に書ける：
// primary が失敗したら順にバックアップを試し、最初に成功した応答を返す。
const { text } = await generateText({
  model: "anthropic/claude-opus-4-8",
  prompt,
  maxRetries: 2,                              // 既定値。多層でリトライを重ねない
  abortSignal: AbortSignal.timeout(15_000),   // 15秒で打ち切り
  providerOptions: {
    gateway: {
      models: ["anthropic/claude-opus-4-8", "openai/gpt-5"], // 順にフォールバック
    },
  },
});
```

When you want to control retries yourself (no gateway, custom backoff), retry **transient failures only** with exponential backoff + full jitter.

```ts
// 自前リトライ：一時障害(429/5xx/timeout)だけを、指数バックオフ+フルジッターで。
const RETRYABLE = new Set([429, 500, 502, 503, 504]);

async function withRetry<T>(fn: () => Promise<T>, retries = 2, baseMs = 500): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const status = (err as { statusCode?: number }).statusCode;
      // 恒久エラー(4xx)はリトライしない。上限に達したら諦める（無限リトライ=コスト暴走）。
      if (attempt >= retries || status === undefined || !RETRYABLE.has(status)) throw err;
      const backoff = Math.min(baseMs * 2 ** attempt, 20_000);
      const wait = retryAfterMs(err) ?? Math.random() * backoff; // Retry-After優先、無ければfull jitter
      await new Promise((resolve) => setTimeout(resolve, wait));
    }
  }
}
```

## Circuit breaker: don't keep hammering a downed provider

When transient failures **persist**, retries backfire. Hammering a failing provider only invites **cascading failures** and cost waste. Enter the **circuit breaker** — when consecutive failures exceed a threshold, trip the circuit **open** (fail fast), then try once in **half-open** after a cooldown to confirm recovery (Martin Fowler).

For testability, **inject the clock as an argument** (don't depend on `Date.now()` — so you can unit-test deterministically).

```ts
// サーキットブレーカ：連続失敗が閾値超で一定時間 open（即失敗）。
// now を注入してテスト可能に（closed → open → half-open）。
class CircuitBreaker {
  private failures = 0;
  private openedAt: number | null = null;
  constructor(
    private readonly threshold = 5,
    private readonly cooldownMs = 30_000,
  ) {}

  allow(now: number): boolean {
    if (this.openedAt === null) return true;                 // closed：通常通り
    if (now - this.openedAt >= this.cooldownMs) return true; // half-open：試行を1回許可
    return false;                                            // open：即失敗（叩かない）
  }
  onSuccess(): void {
    this.failures = 0;
    this.openedAt = null;
  }
  onFailure(now: number): void {
    if (++this.failures >= this.threshold) this.openedAt = now;
  }
}
```

## Multi-provider fallback: don't stop for one vendor's outage

The heart of resilience. **When the primary provider fails, switch to the secondary.** With Vercel AI Gateway you can write it declaratively via the aforementioned `providerOptions.gateway.models` (model order) or `order` (provider order). If you manage it yourself, build a fallback chain combining the circuit breaker and retries.

```ts
// 多プロバイダ・フォールバック：primary が落ちたら次へ。open 中のプロバイダはスキップ。
const CHAIN = ["anthropic/claude-opus-4-8", "openai/gpt-5"] as const;
const breakers = new Map(CHAIN.map((model) => [model, new CircuitBreaker()]));

async function generateResilient(prompt: string, now: () => number): Promise<string> {
  let lastErr: unknown;
  for (const model of CHAIN) {
    const breaker = breakers.get(model)!;
    if (!breaker.allow(now())) continue; // サーキットが open ならこのプロバイダは飛ばす
    try {
      // withRetry が再試行を担うので、SDK側は maxRetries:0（多層リトライを避ける）
      const { text } = await withRetry(() =>
        generateText({ model, prompt, maxRetries: 0, abortSignal: AbortSignal.timeout(15_000) }),
      );
      breaker.onSuccess();
      return text;
    } catch (err) {
      lastErr = err;
      breaker.onFailure(now());
    }
  }
  throw new Error(`all providers failed: ${String(lastErr)}`);
}
```

The catch is **quality consistency.** The fallback target is a different model, so output quality/format drifts. To keep quality across the switch, validate with a schema ([reliable structured output](/blog/structured-output-reliability-constrained-decoding-semantic-validation)) and measure continuously with [evaluation (eval)](/blog/llm-application-evaluation-testing-llm-as-judge-ci-guide).

> **Beware retry amplification.** If the SDK, your own retry, the gateway, and the upstream service each hold retries, the retry count **multiplies** and explodes the load during a failure. Google SRE's principle is clear: **"limit retries per request."** Concentrate retries in one layer and set the others to 0.

## Idempotency: don't double-apply side effects on retries

Retries and fallback are a **double-edged sword for side-effecting operations.** If the first attempt actually succeeded (only the response was lost), the retry is a **double execution.** LLM agents may call a tool multiple times with the same intent, raising the risk further.

The fix is an **idempotency key.** Guarantee downstream that an operation with the same key runs exactly once (same as [MCP tool-execution design](/blog/mcp-model-context-protocol-production-server-guide) and payments platforms).

```ts
// 副作用のある操作は冪等キーで保護。リトライ/二重呼び出しでも一度きり。
async function chargeIdempotent(orderId: string, amountJpy: number): Promise<Receipt> {
  const key = `charge:${orderId}:${amountJpy}`; // 意図が同じなら同じキー
  const existing = await receipts.get(key);
  if (existing) return existing;                // 既に処理済みなら再実行しない
  const receipt = await paymentApi.charge(orderId, amountJpy, { idempotencyKey: key });
  await receipts.put(key, receipt);
  return receipt;
}
```

## Rate limiting and cost resilience

Reliability is also a **cost** problem. If retries or loops run away, it's not availability that breaks — it's your bill. The countermeasures are the same as [OWASP LLM10 "Unbounded Consumption"](/blog/llm-application-security-owasp-top-10-prompt-injection-guide) in the security article — **output-token caps (`maxOutputTokens`), per-user rate limits, timeouts, and a cap on retry count.** Google SRE's "serve degraded responses under overload (load shedding / graceful degradation)" is also a valid option.

## Operate resilience with observability

These mechanisms **only work once you measure them.** Record failure rate, **fallback rate**, circuit open-time, p95 latency, retry count, and cost, and detect anomalies. Metrics reveal design holes — e.g. "fallback firing constantly = the primary is chronically unhealthy." For the foundation, see [production observability with OpenTelemetry](/blog/opentelemetry-observability-production-tracing-metrics-logs).

## Adoption checklist (before going to production)

- [ ] **Timeouts** — is every LLM call given an `abortSignal` (`AbortSignal.timeout`)?
- [ ] **Retry scope** — do you retry only transient failures (429/5xx/timeout) and only idempotent calls?
- [ ] **Backoff + jitter** — exponential backoff with full jitter, preferring Retry-After?
- [ ] **Retry amplification** — retries concentrated in one layer, not stacked (with a cap)?
- [ ] **Circuit breaker** — do you temporarily cut off a provider that keeps failing?
- [ ] **Multi-provider fallback** — do you survive one vendor's outage? Is fallback quality assured by evaluation?
- [ ] **Idempotency** — are side-effecting operations protected by an idempotency key against double execution?
- [ ] **Cost resilience** — are token caps, rate limits, and retry caps in place against blow-ups?
- [ ] **Observability** — do you monitor failure rate, fallback rate, latency, and cost to detect degradation?

## Summary

The reliability of a production LLM app is no special magic — it's **applying the resilience design of distributed systems to LLMs.** Accept that **"the LLM is a failing external dependency," and contain the damage with timeouts, (transient-only) retries, circuit breakers, multi-provider fallback, and idempotency.**

- **Retry only transient, only idempotent, with jitter.** Don't stack across layers (retry amplification).
- **Don't hammer a downed provider with a circuit breaker.** Survive a vendor outage with multi-provider fallback.
- **Prevent double execution with idempotency keys, and cap runaway with rate limits and cost caps.**

Security, MCP, evaluation, privacy, and **reliability** — with these five pillars, generative AI becomes a production system that is "fast to build, secure, unbreakable, trustworthy, and — **it doesn't go down.**" Building fast and staying up are compatible when you design correctly.