# LLM App Evaluation & Testing Guide 2026 | LLM-as-Judge, Regression Testing, CI Gates

> A practical guide to evaluating and testing production LLM apps so they don't break. How to measure non-deterministic output, how to build eval datasets (golden sets), when to use deterministic grading vs LLM-as-judge, RAG metrics (RAGAS), and how to gate regressions in CI — with primary-source-faithful TypeScript code.

- Published: 2026-07-23
- Author: 友田 陽大
- Tags: 生成AI, LLM, 評価, テスト, LLM-as-judge, RAG, CI/CD, AI駆動開発, 品質保証, アーキテクチャ設計
- URL: https://tomodahinata.com/en/blog/llm-application-evaluation-testing-llm-as-judge-ci-guide
- Category: 本番LLMアプリ実装（信頼できるAI）
- Pillar guide: https://tomodahinata.com/en/blog/production-llm-application-engineering-guide

## Key points

- LLM app quality can't be protected by eyeballing. Make evaluation an engineering discipline: dataset + automated grading + CI gate. Look at score distributions and pass rates, not pass/fail.
- Use three grading methods: measure deterministically what you can (exact match, JSON schema, numeric tolerance); grade free-form quality with LLM-as-judge; keep humans as the last resort.
- A strong LLM judge agrees with humans about as well as humans agree with each other (>80%), but has position/verbosity/self-enhancement biases. Fix the rubric, swap order, and scale down only after confirming agreement with humans.
- For RAG, measure retrieval and generation separately: faithfulness, answer relevancy, context precision/recall (RAGAS).
- Evaluation only protects quality once it's in CI. Block deploys when the fixed-dataset pass rate drops below a threshold (offline eval). Continuously evaluate a sample of production traffic to catch drift (online eval).

---

"It worked perfectly in the demo, but broke the moment real users touched it in production" — the most common failure with LLM apps. The reason is simple: **you can't protect LLM quality by eyeballing it.** Change one line of a prompt and a case you'd fixed regresses while a broken case gets fixed. Where deterministic code is protected by unit tests, LLMs require a different discipline — **Evaluation**.

Bottom line up front: the quality of a production LLM app comes down to three pillars.

1. **An evaluation dataset (golden set)** — the "exam questions": representative, edge, and adversarial inputs.
2. **Automated grading** — score the output with deterministic checks + LLM-as-judge.
3. **A CI gate** — block deploys on the eval score. Catch regressions with the pipeline, not the human eye.

This article, written from the standpoint of running generative-AI pipelines in production, adheres to **primary sources (OpenAI / Anthropic eval guides, the origin paper for LLM-as-judge, RAGAS)** and designs an evaluation that "runs in production," in real TypeScript. It's the third pillar of **trustworthy production AI**, alongside [LLM app security](/blog/llm-application-security-owasp-top-10-prompt-injection-guide) and [MCP production servers](/blog/mcp-model-context-protocol-production-server-guide).

## Why testing an LLM app is "different"

Traditional unit testing is `assert(f(x) === expected)`. There's a **single correct answer** for an input, and you judge equality as a boolean. With LLMs, three things break.

- **Non-deterministic** — the output varies for the same input. A single pass might be luck.
- **Free-form output** — there's no unique "correct" answer. The wording can differ yet still be correct.
- **Quality is continuous** — not pass/fail but "how good." So you need **evaluation (scoring), not assertion.**

So shift your thinking. **Look at the pass rate and score distribution across the whole dataset, not a 1/0 pass/fail per case.** Instead of each test being red/green, gate on "does 90% of cases meet the bar?"

## Building an evaluation dataset (golden set)

Every evaluation starts from "exam questions." Cut corners here and no amount of grading precision matters.

- **Mix representative, edge, and adversarial inputs.** OpenAI, too, says test data should include "typical cases, edge cases, and adversarial cases." Irrelevant input, extremely long input, harmful input, genuinely ambiguous input — include them on purpose.
- **Start small and grow iteratively.** Authoritative primary sources **don't prescribe a minimum count.** In practice, start with representative examples and add each production failure one at a time. OpenAI describes eval data as "a dynamic space you expand over time."
- **Prioritize volume.** Anthropic is explicit: **"volume over quality"** — many auto-gradable questions with slightly lower signal beat a few high-quality hand-graded ones. The reason connects directly to the next section.

## Use three grading methods, appropriately

How do you grade the output? Three methods, with one iron rule: **measure deterministically whatever you can** (cheap, fast, reproducible).

| Method | What it measures | Cost | Where to use |
|---|---|---|---|
| **Deterministic (code-based)** | Exact match, JSON-schema conformance, regex, numeric tolerance, keyword inclusion | Tiny | Format, structure, part of the facts. **First choice** |
| **LLM-as-judge** | Semantic quality of free-form text (faithfulness, relevance, tone) | Medium | The "goodness" you can't measure deterministically |
| **Human** | Final validity, subtle judgment | Large | A few important cases; calibrating the judge |

Deterministic grading is just code. Measure the **structure** of the output with a Zod schema, and the **content** with simple checks.

```ts
import { z } from "zod";

type Case = { input: string; expected?: string; mustInclude?: string[] };
type Score = { name: string; pass: boolean; value: number };

// 完全一致（期待値がある設問だけ判定）
const exactMatch = (out: string, c: Case): Score => {
  const pass = c.expected === undefined || out.trim() === c.expected.trim();
  return { name: "exact_match", pass, value: pass ? 1 : 0 };
};

// 必須キーワードの包含（事実の欠落を安く検知）
const containsAll = (out: string, c: Case): Score => {
  const missing = (c.mustInclude ?? []).filter((k) => !out.includes(k));
  return { name: "contains_all", pass: missing.length === 0, value: missing.length ? 0 : 1 };
};

// 出力が期待するJSON構造かをスキーマで判定（形式崩れを回帰として捕まえる）
const matchesSchema = <T>(out: string, schema: z.ZodType<T>): Score => {
  const parsed = schema.safeParse(safeJson(out));
  return { name: "schema", pass: parsed.success, value: parsed.success ? 1 : 0 };
};
```

## Using LLM-as-judge "correctly"

For free-form quality, having a strong LLM grade — **LLM-as-judge** — is the practical solution. The origin paper (Zheng et al., 2023) showed that **a GPT-4-class judge approximates human preferences about as well as humans agree with each other (>80%)**. Scalable and explainable — but **the same paper reports clear biases.**

- **Position bias** — favors whichever answer is shown first.
- **Verbosity bias** — over-rates longer answers.
- **Self-enhancement bias** — prefers its own (same-family) outputs.
- **Weak reasoning** — poor at grading math/logic.

Fix these by design. **(1) Fix the rubric first** to remove ambiguity; **(2) in pairwise comparisons, swap the order and grade both directions** to cancel position bias; **(3) always structure the verdict with a schema**; **(4) start with a strong model, and scale down to a lighter one only after confirming it agrees with human judgments** (OpenAI's official recommendation).

```ts
import { generateObject } from "ai";
import { z } from "zod";

// judge の出力は必ず構造化する（自由文で"良い"と言わせない）
const Judgment = z.object({
  faithful: z.boolean(), // 与えた文脈に忠実か（ハルシネーションなし）
  relevant: z.boolean(), // 質問に答えているか
  score: z.number().int().min(1).max(5), // rubric に基づく5段階
  reason: z.string().max(300),
});

async function judge(input: string, output: string, context: string) {
  const { object } = await generateObject({
    // 判定は強モデルで開始。人間と一致したら軽量モデルへ置換してコストを下げる。
    model: "anthropic/claude-opus-4-8",
    schema: Judgment,
    prompt:
      "次の回答を rubric で採点。基準: (1)文脈に忠実か (2)質問に関連するか。\n" +
      `# 質問\n${input}\n# 文脈\n${context}\n# 回答\n${output}`,
  });
  return object; // 検証済みの構造化スコア
}
```

## RAG-specific evaluation: measure retrieval and generation separately

RAG (retrieval-augmented generation) mixes **retrieval failures** and **generation failures**. Without separating them, you can't tell what to fix. The standard RAG-eval metrics (official RAGAS definitions):

- **Faithfulness** — how factually consistent the answer is **with the retrieved context** (hallucination detection).
- **Answer / Response Relevancy** — how relevant the answer is to the user's question.
- **Context Precision** — did the retriever rank relevant chunks above irrelevant ones (retrieval precision)?
- **Context Recall** — how much of the necessary relevant material was retrieved without misses (retrieval recall)?

If faithfulness is low, suspect generation (prompt/model); if context recall is low, suspect retrieval (embeddings/chunking/authorization filter). For RAG accuracy, see [production RAG anti-patterns](/blog/production-rag-pitfalls-accuracy-improvement-guide), and for retrieval design, [production RAG with pgvector](/blog/pgvector-postgres-production-rag-hybrid-search).

## Putting regression testing into CI

Evaluation only becomes something that **protects** quality once it's in CI. Being good once locally means nothing. **Gate deploys on the eval score.** When the pass rate against a fixed dataset drops below a threshold, fail the build.

```ts
// 集計とCIゲート：合格率が閾値未満なら exit 1 でデプロイを止める
type Grader = (out: string, c: Case) => Score;

async function runEval(
  cases: Case[],
  runApp: (input: string) => Promise<string>,
  graders: Grader[],
  threshold = 0.9,
): Promise<void> {
  let passed = 0;
  for (const c of cases) {
    const out = await runApp(c.input);
    // 決定的グレーダーは全通過を要求（ここに judge を足して重み付けもできる）
    const ok = graders.every((g) => g(out, c).pass);
    if (ok) passed++;
  }
  const rate = passed / cases.length;
  console.log(`pass rate: ${(rate * 100).toFixed(1)}% (${passed}/${cases.length})`);
  if (rate < threshold) {
    console.error(`❌ eval ${(rate * 100).toFixed(1)}% < 閾値 ${threshold * 100}% — デプロイを中止`);
    process.exit(1); // CI が赤 → マージ/デプロイされない
  }
}
```

For tooling, **promptfoo** is practical. Its GitHub Action **evaluates the before/after of a prompt change on a PR and posts the results to the PR** — a "quality diff" at the same granularity as code review. You can build the CI foundation on the same idea as [keyless CI/CD with GitHub Actions OIDC](/blog/github-actions-oidc-keyless-cicd-aws-gcp-guide).

> Note: this is a different layer from [AI-driven development quality gates](/blog/ai-driven-development-quality-gates-ci-type-safety-test-security) (protecting the quality of the code the AI *writes*). This article is about protecting the quality of the output the AI app *generates*. You need both for production quality.

## Offline and online evaluation: run both

- **Offline evaluation (pre-deploy)** — measure quality, edge cases, and robustness against a fixed evaluation dataset **before** shipping. The CI gate lives here.
- **Online evaluation (production)** — **sample production traffic** and evaluate continuously to **detect drift and degradation.** Combine user feedback (👍/👎) with sampled judge-grading of production output.

Offline is "don't ship what's broken"; online is "notice degradation after shipping." Neither alone is enough. Online evaluation builds on logs and traces, so design it together with [production observability via OpenTelemetry](/blog/opentelemetry-observability-production-tracing-metrics-logs). For runtime validation of LLM output (structured output), see [designing reliable structured output](/blog/structured-output-reliability-constrained-decoding-semantic-validation).

## Adoption checklist (before going to production)

- [ ] **Golden set** — do you have eval data with typical / edge / adversarial inputs, and are you growing it with failures?
- [ ] **Deterministic grading first** — are format/structure/keywords/numbers measured with deterministic graders (cheap, reproducible)?
- [ ] **LLM-as-judge** — is the rubric fixed, the verdict schema-structured, and order-swapping used to suppress bias? Did you start with a strong model?
- [ ] **Separate RAG evaluation** — do faithfulness / relevancy / context precision & recall separate retrieval from generation?
- [ ] **CI gate** — do you block deploys when the pass rate is below threshold? Is the quality diff visible on the PR?
- [ ] **Online evaluation** — do you sample production traffic, evaluate continuously, and detect drift?
- [ ] **Observability** — do you record inputs/outputs, scores, latency, and cost to track degradation?

## Summary

Quality assurance for LLM apps is **engineering**, not intuition or eyeballing. **"Score the non-deterministic, free-form output with a dataset and automated grading, and gate it in CI"** — this discipline separates a demo from production.

- **Measure deterministically what you can.** Grade the rest with **LLM-as-judge** (fixed rubric, bias mitigation, start with a strong model).
- **Measure RAG by separating retrieval from generation.**
- **Evaluation protects quality only in CI.** Run offline (a pre-deploy gate) and online (production monitoring) as two wheels.

Security, MCP, evaluation — with all three, generative AI becomes a production system that is "fast to build, secure, and unbreakable." Building fast and being defensibly solid are compatible when you design correctly.