LLM App Evaluation & Testing Guide 2026 | LLM-as-Judge, Regression Testing, CI Gates

"It worked perfectly in the demo, but broke the moment real users touched it in production" — the most common failure with LLM apps. The reason is simple: you can't protect LLM quality by eyeballing it. Change one line of a prompt and a case you'd fixed regresses while a broken case gets fixed. Where deterministic code is protected by unit tests, LLMs require a different discipline — Evaluation.

Bottom line up front: the quality of a production LLM app comes down to three pillars.

1. An evaluation dataset (golden set) — the "exam questions": representative, edge, and adversarial inputs.
2. Automated grading — score the output with deterministic checks + LLM-as-judge.
3. A CI gate — block deploys on the eval score. Catch regressions with the pipeline, not the human eye.

This article, written from the standpoint of running generative-AI pipelines in production, adheres to primary sources (OpenAI / Anthropic eval guides, the origin paper for LLM-as-judge, RAGAS) and designs an evaluation that "runs in production," in real TypeScript. It's the third pillar of trustworthy production AI, alongside LLM app security and MCP production servers.

Why testing an LLM app is "different"

Traditional unit testing is assert(f(x) === expected). There's a single correct answer for an input, and you judge equality as a boolean. With LLMs, three things break.

• Non-deterministic — the output varies for the same input. A single pass might be luck.
• Free-form output — there's no unique "correct" answer. The wording can differ yet still be correct.
• Quality is continuous — not pass/fail but "how good." So you need evaluation (scoring), not assertion.

So shift your thinking. Look at the pass rate and score distribution across the whole dataset, not a 1/0 pass/fail per case. Instead of each test being red/green, gate on "does 90% of cases meet the bar?"

Building an evaluation dataset (golden set)

Every evaluation starts from "exam questions." Cut corners here and no amount of grading precision matters.

• Mix representative, edge, and adversarial inputs. OpenAI, too, says test data should include "typical cases, edge cases, and adversarial cases." Irrelevant input, extremely long input, harmful input, genuinely ambiguous input — include them on purpose.
• Start small and grow iteratively. Authoritative primary sources don't prescribe a minimum count. In practice, start with representative examples and add each production failure one at a time. OpenAI describes eval data as "a dynamic space you expand over time."
• Prioritize volume. Anthropic is explicit: "volume over quality" — many auto-gradable questions with slightly lower signal beat a few high-quality hand-graded ones. The reason connects directly to the next section.

Use three grading methods, appropriately

How do you grade the output? Three methods, with one iron rule: measure deterministically whatever you can (cheap, fast, reproducible).

Deterministic grading is just code. Measure the structure of the output with a Zod schema, and the content with simple checks.

import { z } from "zod";

type Case = { input: string; expected?: string; mustInclude?: string[] };
type Score = { name: string; pass: boolean; value: number };

// 完全一致（期待値がある設問だけ判定）
const exactMatch = (out: string, c: Case): Score => {
  const pass = c.expected === undefined || out.trim() === c.expected.trim();
  return { name: "exact_match", pass, value: pass ? 1 : 0 };
};

// 必須キーワードの包含（事実の欠落を安く検知）
const containsAll = (out: string, c: Case): Score => {
  const missing = (c.mustInclude ?? []).filter((k) => !out.includes(k));
  return { name: "contains_all", pass: missing.length === 0, value: missing.length ? 0 : 1 };
};

// 出力が期待するJSON構造かをスキーマで判定（形式崩れを回帰として捕まえる）
const matchesSchema = <T>(out: string, schema: z.ZodType<T>): Score => {
  const parsed = schema.safeParse(safeJson(out));
  return { name: "schema", pass: parsed.success, value: parsed.success ? 1 : 0 };
};

Using LLM-as-judge "correctly"

For free-form quality, having a strong LLM grade — LLM-as-judge — is the practical solution. The origin paper (Zheng et al., 2023) showed that a GPT-4-class judge approximates human preferences about as well as humans agree with each other (>80%). Scalable and explainable — but the same paper reports clear biases.

• Position bias — favors whichever answer is shown first.
• Verbosity bias — over-rates longer answers.
• Self-enhancement bias — prefers its own (same-family) outputs.
• Weak reasoning — poor at grading math/logic.

Fix these by design. (1) Fix the rubric first to remove ambiguity; (2) in pairwise comparisons, swap the order and grade both directions to cancel position bias; (3) always structure the verdict with a schema; (4) start with a strong model, and scale down to a lighter one only after confirming it agrees with human judgments (OpenAI's official recommendation).

import { generateObject } from "ai";
import { z } from "zod";

// judge の出力は必ず構造化する（自由文で"良い"と言わせない）
const Judgment = z.object({
  faithful: z.boolean(), // 与えた文脈に忠実か（ハルシネーションなし）
  relevant: z.boolean(), // 質問に答えているか
  score: z.number().int().min(1).max(5), // rubric に基づく5段階
  reason: z.string().max(300),
});

async function judge(input: string, output: string, context: string) {
  const { object } = await generateObject({
    // 判定は強モデルで開始。人間と一致したら軽量モデルへ置換してコストを下げる。
    model: "anthropic/claude-opus-4-8",
    schema: Judgment,
    prompt:
      "次の回答を rubric で採点。基準: (1)文脈に忠実か (2)質問に関連するか。\n" +
      `# 質問\n${input}\n# 文脈\n${context}\n# 回答\n${output}`,
  });
  return object; // 検証済みの構造化スコア
}

RAG-specific evaluation: measure retrieval and generation separately

RAG (retrieval-augmented generation) mixes retrieval failures and generation failures. Without separating them, you can't tell what to fix. The standard RAG-eval metrics (official RAGAS definitions):

• Faithfulness — how factually consistent the answer is with the retrieved context (hallucination detection).
• Answer / Response Relevancy — how relevant the answer is to the user's question.
• Context Precision — did the retriever rank relevant chunks above irrelevant ones (retrieval precision)?
• Context Recall — how much of the necessary relevant material was retrieved without misses (retrieval recall)?

If faithfulness is low, suspect generation (prompt/model); if context recall is low, suspect retrieval (embeddings/chunking/authorization filter). For RAG accuracy, see production RAG anti-patterns, and for retrieval design, production RAG with pgvector.

Putting regression testing into CI

Evaluation only becomes something that protects quality once it's in CI. Being good once locally means nothing. Gate deploys on the eval score. When the pass rate against a fixed dataset drops below a threshold, fail the build.

// 集計とCIゲート：合格率が閾値未満なら exit 1 でデプロイを止める
type Grader = (out: string, c: Case) => Score;

async function runEval(
  cases: Case[],
  runApp: (input: string) => Promise<string>,
  graders: Grader[],
  threshold = 0.9,
): Promise<void> {
  let passed = 0;
  for (const c of cases) {
    const out = await runApp(c.input);
    // 決定的グレーダーは全通過を要求（ここに judge を足して重み付けもできる）
    const ok = graders.every((g) => g(out, c).pass);
    if (ok) passed++;
  }
  const rate = passed / cases.length;
  console.log(`pass rate: ${(rate * 100).toFixed(1)}% (${passed}/${cases.length})`);
  if (rate < threshold) {
    console.error(`❌ eval ${(rate * 100).toFixed(1)}% < 閾値 ${threshold * 100}% — デプロイを中止`);
    process.exit(1); // CI が赤 → マージ/デプロイされない
  }
}

For tooling, promptfoo is practical. Its GitHub Action evaluates the before/after of a prompt change on a PR and posts the results to the PR — a "quality diff" at the same granularity as code review. You can build the CI foundation on the same idea as keyless CI/CD with GitHub Actions OIDC.

Note: this is a different layer from AI-driven development quality gates (protecting the quality of the code the AI writes). This article is about protecting the quality of the output the AI app generates. You need both for production quality.

Offline and online evaluation: run both

• Offline evaluation (pre-deploy) — measure quality, edge cases, and robustness against a fixed evaluation dataset before shipping. The CI gate lives here.
• Online evaluation (production) — sample production traffic and evaluate continuously to detect drift and degradation. Combine user feedback (👍/👎) with sampled judge-grading of production output.

Offline is "don't ship what's broken"; online is "notice degradation after shipping." Neither alone is enough. Online evaluation builds on logs and traces, so design it together with production observability via OpenTelemetry. For runtime validation of LLM output (structured output), see designing reliable structured output.

Adoption checklist (before going to production)

• Golden set — do you have eval data with typical / edge / adversarial inputs, and are you growing it with failures?
• Deterministic grading first — are format/structure/keywords/numbers measured with deterministic graders (cheap, reproducible)?
• LLM-as-judge — is the rubric fixed, the verdict schema-structured, and order-swapping used to suppress bias? Did you start with a strong model?
• Separate RAG evaluation — do faithfulness / relevancy / context precision & recall separate retrieval from generation?
• CI gate — do you block deploys when the pass rate is below threshold? Is the quality diff visible on the PR?
• Online evaluation — do you sample production traffic, evaluate continuously, and detect drift?
• Observability — do you record inputs/outputs, scores, latency, and cost to track degradation?

Summary

Quality assurance for LLM apps is engineering, not intuition or eyeballing. "Score the non-deterministic, free-form output with a dataset and automated grading, and gate it in CI" — this discipline separates a demo from production.

• Measure deterministically what you can. Grade the rest with LLM-as-judge (fixed rubric, bias mitigation, start with a strong model).
• Measure RAG by separating retrieval from generation.
• Evaluation protects quality only in CI. Run offline (a pre-deploy gate) and online (production monitoring) as two wheels.

Security, MCP, evaluation — with all three, generative AI becomes a production system that is "fast to build, secure, and unbreakable." Building fast and being defensibly solid are compatible when you design correctly.