Production LLM/AI App Security Guide 2026 | OWASP LLM Top 10 & Prompt Injection Defense

More and more teams are putting generative AI into production. But "is the model smart?" and "is the app secure?" are entirely separate questions. Securing a production LLM app is not solved by clever prompting. Bottom line up front: it comes down to three architectural principles.

1. Don't trust the model's output (zero trust) — treat output as "another user's input" and always validate it before passing it downstream.
2. Minimize tool (integration) permissions — never hand an agent destructive execution rights in the first place.
3. Require human approval for high-impact actions — refunds, deletions, transfers, and permission changes go through deterministic code and a human gate.

This article is written from the standpoint of running generative-AI pipelines in production while specializing in application security (Next.js × Supabase app defense and threat modeling). Aligned with the OWASP Top 10 for LLM Applications (2025) and NIST AI RMF, it shows only mitigations that work in production, with real TypeScript / Zod / Vercel AI SDK code. Not abstractions — designs you can put in tomorrow's PR.

⚠️ The attack examples here are limited to conceptual explanation for understanding the defenses; no complete, weaponizable payloads are included. Always test in an environment you own.

Why LLM app security is "different"

Traditional web security draws a clear boundary between "untrusted input (the user)" and "trusted code (us)", and validates/escapes at that boundary. In LLM apps, three of those assumptions break.

• It's not deterministic. The same input can yield different output. You cannot build a complete whitelist that "only allows this pattern."
• You can't trust the output. The model's output is, in effect, something the user can indirectly steer via the prompt. Trusting LLM output and flowing it downstream is the same as passing user input to SQL without validation.
• Trust boundaries blur. RAG pours external documents, and agents pour tool results, into the prompt one after another. Without deliberate design, "how far is trusted context?" is undefined.

So defense is not about "how to make the model smarter." It's about treating the model as a probabilistic, untrusted component, and guaranteeing safety with the architecture around it. That's the single thread running through this article.

OWASP LLM Top 10 (2025) at a glance

First, the big picture. Details and defensive code for each item follow.

Key changes in the 2025 edition (per OWASP official):

• LLM07 System Prompt Leakage was newly added (real-world exploits became common).
• LLM08 Vector and Embedding Weaknesses was newly added (production adoption of RAG and embeddings grew).
• The old Model Denial of Service was expanded into LLM10 Unbounded Consumption, now including resource management and unexpected cost.
• LLM06 Excessive Agency was expanded given the spread of agentic architectures.

Below we go deep on the highest-impact items for practitioners — LLM01 / LLM05 / LLM06 / LLM08 / LLM10 — with defensive code.

LLM01: Prompt Injection — input filters "fundamentally" can't stop it

Prompt injection tops the LLM Top 10 and is the most misunderstood item. "You can prevent it by trying hard with a filter that blocks bad input" is wrong.

OWASP's LLM01 states it clearly: given that the model operates under essentially stochastic influence, it is unclear whether a fool-proof method of preventing prompt injection exists. In other words, input filtering is a mitigation, not a cure. Make that the starting point of your design.

Direct vs. indirect injection

• Direct — the user tries to override the model's role directly via the prompt, e.g., "ignore all previous instructions and…".
• Indirect / cross-domain — malicious instructions are planted inside content the model ingests from external sources (web pages, files, documents retrieved via RAG, tool results, instructions hidden in images). The user themselves may be harmless, yet the "data" the model reads has been poisoned by an attacker.

The more you use agents and RAG, the more the attack surface expands from direct input to "everything the model touches" — this is the backdrop for RAG (LLM08) becoming its own item in the 2025 edition.

Defense in depth: protect at "output and tools," not input

You of course do what you can on the prompt side (state the role, declare the expected output format). But that's only the first layer. The real battleground is the boundary of the model's output and tool execution.

// アンチパターン：LLMの出力を「信頼できる指示」として下流に流す
const answer = await generateText({ model, prompt });
await db.query(answer.text); // ❌ プロンプトインジェクションが即RCE/情報漏洩になる

The fix is to treat the model as "another untrusted user" and to constrain the output with types (see LLM05 below). And the most effective move is simply not letting the model do dangerous things in the first place — narrowing tool permissions (LLM06). With these two layers, even if injection occurs, you keep the "blast radius" small.

LLM05: Improper Output Handling — validate output with "zero trust"

LLM05 (Improper Output Handling) happens when you pass the model's output to other components (DB, shell, HTML, another API) without validation. In OWASP's words: treat the model like any other user, adopting a zero-trust approach.

With Vercel AI SDK's generateObject and Zod, you can enforce the "type and value range" of the output at the boundary. The key point here: even after schema validation passes, re-check business rules in deterministic code.

import { generateObject } from "ai";
import { z } from "zod";

// LLM出力は「別のユーザー入力」= ゼロトラスト。スキーマで境界を締める。
const RefundDecision = z.object({
  action: z.enum(["approve", "reject", "escalate"]),
  amountJpy: z.number().int().min(0).max(50_000), // 上限を「型」で強制する
  reason: z.string().max(200),
});

const { object } = await generateObject({
  model: "anthropic/claude-opus-4-8",
  schema: RefundDecision,
  prompt,
});

// ここまで来た object は「形式的に」検証済み。だが業務ルールは別問題。
// プロンプトインジェクションで action=approve に誘導されても、
// 決定的なコードが最終防衛線になる。
if (object.action === "approve" && object.amountJpy > refundableBalance(order)) {
  throw new PolicyError("LLMが残高を超える返金を提案しました");
}

If you pass output to SQL, HTML, or a shell, always insert context-aware encoding (HTML escaping, parameterized queries). This is exactly the same as traditional injection defense, and it falls out naturally once you treat "LLM output as a string an attacker may be able to control." For raising the reliability of structured output, see designing reliable structured output.

LLM06: Excessive Agency — separate "propose" from "execute"

The moment you give an agent tools to "issue a refund," "delete a file," or "send an email," prompt injection turns into actual damage. OWASP calls this Excessive Agency and decomposes the cause into excessive functionality, excessive permissions, and excessive autonomy.

The recommended defenses are clear: (1) minimize tools and their functionality to only what is required, (2) least privilege, (3) human-in-the-loop approval for high-impact actions.

The crux of the design is to never hand the agent execution rights. Let the agent go as far as "proposing," and perform the actual destructive operation on the code side, through deterministic business rules and human approval.

import { tool } from "ai";
import { z } from "zod";

// 破壊的アクションは「実行」せず「提案」を返すツールにする。
// ツール自体は最小権限（読み取り＋提案レコード作成のみ）。
const proposeRefund = tool({
  description: "返金の提案を作成する（実行はしない）",
  inputSchema: z.object({
    orderId: z.string().uuid(),
    amountJpy: z.number().int().positive(),
  }),
  execute: async ({ orderId, amountJpy }) => {
    // 副作用は「提案レコードの作成」だけ。ここで実際の送金はしない。
    const proposalId = await createRefundProposal({ orderId, amountJpy });
    return { proposalId };
  },
});

// 高影響アクションは human-in-the-loop（OWASP LLM06 の推奨）。
// 実行は決定的コード側で、業務ルール検証 + 冪等キー付きで行う。
async function executeRefund(proposalId: string, approver: Admin) {
  const p = await getProposal(proposalId);
  assertWithinPolicy(p, approver); // 残高・上限・権限を決定的に再検証
  // 冪等キーで「二重返金」を構造的に防ぐ（分散システムの基本）
  await refundIdempotent(p.orderId, p.amountJpy, { idempotencyKey: p.proposalId });
}

To add context, this "propose/execute separation + idempotency key" is not a new idea. It's exactly the reliability and idempotency design I've used to keep production double-charges at zero on a payments platform. Security in the LLM-agent era is nearly continuous with the reliability design of distributed systems — you surround a probabilistic, untrusted component (the LLM) with deterministic guards. For agent design in general, see designing AI-agent tool execution.

LLM08 / LLM02: RAG data leakage — embed authorization into "retrieval"

RAG (retrieval-augmented generation) is an effective way to reduce misinformation (LLM09), but it brings two new attack surfaces.

1. Indirect prompt injection (LLM08) — malicious instructions are planted in the retrieved documents.
2. Data leakage from weakly-authorized retrieval (LLM02) — the vector search lacks a permission filter, so confidential documents of other tenants or departments end up in the context.

Case (2) especially happens all the time in production. "Clever prompts" can't stop it. Embedding authorization into the search query itself is the only correct answer.

// アンチパターン：全社ナレッジをそのまま検索 → テナント越境・情報漏洩
// const docs = await vectorStore.search(queryEmbedding, { topK: 8 });

// 正：リトリーバルに「認可フィルタ」を必須で埋め込む。
// アプリ層で後からフィルタするのではなく、検索そのものを認可でスコープする。
const docs = await vectorStore.search(queryEmbedding, {
  topK: 8,
  filter: {
    tenantId: session.tenantId,
    visibility: { $in: session.scopes }, // 閲覧可能な範囲だけを検索対象に
  },
});

// 取得したドキュメントは「信頼できない外部入力」。
// プロンプトに入れる前に来歴を明示し、"指示ではなくデータ" として扱う。
const context = docs
  .map((d) => `<context source="${d.source}">\n${d.text}\n</context>`)
  .join("\n\n");

The important thing here is not to misread this as "wrapping in <context> tags prevents injection." The wrapper only helps convey the intent that "this is data, not instructions" to the model. The real cure for indirect injection is still to contain the damage with LLM05 (output validation) + LLM06 (least-privilege tools). RAG authorization design is also covered in production RAG with pgvector.

LLM07: System Prompt Leakage — don't put secrets in the prompt

A new item in the 2025 edition. Design your system prompt on the assumption that it will leak. Attackers often succeed in making the model dump its prompt, and if it contains API keys, DB names, internal roles, or authorization logic, that becomes a foothold for the next attack.

There are two principles:

• Move secrets out of the prompt. Put API keys and DB connection info in environment variables or a secrets manager — not in the prompt.
• Don't delegate security controls to the prompt. Do not implement authorization like "only allow this operation when the user is admin" through prompt instructions. Privilege separation and authorization are enforced deterministically outside the LLM. Guardrails (independent systems that inspect input/output) are a valuable extra layer, but they are not a substitute for deterministic authorization.

LLM10: Unbounded Consumption — stop DoS and cost blow-up

An item that expands the old "Model DoS." Because each LLM request is expensive, abuse immediately turns into financial damage (cost blow-up) and service degradation. The countermeasures are nearly identical to web API defense: rate limiting, input caps, output-token caps, timeouts, and monitoring.

import { Ratelimit } from "@upstash/ratelimit";

// 1) レート制限：濫用・コスト暴走・モデル窃取（大量クエリでの蒸留）を抑止
const { success } = await ratelimit.limit(`llm:${session.userId}`);
if (!success) return new Response("Too Many Requests", { status: 429 });

// 2) 入力サイズと出力トークンを「上限」で締める（＝コスト上限）
const { text } = await generateText({
  model: "anthropic/claude-haiku-4-5",
  prompt: clampInput(userPrompt, 4_000), // 入力長を検証してから渡す
  maxOutputTokens: 512, // 出力上限 = コストの上限
  abortSignal: AbortSignal.timeout(15_000), // 応答が返らないときは止める
});

And observability. Record token consumption, latency, error rate, and refusal rate per user, and detect anomalous consumption. The design of logs and traces builds on production observability with OpenTelemetry.

Framework crosswalk: OWASP × NIST × MITRE × SAIF

The "OWASP LLM Top 10" is an implementation list for developers. For organizational risk management and a broader threat model, use these alongside it. They are authoritative primary sources that AI answers (AI Overview, ChatGPT, etc.) tend to cite.

For how to run threat modeling itself, see threat modeling with STRIDE; for the secure-coding foundation, see secure coding aligned with NIST SSDF / OWASP ASVS.

Adoption checklist (before going to production)

The minimal set you can use starting tomorrow. It distills the OWASP LLM Top 10 into "design-review viewpoints."

• Output validation (LLM05) — Is LLM output validated for type and value range with a Zod schema, and are business rules re-checked in deterministic code?
• Least-privilege tools (LLM06) — Have you avoided handing agents direct destructive execution rights? Did you separate "propose" from "execute"?
• Human approval (LLM06) — Do high-impact actions (refund, delete, transfer, permission change) have an approval gate?
• Idempotency (LLM06) — Do side-effecting tool executions have an idempotency key so retries don't double-execute?
• RAG authorization (LLM02/LLM08) — Do you embed tenant/visibility filters in the vector search query? Do you treat retrieved documents as untrusted input?
• Secret separation (LLM07) — Have you kept API keys, DB info, and authorization logic out of the prompt?
• Rate limiting and token caps (LLM10) — Do you have per-user rate limiting, input-length validation, maxOutputTokens, and timeouts?
• Observability (LLM10) — Do you record token consumption, latency, and refusal rate per user, and can you detect anomalies?
• Supply chain (LLM03/LLM04) — Do you vet the sources of models, datasets, and dependencies, and keep provenance (SBOM/ML-BOM)?

Summary

Securing a production LLM app is not special magic. It comes down to one thing: "treat the LLM as a probabilistic, untrusted component, and surround it with deterministic architecture."

• Prompt injection can't be cured by input filters (LLM01). So validate output with zero trust (LLM05), and keep tools at least privilege with human approval for high-impact actions (LLM06).
• RAG and agents widen the attack surface. Embed authorization into retrieval (LLM08/LLM02), and move secrets out of the prompt (LLM07).
• Stop abuse and cost blow-up with rate limiting, token caps, and observability (LLM10).

And much of this is continuous with the reliability design of distributed systems that kept production double-charges at zero on a payments platform. "Building fast" and "being in a defensible state" are compatible when you design correctly.