Skip to main content
友田 陽大
Generative AI, LLMs & RAG
Next.js
Vercel
AI
セキュリティ
アーキテクチャ設計
コスト最適化

Protecting a Login-Free AI Chat from Bill Bankruptcy — Layered Edge Defense and Picking a Datastore by Consistency Model

A login-free AI chat is, to an attacker, an LLM API they can hammer on your credit card. This guide designs a four-layer defense (Vercel WAF, Proxy, Route Handler, provider budget) in real code, and explains why rate limits belong in an eventually consistent store while billing needs strong consistency — grounded in the architectures of Redis, CockroachDB, and DynamoDB.

Published
Reading time
38 min read
Author
友田 陽大
Share

Let me start with the conclusion. The biggest risk in a login-free generative AI chat is not that your service goes down — it is that your invoice goes vertical. An attacker does not need to take your servers offline. They script requests against an inference endpoint that has no login in front of it, and keep going. At a few cents per request, a hundred requests per second overnight is a five-figure morning. Availability is untouched; only your finances break.

And as of August 2026, most writing on this subject gets three premises wrong.

  1. Vercel KV no longer exists. It was discontinued as a product, and existing stores were automatically migrated to Upstash Redis in December 2024. If you are building today, you install Upstash Redis through the Marketplace.
  2. middleware.ts was deprecated in Next.js 16. It was renamed to proxy.ts, and the default runtime is Node.js, not Edge. The docs also explicitly say the feature is "recommended to be used as a last resort."
  3. "Defend against DDoS at the edge" does not have your code as its subject. If you absorb a volumetric flood in your own proxy, you pay for every rejected request's execution. Blanket defense belongs to Vercel WAF rate limiting — the layer that drops traffic without ever invoking a function.

This article designs a layered defense for anonymous chat using a correct 2026 parts list. The first half covers the defense architecture (four layers: WAF, Proxy, Route Handler, provider budget) and its implementation; the second half explains why rate limits should be eventually consistent while billing must be strongly consistent, working from the architectures of Redis, CockroachDB, and DynamoDB.

The fundamentals of rate limiting itself — fixed vs. sliding windows, atomicity, the x-forwarded-for trap — are covered in Rate Limiting That Actually Works in Next.js. This article builds on that and focuses on what changes when the workload is anonymous, expensive, and powered by an LLM.


1. The threat model: this is cost DoS, not DoS

TL;DR: When usage-based cloud billing meets token-based LLM pricing, the attacker's goal shifts from "take it down" to "make it run." Your availability metrics — error rate, latency — stay green while the damage accrues, so your monitoring has to change too.

OWASP promoted this threat to its own entry in the Top 10 for LLM Applications: LLM10:2025 Unbounded Consumption. In classic API security it maps to API4:2023 Unrestricted Resource Consumption. Both make the same point: serving a request consumes bandwidth, CPU, and memory — but also money, in the form of metered third-party APIs. Leave that consumption uncapped and both availability and cost become attack surfaces.

Lay out what is specific to a login-free chat and the risk becomes obvious.

ConditionOrdinary APILogin-free LLM chat
Cost per requestUnder $0.00001 (a DB read)Cents to tens of cents (input + output tokens)
Setup the attacker needsSteal or mass-produce credentialsNone (just the URL)
Response timeTens of msSeconds to tens of seconds (billed execution time)
Attacker's payoffStolen dataFree LLM access (distillation, resale)
Ease of detectionError rate climbsNothing breaks (only the invoice grows)

That last row is the crux. To an attacker your chat is a free LLM proxy, so they actively want it healthy. CPU, error rate, and latency all stay normal — and then the bill arrives. A green dashboard guarantees nothing against this attack.

There is a second, chat-specific risk: model distillation, where an attacker harvests large volumes of high-quality output to train their own model. That attack is measured in total tokens rather than cost per call, so request-count limits alone will miss it. This is why the token bucket's rate option — weighted consumption — matters later in this article.

When I designed and built a generative-AI voice concierge running on unattended kiosks, the first thing I designed was not the RAG pipeline or the prompts. It was the consumption boundary: how many model calls a single kiosk and a single session may make, whoever the visitor is. Shipping unattended, login-free generative AI to production is less a question of wiring up a model than of deciding where the ceiling on consumption goes.


2. The correct 2026 parts list — three premises that changed

TL;DR: Vercel KV is gone, middleware.ts became proxy.ts with a Node.js default, and blanket DDoS defense is the WAF's job. Miss any of the three and you ship either code that does not run or code that is needlessly expensive.

2.1 Vercel KV is discontinued — use Upstash Redis from the Marketplace

Plenty of articles still show @vercel/kv and kv.get(), but Vercel KV is not offered as a product anymore. Existing KV stores were automatically migrated to Upstash Redis in December 2024, and the path today is to install Upstash for Redis from the Vercel Marketplace. Adding the integration injects credentials (UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN) into your project's environment variables.

The code change is simple: drop @vercel/kv and use @upstash/redis and @upstash/ratelimit directly. Historically @vercel/kv was a wrapper around Upstash, so migrating is effectively "peel off the wrapper."

2.2 middleware.ts is deprecated — use proxy.ts (Node.js runtime by default)

Next.js 16 renamed the middleware file convention to proxy, and middleware.ts is deprecated and slated for removal. The exported function changes from middleware to proxy as well. There is an official codemod.

npx @next/codemod@canary middleware-to-proxy .

The execution model changed at the same time. The version history states v16.0.0: Middleware is deprecated and renamed to Proxy. Proxy defaults to the Node.js runtime, and the runtime config option is unavailable in Proxy — setting it throws. The old habit of writing runtime: 'edge' to "run it at the edge" is now literally an exception in Proxy.

More important is how the Next.js team frames Proxy:

we are moving away from Middleware ... this feature is recommended to be used as a last resort

And a design constraint:

you should not attempt relying on shared modules or globals

That warning bears directly on ephemeralCache (a module-scoped Map) below. It is an optimization, not a correctness mechanism — you have to understand it that way.

Note that Vercel's product is still called Routing Middleware. It is built on Fluid Compute and still runs globally, ahead of the cache. Keep the two ideas separate: the Next.js file convention became proxy.ts; the Vercel platform feature is still Routing Middleware.

2.3 Blanket DDoS defense is the WAF's job, not your code's

This is the most commonly misunderstood point in the whole design. When your proxy rate limits and returns 429, that proxy execution is still billed. Routing Middleware is priced on the Fluid Compute model — you pay for compute resources consumed. Absorb ten thousand requests per second of flood in your own code and you are paying for ten thousand executions while defending.

Vercel WAF rate limiting drops requests before they reach a function. The specifics are worth knowing precisely.

ItemHobbyProEnterprise
Counting keysIP, JA4 DigestIP, JA4 Digestplus User Agent, arbitrary headers
AlgorithmFixed windowFixed windowplus Token bucket
Window10s to 10min10s to 10min10s to 1hr
Rules1 per project40 per project1000 per project
ActionsLog / Deny / Challenge / default 429SameSame

And a limitation the docs state explicitly: rate limit counters are tracked per region, so traffic spread across regions can exceed your configured limit in aggregate. The WAF is a coarse net, not a precise cap. That is exactly why you pair it with application-layer limits.

Put together, defense is not one wall but four layers with different jobs.

LayerWhat it stopsImplementationCost of a false positiveBilling
L1 PlatformVolumetric floods, known-bad IPsVercel automatic DDoS mitigation + WAF rate limitingHigh (legitimate users drop too)No function invoked
L2 Proxy (formerly middleware)Obvious abuse, coarse per-IP caps, rejection before reading the bodyproxy.ts + @upstash/ratelimitMediumProxy execution
L3 Route HandlerPrecise per-session caps, bot verification, input validationapp/api/chat/route.ts + TurnstileLow (a good UI lets them retry)Function execution
L4 ProviderThe global ceiling (last resort)AI Gateway / provider budget and usage caps

Do not skip L4. L1 through L3 all work only if your code is running correctly. A bad deploy drops the proxy matcher, an environment variable disappears, Redis goes down and you fail open — in those moments the only thing that can stop the spending is a hard cap on the provider side. Put a ceiling outside your code that your code cannot raise. That is not redundancy; it is building an independent failure domain.


3. What the edge buys you, and what it does not

TL;DR: The value of the edge (running ahead of the cache, everywhere) is that an attack ends near the user and never touches your origin or your database. But the latency win evaporates if the datastore you consult is far away.

In a centralized architecture, a request from an attacker in Tokyo travels all the way to your API server in Virginia before being rejected. The round-trip latency, the bandwidth, and the compute to make the rejection are all borne at one point you chose, not the one the attacker chose. A single point of concentration is simultaneously a bottleneck and a single point of failure.

Edge deployment inverts that. Routing Middleware runs at points of presence worldwide ahead of the cache, so the rejection happens next to the attacker and never reaches the origin or the database. That is what it means for defense to go from a point to a plane.

But here is the edge's biggest trap, stated plainly in Vercel's docs:

If your Routing Middleware depends on a database far away from one of our supported regions, the overall latency of API requests could be slower than expected

Judge in a Singapore POP but keep your Redis in us-east-1, and every decision crosses the Pacific. Classic failure: you moved it to the edge and it got slower. Three remedies:

  1. Choose a store that is globally readable (or replicated) — Upstash read replicas, Edge Config.
  2. Carry the decision data in the request itself — a signed cookie (the session ID below) is verifiable without a single network round trip.
  3. Close hot rejections locally with ephemeralCache — an identifier already known to be over its limit can be dropped from a local Map without asking Redis at all.

On that third point, Upstash describes ephemeralCache as caching data while the function is hot so that Redis is only consulted when it is cold, and specifies that a response served this way carries reason: "cacheBlock". Next.js, meanwhile, tells you to avoid relying on globals in Proxy. The two are not in conflict. If ephemeralCache vanishes, correctness is unaffected (you just ask Redis again); if it survives, cost drops. It is a one-way optimization. Miss that asymmetry — "we have a local cache, so we're fine" — and things break.


4. Why an edge KV store has to speak HTTP

TL;DR: Redis's native protocol is connection-based and assumes long-lived connections. In serverless and at the edge, where thousands of disposable functions spin up at once, that assumption shows up as connection exhaustion. This is why Upstash offers a REST API.

A traditional server creates one connection pool at startup and reuses it for the life of the process. A hundred servers holding ten connections each is a thousand connections. Manageable.

Serverless is different. Function instances are disposable, hundreds or thousands run concurrently, and each is an independent process. There is nobody to share a pool with. And in edge runtimes (Cloudflare Workers, WebAssembly, Fastly Compute@Edge) raw TCP sockets are not available at all. Upstash puts the constraint succinctly: the Redis protocol is connection based, whereas the REST API is request based.

AspectTCP-based RedisHTTP-based KV (Upstash REST)
Connection modelLong-lived, pooledStateless (per request)
Edge runtimesDoes not work (no TCP)Works (fetch only)
10,000 concurrentExhaustion, ERR max number of clientsFine (HTTP multiplexing)
Cold startHandshake + auth round tripsOne HTTP round trip
AuthAUTH commandBearer token
Best fitLong-running servers, heavy pipeliningServerless, edge, functions

Because @upstash/redis uses fetch under the hood, the same code runs on Node.js, Edge, and Workers. That is more than a convenience: it means your rate limiting logic can be shared between the proxy and the Route Handler without branching per runtime. It keeps you DRY.


5. Designing the limits — the composite-key trap, and two limiters

TL;DR: Concatenating "IP + session ID" into one string key is a hole, because an attacker can rotate the session ID and mint unlimited buckets. Keep a coarse network limiter and a fine session limiter independent, and only allow a request that passes both.

5.1 Why ip:sessionId as one key falls apart

The naive reasoning goes: "IP alone punishes everyone in the office. Session alone dies when they clear cookies. So let's combine them." And you write ratelimit.limit(ip + ":" + sessionId).

This is very nearly meaningless as a security control. A rate limit key identifies a bucket; change the key and you get a fresh bucket. The attacker simply sends no cookie (or a random ID each time) and receives a brand-new bucket per request. Including the IP constrained nothing.

The correct design is two independent limiters, ANDed together.

LimiterIdentifierLimit (example)What it protectsFalse-positive risk
Coarse netFull address for IPv4, /64 prefix for IPv660/minVolume from a single originMedium (NAT, universities, offices)
Fine netSession ID from a signed cookie10/min (burst 20)One legitimate user's experienceLow (only their own session)

The coarse net is the backbone — it still works when the attacker throws cookies away. The fine net is the fairness device that keeps one user from taking down others behind the same IP. Different jobs, so different limits.

Rounding IPv6 to /64 is mandatory in practice. In IPv6 even an end user is typically assigned a /64 (18,446,744,073,709,551,616 addresses); key on the full address and an attacker evades your limit by incrementing. Carry IPv4 intuitions over unchanged and you will step on this mine.

5.2 Getting the client IP — on Vercel, you can trust it

As a general rule x-forwarded-for is client-controlled and should only be read behind a trusted proxy. On Vercel the situation is different, and the docs are explicit:

we currently overwrite the X-Forwarded-For header and do not forward external IPs. This restriction is in place to prevent IP spoofing.

So as long as you deploy on Vercel, x-forwarded-for (and its equivalents x-real-ip and x-vercel-forwarded-for) is a platform-guaranteed value. You do not need to write "take the nth entry from the left" parsing. If you run your own proxy in front of Vercel, however, x-forwarded-for may be overwritten — read x-vercel-forwarded-for in that case.

5.3 Why token bucket

@upstash/ratelimit ships three algorithms. Which fits an LLM chat?

AlgorithmBehaviorBurst handlingCostFit for chat
fixedWindow(limit, window)Counts per window2x can pass at the boundaryCheapestPoor (boundary problem)
slidingWindow(limit, window)Weighted against the previous windowSmoothMediumGood
tokenBucket(refillRate, interval, maxTokens)Refills at a fixed rate, passes while tokens remainExplicitly designableHigh (computation)Best

Chat usage is inherently bursty: a user reads quietly, then fires three or four questions back to back. Squeeze that into "10 per minute" with a fixed window and you reject normal behavior. A token bucket lets you design burst tolerance with maxTokens and long-run average rate with refillRate independently.

// Refill 10 tokens every 60 seconds, bucket capacity 20
// -> averages 10/min, but allows 20 back-to-back after a pause
Ratelimit.tokenBucket(10, "60 s", 20)

Mind the argument order: tokenBucket(refillRate, interval, maxTokens) — different from fixedWindow and slidingWindow, where limit comes first. Upstash also documents that token bucket is expensive in terms of computation and is not yet supported for MultiRegionRatelimit; if you need global distribution, plan to switch to sliding window.

And for the LLM-specific requirement, limit() accepts a rate option for weighted consumption.

// Long prompts and higher-tier models drain more tokens
await limiter.limit(identifier, { rate: estimateCost(input, model) });

You are draining the bucket by cost, not by request count. That is your counter to attacks — such as model distillation — that hit you on total token volume.


6. Implementation — four type-safe, fail-safe modules

The code below assumes TypeScript strict mode and uses no any and no type assertions (as). Every external input — environment variables, request bodies, third-party API responses — is validated and narrowed at the boundary with Zod.

6.1 Foundations — validating env vars and structured logging

Design intent: process.env.FOO! is a lie told to the type system. A missing variable survives as undefined until runtime, where it becomes Cannot read properties of undefined in production. Validate once at startup, then circulate only fully typed values (fail fast).

// lib/env.ts
import "server-only";
import { z } from "zod";

/**
 * Server-only environment variables. Validated once at module load;
 * an invalid environment prevents the process from starting (fail fast).
 */
const serverEnvSchema = z.object({
  // Injected automatically by the Upstash integration from the Vercel Marketplace
  UPSTASH_REDIS_REST_URL: z.url(),
  UPSTASH_REDIS_REST_TOKEN: z.string().min(1),

  // Cloudflare Turnstile secret (must never reach the client)
  TURNSTILE_SECRET_KEY: z.string().min(1),

  // Signing key for anonymous session IDs. Use 32+ random bytes.
  SESSION_SECRET: z.string().min(32),

  // Inference provider. Via Vercel AI Gateway this is a "provider/model" string.
  CHAT_MODEL: z.string().min(1).default("anthropic/claude-sonnet-4.5"),

  // Environment identity (used as a field in structured logs)
  VERCEL_ENV: z.enum(["production", "preview", "development"]).default("development"),
});

const parsed = serverEnvSchema.safeParse(process.env);

if (!parsed.success) {
  // Never print the values themselves — report only the offending key names.
  const invalidKeys = [...new Set(parsed.error.issues.map((issue) => issue.path.join(".")))];
  throw new Error(`[env] Invalid server environment variables: ${invalidKeys.join(", ")}`);
}

export const serverEnv = parsed.data;
// lib/env.client.ts
import { z } from "zod";

/**
 * Public values only — safe to expose to the client.
 * NEXT_PUBLIC_* is statically replaced, so the property access must be written literally.
 */
const clientEnvSchema = z.object({
  NEXT_PUBLIC_TURNSTILE_SITE_KEY: z.string().min(1),
});

export const clientEnv = clientEnvSchema.parse({
  NEXT_PUBLIC_TURNSTILE_SITE_KEY: process.env.NEXT_PUBLIC_TURNSTILE_SITE_KEY,
});

Why two files? Merge them and the moment a client component imports the module for the public key, the secret key names in the schema become part of the client bundle's static analysis. import "server-only" is the safety net that fails the build if a server-only module leaks into the client.

Then logging. console.error("it failed") is worthless in production. Structure it from day one, assuming you will eventually ship it to Sentry or a Log Drain. Confine the output destination to one place and the migration stays inside this file (SRP).

// lib/observability.ts
type LogLevel = "info" | "warn" | "error";

/**
 * One structured log record. Values are limited to string, number, boolean, null.
 * Allowing objects invites the accident of dumping an entire request body.
 */
interface LogFields {
  readonly level: LogLevel;
  /** Dot-separated event name (e.g. "rate_limit.blocked"). This is the aggregation key. */
  readonly event: string;
  readonly [key: string]: string | number | boolean | null | undefined;
}

/**
 * Keeping PII and secrets out is the caller's responsibility.
 * This function carries only an identifier of what happened plus non-PII measurements.
 */
export function logEvent(fields: LogFields): void {
  const line = JSON.stringify({ ...fields, timestamp: new Date().toISOString() });

  // When Sentry arrives later, only this branch needs to change.
  if (fields.level === "error") {
    console.error(line);
    return;
  }
  if (fields.level === "warn") {
    console.warn(line);
    return;
  }
  console.info(line);
}

6.2 lib/rate-limit.ts — two limiters and an observable verdict

Design intent: separate the rate limiting policy from its execution (SRP). The policy lives in one place as constants, eliminating magic numbers. The execution encloses the failure behavior (fail open) so callers are not forced to write exception handling.

// lib/rate-limit.ts
import "server-only";
import { Ratelimit, type Duration } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";
import { serverEnv } from "./env";
import { logEvent } from "./observability";

const redis = new Redis({
  url: serverEnv.UPSTASH_REDIS_REST_URL,
  token: serverEnv.UPSTASH_REDIS_REST_TOKEN,
});

/**
 * Remembers already-rejected identifiers inside a hot instance, saving a
 * Redis round trip.
 * Note: this is an optimization, not a correctness mechanism. Next.js Proxy
 * explicitly forbids relying on global state, and behavior is unchanged if
 * this disappears (it just asks Redis again). Treat it as a one-way optimization.
 */
const ephemeralCache = new Map<string, number>();

interface TokenBucketPolicy {
  /** Tokens refilled per interval (i.e. the long-run average rate) */
  readonly refillRate: number;
  /** The refill interval */
  readonly interval: Duration;
  /** Bucket capacity (i.e. the burst size you tolerate) */
  readonly maxTokens: number;
  /** Redis key namespace. Always distinct per layer. */
  readonly prefix: string;
}

/** Single source of truth for the limits. Tuning in production touches only this object. */
const POLICIES: Readonly<Record<"network" | "session", TokenBucketPolicy>> = {
  // Coarse net: per originating network. The backbone that survives cookie deletion.
  network: { refillRate: 60, interval: "60 s", maxTokens: 90, prefix: "rl:net" },
  // Fine net: per session. The fairness device protecting one user's experience.
  session: { refillRate: 10, interval: "60 s", maxTokens: 20, prefix: "rl:sess" },
};

function createLimiter(policy: TokenBucketPolicy): Ratelimit {
  return new Ratelimit({
    redis,
    limiter: Ratelimit.tokenBucket(policy.refillRate, policy.interval, policy.maxTokens),
    prefix: policy.prefix,
    ephemeralCache,
    // If Redis does not answer, requests past this deadline are allowed (fail open).
    // Availability > strictness: a rate limiting outage must not stop the service.
    timeout: 1_000,
    // Analytics only in production; it consumes extra Redis commands.
    analytics: serverEnv.VERCEL_ENV === "production",
  });
}

const limiters: Readonly<Record<keyof typeof POLICIES, Ratelimit>> = {
  network: createLimiter(POLICIES.network),
  session: createLimiter(POLICIES.session),
};

export interface RateLimitVerdict {
  readonly allowed: boolean;
  /** Which layer rejected it. Used for monitoring and response headers. */
  readonly limitedBy: keyof typeof POLICIES | null;
  /** Seconds until a retry may succeed (for Retry-After) */
  readonly retryAfterSeconds: number;
  /** Async work that must be allowed to finish (analytics writes, etc.) */
  readonly pending: readonly Promise<unknown>[];
}

const ALLOWED: RateLimitVerdict = {
  allowed: true,
  limitedBy: null,
  retryAfterSeconds: 0,
  pending: [],
};

export interface RateLimitInput {
  /** Coarse identifier (full IPv4 address, or the /64 prefix for IPv6) */
  readonly networkId: string;
  /** Fine identifier (session ID from the signed cookie) */
  readonly sessionId: string;
  /** Tokens consumed by this request; weighted by input length and model */
  readonly cost: number;
}

/**
 * Evaluates both limiters with AND semantics. The coarse net runs first so
 * attack traffic is rejected at the lowest possible cost (fail fast).
 */
export async function checkRateLimit(input: RateLimitInput): Promise<RateLimitVerdict> {
  const checks: readonly [keyof typeof POLICIES, string][] = [
    ["network", input.networkId],
    ["session", input.sessionId],
  ];

  const pending: Promise<unknown>[] = [];

  for (const [layer, identifier] of checks) {
    try {
      const result = await limiters[layer].limit(identifier, { rate: input.cost });
      pending.push(result.pending);

      if (!result.success) {
        return {
          allowed: false,
          limitedBy: layer,
          retryAfterSeconds: Math.max(1, Math.ceil((result.reset - Date.now()) / 1_000)),
          pending,
        };
      }
    } catch (error) {
      // Fail open: an outage in the rate limiting infrastructure must not become
      // an outage of the service. Never swallow it silently, though — record it
      // as an anomaly (the L4 provider cap is the last line of defense).
      logEvent({
        level: "error",
        event: "rate_limit.unavailable",
        layer,
        message: error instanceof Error ? error.message : "unknown error",
      });
      return ALLOWED;
    }
  }

  return { ...ALLOWED, pending };
}

On whether to fail open. Whether a broken rate limiter should allow requests (open) or block them (closed) is a calculation, not a philosophy. Here it fails open, because "the loss from chat being down during a Redis outage" outweighs "the inference cost of a few minutes of abuse." But that judgment presumes the L4 provider budget cap exists. Without a ceiling, fail closed. Failing open is only defensible when the ceiling lives somewhere else.

6.3 lib/session.ts — a tamper-proof anonymous session

Design intent: even without login you need an identifier for "probably the same user." A plaintext random ID is not enough — an attacker just changes the value and mints buckets. An HMAC signature makes the ID verifiably server-issued, so forged IDs never reach the session-layer limiter.

// lib/session.ts
import "server-only";
import { serverEnv } from "./env";

const SESSION_COOKIE = "aichat_sid";
const SESSION_MAX_AGE_SECONDS = 60 * 60 * 24; // 24 hours

/** Web Crypto only, so the same code runs on both the Node.js and Edge runtimes. */
async function hmac(payload: string): Promise<string> {
  const key = await crypto.subtle.importKey(
    "raw",
    new TextEncoder().encode(serverEnv.SESSION_SECRET),
    { name: "HMAC", hash: "SHA-256" },
    false,
    ["sign"],
  );
  const signature = await crypto.subtle.sign("HMAC", key, new TextEncoder().encode(payload));
  return Array.from(new Uint8Array(signature))
    .map((byte) => byte.toString(16).padStart(2, "0"))
    .join("");
}

/** Constant-time comparison to avoid timing attacks. */
function timingSafeEqual(a: string, b: string): boolean {
  if (a.length !== b.length) return false;
  let diff = 0;
  for (let i = 0; i < a.length; i += 1) {
    diff |= a.charCodeAt(i) ^ b.charCodeAt(i);
  }
  return diff === 0;
}

export async function issueSessionId(): Promise<string> {
  const id = crypto.randomUUID();
  return `${id}.${await hmac(id)}`;
}

/** Returns the ID only when the signature checks out; forged or missing yields null. */
export async function verifySessionId(value: string | undefined): Promise<string | null> {
  if (!value) return null;
  const separatorIndex = value.lastIndexOf(".");
  if (separatorIndex <= 0) return null;

  const id = value.slice(0, separatorIndex);
  const signature = value.slice(separatorIndex + 1);
  const expected = await hmac(id);

  return timingSafeEqual(signature, expected) ? id : null;
}

interface SessionCookieSpec {
  readonly name: string;
  readonly options: {
    readonly httpOnly: true;
    readonly secure: true;
    /** JS never reads the cookie (httpOnly). lax narrows the CSRF surface. */
    readonly sameSite: "lax";
    readonly path: "/";
    readonly maxAge: number;
  };
}

export const sessionCookie: SessionCookieSpec = {
  name: SESSION_COOKIE,
  options: {
    httpOnly: true,
    secure: true,
    sameSite: "lax",
    path: "/",
    maxAge: SESSION_MAX_AGE_SECONDS,
  },
};

6.4 proxy.ts — early rejection and session issuance

Design intent: this is the "rate limit at the entrance" the brief asked for, implemented against the correct Next.js 16 file convention (proxy.ts / export function proxy). It has exactly two jobs: (1) coarse early rejection, (2) issuing the anonymous session. Body parsing and inference-related decisions belong to the Route Handler (SRP).

// proxy.ts (Next.js 16; formerly middleware.ts — the function name changes too)
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";
import { checkRateLimit } from "@/lib/rate-limit";
import { issueSessionId, sessionCookie, verifySessionId } from "@/lib/session";
import { clientIp, networkKey } from "@/lib/client-identifier";
import { logEvent } from "@/lib/observability";

/**
 * State the protected surface explicitly. The wider the matcher, the more
 * billed executions land on static assets. Here: the inference endpoint and
 * the chat page only.
 */
export const config = {
  matcher: ["/api/chat", "/chat"],
};

/** Baseline cost of one request; the real weighting is re-evaluated in the Route Handler. */
const BASE_COST = 1;

export async function proxy(request: NextRequest): Promise<NextResponse> {
  const networkId = networkKey(clientIp(request.headers));

  // A cookie that fails signature verification is treated as absent, and reissued.
  const existingSessionId = await verifySessionId(
    request.cookies.get(sessionCookie.name)?.value,
  );
  const sessionId = existingSessionId ?? (await issueSessionId());

  // A freshly issued session has by definition drained no bucket, so the session
  // layer always passes. Catching an attacker who keeps discarding cookies is
  // always the network layer's job — which is why the two are independent.
  const verdict = await checkRateLimit({
    networkId,
    // Key on the raw ID, not the signed string
    sessionId: existingSessionId ?? sessionId.split(".")[0],
    cost: BASE_COST,
  });

  if (!verdict.allowed) {
    logEvent({
      level: "warn",
      event: "rate_limit.blocked",
      layer: verdict.limitedBy,
      path: request.nextUrl.pathname,
    });

    return NextResponse.json(
      {
        error: "rate_limited",
        message: "Too many requests. Please wait a moment and try again.",
        retryAfterSeconds: verdict.retryAfterSeconds,
      },
      {
        status: 429,
        headers: { "Retry-After": String(verdict.retryAfterSeconds) },
      },
    );
  }

  const response = NextResponse.next();

  if (!existingSessionId) {
    response.cookies.set({
      name: sessionCookie.name,
      value: sessionId,
      ...sessionCookie.options,
    });
  }

  return response;
}
// lib/client-identifier.ts
const IPV6_PREFIX_SEGMENTS = 4; // /64 = 16 bits x 4

/**
 * The client's real IP. Vercel overwrites x-forwarded-for and does not forward
 * externally supplied values, so this header can be treated as platform-guaranteed.
 * If you run your own proxy in front of Vercel, prefer x-vercel-forwarded-for.
 *
 * Returns an address safe to pass to an external API, or null if unavailable.
 */
export function clientIp(headers: Headers): string | null {
  const raw =
    headers.get("x-vercel-forwarded-for") ??
    headers.get("x-forwarded-for") ??
    headers.get("x-real-ip");

  if (!raw) return null;

  const address = raw.split(",")[0].trim();
  return address.length > 0 ? address : null;
}

/**
 * Rounds an IP down for use as a rate limit key. Kept separate from the real IP
 * because the rounded value is an incomplete address and must never be sent to
 * an external API.
 *
 * IPv6 is rounded to /64: keyed on the full address, even an end user holds
 * 2^64 addresses and can evade the limit indefinitely.
 */
export function networkKey(ip: string | null): string {
  if (!ip) return "unknown";
  if (!ip.includes(":")) return ip;
  return ip.split(":").slice(0, IPV6_PREFIX_SEGMENTS).join(":");
}

Projects still running middleware.ts can use the same code by renaming the export to export function middleware(...) — deprecated in Next.js 16 and slated for removal. The migration is a one-line codemod.

6.5 lib/turnstile.ts — never trust a third-party response

Design intent: the siteverify payload is external input. Trusting response.json() on the strength of a type annotation reintroduces any in all but name. Validate with Zod, and beyond success being true, confirm that action and hostname match. Skip that and an attacker can reuse a token obtained from a widget with the same site key hosted on another site.

// lib/turnstile.ts
import "server-only";
import { z } from "zod";
import { serverEnv } from "./env";
import { logEvent } from "./observability";

const SITEVERIFY_ENDPOINT = "https://challenges.cloudflare.com/turnstile/v0/siteverify";
const VERIFY_TIMEOUT_MS = 3_000;

/** The documented siteverify response shape. Unknown keys are ignored. */
const siteverifyResponseSchema = z.object({
  success: z.boolean(),
  "error-codes": z.array(z.string()).default([]),
  challenge_ts: z.string().optional(),
  hostname: z.string().optional(),
  action: z.string().optional(),
  cdata: z.string().optional(),
});

export type TurnstileFailureReason =
  | "missing-token"
  | "invalid-token"
  | "expired-or-duplicate"
  | "action-mismatch"
  | "verification-unavailable";

export type TurnstileResult =
  | { readonly ok: true }
  | { readonly ok: false; readonly reason: TurnstileFailureReason };

interface VerifyParams {
  readonly token: string | undefined;
  /** The client's real IP — pass the full address, not the rounded key */
  readonly remoteIp: string | null;
  /** The action set on the widget; always confirm it matches */
  readonly expectedAction: string;
  /** Your own hostname, to block tokens harvested elsewhere */
  readonly expectedHostname: string;
  /** Idempotency key so a retry does not count as double consumption */
  readonly idempotencyKey: string;
}

export async function verifyTurnstile(params: VerifyParams): Promise<TurnstileResult> {
  if (!params.token) return { ok: false, reason: "missing-token" };

  const body = new FormData();
  body.append("secret", serverEnv.TURNSTILE_SECRET_KEY);
  body.append("response", params.token);
  body.append("idempotency_key", params.idempotencyKey);
  // remoteip is optional. An invalid value is rejected as bad-request, so send
  // it only when a real address was available.
  if (params.remoteIp !== null) {
    body.append("remoteip", params.remoteIp);
  }

  let payload: unknown;
  try {
    const response = await fetch(SITEVERIFY_ENDPOINT, {
      method: "POST",
      body,
      signal: AbortSignal.timeout(VERIFY_TIMEOUT_MS),
    });
    payload = await response.json();
  } catch (error) {
    // Cloudflare outage or timeout. Bot verification fails to the safe side: reject.
    // Note this is the opposite call from rate limiting, which favors availability.
    logEvent({
      level: "error",
      event: "turnstile.unavailable",
      message: error instanceof Error ? error.message : "unknown error",
    });
    return { ok: false, reason: "verification-unavailable" };
  }

  const parsed = siteverifyResponseSchema.safeParse(payload);
  if (!parsed.success) {
    logEvent({ level: "error", event: "turnstile.malformed_response" });
    return { ok: false, reason: "verification-unavailable" };
  }

  const result = parsed.data;

  if (!result.success) {
    // timeout-or-duplicate means either "older than 300 seconds" or "token reused".
    // Prompt the user to retry; do not treat it as an attack.
    const isReuse = result["error-codes"].includes("timeout-or-duplicate");
    logEvent({
      level: "warn",
      event: "turnstile.rejected",
      codes: result["error-codes"].join(","),
    });
    return { ok: false, reason: isReuse ? "expired-or-duplicate" : "invalid-token" };
  }

  // success: true is not enough. The token only means something once you confirm
  // which widget issued it and on which domain.
  if (result.action !== params.expectedAction || result.hostname !== params.expectedHostname) {
    logEvent({
      level: "warn",
      event: "turnstile.context_mismatch",
      action: result.action ?? "none",
      hostname: result.hostname ?? "none",
    });
    return { ok: false, reason: "action-mismatch" };
  }

  return { ok: true };
}

Three points of the Turnstile spec are non-negotiable. A token expires 300 seconds (5 minutes) after issuance and validates exactly once; reuse is rejected as timeout-or-duplicate. So the client must refresh an expired token automatically (the expired-callback below). And idempotency_key is the mechanism that lets you safely re-verify with the same token when a network failure loses the response.

6.6 app/api/chat/route.ts — finish everything before calling inference

Design intent: the LLM call is the most expensive operation in this system, so exhaust every reason to reject before reaching it (fail fast). Order the checks from cheapest to most expensive: body size, then schema, then Turnstile (one external round trip), then the session rate limit, then inference.

// app/api/chat/route.ts
import { convertToModelMessages, createUIMessageStreamResponse, streamText, toUIMessageStream } from "ai";
import { z } from "zod";
import { after } from "next/server";
import { serverEnv } from "@/lib/env";
import { verifyTurnstile } from "@/lib/turnstile";
import { checkRateLimit } from "@/lib/rate-limit";
import { verifySessionId, sessionCookie } from "@/lib/session";
import { clientIp, networkKey } from "@/lib/client-identifier";
import { logEvent } from "@/lib/observability";

export const maxDuration = 30;

const TURNSTILE_ACTION = "chat-message";
const MAX_BODY_BYTES = 16 * 1024;
const MAX_MESSAGES = 40;

/** The accepted body shape: only the minimum of a UI message is permitted. */
const requestSchema = z.object({
  turnstileToken: z.string().min(1).max(2048),
  messages: z
    .array(
      z.object({
        id: z.string().min(1).max(128),
        role: z.enum(["user", "assistant", "system"]),
        parts: z
          .array(z.object({ type: z.literal("text"), text: z.string().min(1).max(4_000) }))
          .min(1)
          .max(8),
      }),
    )
    .min(1)
    .max(MAX_MESSAGES),
});

/** Estimate bucket consumption from total input length (weighted charging). */
function estimateCost(text: string): number {
  return Math.max(1, Math.ceil(text.length / 500));
}

function errorResponse(status: number, code: string, message: string, retryAfter?: number): Response {
  return Response.json(
    { error: code, message },
    {
      status,
      headers: retryAfter === undefined ? undefined : { "Retry-After": String(retryAfter) },
    },
  );
}

export async function POST(request: Request): Promise<Response> {
  // --- Check 1: body size (cheapest check first) ---
  // This is only an early rejection for well-behaved clients (headers can be forged).
  // The effective limit is enforced by the schema below (length and array caps).
  const contentLength = Number(request.headers.get("content-length") ?? "0");
  if (contentLength > MAX_BODY_BYTES) {
    return errorResponse(413, "payload_too_large", "That message is too long.");
  }

  // --- Check 2: schema (fix the structure before calling inference) ---
  const parsed = requestSchema.safeParse(await request.json().catch(() => null));
  if (!parsed.success) {
    return errorResponse(400, "invalid_request", "The request format is invalid.");
  }
  const { turnstileToken, messages } = parsed.data;

  // --- Check 3: session authenticity (forged cookies die here) ---
  const cookieHeader = request.headers.get("cookie") ?? "";
  const rawSession = cookieHeader
    .split(";")
    .map((part) => part.trim())
    .find((part) => part.startsWith(`${sessionCookie.name}=`))
    ?.slice(sessionCookie.name.length + 1);

  const sessionId = await verifySessionId(rawSession);
  if (!sessionId) {
    return errorResponse(401, "invalid_session", "Your session is invalid. Please reload the page.");
  }

  const ip = clientIp(request.headers);
  const networkId = networkKey(ip);

  // --- Check 4: Turnstile (one external round trip; everything above was local) ---
  const turnstile = await verifyTurnstile({
    token: turnstileToken,
    remoteIp: ip,
    expectedAction: TURNSTILE_ACTION,
    expectedHostname: new URL(request.url).hostname,
    idempotencyKey: `${sessionId}:${messages[messages.length - 1].id}`,
  });

  if (!turnstile.ok) {
    const status = turnstile.reason === "verification-unavailable" ? 503 : 403;
    return errorResponse(status, turnstile.reason, "Verification failed. Please try again.");
  }

  // --- Check 5: weighted rate limit (the last gate before inference) ---
  const inputText = messages
    .flatMap((message) => message.parts.map((part) => part.text))
    .join("");
  const cost = estimateCost(inputText);

  const verdict = await checkRateLimit({ networkId, sessionId, cost });

  if (!verdict.allowed) {
    return errorResponse(
      429,
      "rate_limited",
      "We're seeing heavy usage right now. Please wait a moment and try again.",
      verdict.retryAfterSeconds,
    );
  }

  // Let post-processing (analytics) finish without holding up the response
  after(() => Promise.allSettled(verdict.pending));

  // --- Only now do we call the operation that costs money ---
  logEvent({ level: "info", event: "chat.accepted", cost });

  const result = streamText({
    model: serverEnv.CHAT_MODEL,
    messages: await convertToModelMessages(messages),
    onError: ({ error }) => {
      logEvent({
        level: "error",
        event: "chat.inference_failed",
        message: error instanceof Error ? error.message : "unknown error",
      });
    },
  });

  return createUIMessageStreamResponse({
    stream: toUIMessageStream({ stream: result.stream }),
  });
}

Streaming does not require runtime = "edge". This misconception is everywhere, but ReadableStream, SSE, and AI token streaming all work on the default Node.js runtime (Fluid Compute) with zero configuration. Choosing Edge costs you the Node.js APIs and tightens your execution limits — it is actively worse here.

6.7 components/chat-input.tsx — a UI with accessibility and retry built in

Design intent: because Turnstile is invisible, the user has no idea what happened when it fails. This component's job is to distinguish, communicate, and recover from token expiry (300 seconds), verification errors, and rate limits (429).

// components/chat-input.tsx
"use client";

import { useCallback, useEffect, useId, useRef, useState } from "react";
import Script from "next/script";
import { clientEnv } from "@/lib/env.client";

const TURNSTILE_SCRIPT = "https://challenges.cloudflare.com/turnstile/v0/api.js?render=explicit";
const TURNSTILE_ACTION = "chat-message";

/** Minimal types for explicit rendering, declared here to avoid `any`. */
interface TurnstileRenderOptions {
  sitekey: string;
  action: string;
  appearance: "always" | "execute" | "interaction-only";
  callback: (token: string) => void;
  "error-callback": () => void;
  "expired-callback": () => void;
}

interface TurnstileApi {
  render: (container: HTMLElement, options: TurnstileRenderOptions) => string;
  reset: (widgetId: string) => void;
  remove: (widgetId: string) => void;
}

declare global {
  interface Window {
    turnstile?: TurnstileApi;
  }
}

type Status = "loading-challenge" | "ready" | "sending" | "error";

interface ChatInputProps {
  readonly onSend: (input: { text: string; turnstileToken: string }) => Promise<void>;
  readonly disabled?: boolean;
}

export function ChatInput({ onSend, disabled = false }: ChatInputProps) {
  const [status, setStatus] = useState<Status>("loading-challenge");
  const [errorMessage, setErrorMessage] = useState<string | null>(null);
  const [value, setValue] = useState("");

  const tokenRef = useRef<string | null>(null);
  const widgetIdRef = useRef<string | null>(null);
  const containerRef = useRef<HTMLDivElement | null>(null);

  const inputId = useId();
  const errorId = useId();

  const renderWidget = useCallback(() => {
    const container = containerRef.current;
    const turnstile = window.turnstile;
    if (!container || !turnstile || widgetIdRef.current !== null) return;

    widgetIdRef.current = turnstile.render(container, {
      sitekey: clientEnv.NEXT_PUBLIC_TURNSTILE_SITE_KEY,
      action: TURNSTILE_ACTION,
      // Show the widget only when interaction is required (invisible otherwise)
      appearance: "interaction-only",
      callback: (token) => {
        tokenRef.current = token;
        setStatus("ready");
        setErrorMessage(null);
      },
      "error-callback": () => {
        tokenRef.current = null;
        setStatus("error");
        setErrorMessage("Security check failed. Please reload the page.");
      },
      // Tokens expire after 300 seconds. Always re-issue so nothing breaks silently.
      "expired-callback": () => {
        tokenRef.current = null;
        setStatus("loading-challenge");
        if (widgetIdRef.current !== null) window.turnstile?.reset(widgetIdRef.current);
      },
    });
  }, []);

  useEffect(() => {
    return () => {
      const widgetId = widgetIdRef.current;
      if (widgetId !== null) window.turnstile?.remove(widgetId);
    };
  }, []);

  const isBusy = status === "sending" || status === "loading-challenge";
  const canSubmit = status === "ready" && value.trim().length > 0 && !disabled;

  const handleSubmit = async (event: React.FormEvent<HTMLFormElement>) => {
    event.preventDefault();
    const token = tokenRef.current;
    if (!canSubmit || !token) return;

    setStatus("sending");
    setErrorMessage(null);

    try {
      await onSend({ text: value.trim(), turnstileToken: token });
      setValue("");
    } catch (error) {
      setStatus("error");
      setErrorMessage(
        error instanceof Error ? error.message : "Sending failed. Please try again shortly.",
      );
    } finally {
      // Tokens are single-use. Re-issue on success and failure alike.
      tokenRef.current = null;
      if (widgetIdRef.current !== null) {
        window.turnstile?.reset(widgetIdRef.current);
        setStatus("loading-challenge");
      }
    }
  };

  return (
    <form onSubmit={handleSubmit} className="flex flex-col gap-2">
      <Script src={TURNSTILE_SCRIPT} strategy="afterInteractive" onReady={renderWidget} />

      <label htmlFor={inputId} className="text-sm font-medium">
        Message
      </label>

      <textarea
        id={inputId}
        value={value}
        onChange={(event) => setValue(event.target.value)}
        onKeyDown={(event) => {
          // Enter submits, Shift+Enter inserts a newline; never submit mid-IME-composition
          if (event.key === "Enter" && !event.shiftKey && !event.nativeEvent.isComposing) {
            event.preventDefault();
            event.currentTarget.form?.requestSubmit();
          }
        }}
        rows={3}
        maxLength={4000}
        disabled={disabled || status === "sending"}
        aria-describedby={errorMessage ? errorId : undefined}
        aria-invalid={status === "error"}
        className="w-full resize-none rounded-md border border-neutral-300 p-3 disabled:opacity-60"
        placeholder="Ask a question"
      />

      {/* Mount point for the invisible challenge; renders only when interaction is needed */}
      <div ref={containerRef} />

      {/* Surface errors to assistive technology immediately */}
      <p id={errorId} role="alert" aria-live="assertive" className="min-h-5 text-sm text-red-600">
        {errorMessage}
      </p>

      <button
        type="submit"
        disabled={!canSubmit}
        aria-busy={isBusy}
        className="self-end rounded-md bg-neutral-900 px-4 py-2 text-white disabled:opacity-50"
      >
        {status === "sending" ? "Sending..." : status === "loading-challenge" ? "Checking..." : "Send"}
      </button>
    </form>
  );
}

Four accessibility essentials: an explicit <label> association, errors tied to the input via aria-invalid and aria-describedby, role="alert" plus aria-live="assertive" so users who are not looking at the screen still hear the error, and aria-busy to convey the in-flight state. The isComposing check prevents an Enter keypress that confirms an IME conversion from firing an accidental submit — effectively mandatory for Japanese and other IME-based input.


7. Data gravity — why rate limits are allowed to drift

TL;DR: Stronger consistency is not simply better. Strong consistency always implies a network round trip for consensus, and you pay that latency on every request. The deciding question is what a single leaked request costs you.

7.1 Reading CAP the practical way

CAP is usually summarized as "pick two of consistency, availability, and partition tolerance," but in practice it means something simpler. Network partitions will happen, so P is not a choice. The real decision is what you do during a partition: answer with stale data (AP), or return an error to protect correctness (CP).

The biggest mistake is making that decision once, for the whole application. You make it per kind of data.

DataCost of getting it wrong onceCorrect choiceExample
Rate limit countersOne inference call (cents)AP (eventual)Upstash Redis
Chat history, logsA view is a few hundred ms staleAPRedis / DynamoDB
Billing, balances, creditsReal loss (double charge, negative balance)CP (strong)CockroachDB / an RDBMS
Authentication, permissionsSecurity breachCPSame

"Managing rate limits in CockroachDB with strong consistency" is technically possible, but it means adding a Raft consensus round (tens of ms and up) to every chat request — clear over-engineering for something worth cents. Conversely, "keeping credit balances in Redis" lets the balance go negative inside the replication lag window. That is under-engineering.

7.2 Comparing three stores by architectural principle

Not "fast or slow," but why.

Redis (Upstash) — in-memory, serial execution

Data lives in RAM, so disk I/O is not on the decision path. And because commands execute one at a time, INCR and Lua scripts are atomic without locks. That property is exactly why a rate limit counter update (read, add, write) does not corrupt under concurrent requests.

The price is memory cost and durability. RAM is orders of magnitude more expensive than SSD, so TTLs and maxmemory eviction policies (allkeys-lru, volatile-ttl, and friends) are mandatory design work. For rate limiting, TTLs fall out naturally, which pairs well with the volatile-ttl family. A counter can be lost in the worst case and nothing breaks — the next window simply rebuilds it. That "lossy but not broken" property is the reason rate limits can live in Redis.

CockroachDB — distributed SQL on Raft plus HLC

Data is split into ranges, each distributed as multiple replicas. The replicas of a range form a Raft group, and a write commits only after a majority (quorum) agrees. The default replication factor is 3, tolerating one node failure (5 tolerates two).

Reads are served by a single replica, the leaseholder; Leader Leases keep the leaseholder and the Raft leader aligned so strongly consistent reads need no extra consensus round. On the write side, Parallel Commits cuts commit latency from two rounds of consensus down to one.

For the hardest problem in distributed systems — which event came first — CockroachDB uses HLC (Hybrid Logical Clocks): a timestamp combining a physical component (always close to local wall time) and a logical counter (to distinguish events sharing a physical component). Where Google Spanner depends on TrueTime backed by atomic clocks and GPS, CockroachDB relies on semi-synchronized clocks with bounded uncertainty. The default maximum clock offset is 500ms (250ms is recommended for multi-region clusters), and when a node detects that its clock is out of sync with at least half the other nodes by 80% of the maximum offset, it crashes immediately. That behavior is the CP stance made concrete: sacrifice availability to preserve consistency.

DynamoDB — disk-backed, multi-AZ replication

Data sits on SSD and is replicated across multiple Availability Zones. The single most important fact here is that eventual consistency is the default.

  • Default reads are eventually consistent and may not reflect a just-completed write.
  • Setting ConsistentRead: true gives a strongly consistent read, but it costs twice as much as an eventually consistent one.
  • Strongly consistent reads are available only on tables and LSIs. Reads from GSIs and streams are always eventually consistent.
  • Global tables default to MREC (multi-Region eventual consistency, typically propagating within a second). MRSC (multi-Region strong consistency) with synchronous replication is also available.

So DynamoDB lets you buy consistency strength per read, in exchange for cost. The constraint that GSIs have no strong consistency will absolutely shape your design.

7.3 The selection table

AspectUpstash RedisCockroachDBDynamoDB
Data placementIn-memoryDisk (distributed)SSD (multi-AZ)
Consistency modelEffectively atomic in a single region; replicas are eventualSerializableEventual by default; strong selectable per read
ConsensusNone (single primary)Raft (majority quorum)Internal replication
Clock dependenceNoneHLC (default max-offset 500ms)None
Dominant latency sourceNetwork RTT onlyConsensus roundStorage plus replication
Reachable from the edgeHTTP (ideal)Needs a driver (TCP)Needs an SDK / HTTP API
QueriesKV and data structuresSQL, joins, transactionsKV, limited queries
Role in this systemRate limits, sessions, cacheBilling, credit balances, auditConversation logs, high-volume appends

Using them together is the right answer. "One database for everything" simplifies operations but guarantees you are over- or under-engineered somewhere. Given data gravity — data is heavy and hard to move — classifying data by its properties first and assigning each an appropriate store is what saves you the migration cost later.


8. Pre-production checklist

  • Is there an L4 ceiling? A budget or usage cap is set on the provider side (AI Gateway or similar). Even if all your code fails, spending stops.
  • No @vercel/kv. Environment variables come from the Upstash Marketplace integration.
  • proxy.ts, not middleware.ts (Next.js 16). The export is export function proxy.
  • The Proxy matcher is narrowed to the protected surface (no billed executions on static assets).
  • Rate limit keys are two independent limiters ANDed together, not an IP-plus-session concatenation.
  • IPv6 is rounded to /64.
  • Turnstile is verified server-side via siteverify, and action and hostname are checked in addition to success.
  • The Turnstile expired-callback (300 seconds) is implemented, and reset() runs regardless of send success.
  • Rate limiting fails open; bot verification fails closed. You can explain why that asymmetry is deliberate.
  • 429 responses carry Retry-After, and the UI shows the wait.
  • Structured logs are emitted for rejections and failures (no PII, no secrets).
  • Vercel WAF rate limit rules ran in Log mode for several days before switching to Deny.
  • You alert on spend itself. Error rate and latency do not move during this attack.

That last item deserves emphasis. The only reliable detector for this attack is the money. Chart request counts, token consumption, and estimated cost over time, and alert on a threshold such as three times baseline.


9. Summary

Protecting a login-free generative AI chat is not about controlling who can use it. It is about placing a ceiling on how much can be used, in several independent places.

  1. The threat is the invoice. Availability metrics stay green while damage accrues. Monitor money.
  2. Update your parts list for 2026. Vercel KV is gone, middleware.ts is now proxy.ts (Node.js by default), blanket defense belongs to the WAF.
  3. Split into four layers. WAF (no function invoked), Proxy (coarse net), Route Handler (fine net plus bot verification), provider budget (last resort).
  4. Never concatenate a composite key. Keep the coarse and fine nets independent and AND them.
  5. Choose consistency per kind of data. Eventual is correct for rate limits; strong is required for billing. The deciding question is what one leak costs.
  6. Make your failure directions asymmetric. Mechanisms that exist for availability (rate limiting) fail open; mechanisms that exist for safety (bot verification) fail closed.

Most important of all: recognize that every one of these defenses is conditional on your code running correctly. Bad deploys, vanished environment variables, dependency outages — all of them happen. So put a ceiling outside your code, on the provider side, that your code cannot raise. That is the last design decision standing between you and waking up to an invoice instead of an alert.

Frequently asked questions

Doesn't requiring login solve this?
It mitigates it; it does not solve it. If sign-up is free, attackers mass-produce accounts and do the same thing (credential stuffing and disposable email addresses work the same way). Authentication improves the quality of your identifier — it is not a cap on consumption. With or without login you still need per-identifier limits, a provider-side budget ceiling, and anomaly detection. The real benefit of login is that you can rate limit on a stable identifier (a user ID) instead of an anonymous IP.
Is Vercel KV really gone?
Yes. Vercel KV was discontinued as a product, and existing stores were automatically migrated to Upstash Redis in December 2024. Today you install Upstash for Redis (or another store) from the Vercel Marketplace, and credentials are injected into your project's environment variables automatically. In code, the standard as of August 2026 is @upstash/redis and @upstash/ratelimit — not @vercel/kv.
Should rate limiting live in middleware.ts or in the Route Handler?
If you are protecting a single expensive endpoint (/api/chat), the Route Handler is the more natural home. Next.js 16 deprecated middleware.ts (renamed to proxy.ts) and the docs explicitly recommend using the feature as a last resort. Proxy earns its place when you need one entry policy across many routes, or when you want to reject a request before reading its body to shave function execution time. Either way, the proxy or function invocation itself is still billed — which is why volumetric floods belong in the WAF.
Cloudflare Turnstile or Vercel BotID?
Choose BotID if you want everything inside Vercel; choose Turnstile if you already run Cloudflare or want to own the token verification logic yourself. BotID Basic is free on all plans, and Deep Analysis (powered by Kasada) costs $1 per 1,000 checkBotId() calls on Pro. Turnstile has a generous free tier and, because you call siteverify yourself, gives you full control over verification (matching action and hostname, idempotency keys). Both share the same caveat: bot detection is probabilistic and can be defeated, so neither is a reason to drop rate limiting.
With eventually consistent rate limiting, won't requests exceed the limit?
They will — and usually that is the correct design. A single-region Upstash Redis setup is effectively atomic, but multi-region read replicas and Vercel WAF rate limiting (whose counters are tracked per region) can let the global total exceed your configured limit. The deciding question is what one leaked request costs you. If it is one inference call (cents), accept it. Reserve strongly consistent stores for the things where a single leak becomes a real financial loss: billing, balances, inventory.

References

友田

友田 陽大

Developer of a METI Minister's Award–winning product. With TypeScript + Python + AWS, I deliver SaaS, industry DX, and production-grade generative AI (RAG) end to end — from requirements to infrastructure and operations — single-handedly.

I can take on the implementation from this article as an engagement

Abuse and runaway-cost defense for generative AI endpoints — design, implementation, and monitoring

I implement the layered defense that lets you ship a login-free AI chat or a public LLM feature without breaking your invoice: volumetric control at the WAF, two-tier rate limiting that keeps the network and session identifiers independent, bot exclusion via Turnstile or BotID, a hard budget ceiling on the provider side, and monitoring that watches spend rather than error rate. Having run an unattended, login-free generative AI voice concierge in production — where you cannot control who walks up and talks to it — I design the consumption boundary first, so the thing stays up, stays cheap, and stays safe.

Available for both project-based (contract) and advisory engagements. Start with a free 30-minute consult.

最短ルート:カレンダーから直接予約

相談内容が固まっている方は、フォーム送信よりその場で日程を確定する方がスムーズです。下記から空き時間をお選びください。

  • 30分のオンライン無料相談
  • Google Meet / Zoom / Microsoft Teams
  • NDA 商談前締結可・無理な営業はいたしません
無料相談の空き枠を予約する

Also worth reading