Qwen3-8B-AWQ practical guide: self-hosting a 'reasoning LLM' on a single GPU with 4-bit quantization

The goal of this article

In the big picture of putting an open-weight LLM into production, I wrote "for self-hosting, vLLM" and "what decides the cost is quantization and utilization." This piece is the definitive concrete example of that. The subject is Qwen3-8B-AWQ — an Apache-2.0 open weight where Alibaba's Qwen team's 8B model is quantized to 4-bit with AWQ.

What's interesting about this model comes down to the single point that it's "small and cheap, yet 'thinks.'" Since AWQ compresses the weights to about 1/3, it fits on a single 24GB GPU, and moreover you can switch a reasoning mode and a fast-dialogue mode in one model. That is, "o1-class staged reasoning, on your own server, cheaply" becomes a reality.

The goal isn't to be able to type vllm serve, but to reach a state where, staying faithful to the official documentation, you understand in which scene to use what, and what crumbles the quality if you remove it. All code is shown in a form that runs on your machine.

Disclosure of credibility: the difficulty of running an LLM in production on a GPU (VRAM, throughput, resilience) is an area I actually stepped on at the video-AI-localization platform. This article's specs, commands, and sampling values are all based on official primary sources — the Qwen3-8B-AWQ model card, the Qwen official documentation, and the vLLM documentation. I clearly state that the actual numbers of VRAM and throughput are environment-dependent and need benchmarking.

The 30-second conclusion: when to choose Qwen3-8B-AWQ

The essence is this one line: Qwen3-8B-AWQ is an option for "running a small model that can think, without sending data outside, cheaply on a single GPU."

• ✅ Suited: inference on-prem / inside a VPC, projects where sensitive data can't go outside, optimizing the cost of the "reasoning role" of RAG or an agent, running verification to small-scale production on 1 GPU.
• ❌ Not suited: you need the absolute highest quality (go to a bigger model) / want to start with zero operation (start with an API first). For the overall view of the judgment axes, align with the break-even of API vs. self-host.

What Qwen3-8B-AWQ is (the official specs accurately)

First, let me lay out the official facts without inflating them (source: the model card).

The official states that Qwen3 surpasses the past QwQ and Qwen2.5 family in reasoning ability, and that the alignment with human preferences in creative writing, role-play, and instruction-following also improved (for specific scores, see the technical report). This piece concentrates not on claiming bench numbers but on "running it in production exactly per the official spec."

🔎 Beware of derivatives: Hugging Face lines up Qwen/Qwen3-8B (full precision), Qwen/Qwen3-8B-FP8, Qwen/Qwen3-8B-Base, the community MLX version, etc. What this article handles is the official Qwen/Qwen3-8B-AWQ (4-bit). GQA is Q32/KV8 — a design with few KV heads has the implication of a light KV cache and being easy to load concurrency onto = self-host-friendly.

What AWQ (4-bit quantization) is: why it fits on "a single GPU"

AWQ = Activation-aware Weight Quantization. The official explains it as "hardware-friendly low-bit weight quantization." What's important is activation-aware — since it quantizes so as not to break the weights that matter to the output more, it suppresses quality degradation even at 4-bit. The AutoAWQ implementation shows about 3x memory reduction and about 3x speedup vs. FP16 (the AWQ documentation).

A rough VRAM estimate makes "why it fits on one" click.

FP16 weights  ≈ 8.2B × 2 byte ≈ 16.4 GB
AWQ 4bit      ≈ 8.2B × 0.5 byte ≈ 4.1 GB (+ scale/zero-point, ~6GB measured)

In other words, the weights are roughly around 6GB. You can turn the remaining VRAM toward the KV cache (which decides concurrency and context length). With a single 24GB GPU, it's the picture of weights 6GB + plenty of KV room (always fix the actual concurrency and throughput with a load test).

💡 Use distinction: if you have an FP8-supported GPU like H100, FP8 is also strong (see the episode of serving Llama in FP8). On the other hand, for "just load it and run it cheaply" on a 24GB GPU like L4 / A10 / RTX 4090, AWQ 4-bit is the royal road. The choice among AWQ, GPTQ, FP8, and GGUF is detailed in how to choose a quantization method.

The killer feature: hybrid thinking (thinking / non-thinking)

Qwen3's biggest differentiator is being able to switch "think" and "answer immediately" in one model.

• Thinking mode: reasons in stages inside <think> … </think> before answering. Strong at math, code, and complex judgment.
• Non-thinking mode: answers immediately without emitting a reasoning block. For uses where speed and cost matter, like classification, extraction, and routine dialogue.

Switching is in two ways (official spec)

1. Hard switch: apply_chat_template(..., enable_thinking=True/False) (in the API, chat_template_kwargs).
2. Soft switch: in the enable_thinking=True state, writing /think / /no_think in the user utterance overrides it per turn.

The recommended sampling per mode (it breaks if you remove it)

Note that the official specifies different sampling per mode.

⚠️ The official's clear warning: in thinking mode, you must not use greedy decoding (equivalent to temperature=0). It invites infinite loops, repetition, and performance degradation. "I want reasoning output deterministically, so temp=0" is counterproductive.

There's also an official guideline for output length: most queries are 32,768 tokens, and complex problems like math or programming should be expected up to 38,912 tokens. Since thinking mode "thinks" for a long time before the answer, if you skimp on max_tokens, it cuts off before reaching the conclusion.

Running it ①: minimal confirmation with transformers (for development)

First, minimal confirmation locally. Running AWQ weights needs the awq kernel, so install autoawq (for production, vLLM described later is recommended).

# pip install -U transformers accelerate autoawq
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-8B-AWQ"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "9.11 と 9.9 はどちらが大きい？理由も。"}]

# enable_thinking=True で <think>...</think> による段階推論を有効化（ハードスイッチ）
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

# 思考モードの公式推奨サンプリング（greedy は使わない）
generated = model.generate(
    **inputs, max_new_tokens=32768,
    temperature=0.6, top_p=0.95, top_k=20,
)
output_ids = generated[0][len(inputs.input_ids[0]):].tolist()
print(tokenizer.decode(output_ids, skip_special_tokens=True))

For a single-machine try in development, this is enough. But transformers's generate doesn't have continuous batching, so it's not suited for many simultaneous requests in production. So I move to vLLM.

Running it ②: production serving with vLLM (OpenAI-compatible)

Production is vLLM. With continuous batching and PagedAttention, it doesn't idle the GPU, and it exposes an OpenAI-compatible endpoint, so you can swap only the destination without changing the app-side code (the whole of building the foundation is consolidated in the vLLM production-self-host operation log).

# Qwen3-8B-AWQ を OpenAI 互換でサーブ。思考の分離パースを有効化
vllm serve Qwen/Qwen3-8B-AWQ \
  --reasoning-parser qwen3 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --port 8000

• --reasoning-parser qwen3: the specification for vLLM 0.9.0 and later. It separates the <think> block from the body and adds a reasoning_content field to the response. On older vLLM, use --enable-reasoning --reasoning-parser deepseek_r1.
• --max-model-len 32768: the native upper limit. Narrowing to the length you actually need is the iron rule of cost and latency (don't make it unlimited).
• --gpu-memory-utilization 0.90: the reservation ratio. Too high = OOM, too low = concurrency decreases.

Once it's up, you can hit it with an OpenAI client as-is. Control thinking on/off with extra_body.chat_template_kwargs.enable_thinking.

from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

resp = client.chat.completions.create(
    model="Qwen/Qwen3-8B-AWQ",
    messages=[{"role": "user", "content": "在庫が負になりうる箇所をこのSQLから指摘して"}],
    temperature=0.6, top_p=0.95, extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},  # 思考モード
        # 量子化モデルで“終わらない繰り返し”が出たら presence_penalty を 0〜2 で（推奨 1.5）
        "presence_penalty": 1.5,
    },
)
msg = resp.choices[0].message
print("思考:", getattr(msg, "reasoning_content", None))  # <think> の中身（ログ/監査用）
print("回答:", msg.content)                               # ユーザーに返す最終回答

reasoning_content (the thinking process) and content (the final answer) being structurally separated pays off in production. Return only content to the user, and handle reasoning_content internally for audit/evaluation — this separation becomes the foundation of the type safety and observability described later.

Long context: 32K → 131K with YaRN (only when needed)

The native is 32,768. Only when you need more context, extend with YaRN. You can also declare it on the CLI.

vllm serve Qwen/Qwen3-8B-AWQ \
  --reasoning-parser qwen3 \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' \
  --max-model-len 131072

The form of writing the equivalent in config.json (factor:4.0, original_max_position_embeddings:32768) is also shown officially.

⚠️ Don't enable YaRN all the time. The official clearly states "only when needed." Applying long-context scaling even to short inputs has the side effect of lowering the quality of short text. If 32K is enough, remove YaRN — "max just in case" is counterproductive. The longer the context, the more the KV cache swells and VRAM and latency increase, so set max-model-len to match actual demand.

Tool calling / agents

Qwen3 supports function calling (tool use). In vLLM, enable it with a flag.

vllm serve Qwen/Qwen3-8B-AWQ \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

After that, if you pass OpenAI-compatible tools, the model returns a function call as needed. If you build agent-like tool integration, using the official Qwen-Agent (which has templates for tool definition and parsing) saves you from writing the tool-call handling yourself. The implementation of a safe loop including Zod validation of arguments, an iteration upper limit, and idempotent side effects is concretized in making Qwen3 an agent. For the crux of the design — "separate the judgment you leave to the LLM and the execution you lean toward deterministic code" — align with the design of tool use / function calling.

In what scenes to use it: a type-safe production client (application)

From here is the application. Qwen3-8B-AWQ's strengths are "can think," "cheap," "self-hosted." Wrapping this in type safety, idempotency, and resilience makes it a component that earns in production. A representative application is a self-hosted RAG that doesn't send sensitive data outside.

Design policy (as in the principles of CLAUDE.md, make SRP / KISS / type-safe boundaries effective):

1. Route the mode by difficulty — easy tasks to non-thinking (fast, cheap), only the hard parts to thinking (the main thrust of cost optimization).
2. Separate the thinking process and the answer — reasoning_content to logs, validate and return only content.
3. Validate structured output with Zod at the boundary — LLM output is external input. Reject the invalid with parse (the discipline of type safety). For a two-stage setup with guided decoding that doesn't let invalid JSON be generated at the generation stage, go to type-safe structured output.
4. Timeout + idempotent cache — don't generate the same input twice = the cost is also stable.

// lib/qwen-client.ts — 思考/非思考をルーティングし、出力を境界で型検証する薄いクライアント
import OpenAI from "openai";
import { z } from "zod";
import { createHash } from "node:crypto";

const client = new OpenAI({
  baseURL: process.env.QWEN_BASE_URL, // 例: http://qwen-internal:8000/v1（privateで公開しない）
  apiKey: "internal",
  timeout: 60_000, // 思考モードは長い。短すぎる timeout は“正常な熟考”を切る
});

/** タスク難度 → モードと公式推奨サンプリングを決める（SSoT） */
const MODE = {
  fast: { enable_thinking: false, temperature: 0.7, top_p: 0.8, top_k: 20 },
  think: { enable_thinking: true, temperature: 0.6, top_p: 0.95, top_k: 20 },
} as const;
type Difficulty = keyof typeof MODE;

/** 期待する構造化出力。LLMの“それっぽい文字列”を信用せず、ここで弾く */
const RiskFinding = z.object({
  hasRisk: z.boolean(),
  severity: z.enum(["low", "medium", "high"]),
  reason: z.string().min(1),
});
type RiskFinding = z.infer<typeof RiskFinding>;

const cache = new Map<string, RiskFinding>(); // 実運用は Redis 等に置換

/** 入力が同じなら生成も同じ（冪等）。連打・リトライ・重複依頼でコストを無駄にしない */
const keyOf = (prompt: string, d: Difficulty) =>
  createHash("sha256").update(`qwen3-8b-awq:${d}:${prompt}`).digest("hex");

export async function assessRisk(prompt: string, difficulty: Difficulty = "think"): Promise<RiskFinding> {
  const key = keyOf(prompt, difficulty);
  const hit = cache.get(key);
  if (hit) return hit;

  const m = MODE[difficulty];
  const resp = await client.chat.completions.create({
    model: "Qwen/Qwen3-8B-AWQ",
    messages: [
      { role: "system", content: 'JSONのみで返答: {"hasRisk":bool,"severity":"low|medium|high","reason":string}' },
      { role: "user", content: prompt },
    ],
    temperature: m.temperature,
    top_p: m.top_p,
    response_format: { type: "json_object" },
    presence_penalty: 1.5, // 量子化モデルの繰り返し対策（公式推奨レンジ 0〜2・OpenAI互換）
    // vLLM拡張はトップレベルで送る。Node SDKは未知キーも本文へ転送し、spreadなので型エラーも出ない。
    // ※ Python SDK の extra_body は TS SDK には存在しないので使わない。
    ...{ top_k: m.top_k, chat_template_kwargs: { enable_thinking: m.enable_thinking } },
  });

  // content だけを検証対象に。reasoning_content（思考過程）は監査ログへ（PIIは載せない）
  const raw = resp.choices[0]?.message.content ?? "{}";
  const finding = RiskFinding.parse(JSON.parse(raw)); // 不正な形なら throw → 上位で握る

  cache.set(key, finding);
  return finding;
}

The crux of this client is that it never takes the output of a "reasoning LLM" at face value. Keep reasoning_content for human audit and evaluation, and what you return to the user is only the type that passed through Zod. Furthermore, by taking enable_thinking in and out by task difficulty, the cost design of the easy 80% cheaply in non-thinking, only the hard 20% accurately in thinking holds in one model.

Production build-out (avoiding duplication, just the key points)

Observability, auto-scaling, graceful drain, resilience, and network isolation are the same practice whether the model is Qwen or Llama. Without duplicating the details, I leave them to the vLLM production-self-host operation log. Only the points that especially matter for Qwen3-8B-AWQ:

• Observability: monitor num_requests_waiting / gpu_cache_usage_perc / TTFT with vLLM's /metrics (Prometheus). Since thinking mode tends to produce long output, visualizing reasoning's token consumption per use gives a sense of the cost (the correlation design of OpenTelemetry).
• Resilience: don't let a self-node failure become a total failure. Timeout + retry + fallback. If the 8B goes down, escape to another node / a higher model (retry, circuit breaker).
• Security: vLLM's OpenAI-compatible server has no strong authentication itself. Don't expose it directly to the internet. Place it in a private VPC, and authenticate, rate-limit, and audit at the front API Gateway. Don't keep the prompt body (PII) in logs — after all, "not sending it outside" is the motive of self-hosting.

Gotchas & official best practices

From the official documentation, let me extract what crumbles the quality if you remove it.

• 🔴 Greedy decoding forbidden in thinking mode. Don't set temp=0. It causes repetition and infinite loops.
• 🟠 presence_penalty ≈ 1.5 against repetition in quantized models (range 0–2). But raising it too much rarely causes language mixing or a slight performance drop, so apply it to the degree that it works when it appears.
• 🟠 Don't keep thinking content (<think>) in the history in multi-turn. Stack only the final answer into the next turn's input. Continuing to stack thinking pollutes the context and worsens both quality and cost.
• 🟠 Don't narrow the output length too much. Thinking is long before the answer. Secure max_tokens with a guideline of 32,768 normally / 38,912 tokens for complex problems.
• 🟢 For math, specify the output format: "Please reason step by step, and put your final answer within \boxed{}." makes the final solution easier to machine-extract.
• 🟢 YaRN only when needed. Enabling it all the time lowers the quality of short text.
• 🟢 AWQ depends on the awq kernel with transformers. Lean production throughput toward vLLM (continuous batching).

Summary

Qwen3-8B-AWQ straightforwardly satisfies the requirement that was previously hard to reconcile: "running a small model that can think, without sending data outside, cheaply on a single GPU."

1. With AWQ 4-bit, weights ≈6GB → fits on a single 24GB GPU (Apache-2.0, commercial OK).
2. Route hybrid thinking by difficulty — easy cheaply in non-thinking, hard accurately in thinking.
3. OpenAI-compatible serving with vLLM (--reasoning-parser qwen3), separate thinking and answer with reasoning_content.
4. Strictly observe the official sampling (thinking: 0.6/0.95/20, greedy forbidden, presence_penalty≈1.5, don't keep thinking in the history).
5. Type-validate output at the boundary (Zod) + idempotent cache. The production practice is consolidated in the vLLM operation log.

I build a "reasoning LLM" on-prem / inside a VPC, including observability, resilience, and type safety. See my GPU-production track record and consult me on model selection, cost design, and self-host migration. With one person × generative AI, fast, cheap, and safe.

Sources / official resources

• Qwen/Qwen3-8B-AWQ — Hugging Face model card — specs, mode switching, recommended sampling, best practices
• Qwen official documentation (vLLM deployment) — reasoning-parser, tool-call, YaRN flags
• Qwen official documentation (AWQ quantization) — how AWQ works and how to use it
• vLLM official documentation — server, metrics, quantization
• QwenLM/Qwen3 (GitHub) — reference implementation
• Qwen3 Technical Report (arXiv:2505.09388)

※ Specs, flags, and sampling recommendations get updated. VRAM and throughput are environment-dependent and need benchmarking. Before implementing, always confirm with the primary sources and your own bench.