Skip to main content
NVIDIA, GPUs & AI chips
AI
インフラ
技術選定
コスト最適化
パフォーマンス
Python

CPU vs GPU vs TPU: how they differ and how to choose — systolic arrays, memory bandwidth and cost per 1M tokens

What a TPU is, how it differs from CPUs and GPUs, why the TPU v1 "15-30x" figure is outdated, and a cost-per-token model that flips with batch size and SLOs.

Published
Reading time
42 min read
Author
友田 陽大
Share

"What is a TPU?" "Is it faster than a GPU?" "Which one is cheapest for inference?" These questions come up whenever a team picks AI infrastructure.

The short answer first. A TPU (Tensor Processing Unit) is an ASIC (application-specific integrated circuit) that Google designed for machine learning. CPUs, GPUs and TPUs can all multiply matrices. What differs is how each one splits its budget between flexibility and efficiency. A CPU can do anything but multiplies matrices relatively inefficiently; a TPU specializes in matrix multiplication and has clear weak spots in return. A GPU sits in between: it speeds up matrix multiplication while staying general-purpose.

And which one is cheapest is not decided by chip performance alone. Inference cost comes down to "cost per 1M tokens = hourly price ÷ (measured tokens per second × utilization)". Change the batch size, the demand, or the latency target (SLO, service level objective — in this article, a cap such as the time between generated tokens), and which of two machines is cheaper can flip.

This article first walks through how each processor works. For the TPU's core, the systolic array, we trace the data flow with a small example and code. It then compares numeric precision, memory, interconnects, software, benchmarks and power, and closes with a model for cost per 1M tokens (Cost/Token) and a step-by-step selection procedure.

About the numbers in this article. Performance figures are either the theoretical peaks (published specs) that each vendor lists on its official pages, or my own calculations from them (marked "theoretical"). None are my own measurements. The machine figures used in the Cost/Token calculation are assumptions made for illustration, not real products or prices. For context, my own production work is at the scale of an inference service running vLLM and AWQ-quantized Qwen3-8B on a single Tesla T4; I have not run TPUs in production.

If you only want the selection verdict, read the procedure and tables in section 15 first.

1. Why compare three kinds of processor?

Most of the arithmetic in AI is multiplying and adding large matrices. I covered this in detail, including the cost of a matrix multiply (2MNK2MNK), in Why NVIDIA GPUs are so strong at AI.

So why are there three kinds of processor for the same matrix multiply? Because they make different choices about where to spend chip area and power.

A kitchen analogy:

  • A CPU is a kitchen with a handful of master chefs who can cook anything. Complex orders and mid-course changes are no problem. Making the same dish ten thousand times is not their strength.
  • A GPU is a kitchen full of line cooks who all follow the same recipe at once. When everyone does the same steps, they are overwhelmingly fast — and you can still rewrite the recipe.
  • A TPU is a dedicated processing line where ingredients arrive on a conveyor belt. It handles its fixed process more efficiently than anyone, but it cannot make a dish the line wasn't built for.

The TPU v1 paper (Jouppi et al., ISCA 2017) states the difference plainly. It lists the mechanisms CPUs and GPUs carry to raise average throughput — caches, branch prediction, out-of-order execution, multithreading, speculative prefetching — and says the TPU has none of them, which is why it could be a small, low-power chip. Control logic took just 2% of the TPU v1 die.

Do you actually know whether self-hosting beats the API?

Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing

2. The common ground: matrix multiplication and the memory wall

The shared yardstick for comparing the three is arithmetic intensity: the number of operations performed per byte read from memory. Speed is capped by the smaller of compute throughput and memory bandwidth (the Roofline model), and compute ÷ bandwidth is called the ridge point. Work with lower intensity is limited by bandwidth, work with higher intensity by compute. The formulas are in section 6 of the series pillar.

The TPU v1 paper compares the three processors with exactly this model. In the paper's figures, the ridge points per byte of weights are 13 for Haswell (CPU), 9 for the K80 (GPU, one die) and 1,350 for TPU v1. TPU v1 had a lot of arithmetic but slow memory (DDR3 at 34 GB/s), which pushed its ridge point far to the right. Indeed, according to the paper, four of its six applications were limited by memory bandwidth on the TPU.

"Fast arithmetic, memory can't keep up" is not just a 2015 TPU v1 problem. Today's TPUs and GPUs both carry HBM (High Bandwidth Memory: memory dies stacked right next to the processor and connected by a very wide bus), an expensive kind of memory — and many inference workloads are still bandwidth-bound.

3. CPU: the general-purpose processor that does everything

A CPU spends its die on large caches, branch prediction with speculative execution, and out-of-order execution, which speed up work whose steps depend on the data (branchy programs, preprocessing, tokenization). These do little for regular computation like matrix multiplication, but modern CPUs do have matrix features: SIMD instructions, and Intel's AMX, which the documentation of Intel's oneDNN library lists as an instruction set handling INT8, BF16 and more. Pipelining, branch prediction and out-of-order execution are covered in depth in sections 4-5 of How CPUs and GPUs differ in architecture.

Google's Cloud TPU documentation lists cases where CPUs fit: quick prototyping that needs maximum flexibility, simple models that don't take long to train, small models with small effective batch sizes, and models limited by the host system's I/O or network bandwidth.

4. GPU: thousands of threads at once with SIMT

GPU threads are grouped 32 at a time into a warp, and a warp runs the same instruction in the SIMT (Single-Instruction, Multiple-Thread) model (NVIDIA's CUDA Programming Guide). The programmer writes code for a single thread; the performance catch is that when threads in a warp branch differently, the warp runs both paths in turn (warp divergence). Section 10 of How CPUs and GPUs differ in architecture explains this in detail. Data-center GPUs add dedicated matrix units (NVIDIA's Tensor Cores) and HBM (section 2 of the pillar).

The GPU's strength is that it multiplies matrices fast while remaining general-purpose. You can write custom kernels (programs that run on the GPU), so new operations and models that mix in non-matrix work are fine. Google Cloud's documentation itself lists models with many custom PyTorch/JAX operations, and medium-to-large models with larger effective batch sizes, as good fits for GPUs.

5. TPU: turning matrix multiplication into an assembly line with a systolic array

A grid of connected multiply-accumulate units

A TPU is an ASIC that Google designed for machine learning — an application-specific integrated circuit, a chip whose circuits are built for a fixed kind of computation rather than general-purpose work. Its heart is the MXU (Matrix Multiply Unit). According to Google Cloud's documentation, an MXU is a systolic array — multiply-accumulate units (each does one multiply and one add) laid out in a grid — with 256×256 units on TPU v6e and TPU7x and 128×128 on earlier generations.

"Systolic" comes from the heart's contraction (systole): data pulses through the grid at a steady rhythm. As the documentation describes it, the TPU first loads the parameters (weights) from HBM into the MXU, then loads the data. Each time a multiplication runs, its result is passed to the next multiply-accumulator. No memory access is needed during the matrix multiply.

Google Cloud's documentation presents this as the difference from CPUs and GPUs, saying that "for every calculation in the thousands of ALUs, a GPU must access registers or shared memory." That holds for CPU SIMD units and GPU CUDA cores. But a GPU's Tensor Cores also chain multiply-accumulates internally within a small matrix tile; they don't write back to registers after every operation. The real difference is the size of the reuse unit. An MXU keeps weights fixed across a whole 128×128 grid (256×256 on v6e and TPU7x) and passes partial sums from one edge of the grid to the other, which sharply cuts register and SRAM reads and writes and the power they consume. The TPU v1 paper says the same: because reading a large SRAM costs much more power than arithmetic, the matrix unit uses systolic execution to save energy by reducing reads and writes.

Tracing the data flow in a small example

Following the TPU v1 paper's description (data flows in from the left, weights are preloaded from the top, and each multiply-accumulate moves through the grid as a diagonal wavefront), let's trace a weight-stationary systolic array with a small 2×2 example.

We compute Y=XWY = XW:

X=(123456),W=(1101001000)X = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix},\quad W = \begin{pmatrix} 1 & 10 \\ 100 & 1000 \end{pmatrix}

We feed each row of XX (three inputs) through a 2×2 grid. There are only three rules:

  1. Cell (k,n)(k, n) holds weight WknW_{kn}, placed there up front (it never moves).
  2. Every cycle, each cell takes "input xx from the left" and "partial sum pp from above", passes p+x×Wknp + x \times W_{kn} down and xx to the right.
  3. Column kk of the input enters the left edge kk cycles late (to line up the timing).

The values leaving the bottom row are finished elements of YY. The Python below executes these rules literally, one cycle at a time.

  • Language / dependencies: Python 3.9+, NumPy 1.24+
  • Runs on: any CPU (I ran it on an Apple M4 Pro with Python 3.9 and NumPy 2.0.2)
  • Run: python systolic.py
# systolic.py — simulate a weight-stationary systolic array one cycle at a time
# Python 3.9+ / NumPy 1.24+. Run: python systolic.py
import numpy as np


def systolic_matmul(X, W, trace=False):
    """Compute Y = X @ W on a K×N systolic array and return (Y, number of cycles).

    X: (B, K) inputs (flowing in from the left)
    W: (K, N) weights (one preloaded into each cell)
    Every cycle, cell (k, n) takes "x from the left" and "the partial sum from above",
    passes partial sum + x * W[k, n] down and x to the right.
    """
    B, K = X.shape
    K2, N = W.shape
    assert K == K2
    a_reg = [[None] * N for _ in range(K)]  # input value each cell passes to the right
    p_reg = [[None] * N for _ in range(K)]  # partial sum each cell passes down
    Y = np.zeros((B, N))
    cycles = B + K + N - 2                  # total cycles from the first input to the last output
    for t in range(cycles):
        new_a = [[None] * N for _ in range(K)]
        new_p = [[None] * N for _ in range(K)]
        for k in range(K):
            for n in range(N):
                if n == 0:                  # left edge: row k gets x[b, k] delayed by k cycles
                    b = t - k
                    a_in = X[b, k] if 0 <= b < B else None
                else:                       # otherwise: what the left neighbor held last cycle
                    a_in = a_reg[k][n - 1]
                if a_in is None:
                    continue
                p_in = 0.0 if k == 0 else p_reg[k - 1][n]
                new_a[k][n] = a_in
                new_p[k][n] = p_in + a_in * W[k, n]
                if trace:
                    b = t - k - n
                    print(f"  t={t} cell({k},{n}): x[{b},{k}]={a_in:g} * w={W[k, n]:g}"
                          f" + {p_in:g} -> {new_p[k][n]:g}")
        for n in range(N):                  # a partial sum leaving the bottom row is a finished output
            if new_p[K - 1][n] is not None:
                b = t - (K - 1) - n
                Y[b, n] = new_p[K - 1][n]
        a_reg, p_reg = new_a, new_p
    return Y, cycles


if __name__ == "__main__":
    # 1) Small example: feed 3 inputs through 2x2 weights
    X = np.array([[1, 2], [3, 4], [5, 6]], dtype=float)
    W = np.array([[1, 10], [100, 1000]], dtype=float)
    Y, cycles = systolic_matmul(X, W, trace=True)
    print("Y =", Y.tolist(), " cycles =", cycles)
    print("X @ W =", (X @ W).tolist())

    # 2) Check agreement with NumPy on larger random matrices
    rng = np.random.default_rng(0)
    for B, K, N in [(1, 16, 16), (16, 16, 16), (256, 16, 16)]:
        X = rng.standard_normal((B, K))
        W = rng.standard_normal((K, N))
        Y, cycles = systolic_matmul(X, W)
        err = np.abs(Y - np.einsum("bk,kn->bn", X, W)).max()  # difference from a NumPy reference
        util = B * K * N / (K * N * cycles)  # useful MACs ÷ (cells × cycles)
        print(f"B={B:3d} K={K} N={N}: cycles={cycles:3d} max_err={err:.1e} utilization={util:.1%}")

Output (copied verbatim from my run):

  t=0 cell(0,0): x[0,0]=1 * w=1 + 0 -> 1
  t=1 cell(0,0): x[1,0]=3 * w=1 + 0 -> 3
  t=1 cell(0,1): x[0,0]=1 * w=10 + 0 -> 10
  t=1 cell(1,0): x[0,1]=2 * w=100 + 1 -> 201
  t=2 cell(0,0): x[2,0]=5 * w=1 + 0 -> 5
  t=2 cell(0,1): x[1,0]=3 * w=10 + 0 -> 30
  t=2 cell(1,0): x[1,1]=4 * w=100 + 3 -> 403
  t=2 cell(1,1): x[0,1]=2 * w=1000 + 10 -> 2010
  t=3 cell(0,1): x[2,0]=5 * w=10 + 0 -> 50
  t=3 cell(1,0): x[2,1]=6 * w=100 + 5 -> 605
  t=3 cell(1,1): x[1,1]=4 * w=1000 + 30 -> 4030
  t=4 cell(1,1): x[2,1]=6 * w=1000 + 50 -> 6050
Y = [[201.0, 2010.0], [403.0, 4030.0], [605.0, 6050.0]]  cycles = 5
X @ W = [[201.0, 2010.0], [403.0, 4030.0], [605.0, 6050.0]]
B=  1 K=16 N=16: cycles= 31 max_err=1.1e-15 utilization=3.2%
B= 16 K=16 N=16: cycles= 46 max_err=1.8e-15 utilization=34.8%
B=256 K=16 N=16: cycles=286 max_err=3.6e-15 utilization=89.5%

Reading the trace shows the mechanism:

  • At t=1, cell (1,0) receives the partial sum "1" (1×11 \times 1) that the cell above, (0,0), computed at t=0. It multiplies its own input x0,1=2x_{0,1} = 2 by the weight 100, adds, and outputs 1+200=2011 + 200 = 201. That is Y0,0Y_{0,0}.
  • At t=2, all four cells work at once. The three inputs pass through the grid shifted diagonally — the paper's "diagonal wavefront".
  • No cell writes an intermediate result back to memory. Partial sums go down and inputs go right, handed directly to the neighbor.

With small batches, the array sits idle

The second half of the output shows something else that matters. Feeding BB inputs through a K×NK \times N array takes B+K+N−2B + K + N - 2 cycles from the first input entering to the last output leaving. The useful multiply-accumulates in that time number BKNBKN, so the array's utilization is

utilization=BKNKN (B+K+N−2)=BB+K+N−2\text{utilization} = \frac{BKN}{KN\,(B + K + N - 2)} = \frac{B}{B + K + N - 2}

On a 16×16 array, one input gives 3.2%, 16 inputs give 34.8% and 256 inputs give 89.5%. The time to fill the grid and drain it is fixed regardless of the number of inputs, so with few inputs most of the array is idle.

This is a simplified teaching model. A real TPU overlaps draining one matrix multiply with filling the next (the TPU v1 paper says a B×256B \times 256 input is processed in BB pipelined cycles). On the other hand, the model ignores the time to load weights into the grid, which is large on real hardware: according to Table 3 of the TPU v1 paper, averaged over its six applications, waiting for weights to arrive (weight stall) took 43% of cycles and shifting weights into the grid (weight shift) took 12%. Either way, the property holds: you have to stream enough data to keep the array busy.

Shapes that don't fit need padding

The other constraint is matrix shape. Google Cloud's documentation explains that the XLA compiler tiles a matrix multiply into 128×128 chunks for the MXU, and pads tensors with zeros when a computation can't fill an MXU. It therefore recommends making either the batch size or a feature dimension a multiple of 128, and ideally both multiples of 8. That guidance assumes 128×128 MXUs, though; the page doesn't state the best multiple for the 256×256 MXUs on v6e and TPU7x.

For example, a dimension of width 100 placed in a 128-wide tile uses only 100/128≈78%100 / 128 \approx 78\% of it (theoretical). Padding not only wastes arithmetic units but also takes extra memory.

6. Don't apply TPU v1's results to today's TPUs

Whenever TPUs come up, the TPU v1 paper's "15-30x faster than the GPU or CPU, with 30-80x better performance per watt" gets quoted. Those numbers really are in the paper's abstract, but you must not use them to compare today's TPUs and GPUs. The reason lies in the conditions behind them.

ConditionTPU v1 paper's comparisonWhat a comparison today asks about
UseInference only (the paper says training ran on purchased GPUs)Both training and inference
Number format8-bit integers (16-bit ran at a quarter of the speed)BF16, FP8 and others
TimingIn data centers since 20152026 products
CompetitorsNVIDIA K80 (Boost mode disabled) and Intel HaswellBlackwell, Rubin generations and others
WorkloadSix of Google's internal models (MLP, LSTM, CNN)Transformers, LLMs and others
Constraint7 ms 99th-percentile response time (for MLP0)Varies by service

The last row, the response-time constraint, matters most. According to Table 4 of the paper, running the model MLP0 under the 7 ms limit forced the CPU and GPU to cap their batch size at 16, reaching only 42% and 37% of their maximum throughput respectively. The TPU ran at a batch size of 200 and reached 80% of its maximum. Part of TPU v1's large margin came from the CPU and GPU being unable to use their full performance under a latency limit.

There are still lessons from the paper that hold today:

  • Inference often prioritizes response time over throughput. The latency target decides the usable batch size, and that decides real performance.
  • Memory bandwidth often decides performance. The paper estimates the TPU would have been faster still with the K80's GDDR5 memory.
  • Prices are often not published. Saying that chip prices are negotiated and can't be disclosed, the paper uses performance per watt as a proxy for TCO (total cost of ownership).

TPUs have changed a lot since. According to the TPU v4 paper (ISCA 2023), TPU v4, deployed in 2020, has a theoretical peak of 275 TFLOPS in BF16 and INT8 and 32 GiB of HBM2 (1,200 GB/s), and is a training supercomputer that links up to 4,096 chips through optical circuit switches (OCS). The paper reports that, for similar-sized systems, it was 1.2-1.7x faster than the NVIDIA A100 and used 1.3-1.9x less power. This, too, is a comparison by Google's authors, against an A100 that is a 2020 product. For any comparison, check when, which chips and under what conditions.

7. Numeric precision: how many bits can it compute in?

AI computation usually runs in number formats smaller than 32-bit floating point (FP32). Smaller formats help the number of arithmetic units, the amount of memory and the bandwidth all at once (the pillar's section 3 explains the formats). Here is where each processor stands.

CPUGPU (NVIDIA data center)TPU (Google Cloud)
Main matrix formatsFP32; INT8 and BF16 via AMX and similarFP16, BF16, FP8, FP4 and more (by generation)BF16 (inputs) + FP32 (accumulation)
8-bitDepends on the instruction setBy generationINT8 (v5e, v6e), FP8 (v5p, TPU7x)
Double precision (FP64)NativeDepends on the product (below)Google says workloads that need high-precision arithmetic are not a fit

The TPU 8-bit row follows Google Cloud's TPU architecture page. However, the generation-comparison table on Google Cloud's own TPU7x page lists FP8 918 TFLOPS in the v6e row, so the two pages disagree.

According to Google Cloud's documentation, the TPU's default is to multiply in bfloat16 and accumulate in FP32. The XLA compiler inserts the FP32→bfloat16 conversion automatically, rounding to nearest even, and bfloat16 on Cloud TPU does not support subnormals (tiny values extremely close to zero); they are flushed to zero. Google says this approach shortens time to convergence without losing accuracy, but that is Google's claim, and some models may need checking.

Nor can you say "GPUs are good at double precision" across the board. In NVIDIA's datasheets, the HGX B200 delivers 37 TFLOPS of FP64 per GPU, while its successor, the HGX B300, delivers 1.2 TFLOPS. The B300 strengthens low-precision AI math at the cost of most of its double precision. For work that needs double precision, such as scientific computing, you have to check SKU by SKU even on GPUs.

8. Memory: HBM capacity and bandwidth

Lining up the published specs

Here are per-chip published figures for current TPUs and NVIDIA GPUs, taken from official sources (the Google Cloud pages for TPUs, NVIDIA's H100 whitepaper for the H100, and the Blackwell datasheet for the B200).

ChipBF16 (TFLOPS)8-bit (TFLOPS/TOPS)HBM capacityHBM bandwidth (GB/s)Chip-to-chip (bidirectional, GB/s)
TPU v5e197INT8 39316 GiB819ICI 400
TPU v6e (Trillium)918INT8 1,83632 GiB1,638ICI 800
TPU v5p459FP8 45995 GiB (96 physical)2,765ICI 1,200
TPU7x (Ironwood)2,307FP8 4,614192 GiB7,380ICI 1,200
H100 SXM (dense)989.4FP8 1,97980 GB3,352NVLink 900
HGX B200 (dense)2,250FP8 4,500180 GB7,700NVLink 1,800

Some cautions when reading it:

  • NVIDIA figures are normalized to dense matrices. The big numbers on NVIDIA's product pages and datasheets use sparsity and are halved for dense matrices (why: section 5 of the pillar). Google Cloud's pages don't say whether their figures are dense or sparse; the table shows them as printed.
  • Capacity units differ. NVIDIA uses GB (10^9 bytes). Google mostly uses GiB (2^30 bytes), but the v6e spec table alone says "32 GB" while a note on the same page says 32 GiB. The TPU pages also note that usable HBM, after what the runtime reserves, is smaller than physical capacity (31.25 GiB of 32 GiB on v6e).
  • v5p's FP8 figure is printed as 459 TFLOPS, the same as BF16.
  • All chip-to-chip figures are bidirectional totals. The H100's NVLink 900 GB/s is 450 GB/s in each direction. NVIDIA's NVLink page lists per-GPU bandwidth for generations 4-6 in one table as 900, 1,800 and 3,000 GB/s, and describes the sixth generation's 3 TB/s as "bidirectional".

On published specs alone, TPU7x and the B200 are at the same level in BF16 and FP8 compute and HBM bandwidth. But similar published specs do not mean similar real-world speed. Section 11 covers how to compare actual speed.

Computing the ridge point

From the table we can compute the theoretical BF16 ridge point (compute ÷ HBM bandwidth).

ChipRidge point (BF16, theoretical)
TPU v5eabout 241 FLOP/byte
TPU v6eabout 560 FLOP/byte
TPU v5pabout 166 FLOP/byte
TPU7xabout 313 FLOP/byte
H100 SXMabout 295 FLOP/byte
HGX B200about 292 FLOP/byte

Every chip needs hundreds of operations per byte read from memory to saturate its arithmetic units. LLM decode, which generates one token at a time, does roughly one operation per byte at batch size 1. Whatever the chip, decode tends to be bandwidth-bound. v6e's high ridge point means its bandwidth is small relative to its compute, so on bandwidth-bound work it won't deliver speed in proportion to its compute.

Who manages on-chip memory?

On a CPU, hardware manages the on-chip caches automatically. TPUs are different. The TPU v1 paper calls its 28 MiB of on-chip memory "software-managed", and the TPU7x documentation describes VMEM, an on-chip SRAM, as a scratchpad with far higher bandwidth to the MXU than HBM. The compiler or kernels decide which data sits on the chip and when — one reason the compiler (XLA) matters so much on TPUs.

Training a large model means hundreds or thousands of chips continuously exchanging gradients and parameters. Chip-to-chip communication matters as much as compute.

TPU: ICI and the torus

TPU chips connect over a dedicated link called ICI (Inter-Chip Interconnect). According to Google Cloud's documentation, each TPU7x chip has six ICI links of 200 GB/s bidirectional each, 1,200 GB/s bidirectional in total. Each chip connects directly to its neighbors in all three dimensions, and slices of 64 chips (a 4×4×4 "cube") or more form a torus (a grid whose edges wrap around). Cubes are joined by optical fiber and reconfigurable optical circuit switches (OCS), and one TPU7x Pod scales to 9,216 chips. Beyond a Pod, Multislice links multiple slices over the data-center network (DCN).

NVIDIA GPUs connect over NVLink. Rather than wiring neighbors together like a torus, GPUs in the same "NVLink domain" talk to each other through NVLink Switches. According to NVIDIA's datasheet, the GB200 NVL72 puts 72 GPUs in one NVLink domain with 130 TB/s of total bandwidth. Outside the domain, InfiniBand or Ethernet networks connect them.

Which is better?

Per-chip bandwidth is in the same order of magnitude, but because the wiring differs, which one fits depends on the communication pattern.

  • According to the TPU v4 paper, All-Reduce used in backpropagation (summing every chip's gradients and handing the result back to all) maps well to 2D and 3D tori.
  • Communication in which every chip exchanges data with every other chip is called All-to-All. It arises with embeddings in recommendation models, and routing tokens to experts in an MoE produces the same shape of traffic.
  • The same paper says All-to-All strains the torus's bisection bandwidth (the bandwidth across a cut that splits the network in half). That is one reason TPU v4 lets you choose a "twisted torus" through OCS.

When comparing interconnects, look beyond per-chip bandwidth to how many chips can be tightly connected at once (72 for NVL72, 9,216 for a TPU7x Pod) and which communication patterns your model uses most. This comparison matters for inference too, whenever a model too large for one chip is split across several.

10. Software: CUDA, ROCm/HIP and XLA

Hardware performance is only usable to the extent software can extract it. And software is what decides the cost of switching.

CUDA (NVIDIA)

NVIDIA GPUs run on the CUDA programming model and the libraries on top of it (cuBLAS, cuDNN, NCCL and others). Major frameworks such as PyTorch call these internally. Section 11 of the pillar explains how much of NVIDIA's strength comes from this deep CUDA stack.

ROCm and HIP (AMD)

AMD's GPU software platform is ROCm. AMD's documentation (ROCm 10.1.0 as of November 17, 2026) describes ROCm as "the AMD open software stack for GPU-accelerated computing"; it includes hipBLAS and rocBLAS for matrix math, MIOpen for deep learning and RCCL for collective communication.

To run CUDA code on AMD GPUs there is HIP, a C++ API and kernel language, plus a tool (HIPIFY) that converts CUDA API calls to HIP. But AMD's own documentation says HIP is not intended to be a drop-in replacement for CUDA, and porting requires some manual coding and performance tuning. According to the PyTorch documentation, the ROCm build of PyTorch intentionally reuses the torch.cuda interfaces, so code written at the PyTorch level moves over relatively easily. Parts that depend on custom CUDA kernels still need porting work.

XLA, JAX and TorchTPU (Google)

Code that runs on a TPU has to be compiled by XLA (Accelerated Linear Algebra). Google Cloud's documentation describes XLA as a just-in-time compiler: it takes the graph emitted by the framework and compiles the linear algebra, loss and gradient parts into TPU machine code. XLA itself is open source; as the OpenXLA project it is developed with Google, NVIDIA, AMD, Intel, Meta and others, and it also runs on GPUs and CPUs.

Depending on a compiler has both benefits and constraints.

  • Benefit: the compiler can optimize operator fusion and memory placement as a whole.
  • Constraint: if tensor shapes change from run to run, each change needs a compile. Google Cloud's documentation says models with dynamically shaped tensors aren't well suited to TPUs, and the JAX documentation explains that changing the value of a static argument triggers recompilation. Workloads such as LLM inference, where input lengths vary, need techniques like bucketing lengths into a few fixed sizes.

The most natural framework on TPUs is JAX. For PyTorch, something big changed in November 2026. Google Cloud's documentation (updated November 10, 2026) points to TorchTPU, co-developed with the PyTorch team at Meta, as the foundation for running PyTorch on TPUs. TorchTPU uses standard PyTorch with torch.device("tpu"), runs eagerly by default, and compiles through OpenXLA when you use torch.compile. The older PyTorch/XLA is deprecated. Its documentation announced that the repository would be archived on November 15, 2026, but as of November 17, 2026 the GitHub repository had not yet been archived. For inference, TorchTPU versions of vLLM and SGLang are also offered.

The TPU7x documentation also states plainly that JAX and PyTorch work on TPU7x and TensorFlow is not supported.

TorchTPU is a brand-new foundation. You won't know whether your model runs, or performs, until you try it. Planning on the assumption that "it's written in PyTorch, so it'll just run on TPUs" is risky.

11. Comparing with measurements: how to read MLPerf

Published specs are only upper bounds. The most widely used way to compare chips' actual speed is MLPerf from the industry consortium MLCommons. The basics of reading it — divisions, availability categories, result IDs — are in section 10 of the pillar.

MLPerf Training v6.1: TPU7x and Rubin debut

According to MLCommons, MLPerf Training v6.1, published on November 16, 2026, includes results on NVIDIA's Rubin-generation GPUs and Google's TPUv7 (Ironwood) for the first time. Checking the submitter-written system descriptions in MLCommons' public results repository, Google's TPU7x systems (for example tpu7x-8x16x16, 512 nodes × 4 chips) are in the Closed division recorded as "Available cloud", and NVIDIA's Rubin GPU system (Hecate (8x NVIDIA VR NVL72)) is in the Closed division recorded as "Available on-premise". This article does not compare which is faster.

MLPerf Inference and latency targets

For inference comparisons, the thing to notice is that MLPerf Inference measures throughput under latency constraints. Under the MLPerf Inference rules, the Server scenario sends requests at random (Poisson) intervals and measures the highest throughput that still meets the latency constraint. For LLMs there are two constraints: time to first token (TTFT) and the average time per output token (TPOT). For Llama2-70b, for example, the conversational category requires TTFT 2,000 ms and TPOT 200 ms, and the interactive category TTFT 450 ms and TPOT 40 ms.

This measures exactly the issue the TPU v1 paper raised: whether inference can deliver performance under a response-time limit. In your own comparisons, too, fix the SLO first, then measure throughput while meeting it. The model in section 13 uses the same idea.

12. Power and TCO

Don't mix rated power with measured power

Chip power figures come in different kinds.

  • TDP (thermal design power): the upper bound that cooling is designed for. NVIDIA's published figures are up to 700 W for the H100 SXM and up to 1,000 W for the HGX B200.
  • Measured power: the TPU v4 paper reports the power of the chip and HBM while running production applications as 170 W on average (121 W minimum, 192 W maximum), and lists TDP as "N.A.". In the TPU v1 paper, TDP was 75 W and measured power while busy was 40 W.
  • MLPerf Power: according to MLCommons, it measures the average AC power the whole system draws from the wall during the benchmark run; any other power figure, such as TDP, is not measured or validated by MLCommons.

You can't put TPU v4's 170 W average next to the H100's 700 W TDP and say "a TPU uses a quarter of the power." One is a measured average, the other an upper bound, and they are different generations. Also, as of November 17, 2026, Google Cloud's TPU7x page does not list chip power.

Converting TCO to an hourly figure

If you rent in the cloud, most of TCO is already inside the hourly price. If you buy, the formula below turns ownership into an hourly cost, so you can compare on the same yardstick as the cloud:

chour=CcapexY×8760+PkW×PUE×e+copsc_{\mathrm{hour}} = \frac{C_{\mathrm{capex}}}{Y \times 8760} + P_{\mathrm{kW}} \times \mathrm{PUE} \times e + c_{\mathrm{ops}}
  • CcapexC_{\mathrm{capex}}: purchase cost (servers, networking and installation)
  • YY: depreciation period in years (8760 is the number of hours in a year)
  • PkWP_{\mathrm{kW}}: actual average power draw (a measured value or estimate, not TDP)
  • PUE\mathrm{PUE}: the data center's power usage effectiveness (a multiplier that includes cooling)
  • ee: electricity price per kWh
  • copsc_{\mathrm{ops}}: maintenance and operations staff cost, spread per hour

This chourc_{\mathrm{hour}} becomes the "hourly price" in the next section's Cost/Token formula. Google Cloud's v5e and v6e pages carry a "Performance/TCO (expected)" metric (0.65x for v5e, 1 for v6e), but the pages don't say what the ratio is relative to, so this article doesn't use it for comparison.

13. Original analysis: Cost/Token flips with conditions

The formula

Compare LLM inference cost as cost per 1M tokens:

cost per 1M tokens=chour/3600tokens/s×U×106\text{cost per 1M tokens} = \frac{c_{\mathrm{hour}} / 3600}{\text{tokens/s} \times U} \times 10^{6}

UU is utilization: the share of rented time actually spent processing. The break-even between APIs and self-hosting is covered in Designing Llama inference costs and Generative AI cost and break-even. Here the focus is different: when you compare self-hosted machines against each other, what flips the verdict?

A theoretical model and its assumptions

One important caution first. The "tokens/s" in this formula should really be a value measured on your own model. A value derived from spec-sheet TFLOPS has to assume an efficiency, so it tends to underestimate real cost.

To show how the conclusion shifts with conditions, this article approximates tokens/s with a theoretical model. One decode step (the time for every in-flight request to advance by one token) is modeled Roofline-style as:

t(b)=max⁡(W+b L kηbwBmem, 2 b NparamηfF)t(b) = \max\left(\frac{W + b\,L\,k}{\eta_{\mathrm{bw}} B_{\mathrm{mem}}},\ \frac{2\,b\,N_{\mathrm{param}}}{\eta_{\mathrm{f}} F}\right)

bb is the batch size (concurrent requests), WW the weight size in bytes, LL the context length each request holds, and kk the KV-cache bytes per token. The KV cache is memory that stores the attention keys and values of tokens already generated, and it is read on every generation step (details in section 7 of the pillar). BmemB_{\mathrm{mem}} is memory bandwidth, FF compute throughput and η\eta the efficiency relative to published specs. One machine's throughput is b/t(b)b / t(b) tokens/s, and the time between generated tokens (TPOT) is t(b)t(b).

All inputs are assumptions. Machines A, B and C are not real products; the figures and prices are placeholders for illustration.

ItemAssumed value
Machine A (large, high bandwidth)6.0 USD per hour, 96 GB memory, 4.0 TB/s bandwidth, 2,000 TFLOP/s at 8 bits
Machine B (small, inference-oriented)1.0 USD per hour, 24 GB memory, 0.8 TB/s bandwidth, 400 TFLOP/s at 8 bits
Machine C (CPU server)1.5 USD per hour, 256 GB memory, 0.3 TB/s bandwidth, 20 TFLOP/s
Model8 billion parameters, 8-bit weights = 8 GB
KV cache128 KiB per token, average context of 1,024 tokens
Efficiency70% of bandwidth, 40% of compute
Other90% of memory is usable for weights and KV cache. Prefill (processing the input) is ignored; only output tokens are counted

The calculator code

  • Language / dependencies: Python 3.9+, standard library only
  • Runs on: any CPU (I ran it on an Apple M4 Pro with Python 3.9)
  • Run: python cost_per_token.py

The code's printed labels are left in Japanese so that the code runs exactly as on the Japanese page. In the output, "大型・高帯域" = large, high bandwidth; "小型・推論向け" = small, inference-oriented; "CPUサーバ" = CPU server; "メモリ上の最大バッチ" = largest batch that fits in memory; "メモリ不足" = out of memory; "台" = units; "SLO を満たせない" = cannot meet the SLO.

# cost_per_token.py — compare cost per 1M tokens across batch size, utilization and demand
# Python 3.9+, standard library only. Run: python cost_per_token.py
# Note: every figure for machines A/B/C is an illustrative "assumption", not a real product's spec or price.
import math
from dataclasses import dataclass


@dataclass(frozen=True)
class Machine:
    name: str
    usd_per_hour: float   # hourly price per unit (assumption)
    mem_gb: float         # memory capacity
    bw_tbs: float         # published memory bandwidth [TB/s]
    tflops: float         # published compute at the inference precision (dense) [TFLOP/s]


# ---- Assumptions (replace with your own measurements and quotes) ----
MACHINES = [
    Machine("A 大型・高帯域", usd_per_hour=6.0, mem_gb=96, bw_tbs=4.0, tflops=2000),
    Machine("B 小型・推論向け", usd_per_hour=1.0, mem_gb=24, bw_tbs=0.8, tflops=400),
    Machine("C CPUサーバ", usd_per_hour=1.5, mem_gb=256, bw_tbs=0.3, tflops=20),
]
PARAMS = 8e9              # model parameter count (8B class)
WEIGHT_BYTES = 8e9        # 8-bit weights → 8 GB
KV_BYTES_PER_TOKEN = 128 * 1024   # KV cache per token (assuming an 8B-class model with GQA)
CONTEXT = 1024            # average context length held per request [tokens]
ETA_BW, ETA_FLOP = 0.7, 0.4       # efficiency relative to published specs (assumption)
MEM_USABLE = 0.9          # share of memory usable for weights and KV (assumption)


def step_time(m: Machine, batch: int) -> float:
    """Time for one decode step (every request advances one token) [s]. Roofline-style approximation."""
    bytes_read = WEIGHT_BYTES + batch * CONTEXT * KV_BYTES_PER_TOKEN
    t_mem = bytes_read / (ETA_BW * m.bw_tbs * 1e12)
    t_compute = batch * 2 * PARAMS / (ETA_FLOP * m.tflops * 1e12)
    return max(t_mem, t_compute)


def max_batch_by_memory(m: Machine) -> int:
    free = m.mem_gb * 1e9 * MEM_USABLE - WEIGHT_BYTES
    return max(0, int(free // (CONTEXT * KV_BYTES_PER_TOKEN)))


def best_batch(m: Machine, tpot_slo: float) -> int:
    """Largest batch that fits in memory and keeps the time between tokens (TPOT) within the SLO."""
    b = 0
    for cand in range(1, max_batch_by_memory(m) + 1):
        if step_time(m, cand) <= tpot_slo:
            b = cand
    return b


def usd_per_1m_tokens(usd_per_hour: float, tokens_per_sec: float, utilization: float) -> float:
    """cost = (hourly price / 3600) / (tokens per second × utilization) × 1e6"""
    return usd_per_hour / 3600 / (tokens_per_sec * utilization) * 1e6


if __name__ == "__main__":
    print("[1] バッチサイズ別(稼働率100%)  TPOT=生成間隔, tok/s=1台の出力スループット")
    for m in MACHINES:
        bmax = max_batch_by_memory(m)
        print(f"  {m.name}: メモリ上の最大バッチ={bmax}")
        for b in [1, 8, 32, 128, bmax]:
            if b > bmax:
                print(f"    b={b:4d}: メモリ不足")
                continue
            t = step_time(m, b)
            tps = b / t
            print(f"    b={b:4d}: TPOT={t*1e3:6.1f} ms  tok/s={tps:7.0f}  "
                  f"USD/1M={usd_per_1m_tokens(m.usd_per_hour, tps, 1.0):6.2f}")

    print("\n[2] 稼働率を変えたとき(バッチはメモリ上の最大)")
    for m in MACHINES:
        b = max_batch_by_memory(m)
        tps = b / step_time(m, b)
        row = "  ".join(f"U={u:.0%}:{usd_per_1m_tokens(m.usd_per_hour, tps, u):6.2f}"
                        for u in [1.0, 0.5, 0.2])
        print(f"  {m.name}: {row}")

    for slo in [0.050, 0.020]:
        print(f"\n[3] TPOT の上限 {slo*1e3:.0f} ms・需要量(tok/s)別の 1M トークン原価(USD)")
        for m in MACHINES:
            b = best_batch(m, slo)
            if b == 0:
                print(f"  {m.name}: SLO を満たせない")
                continue
            cap = b / step_time(m, b)          # upper limit one unit can deliver while meeting the SLO
            cells = []
            for demand in [20, 200, 2000, 20000]:
                n = math.ceil(demand / cap)     # units needed (you can only rent whole units)
                util = demand / (n * cap)       # resulting utilization
                cost = usd_per_1m_tokens(n * m.usd_per_hour, demand, 1.0)
                cells.append(f"{demand:>5}:{cost:7.2f}({n}台,U={util:.1%})")
            print(f"  {m.name} (b={b}, 上限{cap:.0f} tok/s/台): " + "  ".join(cells))

Output (copied verbatim from my run; the [3] lines are long, so scroll sideways). Section headers: [1] by batch size at 100% utilization; [2] varying utilization at the largest batch that fits; [3] cost per 1M tokens (USD) by demand (tok/s) with a TPOT cap of 50 ms, then 20 ms.

[1] バッチサイズ別(稼働率100%)  TPOT=生成間隔, tok/s=1台の出力スループット
  A 大型・高帯域: メモリ上の最大バッチ=584
    b=   1: TPOT=   2.9 ms  tok/s=    344  USD/1M=  4.84
    b=   8: TPOT=   3.2 ms  tok/s=   2469  USD/1M=  0.68
    b=  32: TPOT=   4.4 ms  tok/s=   7288  USD/1M=  0.23
    b= 128: TPOT=   9.0 ms  tok/s=  14234  USD/1M=  0.12
    b= 584: TPOT=  30.9 ms  tok/s=  18930  USD/1M=  0.09
  B 小型・推論向け: メモリ上の最大バッチ=101
    b=   1: TPOT=  14.5 ms  tok/s=     69  USD/1M=  4.03
    b=   8: TPOT=  16.2 ms  tok/s=    494  USD/1M=  0.56
    b=  32: TPOT=  22.0 ms  tok/s=   1458  USD/1M=  0.19
    b= 128: メモリ不足
    b= 101: TPOT=  38.5 ms  tok/s=   2624  USD/1M=  0.11
  C CPUサーバ: メモリ上の最大バッチ=1657
    b=   1: TPOT=  38.7 ms  tok/s=     26  USD/1M= 16.14
    b=   8: TPOT=  43.2 ms  tok/s=    185  USD/1M=  2.25
    b=  32: TPOT=  64.0 ms  tok/s=    500  USD/1M=  0.83
    b= 128: TPOT= 256.0 ms  tok/s=    500  USD/1M=  0.83
    b=1657: TPOT=3314.0 ms  tok/s=    500  USD/1M=  0.83

[2] 稼働率を変えたとき(バッチはメモリ上の最大)
  A 大型・高帯域: U=100%:  0.09  U=50%:  0.18  U=20%:  0.44
  B 小型・推論向け: U=100%:  0.11  U=50%:  0.21  U=20%:  0.53
  C CPUサーバ: U=100%:  0.83  U=50%:  1.67  U=20%:  4.17

[3] TPOT の上限 50 ms・需要量(tok/s)別の 1M トークン原価(USD)
  A 大型・高帯域 (b=584, 上限18930 tok/s/台):    20:  83.33(1台,U=0.1%)    200:   8.33(1台,U=1.1%)   2000:   0.83(1台,U=10.6%)  20000:   0.17(2台,U=52.8%)
  B 小型・推論向け (b=101, 上限2624 tok/s/台):    20:  13.89(1台,U=0.8%)    200:   1.39(1台,U=7.6%)   2000:   0.14(1台,U=76.2%)  20000:   0.11(8台,U=95.3%)
  C CPUサーバ (b=18, 上限363 tok/s/台):    20:  20.83(1台,U=5.5%)    200:   2.08(1台,U=55.1%)   2000:   1.25(6台,U=91.9%)  20000:   1.17(56台,U=98.4%)

[3] TPOT の上限 20 ms・需要量(tok/s)別の 1M トークン原価(USD)
  A 大型・高帯域 (b=357, 上限17877 tok/s/台):    20:  83.33(1台,U=0.1%)    200:   8.33(1台,U=1.1%)   2000:   0.83(1台,U=11.2%)  20000:   0.17(2台,U=55.9%)
  B 小型・推論向け (b=23, 上限1162 tok/s/台):    20:  13.89(1台,U=1.7%)    200:   1.39(1台,U=17.2%)   2000:   0.28(2台,U=86.1%)  20000:   0.25(18台,U=95.6%)
  C CPUサーバ: SLO を満たせない

Reading the results: three conditions that flip the verdict

Flip one: batch size. At batch 1, machine B costs 4.03 USD and machine A 4.84 USD, so B is cheaper. Both are bandwidth-bound, so cost is roughly hourly price ÷ bandwidth: 6.0 ÷ 4.0 = 1.5 for A versus 1.0 ÷ 0.8 = 1.25 for B. But raise each to the largest batch that fits in memory and it reverses: A costs 0.09 USD and B 0.11 USD. A's larger memory lets it serve 584 requests at once, while B tops out at 101. From batch 1, A's cost falls by about 55x (4.84 → 0.088). Even between the same two machines, how much work you can batch together decides which one is cheaper.

Flip two: demand and unit granularity (utilization). As [2] shows, cost is inversely proportional to utilization. A at 50% utilization (0.18 USD) costs more than B at 100% (0.11 USD). And utilization is set by the fact that you can only rent whole units. In [3] with a 50 ms cap on time between tokens, at a demand of 2,000 tokens/s, one B runs at 76.2% utilization (0.14 USD), while even a single A runs at 10.6% (0.83 USD). A large machine is cheap when full, but when demand is small you pay for idle time.

Flip three: the SLO. With a 50 ms cap on time between tokens and a demand of 20,000 tokens/s, B needs 8 units at 0.11 USD and A needs 2 units at 0.17 USD, so B is cheaper. Tighten the cap to 20 ms, and B must cut its batch to 23 to meet it; each unit then handles less, 18 units are needed, and cost rises to 0.25 USD. A still meets 20 ms at batch 357, stays at 0.17 USD, and becomes the cheaper option. This is exactly the pattern from the TPU v1 paper: the response-time limit decides the usable batch, and that decides real performance.

The CPU server (machine C) has low compute, so once its batch reaches 32 it becomes compute-bound and throughput plateaus at 500 tokens per second. It can't meet a 20 ms SLO even at batch 1 (38.7 ms). Even at a demand of 20 tokens per second, machine B (13.89 USD) is cheaper under these assumptions. However, for a CPU server you already own and aren't fully using for anything else, the added hourly cost is close to zero, which could change that conclusion.

Limits of this model

This model is a tool for seeing how the verdict flips, not a prediction of real costs.

  • All machine figures and prices are assumptions and correspond to no real product.
  • It ignores prefill. The longer the inputs, the more compute throughput matters.
  • The efficiencies (70% of bandwidth, 40% of compute) are fixed assumptions; in reality they change with batch size and implementation.
  • Context length is fixed at an average of 1,024 tokens. With long contexts, KV-cache reads grow, and raising the batch tends to leave you bandwidth-bound.
  • It ignores demand fluctuating over time, spare units for redundancy, and autoscaling lag.

When you actually choose, replace this code's tokens/s with values measured on your model, with your input/output lengths and SLO. Don't put tokens/s derived from theoretical peaks and measured tokens/s side by side in the same table.

14. Vendor lock-in: what are you actually tied to?

Lock-in is decided less by "which chip" than by which layer of software you depend on.

  1. Custom kernels: kernels written in CUDA need manual work to port to HIP (AMD's own documentation says so) and a rewrite for TPUs. This is the hardest layer to move.
  2. Frameworks: JAX code also runs on GPUs and CPUs through XLA. PyTorch code runs on ROCm builds through torch.cuda as-is, and on TPUs there is a path through TorchTPU. In every case, performance needs re-tuning.
  3. Inference engines and operations: an engine that supports multiple kinds of hardware, such as vLLM, makes it easier to swap the layer below while keeping the layers above the same. A TorchTPU version of vLLM is offered for TPUs as well.
  4. Cloud and contracts: the ways to use TPUs described in the official TPU documentation are Google Cloud's Compute Engine, GKE and Gemini Enterprise Agent Platform. Long-term reservations and discount contracts also add switching cost.

Practical ways to limit lock-in:

  • Save model weights in standard formats. Google Cloud's documentation says checkpoints trained on TPUs can be used for inference on CPUs or GPUs without extensive manual conversion.
  • Use custom kernels only where truly necessary. When you do, record why they're needed and what the alternative is on other hardware.
  • Keep a hardware-independent benchmark procedure. If you can measure the tokens/s that goes into the section 13 calculator the same way on any hardware, switching decisions get faster.

15. A decision framework: the selection procedure

Finally, here is the procedure for choosing among CPU, GPU and TPU. The order matters, because each step narrows the options for the next.

Step 0: First check whether an API is enough. Before owning or renting hardware, check whether pay-per-use APIs would do (for the break-even, see Generative AI cost and break-even).

Step 1: Put numbers on the shape of the work. Training or inference? For inference, define the distribution of input and output token counts, peak demand (tokens per second) and the SLO (caps on TTFT and TPOT). If matrix multiplication isn't central to the work, first check whether a CPU is enough.

Step 2: Narrow by software constraints. If you depend on custom CUDA kernels, NVIDIA GPUs are close to a given. If the code is written in JAX, TPUs are a natural option. For standard PyTorch models, both GPUs and TPUs (TorchTPU) are candidates, but always test function and performance on TPUs. If tensor shapes change often, TPUs need countermeasures such as bucketing.

Step 3: Narrow by memory. How many chips do the weights and KV cache (plus optimizer state, for training) fit on? If they don't fit on one chip, chip-to-chip connectivity (the size of the NVLink domain, the ICI topology) shapes performance.

Step 4: Compute Cost/Token from measured tokens per second. Run your own model on each candidate while meeting the SLO, and measure tokens per second. Feed that into the section 13 formula and compute cost with the unit count and utilization your demand implies. Don't use published TFLOPS at this stage. In practice, a good place to start is designing the procedure for measuring tokens per second on your own model, and estimating VRAM and budget.

Step 5: Add the cost of switching. Estimate porting effort, operational learning curve and contract terms. If the cost difference doesn't exceed this, choosing the platform you already know is often the rational call.

The decision factors in a table:

FactorLeans CPULeans GPULeans TPU
Nature of the workBranchy, little matrix mathMatrix math + custom opsMostly matrix math, fixed shapes
SoftwareExisting general-purpose codeLots of CUDA/PyTorch assetsExisting JAX/XLA assets
Demand and SLOLow volume, loose SLOCovers a wide rangeLarge-scale, steady
ScaleOne machine is enoughA few GPUs up to an NVLink domainWant to scale to Pod size

And by workload:

WorkloadCPUGPUTPU
Preprocessing, tokenization, branchy workGood fitPoor fit (warp divergence)Poor fit (Google's documentation also says branch-heavy work is not suited)
Low-volume inference on small modelsSometimes a good fit (if the SLO is loose)Good fitCan be oversized depending on scale
Research on custom ops and new kernelsGood for prototypesBest fit (you can write custom kernels)Poor fit (Google states custom ops in the main training loop are not suited)
Large-scale training of standard TransformersPoor fitGood fitGood fit (when matrix math dominates and shapes are fixed)
High-volume, steady LLM inferencePoor fitGood fitGood fit (if shape changes can be handled with XLA)
Recommendation models with huge embeddingsPoor fitGood fitGood fit (has dedicated SparseCore circuits)
Double-precision scientific computingGood fitDepends on the SKU (37 TFLOPS on B200, 1.2 TFLOPS on B300)Poor fit

My own production work has been an inference service running vLLM and AWQ-quantized Qwen3-8B on a single T4 (the GPU pipeline behind AI video localization). Even at that scale, the memory constraint in step 3 decided the design: running several jobs at once ran out of VRAM, so GPU work runs one job at a time, in series. Whatever the chip or scale, the order stays the same: put numbers on the shape of the work and its constraints, then compare by measurement.

16. Limits and counterexamples

This comparison has limits:

  • Published specs are theoretical peaks. Every value in the tables is a vendor-published upper bound, not real performance. The TPU figures carry no dense/sparse annotation.
  • Benchmark rankings are not covered. I confirmed that MLPerf Training v6.1 includes TPU7x and Rubin, but this article does not compare the results or rankings.
  • Prices are not covered. Cloud prices vary by region, contract and time, so Cost/Token is shown with assumed numbers.
  • The software landscape changes fast. As the move to TorchTPU shows, what was true six months ago may not be now. This article reflects November 17, 2026.
  • AMD GPUs and other vendors' accelerators are not covered in depth beyond software. The same procedure works for evaluating non-NVIDIA GPU options.

Some counterexamples:

  • If you run a small model at low volume, a CPU you already own can be cheaper than any accelerator.
  • A large machine is the cheapest when full but the most expensive when demand is small ([3] in section 13).
  • Chips with similar published specs can differ widely in real performance if their software maturity differs.

Summary

  • CPUs, GPUs and TPUs can all multiply matrices. The difference is how they split flexibility and efficiency: the CPU spends its die on caches and branch prediction, the GPU on many SIMT threads, the TPU on a systolic array.
  • A systolic array keeps weights fixed across a large grid and passes intermediate results directly to the neighboring multiply-accumulator, cutting memory reads and writes. In return, its units sit idle unless you stream enough data, and mismatched matrix shapes need padding.
  • TPU v1's "15-30x faster" compares an inference-only, 8-bit integer chip from 2015 with the contemporary K80 and Haswell. It can't be used to compare today's TPUs and GPUs.
  • Current-generation published specs put TPU7x and the B200 at the same level. But published specs are not measurements: compare with condition-matched measurements like MLPerf, and check result IDs and categories.
  • Software decides lock-in. As of November 2026, the foundation for running PyTorch on TPUs has moved from PyTorch/XLA to TorchTPU.
  • Cost/Token is hourly price ÷ (tokens per second × utilization), and which machine is cheapest flips with batch size, demand and SLO. Use tokens per second measured while meeting your SLO.

Chip performance numbers are only the starting point. Putting numbers on the shape of the work and its constraints, measuring on the candidate machines, and including the cost of switching is the most reliable way to choose among CPU, GPU and TPU.

The supply-side view — how NVIDIA, AMD, Google and AWS differ in software and supply — is covered in NVIDIA's moat is more than a CUDA monopoly.

Frequently asked questions

What is a TPU, and how is it different from a GPU?
A TPU (Tensor Processing Unit) is an ASIC (application-specific integrated circuit) that Google designed for machine learning. At its core is a systolic array: a grid of multiply-accumulate units that pass intermediate results directly to their neighbors instead of writing them back to memory. A GPU is a general-purpose parallel processor that runs 32 threads at a time on the same instruction (SIMT), and it lets you write custom kernels and non-matrix work freely. The TPU trades that flexibility for matrix-multiply efficiency.
Is a TPU faster than a GPU?
There is no single answer. The often-quoted "15-30x faster" comes from a 2017 paper comparing the inference-only TPU v1, deployed in 2015, with the NVIDIA K80 of the same era; it does not apply to today's TPUs and GPUs. Current-generation published peaks put TPU7x and NVIDIA's B200 at roughly the same level, but peaks are theoretical upper bounds. Real speed depends on the model, precision, chip count and software, so decide from condition-matched measurements like MLPerf and from trials on your own model.
How should I compare AI inference costs?
Compare cost per 1M tokens, not price per hour. The formula is hourly price ÷ 3600 ÷ (tokens per second × utilization) × 1,000,000. For tokens per second, use a value measured on your own model, with your input/output lengths, while meeting your latency target (for example, a cap on the time between generated tokens). A number derived from spec-sheet TFLOPS has to assume an efficiency, so it tends to underestimate real cost. Also note that you rent in whole units, so low demand means low utilization, and bigger machines become relatively expensive.
Can PyTorch models run on TPUs?
There is a way. As of November 2026, Google Cloud's official documentation points to TorchTPU, co-developed with the PyTorch team at Meta, as the foundation for running PyTorch on TPUs. The older PyTorch/XLA is deprecated; its repository was announced to be archived on November 15, 2026, but as of November 17, 2026 it had not yet been archived on GitHub. Models that use custom CUDA kernels will not run as-is. TorchTPU is a new foundation, so verify that your model runs and performs well before committing.
Can I use a CPU for LLM inference?
Sometimes. For a small model with few concurrent requests, where tens of milliseconds per generated token is acceptable, a CPU can be enough — and if you reuse a server you already own, the added cost is close to zero. But a CPU has less memory bandwidth and matrix-multiply throughput than a GPU or TPU, so throughput stops growing quickly as you raise the batch size, and strict latency targets are out of reach. Put numbers on your demand and latency target before deciding.

References

友田

友田 陽大

Developer of a METI Minister's Award–winning product. With TypeScript + Python + AWS, I deliver SaaS, industry DX, and production-grade generative AI (RAG) end to end — from requirements to infrastructure and operations — single-handedly.

Do you actually know whether self-hosting beats the API?

Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing

When to use AWQ / GPTQ / FP8 / GGUF, VRAM estimates that include the KV cache, the break-even point against per-token API pricing, and the license terms of Apache-2.0 / Llama / Qwen / Gemma. From hands-on experience adopting commercially usable open quantized models stage by stage against license, quality, and speed trade-offs, I help you assemble the decision criteria before you implement.

Available for both project-based (contract) and advisory engagements. Start with a free 30-minute consult.

Also worth reading