People say NVIDIA GPUs are strong at AI. Strong at what, exactly, and why?
The usual answer is "because they have lots of compute units". That is half right, and it leaves several things unexplained. Swap in a GPU with ten times the compute and a chatbot may not answer ten times faster. The FP8 figure on the spec sheet is out of reach for almost every workload. Going from one GPU to eight rarely makes training eight times faster.
This article breaks NVIDIA's strength down into four layers that multiply together:
- A compute unit built for matrix multiplication (the Tensor Core)
- Computing in small number formats (FP16, BF16, FP8, FP4)
- Memory that feeds data to the compute units (HBM)
- Communication between GPUs (NVLink, NVSwitch, NCCL)
Across all four sits the software that puts them to work (cuBLAS, cuDNN, NCCL, PyTorch and others).
The short answer: NVIDIA GPUs are strong because these four layers and the software come together. But what usually decides how fast a chatbot answers is memory bandwidth, not compute — and the second half of this article shows why, with formulas.
Each section opens by explaining in words why something happens, then backs it up with formulas and published specs. If you want material for a business decision, read the opening of each section and the decision guide at the end (section 13). Engineers will get the most from the formulas and tables.
About the numbers. Every performance figure here is either a theoretical peak published by NVIDIA (an upper bound computed as clock speed times number of units) or my own arithmetic on those figures. None of it is something I measured on an H100 or B200, and values derived by calculation are labelled "theoretical". For context: my own production work has been inference on a single Tesla T4; I have not operated H100 or B200 clusters.
1. AI is a huge pile of multiplications and additions
At bottom, a neural network is a vast number of multiplications and additions.
One layer of a language model, for example, multiplies an input vector by a weight matrix to produce an output :
If is 4096×4096, each output element takes 4,096 multiplications and 4,096 additions, and there are 4,096 output elements. In practice many tokens are processed together as a matrix-times-matrix product. That is a GEMM (General Matrix Multiply).
The cost of multiplying an matrix by a matrix is counted as
- : the number of tokens processed together (batch × sequence length)
- : the input dimension
- : the output dimension
- The factor 2: each multiply-and-add counts as two FLOPs, one multiplication and one addition
Pushing 2,048 tokens through a 4096×4096 layer costs floating-point operations — about 68.7 billion. That is one layer, and a model has hundreds of them.
For training a whole Transformer, Kaplan et al.'s scaling-laws paper approximates the total compute, with the number of non-embedding parameters and the number of training tokens, as
The forward pass costs about FLOPs per token and the backward pass about twice that, so about in total (the paper writes , where is the number of training tokens). It is an approximation: it drops the term that attention adds in proportion to sequence length. The paper argues that term is small when the model is wide compared with its context length. For long-context models, the approximation underestimates.
The one thing to take from this section: most of AI's arithmetic is the same shape of multiply-and-add, repeated an enormous number of times, independently. Because the pieces are independent, many units can work on them at once. That is where the fit with GPUs starts.
Do you actually know whether self-hosting beats the API?
Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing
2. CUDA cores and Tensor Cores work at different granularity
FMA: multiply and add in one instruction
A GPU's basic arithmetic unit (a CUDA core, in NVIDIA's terms) executes an FMA (fused multiply-add) as one instruction:
An FMA carries the product into the addition without rounding it and rounds only once, at the end. Fewer roundings mean better accuracy than doing the two steps separately, and half the instructions. A dot product inside a matrix multiplication is a long chain of FMAs.
These cores are not counted in the same units as CPU cores. A CPU core is an independent execution unit with branch prediction and out-of-order execution. A CUDA core is closer to a single lane that applies the same instruction to many pieces of data at once. The GPU groups its units of work (threads) into sets of 32 called warps, and every thread in a warp executes the same instruction together. Comparing a CUDA-core count with a CPU-core count therefore says nothing about speed.
Tensor Cores: a whole small matrix product in one instruction
A Tensor Core performs FMAs not on single numbers but on small blocks of matrices. Conceptually it does this in one operation:
If a CUDA core is a clerk with a calculator, a Tensor Core is a machine that fills in a whole multiplication table in one go. It is less general, and far faster at the one job of multiplying matrices.
Tensor Cores first appeared in the Tesla V100 (Volta) in 2017. The V100 has 8 per SM (Streaming Multiprocessor, the GPU's basic compute block), 640 in total, and each performs 64 FMAs per clock. From that we can compute the theoretical peak:
That matches the "125 Tensor TFLOPS" NVIDIA published for the V100. A spec-sheet peak is a product like this one, not the result of running anything.
The H100's "FP32 performance" and "Tensor Core performance" are different things
Here is the same distinction on the H100 SXM5 (the SXM board version of the H100), from Table 1 of NVIDIA's whitepaper v1.04. TFLOPS means a trillion floating-point operations per second. "Dense" means ordinary matrix arithmetic that does not skip zeros (sparsity comes in section 5).
| Operation | Theoretical peak (dense) | Note |
|---|---|---|
| FP64 | 33.5 TFLOPS | CUDA cores |
| FP32 | 66.9 TFLOPS | CUDA cores |
| TF32 Tensor Core | 494.7 TFLOPS | 989.4 published with sparsity |
| FP16 / BF16 Tensor Core | 989.4 TFLOPS | 1,978.9 published with sparsity |
| FP8 Tensor Core | 1,978.9 TFLOPS | 3,957.8 published with sparsity |
Dense FP16 on Tensor Cores is about 15 times FP32 on CUDA cores (). That gap is what a matrix-specialised unit buys. Turned around: work that does not map onto Tensor Cores — branch-heavy code, anything that is not shaped like a matrix product — gets none of that 15x.
The H100's FP16 peak decomposes the same way as the V100's: 132 SMs, 4 Tensor Cores per SM, 4,096 FLOPs per SM per clock (dense FP16), and a 1,830 MHz boost clock for Tensor Core work give
3. Number formats: what you gain and lose with smaller numbers
Why small numbers help
Whether a number takes 32 bits or 8 changes three things:
- How many units fit: a chip of the same area holds more small arithmetic units. As the table shows, the H100's FP8 peak is twice its FP16 peak.
- How much memory a model needs: the same model fits in half or a quarter of the space.
- Memory bandwidth: each transfer carries more numbers. As we will see, in inference this is often the effect that matters most.
The price is precision and range. Fewer bits mean fewer significant digits (mantissa), a narrower range of magnitudes (exponent), or both.
How a floating-point number is built
An IEEE 754-style floating-point number is a sign, an exponent and a mantissa. A normal number is
Numbers very close to zero (subnormals) take the form . FP32 and FP16 are standard IEEE 754 formats; BF16, TF32, FP8 and FP4 follow the same idea but are separate formats, not IEEE 754 standard formats.
- The exponent width sets the range (how large and how small a number can be)
- The mantissa width sets the precision (the spacing between neighbouring values)
The formats common in AI line up like this. I computed the maxima and machine epsilon (the gap between 1 and the next representable number, ) from the bit layouts in Python, and checked FP8 and FP4 against the tables in the original specifications.
| Format | Bits | Exponent | Mantissa | Max | Machine epsilon | Typical use |
|---|---|---|---|---|---|---|
| FP32 | 32 | 8 | 23 | ≈ 3.4×10³⁸ | ≈ 1.2×10⁻⁷ | Reference, master weights |
| TF32 | (19) | 8 | 10 | ≈ 3.4×10³⁸ | ≈ 9.8×10⁻⁴ | Internal Tensor Core mode |
| BF16 | 16 | 8 | 7 | ≈ 3.4×10³⁸ | ≈ 7.8×10⁻³ | Training |
| FP16 | 16 | 5 | 10 | 65,504 | ≈ 9.8×10⁻⁴ | Training, inference |
| FP8 E4M3 | 8 | 4 | 3 | 448 | 0.125 | Weights, activations |
| FP8 E5M2 | 8 | 5 | 2 | 57,344 | 0.25 | Gradients |
| FP4 E2M1 | 4 | 2 | 1 | 6 | 0.5 | Inference (with scaling) |
A few notes:
- BF16 and FP16 are both 16 bits with opposite characters. BF16 has FP32's 8-bit exponent and therefore its range; FP16 has three more mantissa bits and therefore more precision. Training favours BF16 because very small gradients are less likely to collapse to zero.
- TF32 is not a storage format. It is a Tensor Core mode introduced with the A100. As NVIDIA describes it, FP32 inputs are rounded to TF32 (8-bit exponent, 10-bit mantissa) for the multiplication, products are accumulated in FP32, and results are written as FP32; the data in memory stays FP32. NVIDIA's 2021 blog calls TF32 "the default", but PyTorch's flag for using TF32 in matrix multiplication (
torch.backends.cuda.matmul.allow_tf32) has defaulted to off since PyTorch 1.12 (convolutions'torch.backends.cudnn.allow_tf32defaults to on, and newer PyTorch points users towards thefp32_precisionsettings). - There are two FP8 formats. Micikevicius et al. (arXiv:2209.05433) recommend E4M3, with less range and more precision, for weights and activations, and E5M2, with more range, for gradients. E4M3 has no infinities and reuses those bit patterns for ordinary values, which stretches its maximum to 448.
- FP4 (E2M1) can represent just eight non-negative values: 0, 0.5, 1, 1.5, 2, 3, 4 and 6. That alone cannot represent weights, so it is paired with a scale per block.
Keep input, accumulation and storage precision apart
Low-precision arithmetic gets confusing because one matrix multiplication involves three precisions:
- Input precision: the format of the values being multiplied (e.g. FP8)
- Accumulation precision: the format of the running sum of products (e.g. FP32)
- Storage precision: the format the result is written back in (e.g. BF16)
A dot product sums thousands of products, so accumulating in low precision piles up error. Tensor Cores are therefore built to accumulate at higher precision even when the inputs are low-precision; the V100's Tensor Cores already accumulated FP16 inputs into FP32. "Computing in FP8" usually means "the inputs are FP8".
Block scaling: MXFP4 and NVFP4
When a format has as few values as 4-bit does, each short run of values gets its own multiplier (a scale) that brings them back to their real magnitude.
- MXFP4 (the Open Compute Project Microscaling specification v1.0): one 8-bit scale (E8M0, which can only represent powers of two) per 32 values.
- NVFP4 (introduced by NVIDIA with Blackwell): one FP8 (E4M3) scale per 16 values, plus one FP32 scale for the whole tensor.
With smaller blocks and scales that need not be powers of two, NVFP4 holds quantization error down more finely. NVIDIA's blog puts it at about 4.5 bits per value — about 3.5 times smaller than FP16 and 1.8 times smaller than FP8. The same blog reports accuracy within 1% of FP8 on DeepSeek-R1-0528; that is NVIDIA's own evaluation, and the loss on your model and task has to be measured separately.
Choosing a quantization method in practice (AWQ, GPTQ, FP8, GGUF) is outside this article's scope. How to choose a quantization method for Qwen3-8B and The serving economics of quantization: AWQ vs FP8, KV cache and the VRAM budget cover that, with which GPU generation supports what.
4. Mixed-precision training: lower precision without breaking training
Three techniques
Training in 16 bits is faster and uses less memory, but done naively it can break training. Micikevicius et al.'s 2018 paper "Mixed Precision Training" (ICLR 2018) showed that three techniques let FP16 training match FP32 accuracy:
- Keep a master copy of the weights in FP32. A single update is tiny compared with the weight; added in FP16 it can round away entirely. Updates go to the FP32 weights, and the weights are converted to 16 bits only for computation.
- Scale the loss. Gradients that are too small cannot be represented in FP16 and become zero. The smallest positive FP16 value (a subnormal) is , so anything smaller is lost. The loss is therefore multiplied by, say, before backpropagation so the gradients land in range, and divided back out just before the update.
- Accumulate products in FP32. This is the accumulation precision from the previous section: the sums inside dot products are kept at higher precision.
BF16 has the same exponent as FP32, so the second technique, loss scaling, is usually unnecessary with it.
Implementing it in PyTorch
PyTorch provides this through torch.autocast and torch.amp.GradScaler. Below is a complete example that uses FP16 with loss scaling when CUDA is available and BF16 on the CPU otherwise. It compares the low-precision matrix product with FP32 and uses assert to check the error is within the expected range.
- Requirements: Python 3.9 or later, PyTorch 2.3 or later (
pip install torch;torch.amp.GradScalerwas added in 2.3) - Hardware: runs on a CPU alone; with a CUDA-capable NVIDIA GPU it takes the FP16 path automatically.
"""Mixed-precision matmul and training step (PyTorch 2.3 or later).
- FP16 autocast + GradScaler when CUDA is available, otherwise BF16 autocast on CPU.
- Compares the low-precision result with an FP32 reference and asserts the error is acceptable.
"""
import torch
torch.manual_seed(0)
device = "cuda" if torch.cuda.is_available() else "cpu"
amp_dtype = torch.float16 if device == "cuda" else torch.bfloat16
print(f"torch {torch.__version__} / device={device} / autocast dtype={amp_dtype}")
# --- 1. Run only the matmul in low precision and compare with FP32 ---
M, N, K = 512, 512, 512
a = torch.randn(M, K, device=device)
b = torch.randn(K, N, device=device)
ref = a @ b # FP32 reference
with torch.autocast(device_type=device, dtype=amp_dtype):
low = a @ b # ops on autocast's list (matmul) run in low precision
print("autocast matmul の出力 dtype:", low.dtype)
rel_err = ((low.float() - ref).norm() / ref.norm()).item()
print(f"FP32 に対する相対誤差: {rel_err:.2e}")
# Error comes from rounding the inputs to BF16 and rounding the output to BF16 (BF16 machine epsilon is 2^-7)
assert rel_err < 1e-2, "低精度の誤差が想定より大きい"
# --- 2. One mixed-precision training step ---
model = torch.nn.Sequential(
torch.nn.Linear(256, 1024), torch.nn.GELU(), torch.nn.Linear(1024, 10)
).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
# FP16 gradients underflow easily, so scale the loss. Usually unnecessary with BF16.
scaler = torch.amp.GradScaler(device, enabled=(amp_dtype == torch.float16))
x = torch.randn(64, 256, device=device)
y = torch.randint(0, 10, (64,), device=device)
for step in range(3):
opt.zero_grad(set_to_none=True)
with torch.autocast(device_type=device, dtype=amp_dtype):
logits = model(x) # Linear runs in low precision
loss = torch.nn.functional.cross_entropy(logits, y) # the loss is computed in FP32
scaler.scale(loss).backward()
scaler.step(opt)
scaler.update()
print(f"step {step}: loss={loss.item():.4f} (loss dtype={loss.dtype})")
# The weights (master weights) stay in FP32
print("重みの dtype:", next(model.parameters()).dtype)
assert next(model.parameters()).dtype == torch.float32
The output on my machine (Apple M4 Pro, PyTorch 2.8.0, CPU). The printed labels are the Japanese strings from the code: 出力 = output, 相対誤差 = relative error, 重み = weights.
torch 2.8.0 / device=cpu / autocast dtype=torch.bfloat16
autocast matmul の出力 dtype: torch.bfloat16
FP32 に対する相対誤差: 2.87e-03
step 0: loss=2.3306 (loss dtype=torch.float32)
step 1: loss=1.9683 (loss dtype=torch.float32)
step 2: loss=1.6284 (loss dtype=torch.float32)
重みの dtype: torch.float32
Three things are visible. The matmul output is BF16, with a relative error of about 0.3% against FP32. The loss is computed in FP32. The weights stay in FP32. The CUDA path (FP16 with GradScaler) was not run on this machine.
Two points from PyTorch's documentation: torch.cuda.amp.GradScaler and torch.cuda.amp.autocast are deprecated in favour of torch.amp.GradScaler("cuda") and torch.autocast("cuda"); and autocast should wrap only the forward pass and the loss computation, not the backward pass.
5. Sparsity: where the big spec-sheet numbers come from
The most misunderstood figures on NVIDIA's spec sheets are the ones with sparsity.
2:4 structured sparsity, introduced with the A100, splits the weights into groups of four and requires two in each group to be zero. Because the zeros sit in a regular pattern, the Tensor Core can skip them and double its math throughput at most. Figures marked "with sparsity" on a spec sheet have that 2x built in.
For the H100 SXM5 in FP8, 3,957.8 TFLOPS is the sparsity figure and 1,978.9 TFLOPS the dense one. NVIDIA's Japanese product page marks it 「疎性あり」 ("with sparsity"), and the Blackwell datasheet states in a footnote that the figures are sparse and dense is one half.
That 2x becomes real speed only under conditions:
- The weights have to be pruned to 2:4. The recipe in Mishra et al. (arXiv:2104.08378) is: train normally, prune to 2:4, then train again with the same settings to recover accuracy. Training (or part of its phases) has to be run again.
- Only the matrix-multiplication math doubles. A model's total time includes other operations and memory traffic; the same paper says the speedup depends on matrix size and arithmetic intensity.
- In TF32 mode the pattern becomes 1:2. The conditions change with the precision mode.
Decision rule: when comparing GPUs, first make sure you are comparing dense with dense or sparse with sparse. For ordinary training and inference estimates, dense figures are the safe choice.
6. Memory bandwidth: keeping the compute units fed
Compute units sit idle without data
This is the most important part for understanding NVIDIA GPUs.
However fast a unit is, it can do nothing until its data arrives. Hire dozens of fast cooks and give the kitchen a single loading door, and the kitchen stalls waiting for ingredients. A GPU's loading door is its memory bandwidth.
NVIDIA's data-centre GPUs use HBM (High Bandwidth Memory): memory dies stacked vertically, placed right next to the GPU and connected by a very wide bus. The H100 SXM5's HBM3 is 5,120 bits wide and delivers 3,352 GB/s (about 3.35 TB/s). This HBM is the GPU's memory — its VRAM.
The roofline: is it compute or bandwidth?
The roofline model (Williams, Waterman and Patterson, 2009) bounds how fast work can go:
- : attainable performance [FLOP/s]
- : the compute units' theoretical peak [FLOP/s]
- : memory bandwidth [byte/s]
- : arithmetic intensity (the paper calls it operational intensity) = operations performed per byte of traffic to and from memory (reads and writes) [FLOP/byte]
is the performance you can reach while the data supply keeps up. Low-intensity work hits the ceiling no matter how fast the compute units are.
The point where the two lines meet is the ridge point, . Work below it is bandwidth-bound; work above it is compute-bound.
For dense FP16 on the H100 SXM5's Tensor Cores (theoretical):
To keep an H100's compute units busy, you need to do about 295 operations for every byte read from memory.
In LLM inference, prefill and decode hit different limits
Large language model (LLM) inference has two stages with different character:
- Prefill: process the whole prompt in one pass. Hundreds or thousands of tokens are computed as matrix-times-matrix, so each read of the weights supports a lot of arithmetic. Intensity is high and the work tends to be compute-bound.
- Decode: generate the output one token at a time. Each step multiplies one token's vector by the weight matrices (matrix-times-vector), so reading all the weights buys very little arithmetic.
Estimate decode's intensity. At batch size 1, each FP16 weight read (2 bytes) is used for one multiplication and one addition (2 FLOPs):
That is roughly 1/300 of the 295 ridge point. At batch 1, decode can use only a sliver of an H100's compute, and memory bandwidth sets the speed. The vLLM paper (Kwon et al., SOSP '23) likewise says decode's matrix-vector work leaves GPU compute heavily underused and becomes memory-bound.
Theoretical upper bound: generation speed for a 70B model
That gives an upper bound on decode speed. Approximating each generated token as one full read of the weights (true for a dense model; a mixture-of-experts model, which uses only some of its expert networks, reads only the selected experts' weights):
For a 70-billion-parameter (70B) model on one H100 SXM5 (3.352 TB/s) at batch 1, the theoretical upper bounds are:
| Weight format | Weight size | Upper bound (theoretical) |
|---|---|---|
| FP16 | 140 GB | ≈ 24 tokens/s |
| FP8 | 70 GB | ≈ 48 tokens/s |
| 4-bit | 35 GB | ≈ 96 tokens/s |
This table is an upper bound resting on several assumptions:
- 140 GB of FP16 weights does not fit in an H100's 80 GB, so in practice the model must be split across GPUs. The table simplifies to show the bandwidth relationship.
- It ignores reading the KV cache (below). The longer the context, the further real speed falls below the bound.
- The H100 has no FP4 Tensor Cores. The 4-bit row is the bandwidth bound for weight-only quantization: weights stored in 4 bits and expanded to higher precision after reading.
- Raising the batch size serves several requests from the same read of the weights, so total throughput goes up — at the cost of possibly longer waits per request.
The essential point of the table: in decode, reducing the bytes of weights pays off more directly than faster compute. That, before anything about arithmetic speed, is the main reason quantization speeds up inference.
7. KV cache and FlashAttention: attention's memory problem
How big the KV cache is
So that past tokens' information (keys and values) is not recomputed at every step, a Transformer keeps it in memory: the KV cache. Its size per token is
- 2: one key and one value
- : number of layers
- : number of KV heads
- : dimension per head
- : bytes per element (2 for FP16)
Table 3 of the Llama 3 paper (arXiv:2407.21783) gives the 70B model 80 layers, a model dimension of 8,192, 64 attention heads and 8 KV heads. Each head's dimension is , so in FP16
An 8,192-token context comes to about 2.5 GiB per request (theoretical). With all 64 heads keeping their own keys and values (classic multi-head attention, MHA), it would be eight times that, about 20 GiB. GQA (Grouped-Query Attention, Ainslie et al., 2023), which cuts the number of KV heads, exists to save exactly this memory.
Every concurrent request needs its own KV cache, so how many users an inference server can handle at once is often set by the VRAM left over after the weights. The vLLM paper reports that earlier systems used only 20.4–38.2% of their KV-cache memory for actual token state, and reduced that waste with paging (PagedAttention). How to build a VRAM budget, with worked examples, is in The serving economics of quantization and Self-hosting Llama in production with vLLM.
FlashAttention: fewer round trips to HBM
Attention builds an intermediate matrix (the attention scores) of size for sequence length . Implemented naively, that huge matrix is written to HBM and read back, wasting bandwidth.
FlashAttention (Dao et al., NeurIPS 2022) never writes the matrix to HBM. It splits the computation into small tiles processed in the GPU's fast on-chip memory (SRAM), renormalises the softmax incrementally, and recomputes intermediate results in the backward pass instead of storing them. It actually does more arithmetic, but it is faster because it makes fewer trips to HBM. Crucially, it is exact, not an approximation. The speedups the paper reports (such as 3x over the Hugging Face implementation for GPT-2 training) were measured on the A100; FlashAttention-3, optimised for the H100, is a separate paper.
It is a good illustration that the key to GPU performance is less about arithmetic speed than about moving less data.
8. NVLink and NCCL: the communication that ties GPUs together
Why communication becomes the problem
Training a large model does not fit on one GPU. Training with Adam in mixed precision needs 16 bytes per parameter by the ZeRO paper's accounting (Rajbhandari et al., 2020): 2 bytes of FP16 weights, 2 bytes of FP16 gradients, and 12 bytes of FP32 weights, momentum and variance.
Even a 7B (7-billion-parameter) model needs GB, which does not fit on one 80 GB H100 before activations are even counted. Split eight ways as in ZeRO-3, that becomes 14 GB per GPU (both theoretical). So the data or the model is split across GPUs, and the results are combined by communication.
- Data parallelism: each GPU holds the same model, computes on different data, and the gradients are averaged.
- Tensor parallelism: one matrix multiplication is shared across GPUs. In Megatron-LM's scheme, each Transformer layer needs two all-reduces in the forward pass and two in the backward pass.
- Pipeline parallelism: the layers are divided among GPUs.
- ZeRO: the weights, gradients and optimizer state held by each data-parallel GPU are partitioned to remove duplication.
In every scheme, communication between GPUs turns into waiting time for computation.
Estimating all-reduce time
Averaging gradients across data-parallel GPUs is an all-reduce. The documentation of nccl-tests, the test suite for NVIDIA's NCCL communication library, gives the best-case time for an all-reduce built only from point-to-point sends and receives (ring or tree alike) as
- : number of GPUs
- : size of the data being reduced [byte]
- : the bandwidth one GPU can send and receive at [byte/s] (one direction of the bottleneck link)
Each GPU must send and receive at least of data, and Patarasuk and Yuan (2009) showed this is the lower bound when the additions are done on the GPUs. The NVSwitch of the H100 generation, however, can perform the additions inside the switch (NVIDIA SHARP). That changes the premise, so read the NVLink row in the table below as a guide, not a lower bound.
Watch the definition of each bandwidth figure. The H100's NVLink "900 GB/s" is the sum of both directions over 18 links (the whitepaper says each link carries "25 GB/sec … in each direction"); one direction is 450 GB/s. Likewise PCIe Gen5 x16's "128 GB/s" is both directions summed, 64 GB/s each way.
With that, here is the best-case time to all-reduce a 7B model's gradients (14 GB in BF16) across eight GPUs once (theoretical; ignores latency and overlap with computation):
| Link | Bandwidth per direction | Best-case time per all-reduce (theoretical guide) |
|---|---|---|
| NVLink 4 (H100) | 450 GB/s | ≈ 54 ms |
| PCIe Gen5 x16 | 64 GB/s | ≈ 383 ms |
In training this happens every step. If a step's computation takes a few hundred milliseconds, over PCIe the communication could take as long as the computation or longer. That is the value of NVLink, and of the NVSwitch that extends it across many GPUs. A DGX H100 connects eight H100s through four NVSwitches, and each GPU can talk to the others at 900 GB/s (both directions summed).
Note also that the GB200 NVL72's "130 TB/s" is the sum of each of its 72 GPUs' 1.8 TB/s. It is not the same metric as bisection bandwidth (the bandwidth across a cut that splits the system in half), so take care when using it for comparisons.
9. Across generations: compute grew faster than bandwidth
Here are the four layers in NVIDIA's published figures, generation by generation. All are theoretical peaks, with performance normalised to dense arithmetic.
| Generation (SKU) | Year | FP16 Tensor (dense) | Lowest-precision dense | HBM bandwidth | NVLink (both directions) |
|---|---|---|---|---|---|
| V100 SXM2 (Volta) | 2017 | 125 TFLOPS | FP16: 125 TFLOPS | 900 GB/s | 300 GB/s |
| A100 SXM 80GB (Ampere) | 2020 | 312 TFLOPS | FP16: 312 TFLOPS* | 2,039 GB/s | 600 GB/s |
| H100 SXM5 (Hopper) | 2022 | 989.4 TFLOPS | FP8: 1,978.9 TFLOPS | 3,352 GB/s | 900 GB/s |
| HGX B200 (Blackwell) | 2024 | 2,250 TFLOPS | FP4: 9,000 TFLOPS | 7,700 GB/s | 1,800 GB/s |
* FP16/BF16 is the A100's lowest floating-point precision (integer INT8 excluded). The B200 figures are the Blackwell datasheet's per-GPU HGX B200 values (TDP up to 1,000 W), halving the sparsity figures (FP4: 18 PFLOPS) as the datasheet's footnote instructs. The B200 in a GB200 has different published values (8 TB/s HBM, TDP up to 1,200 W). Years are each architecture's announcement year.
I read two things in this table.
First: lowest-precision compute grew about 72x, while bandwidth grew only about 8.6x. From V100 to B200, lowest-precision dense throughput rose times and HBM bandwidth times. The 72x splits into 18x at the same FP16 and a further 4x from moving to a smaller number format, FP16 to FP4 ().
Second: the ridge point keeps moving right. The FP16 ridge point rose from about 139 FLOP/byte on the V100 to about 295 on the H100. At the lowest precision it is about 590 for the H100 in FP8 and about 1,169 for the B200 in FP4 (all theoretical). In other words, each new generation needs higher arithmetic intensity to keep its compute busy.
As a result, bandwidth-bound work such as decode speeds up less than compute does. Compare the decode bound for a 70B model (FP16, batch 1): about 55 tokens/s on the B200 against about 24 on the H100, about 2.3x (theoretical). Under this approximation the ratio is exactly the bandwidth ratio () and has nothing to do with compute growth.
This means NVIDIA's designs are optimised for training and for large-batch inference. It also means small-batch inference tends to leave a new GPU's compute unused.
On the next generation, Rubin, NVIDIA announced at CES in January 2026 that it was "in full production", with partner availability in the second half of 2026. MLPerf Training v6.1, published on 2026-11-16, included results on Rubin-generation GPUs for the first time. As of this writing (2026-11-17) I have not been able to pin down per-GPU published figures or general customer availability, so Rubin is left out of the table.
10. Reading measured benchmarks: MLPerf
A theoretical peak is only a ceiling. The most widely used way to compare real speed is MLPerf from the industry consortium MLCommons. MLPerf Training measures the wall-clock time to train a specified model to a specified quality. The latest round, MLPerf Training v6.1, was published on 2026-11-16, with 250 results from 24 organisations.
When reading MLPerf results, check:
- Division: the Closed division uses the same model as the reference, for fair comparison; the Open division allows changes to the model.
- Availability: results are categorised as Available (purchasable or rentable in the cloud), Preview (to be submitted as Available by the next round) or RDI (research, development or internal).
- Chip count and system configuration: a result on thousands of GPUs cannot be compared with one on eight.
- Source: a vendor blog claiming "fastest on every benchmark" is not MLCommons' official result. MLCommons' guidelines ask citations to give the result ID, version, division and similar details, and comparisons to disclose differences in category (Available and so on) and chip count. Derived metrics such as performance per watt computed from TDP are not official MLPerf metrics.
11. Software: what puts the four layers to work
None of the four layers is usable without software. Application developers almost never write their own code to dispatch matrix products to Tensor Cores, manage FP8 scales, or overlap GPU-to-GPU communication with computation.
- cuBLAS: the linear-algebra library for matrix multiplication and the like. It picks a Tensor Core implementation to suit the matrix shapes and the GPU generation.
- cuDNN: the library of neural-network operations such as convolution and attention.
- NCCL: the collective-communication library behind section 8's all-reduce, aware of the NVLink and network paths.
- Frameworks such as PyTorch: they call these libraries internally. As with
torch.autocastin section 4, users get low-precision arithmetic from a few lines of configuration.
This is why NVIDIA's strength cannot be described by the compute units alone. On some spec-sheet numbers, other vendors' accelerators exceed NVIDIA's GPUs: AMD's Instinct MI300X, for example, is rated by AMD at 192 GB of HBM3 and 5.3 TB/s of memory bandwidth, larger than the H100 SXM5's 80 GB and 3.35 TB/s. Even so, the speed users actually get depends on whether frameworks and libraries can draw that performance out. Other vendors are building out their software too, and the gap is not fixed.
12. Limits and counterexamples: where GPUs are not strong
Turning the argument around shows the conditions under which GPUs disappoint.
- Little parallel work. With small models, little data or sequential processing, most of a GPU's units sit idle. Moving data between CPU and GPU, or launching kernels (the individual programs that run on the GPU), can cost more than the work itself.
- Work other than matrix multiplication dominates. Preprocessing, data loading, tokenization and branch-heavy post-processing get nothing from Tensor Cores. When adding GPUs does not help, the bottleneck is often outside the GPU.
- Bandwidth-bound work. For low-intensity work such as batch-1 decode, a GPU with more compute is faster only by its bandwidth ratio.
- Low-precision error is unacceptable. FP8 and FP4 can lose too much accuracy for some models and tasks, and evaluating that costs effort.
- Communication becomes the bottleneck. The more GPUs, the larger communication's share, so speed does not scale with GPU count.
- Procurement and operating costs. The published TDP (thermal design power, the power ceiling cooling is designed around) is up to 700 W for the H100 SXM5 and up to 1,200 W for the B200 in a GB200. Power, cooling, procurement lead times and software maintenance all belong in the total cost.
13. Decision guide: what to look at when choosing GPUs
Finally, which metrics to look at when choosing a GPU or GPU cloud:
| Workload | Likely bottleneck | Metrics to check |
|---|---|---|
| Large-scale training | Compute, GPU-to-GPU communication | Dense compute (at the precision you use), NVLink bandwidth and how many GPUs it connects |
| Inference prefill (long inputs) | Compute | Dense compute |
| Inference decode (long outputs, small batches) | Memory bandwidth and capacity | HBM bandwidth, VRAM capacity (weights + KV cache) |
| Inference on small models | Outside the GPU (preprocessing, transfers) | First measure whether you need a GPU at all |
The same reasoning applies beyond the H100 and B200, to the inference GPUs that are easy to rent in the cloud, such as the T4, L4 and L40S. The T4 (Turing generation), for example, has no FP8 or BF16 Tensor Cores, so the generation of GPU also decides which number formats and quantization methods you can use.
For buyers and decision-makers, I suggest checking in this order:
- Measure the shape of the work. Training or inference? For inference, what is the distribution of input and output token counts, and how many requests must run at once?
- Find the bottleneck. Estimate, with the roofline in mind, whether that work stops at compute, bandwidth, capacity, communication or something outside the GPU — and measure at small scale if you can.
- Compare on the metric that addresses that bottleneck. Match sparsity, precision and SKU before comparing.
- Decide on total cost. Compare cost per token or per training run, not the hourly GPU price. For the comparison with APIs, with formulas, see Designing Llama inference cost: the API vs self-hosting break-even and Generative AI cost and break-even.
The production work I have done is inference infrastructure at the scale of vLLM serving an AWQ-quantized Qwen3-8B on a single spot Tesla T4 (the GPU pipeline behind AI video localization). I have not run large H100 or B200 clusters. Even so, the bandwidth and capacity constraints covered here shaped that design at the scale of a single T4: running several jobs at once would exhaust the VRAM, so GPU work is executed one job at a time, in series. Whatever the scale, the principle is the same: identify the bottleneck before you invest.
Summary
NVIDIA GPUs are strong at AI for more reasons than having many compute units.
- Most AI arithmetic is matrix multiplication, which Tensor Cores handle in dedicated circuits. The H100's FP16 Tensor Cores have about 15 times the theoretical peak of its FP32 CUDA cores.
- Small number formats help the unit count, memory footprint and bandwidth all at once — but input, accumulation and storage precision have to be kept apart, and FP8 and FP4 presuppose scaling and evaluation.
- Many of the big spec-sheet numbers are with sparsity; dense figures are half.
- In many cases memory bandwidth sets real speed. The H100's ridge point is about 295 FLOP/byte and batch-1 decode's intensity about 1 FLOP/byte — a gap of about 300x.
- Across generations, compute grew far faster than bandwidth, so bandwidth-bound work gains least from a new GPU.
- In large-scale training, communication between GPUs matters as much as computation, and NVLink and NCCL carry it.
Whether these layers are actually used to the full comes down to software such as cuBLAS, cuDNN, NCCL and PyTorch. Theoretical hardware peaks and mature software, together, are what NVIDIA's strength consists of today.
This series digs into these mechanisms from other angles too. The design differences from CPUs are in How CPUs and GPUs really differ; GPU programming in How CUDA works; the comparison with TPUs and cost per token in CPU vs GPU vs TPU; and the competitive landscape and future scenarios in NVIDIA's moat is more than a CUDA monopoly. For the company's history, see A history of NVIDIA; for export controls, The US–China AI chip war and export controls.