Skip to main content
NVIDIA, GPUs & AI chips
アーキテクチャ設計
パフォーマンス
インフラ
AI
技術選定

How CPUs and GPUs really differ — pipelines, out-of-order execution, SIMT and the Roofline model, from formulas and measurements

CPU vs GPU from first principles: out-of-order execution versus SIMT and warps, with Roofline and PCIe math showing when moving work to a GPU pays off.

Published
Reading time
45 min read
Author
友田 陽大
Share
Contents

People often say GPUs are faster than CPUs. Why are they faster — and why aren't they always?

"A CPU has dozens of cores and a GPU has thousands" is only half right, and the other half is misleading, because a CPU core and a CUDA core are not the same unit of counting. This article explains the difference by asking what each processor spends its circuitry on.

The short answer: a CPU spends its circuitry on finishing a few instruction streams, one at a time, as fast as possible. It guesses where a branch will go and runs ahead (branch prediction and speculative execution), and it reorders instructions to fill idle execution units (out-of-order execution). A GPU has none of this. It keeps thousands of threads ready and, while one thread waits on memory, it runs another, so the waiting itself is hidden. The first approach is called latency-oriented, the second throughput-oriented.

TermIn one lineSection
Out-of-order executionA CPU mechanism that runs independent instructions ahead of program order3
Branch prediction / speculative executionGuessing a branch's outcome and running the following instructions before it resolves4
Warp / SIMTThe GPU's way of running the same instruction across a group of 32 threads7–8
RooflineA model of whether compute or memory bandwidth caps performance11

In the second half, the article combines the Roofline model, Amdahl's law and PCIe transfer time to estimate, in theory, whether a given workload is worth moving to a GPU.

The numbers here are vendor specifications or theoretical values I computed from them. The only thing I actually ran on my machine (an Apple M4 Pro) is a C++ program that exercises CPU branch prediction and instruction-level parallelism. The GPU code is written but has not been run yet. My hands-on GPU experience is at the scale of an inference service running vLLM with Qwen3-8B-AWQ on a single Tesla T4.

If you read this from a management or procurement seat, start with the short answer and Sections 11, 12 and 15. There are three takeaways: (1) don't choose hardware by comparing core counts (Section 11); (2) light work whose data sits on the CPU side can get slower when moved to a GPU (Section 12); (3) when someone proposes "100x with GPUs," check the conditions of the comparison (Section 13).

The parts of NVIDIA's AI advantage that come from Tensor Cores, low-precision number formats, HBM and its effect on LLM inference, and NVLink are covered in Why NVIDIA GPUs are so strong at AI. This article stays one step earlier: the basic design difference between CPUs and GPUs.

1. The basic equation for run time: T = IC × CPI / f

First, one distinction. The ISA (instruction set architecture) is the contract with software — which instructions exist and what they do; x86-64 and Armv8/Armv9 are ISAs. The microarchitecture is how a circuit implements that ISA. Intel's optimization manual describes several microarchitectures, such as Golden Cove and Redwood Cove, that implement the same Intel 64 ISA, which is why the same program runs on CPUs of different generations and only its speed changes. Almost everything below is about microarchitecture.

A program's run time is the product of three numbers.

T=IC×CPIfT = \frac{IC \times CPI}{f}
  • TT: run time [s]
  • ICIC: number of instructions executed (instruction count) [instructions]
  • CPICPI: average clock cycles per instruction [cycles/instruction]
  • ff: clock frequency [cycles/s = Hz]

For example, a program of one billion instructions on a 4 GHz CPU with a CPI of 0.5 (two instructions per cycle on average) takes the following (illustrative numbers):

T=109×0.54×109=0.125 sT = \frac{10^9 \times 0.5}{4 \times 10^9} = 0.125\ \mathrm{s}

There are only three ways to go faster: execute fewer instructions (better algorithms, SIMD instructions), lower the CPI (handle more instructions per cycle), or raise the frequency.

Frequency tends to hit a ceiling because of power and heat, so it can't be relied on alone. It helps to view CPUs and GPUs as two different allocations of circuitry between lowering CPI and adding cores and threads.

Do you actually know whether self-hosting beats the API?

Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing

2. Pipelining: turning instructions into an assembly line

A three-stage assembly line

A CPU processes each instruction in stages. The simplest split has three:

  1. Fetch: read the next instruction from memory (in practice, the instruction cache)
  2. Decode: work out what the instruction does and which registers it uses
  3. Execute: compute it in an execution unit

While instruction 1 executes, instruction 2 is decoded and instruction 3 is fetched — like a household with one washer and one dryer that loads the second wash as soon as the first goes into the dryer. This is a pipeline. Ideally one instruction completes per cycle and the CPI approaches 1.

The three hazards that stall a pipeline

The assembly line only flows when consecutive instructions are independent. Anything that stops it is a hazard, and there are three kinds:

  • Structural hazard: two instructions want the same execution unit or memory port at once
  • Data hazard: an instruction needs the result of an earlier one
  • Control hazard: until a branch resolves, the CPU doesn't know which instruction to fetch next

Data hazards split into three more. Consider this sequence (r1 and so on are registers):

(1) r1 = r2 + r3
(2) r4 = r1 * 2      ← reads the r1 written by (1)
(3) r2 = r5 - 1      ← overwrites the r2 that (1) reads
(4) r1 = r6 + r7     ← overwrites the r1 that (1) writes
TypeExampleMeaningReal dependency?
RAW (Read After Write)(1)→(2)read a value after it is writtenReal. Nothing to do but wait for the value
WAR (Write After Read)(1)→(3)must not overwrite before the readOnly a name clash
WAW (Write After Write)(1)→(4)writes out of order change the final valueOnly a name clash

This is the key to what follows. RAW is a dependency in the computation itself; WAR and WAW are apparent dependencies created by reusing a register name. Rename the registers, and WAR and WAW disappear.

3. Superscalar issue and out-of-order execution

Superscalar: several instructions per cycle

To get the CPI below 1, fetch, decode and execute several instructions per cycle. A design with several execution units that run several instructions at once is called superscalar.

As a concrete example, Intel's optimization manual lists these changes in the Golden Cove microarchitecture (the core used in Sapphire Rapids and others) over the previous generation: decoders increased from four to six, allowing up to six instructions to be decoded per cycle; execution ports from 10 to 12; and retirement width from 4 to 8. These are ceilings; how many instructions actually progress together depends on the dependencies between them. How many instructions in one stream can run at the same time is its instruction-level parallelism (ILP).

Register renaming: removing apparent dependencies

The biggest obstacle to ILP is the WAR and WAW hazards from Section 2. The number of register names a program can use (architectural registers) is fixed by the ISA, and compilers reuse those few names.

So the CPU keeps far more physical registers than architectural ones, and every time it decodes an instruction it assigns a fresh physical register as the destination. If the r1 written by (4) gets a different physical register, (1) and (4) no longer need to wait for each other. This is register renaming.

Tomasulo's algorithm: the 1967 prototype

The prototype of this idea is "An Efficient Algorithm for Exploiting Multiple Arithmetic Units," published by IBM's R. M. Tomasulo in 1967. It describes the floating-point unit of the IBM System/360 Model 91, and it has three key pieces:

  1. Reservation stations: instructions wait in front of the execution units until their inputs are ready.
  2. Tags: a value still being computed is represented by a tag saying which unit will produce it.
  3. Common data bus (CDB): a unit broadcasts its result on the bus with its tag, and everyone waiting for that tag receives it at once.

The paper's abstract says this lets independent instructions execute at the same time while preserving the essential precedences in the instruction stream. Renaming values by tag does the same job as modern register renaming.

The reorder buffer: committing in order

Executing out of order makes it unclear, when an exception (division by zero, for example) occurs, which instructions have completed. And when speculation (Section 4) turns out wrong, results computed on the wrong path must be undone. That is why modern CPUs have a reorder buffer (ROB). Instructions enter the ROB in program order and execute in any order, but their results are committed (retired) strictly in order from the head of the ROB. This preserves a state in which, at the point of an exception, every earlier instruction has completed and no later one has been committed — a precise exception. The idea was laid out in work such as Smith and Pleszkun's 1988 paper.

Fetch → Decode → Rename (assign physical registers, enter the ROB in order)
  → Wait for inputs in reservation stations (issue out of order)
  → Execute (out of order)
  → Commit in order from the head of the ROB (retire)

Golden Cove's "retirement width from 4 to 8" refers to that last stage.

The limit of ILP, and SMT

ILP has a limit. If a stream contains a chain of RAW dependencies, no number of execution units can finish faster than the chain is long. So CPUs also use thread-level parallelism (TLP), running separate instruction streams (threads) on multiple cores. With SMT, which runs two threads on one core at once (Intel's implementation is Hyper-Threading), Intel's manual explains that the two logical processors each keep a full set of architectural registers while sharing the execution units and caches; if one stalls on a cache miss, the other can still make progress. "Fill the wait with another thread" is the same idea as the GPU in Section 7, and the difference is scale: SMT on a CPU means about two threads, while a GPU puts dozens of warps (thousands of threads) on one SM.

4. Branch prediction and speculative execution

You can't stop at every branch

A branch resolves in the execute stage; if the CPU couldn't fetch the next instruction until then, the pipeline would sit empty in the meantime. The deeper and wider the pipeline, the more work is lost.

So the CPU predicts the branch outcome and fetches and executes the predicted path ahead of time. This is speculative execution. If the guess is right, the wait is zero. If it is wrong, the CPU throws away every instruction on the wrong path (the ROB makes uncommitted results undoable) and restarts from the right one. That restart cost is the branch misprediction penalty.

A branch predictor guesses from the history of past outcomes, so regular branches, like a loop's exit test, are almost always right, while branches whose outcome changes randomly with the data miss.

The option of removing the branch

Intel's manual lists "eliminate branches whenever possible" first among its branch optimizations, and shows how to replace branches with conditional moves (CMOV) or SETCC. But it attaches clear conditions to that advice:

  • Use it only for unpredictable branches, not predictable ones.
  • CMOV evaluates both paths, so it costs extra execution.
  • Turning a branch into CMOV trades control dependence for data dependence and restricts the out-of-order engine.
  • Branch prediction rates on Intel processors are usually very high; consistently mispredicted branches are rare.

In other words, remove a branch only when the cost of mispredicting is larger than the cost of computing both sides. Compilers make this call automatically, and the experiment in Section 5 shows it happening.

GPUs don't have this machinery

This is the first big difference from GPUs. The CUDA Programming Guide says of the GPU's processing block (the SM):

Unlike CPU cores, SMs issue instructions in order and do not perform branch prediction or speculative execution.

A GPU's SM issues instructions in order and does neither branch prediction nor speculative execution. The area and power a CPU spends on the circuitry described in Section 3 and this section, a GPU spends on something else. Section 7 covers what.

5. Trying it locally: the cost of branch mispredictions and the value of ILP

The C++20 program below runs two experiments:

  • Experiment A (branch prediction): from an array of random values in 0–255, sum only the values of 128 or more, and compare sorted with unsorted data. Once sorted, the branch outcome is "false for the first half, true for the second," so it is predicted almost perfectly.
  • Experiment B (ILP): sum a 16 KiB array (it fits in L1 cache) many times over, splitting the running sum across 1, 2, 4 and 8 accumulator variables.

Prerequisites:

  • Language: C++20, standard library only.
  • Compiler: Clang or GCC. One spot uses asm volatile, a GCC/Clang extension, to keep a branch in place; it will not build with MSVC.
  • Environment: I ran it on an Apple M4 Pro (arm64, macOS 27.0.1, Apple clang 21.0.0). It has not been tested on x86-64 Linux.
  • Build and run: clang++ -std=c++20 -O2 cpu_pipeline_demo.cpp -o cpu_pipeline_demo && ./cpu_pipeline_demo
// cpu_pipeline_demo.cpp — a C++20 program to feel branch prediction and instruction-level parallelism (ILP)
// Build: clang++ -std=c++20 -O2 cpu_pipeline_demo.cpp -o cpu_pipeline_demo
// Run:   ./cpu_pipeline_demo
// Note:  asm volatile is a GCC/Clang extension (not available in MSVC).
#include <algorithm>
#include <chrono>
#include <cstdint>
#include <cstdio>
#include <random>
#include <vector>

namespace {

using Clock = std::chrono::steady_clock;

// Time f reps times and return the median nanoseconds per element
template <class F>
double median_ns_per_elem(F&& f, std::size_t elems, int reps = 7) {
  std::vector<double> t;
  for (int r = 0; r < reps; ++r) {
    const auto t0 = Clock::now();
    f();
    const auto t1 = Clock::now();
    t.push_back(std::chrono::duration<double, std::nano>(t1 - t0).count() / elems);
  }
  std::sort(t.begin(), t.end());
  return t[t.size() / 2];
}

// (A1) The straightforward version. Clang at -O2 vectorizes it and removes the branch
long long sum_naive(const std::vector<std::uint8_t>& d) {
  long long s = 0;
  for (std::uint8_t v : d)
    if (v >= 128) s += v;
  return s;
}

// (A2) A version that keeps the branch. An empty asm volatile cannot be executed speculatively,
//      so the compiler cannot turn the if into a conditional select (csel/cmov) or vectorize it
long long sum_branchy(const std::vector<std::uint8_t>& d) {
  long long s = 0;
  for (std::uint8_t v : d) {
    if (v >= 128) {
      asm volatile("" ::: "memory");
      s += v;
    }
  }
  return s;
}

// (B) Floating-point sum computed with K independent accumulators.
//     Without -ffast-math, the compiler may not reorder the additions on its own
template <int K>
double sum_k_accumulators(const std::vector<double>& a, int repeat) {
  double acc[K] = {};
  for (int r = 0; r < repeat; ++r)
    for (std::size_t i = 0; i < a.size(); i += K)
      for (int k = 0; k < K; ++k) acc[k] += a[i + k];
  double s = 0.0;
  for (int k = 0; k < K; ++k) s += acc[k];
  return s;
}

}  // namespace

int main() {
  // ---- (A) Branch prediction ----
  constexpr std::size_t N = 1u << 24;  // 16,777,216 elements (16 MiB)
  std::mt19937 rng(42);
  std::uniform_int_distribution<int> dist(0, 255);
  std::vector<std::uint8_t> unsorted(N);
  for (auto& v : unsorted) v = static_cast<std::uint8_t>(dist(rng));
  std::vector<std::uint8_t> sorted = unsorted;
  std::sort(sorted.begin(), sorted.end());

  long long check = 0;
  std::printf("[A] branch prediction: N = %zu bytes, values uniform in [0,255]\n", N);
  struct Case { const char* name; long long (*fn)(const std::vector<std::uint8_t>&); };
  const Case cases[] = {{"naive  (compiler removed branch)", sum_naive},
                        {"branchy (branch kept)           ", sum_branchy}};
  for (const auto& c : cases) {
    long long r1 = 0, r2 = 0;
    const double tu = median_ns_per_elem([&] { r1 = c.fn(unsorted); }, N);
    const double ts = median_ns_per_elem([&] { r2 = c.fn(sorted); }, N);
    if (r1 != r2) { std::printf("result mismatch!\n"); return 1; }
    check += r1;
    std::printf("  %s unsorted %.3f ns/elem | sorted %.3f ns/elem | ratio %.1fx\n",
                c.name, tu, ts, tu / ts);
  }

  // ---- (B) Instruction-level parallelism ----
  constexpr std::size_t M = 2048;  // 2048 doubles = 16 KiB (fits in the L1 data cache)
  constexpr int R = 20000;         // 40,960,000 additions in total
  std::vector<double> a(M);
  for (std::size_t i = 0; i < M; ++i) a[i] = 1.0 / static_cast<double>(i + 1);
  const std::size_t adds = M * static_cast<std::size_t>(R);

  std::printf("[B] ILP: %zu FP64 adds over a 16 KiB array\n", adds);
  double s1 = 0, s2 = 0, s4 = 0, s8 = 0;
  const double t1 = median_ns_per_elem([&] { s1 = sum_k_accumulators<1>(a, R); }, adds, 5);
  const double t2 = median_ns_per_elem([&] { s2 = sum_k_accumulators<2>(a, R); }, adds, 5);
  const double t4 = median_ns_per_elem([&] { s4 = sum_k_accumulators<4>(a, R); }, adds, 5);
  const double t8 = median_ns_per_elem([&] { s8 = sum_k_accumulators<8>(a, R); }, adds, 5);
  std::printf("  1 accumulator : %.3f ns/add (1.0x)\n", t1);
  std::printf("  2 accumulators: %.3f ns/add (%.1fx)\n", t2, t1 / t2);
  std::printf("  4 accumulators: %.3f ns/add (%.1fx)\n", t4, t1 / t4);
  std::printf("  8 accumulators: %.3f ns/add (%.1fx)\n", t8, t1 / t8);
  // The addition order differs, so the last digits don't match (FP addition is not associative)
  std::printf("  sums: %.12f %.12f %.12f %.12f\n", s1, s2, s4, s8);
  std::printf("checksum %lld\n", check);
  return 0;
}

One run of the output on my machine (Apple M4 Pro, 14 cores = 10 performance + 4 efficiency, 48 GB memory):

[A] branch prediction: N = 16777216 bytes, values uniform in [0,255]
  naive  (compiler removed branch) unsorted 0.053 ns/elem | sorted 0.053 ns/elem | ratio 1.0x
  branchy (branch kept)            unsorted 3.025 ns/elem | sorted 0.276 ns/elem | ratio 11.0x
[B] ILP: 40960000 FP64 adds over a 16 KiB array
  1 accumulator : 0.554 ns/add (1.0x)
  2 accumulators: 0.301 ns/add (1.8x)
  4 accumulators: 0.154 ns/add (3.6x)
  8 accumulators: 0.067 ns/add (8.2x)
  sums: 164041.575438243395 164041.575436810090 164041.575437169959 164041.575436713174
checksum 3212822356

Across six runs of the same program, the unsorted/sorted ratio for the branch-keeping version ranged from 10.4x to 11.0x, and the speed-up with eight accumulators from 6.5x to 8.3x. The sums and checksum lines were identical every time.

Reading the results

Experiment A. The straightforward sum_naive ran at the same speed whether or not the data was sorted. The generated assembly shows that Clang vectorized the loop and removed the data-dependent branch — the "remove unpredictable branches" advice from Section 4, applied automatically by the compiler. Note, though, that the "compiler removed branch" label in the output reflects what I observed in my environment (Apple clang, arm64); other compilers or x86-64 may keep the branch or replace it with CMOV. In sum_branchy, where the branch cannot be removed, unsorted data was about 11x slower. Its assembly keeps a conditional branch instruction (tbz) for every element.

The difference is about 2.7 ns per element (3.025 − 0.276). The values are spread uniformly over 0–255, so on unsorted data the branch outcome is essentially a coin flip, and the predictor should miss about half the time no matter how clever it is. Under that assumption, each misprediction costs roughly 5.5 ns (2.749 ÷ 0.5). This is an estimate that rests on that assumption. Apple publishes neither the misprediction penalty in cycles nor the clock frequency under load, so I don't convert it to cycles.

Experiment B. With a single accumulator, each acc += a[i] waits for the previous addition — a chain of RAW dependencies — so the adder's latency sets the speed. Splitting the sum creates more independent chains. The adders are pipelined and there are several of them, so independent chains can flow through in parallel (out-of-order execution helps with this at run time). Indeed, the compiler interleaves the independent fadd instructions, and the speed rose almost in proportion to the number of chains. All four versions use the scalar fadd instruction (not NEON vector instructions), so the speed-up comes from ILP, not SIMD.

The last digits of sums differ between versions because floating-point addition is not associative. That is also why the compiler doesn't split the accumulator on its own: the result would change (an option like -ffast-math would allow it).

Limits of this experiment

  • My machine is a shared environment with about ten sessions running at once; the load average during measurement was about 20. The numbers show a trend, not representative M4 Pro performance, and I did not control whether the code ran on performance or efficiency cores.
  • Keeping the branch with asm volatile is an artificial manipulation for the experiment.

6. SIMD: one instruction, many data elements

On top of ILP and TLP, CPUs also extract data-level parallelism (applying the same operation to many data elements) through SIMD (single instruction, multiple data) instructions. One instruction processes several values packed into a wide register.

  • x86 AVX family: AVX/AVX2 use 256-bit registers, AVX-512 uses 512-bit ones. A 512-bit register holds 8 FP64 values or 16 FP32 values.
  • Arm Neon (Advanced SIMD): Arm's ACLE (Arm C Language Extensions) describes it as a 64-bit/128-bit SIMD instruction set defined as part of the Arm architecture. A 128-bit register holds four FP32 values (type float32x4_t).

SIMD has a large effect on a CPU's theoretical peak. A CPU's FP64 peak can be counted as:

Ppeak=(cores)×f×(FMA units)×(lanes)×2P_{\mathrm{peak}} = (\text{cores}) \times f \times (\text{FMA units}) \times (\text{lanes}) \times 2

The final 2 counts an FMA (multiply and add in one instruction) as 2 FLOP.

As a concrete example, take the Intel Xeon Platinum 8480+ (Sapphire Rapids, launched Q1 2023). According to Intel's product specification page, it has 56 cores, a 2.00 GHz base frequency, a 3.80 GHz max turbo frequency and two AVX-512 FMA units. Intel's optimization manual likewise says that server parts with this core (Golden Cove) can execute 512-bit FMAs on two ports. AVX-512 FP64 has 8 lanes, so the theoretical peak for one socket at the base frequency is:

56×2.0×109×2×8×2≈3.58×1012 FLOP/s56 \times 2.0 \times 10^9 \times 2 \times 8 \times 2 \approx 3.58 \times 10^{12}\ \mathrm{FLOP/s}

At the max turbo frequency it is about 6.81 TFLOP/s. FP32 lanes are half as wide, so FP32 is twice each figure: about 7.17 and 13.62 TFLOP/s.

Here the clock basis matters. The H100 whitepaper computes its FP32 and FP64 peaks at the GPU Boost Clock (1,980 MHz), and a footnote says that for data center GPUs the peak clock and boost clock are synonymous. So the GPU figures are "values at the peak clock." A CPU figure computed at the base frequency uses a lower basis; the two are not the same kind of number.

Should the CPU then be computed at max turbo? The frequency at which all cores run while using AVX-512 is not on the product specification page. Intel's manual explains, for Skylake Server and similar generations, that code heavy in AVX-512 may run under a lower maximum frequency (that text is not about Sapphire Rapids). So this article computes with two assumptions, base and max turbo, as a range.

The 8480+ also has AMX for matrix operations ("Yes" on the spec page), but, like the GPU's Tensor Cores, it is left out of the comparison.

7. The GPU philosophy: hide the wait

A CPU reduces the wait; a GPU hides it

Everything the CPU does above serves to reduce the waiting time of a single instruction stream: predict branches instead of waiting, reorder instructions to fill gaps, keep data in large caches instead of waiting on memory. A GPU takes the opposite view. It doesn't need to reduce waiting. Instead, it does other work while waiting.

According to the CUDA C++ Best Practices Guide, CPU threads are "heavyweight" and OS context switches are slow and expensive, while GPU threads are extremely lightweight: every active thread has its own registers, so switching needs no register save and restore. The guide sums up that CPU cores are designed to minimize latency for a small number of threads, while GPUs are designed to maximize throughput with a large number of lightweight threads.

In the guide's example, a server with two 32-core CPUs can run 64 threads concurrently (or a small multiple of that with SMT), while a modern NVIDIA GPU can keep up to 2,048 threads active per SM, which with 80 SMs comes to more than 160,000. The guide also says good performance generally requires thousands or tens of thousands of concurrent threads.

SMs and warp schedulers

The GPU's basic compute unit is the SM (streaming multiprocessor). The H100 SXM5 has 132 SMs (per NVIDIA's whitepaper).

The GPU groups the threads a program launches into sets of 32 called warps. Each cycle, a warp scheduler inside the SM picks one warp that is ready to run its next instruction and issues that instruction. The CUDA Programming Guide explains that each warp's execution context (program counter, registers) stays on-chip for the warp's whole lifetime, so switching between warps costs nothing. According to the Volta (V100) whitepaper, the SM is split into four processing blocks, each with its own warp scheduler.

8. SIMT and warp divergence

SIMT: write the code for one thread

NVIDIA calls the GPU's execution model SIMT (single instruction, multiple threads). According to the CUDA Programming Guide, a warp executes one common instruction at a time. Unlike SIMD, where the vector width is visible to software, a SIMT instruction specifies the behaviour of a single thread. For correctness you can essentially ignore SIMT behaviour, but keeping the threads of a warp on the same path improves performance a lot.

With SIMD, the programmer (or compiler) writes code that "processes 8 elements at once" and has to handle branches with masks. With SIMT you write the code for one thread, and running 32 of them together is the hardware's job.

Warp divergence: a split branch runs both sides

What happens when threads in a warp take different paths? According to the CUDA Programming Guide, when a data-dependent branch splits a warp, the warp executes each path taken, one after another, disabling the threads that are not on that path. This is warp divergence. Branches split only within a warp; different warps run independently.

if (threadIdx.x % 2 == 0) { A(); } else { B(); }

32 threads in a warp:  even → A, odd → B
What happens:  [run A, odd threads disabled] → [run B, even threads disabled]
Time ≈ time of A + time of B

Splitting by even/odd splits every warp. Splitting by warp number ((threadIdx.x / 32) % 2) sends every thread in a warp down the same path, so no warp divergence occurs. The total work is the same, yet the speed changes with nothing but how you split.

Note that the cost behaves differently from a CPU branch. On a CPU, a correctly predicted branch is almost free and a mispredicted one costs a pipeline restart. On a GPU there is no prediction; whether the warp splits is what determines the cost.

Independent Thread Scheduling (Volta and later)

Before compute capability 7.0 (Volta), the 32 threads of a warp shared one program counter, so code in which threads of the same warp waited on each other through a lock could deadlock. From Volta on, Independent Thread Scheduling gives each thread its own program counter and call stack, and threads can diverge and reconverge at finer than warp granularity. In exchange, older code that assumed the threads of a warp always move in lockstep may break and needs explicit synchronization with __syncwarp().

A common misconception lives here. Independent Thread Scheduling does not remove the cost of warp divergence. The Volta whitepaper states that execution is still SIMT: at any given clock cycle, the cores execute the same instruction for all active threads in a warp. When paths split, both are still run in turn. What changed is the range of programs you can write correctly.

9. Occupancy and the competition for registers

Occupancy: how many warps fit on an SM

The CUDA Programming Guide defines occupancy as the ratio of active warps to the maximum number of active warps the SM supports. On the H100 (compute capability 9.0), the maximum is 64 warps (2,048 threads) per SM.

Occupancy is set by the resources each thread uses. An SM has 64K (65,536) 32-bit registers, shared among the active threads. A simplified calculation gives the following (theoretical values that ignore finer constraints such as register allocation granularity):

Registers per threadThreads that fit on an SMOccupancy
3265,536 / 32 = 2,048100%
6465,536 / 64 = 1,02450%
12865,536 / 128 = 51225%

Shared memory usage (Section 10) and the number of threads per block are also constraints.

This shows a characteristic of GPU design. The H100 SXM5's register file is 256 KB per SM and 33,792 KB across the GPU (per the whitepaper) — the same order of magnitude as its 50 MB L2 cache. A GPU devotes a large share of its on-chip storage to registers so that it can hold the state of many threads at once. That contrasts with a CPU, which spends die area on large caches and branch predictors.

Register spilling

A thread can use at most 255 registers. If it needs more variables than that, or if you cap the register count with the -maxrregcount option, the compiler pushes some values out to local memory. This is register spilling.

Despite its name, local memory lives in off-chip device memory and has the same latency and bandwidth as global memory. nvcc's --resource-usage option shows how many registers a kernel uses.

Higher occupancy is not always better

The Best Practices Guide says that raising occupancy from 66% to 100% generally does not bring a matching speed-up. Lower occupancy gives each thread more registers and less spilling, and with enough instruction-level parallelism, latency can sometimes be hidden even at low occupancy.

So on a GPU too, ILP (parallelism within one thread) and the number of warps trade off against each other and compete for registers. The trick from Experiment B in Section 5 — adding independent accumulators — works in GPU kernels as well.

10. The memory hierarchy: caches, HBM, coalescing and bank conflicts

CPU and GPU memory hierarchies

Both processors have a hierarchy from small, fast storage to large, slow storage.

LevelCPU (examples)GPU (H100 SXM5)
Level-1 cachePer core (the M4 Pro performance core's L1 data cache is 128 KiB, sysctl on my machine)256 KB per SM for L1 and shared memory combined (shared memory up to 228 KB)
Level-2/3 cacheXeon 8480+: "Cache 105 MB" on the spec page, plus a 2 MB level-2 cache per core (Intel's manual)L2 50 MB
Main memoryDDR5 (the 8480+ has 8 channels, DDR5-4800)HBM3, 3,352 GB/s

A CPU's theoretical memory bandwidth is channels × transfer rate × 8 bytes. For one 8480+ socket (theoretical):

8×4800×106×8=307.2×109 byte/s8 \times 4800 \times 10^6 \times 8 = 307.2 \times 10^9\ \mathrm{byte/s}

Apple's announcement gives the M4 Pro 273 GB/s of memory bandwidth. The H100 SXM5's HBM3 figure of 3,352 GB/s is from the whitepaper's table (which carries a "Not Finalized for H100" note; NVIDIA's current product page also lists 3.35 TB/s). That is about 10.9x one 8480+ socket (a ratio of theoretical values). Why HBM can offer so much bandwidth, and how that plays out in LLM inference, is covered in Section 6 of Why NVIDIA GPUs are so strong at AI.

Memory coalescing: reading per warp

According to the CUDA Programming Guide, global memory is read and written in 32-byte transactions. If the 32 threads of a warp each read consecutive 4-byte words, the warp reads 128 bytes in four transactions, at 100% efficiency. If neighbouring threads read addresses 32 bytes or more apart, each thread needs its own 32-byte transaction: 1,024 bytes move to deliver 128 used bytes. Traffic is 8x what is needed, and efficiency is 12.5% (87.5% wasted).

This is memory coalescing. It resembles laying out data with the CPU's cache line (128 bytes on the M4 Pro) in mind, but on a GPU what matters is the pattern of addresses that 32 threads read at the same moment.

The short kernel below has neighbouring threads copy locations stride elements apart. At stride 1 the accesses coalesce; at 2 half the transfer is wasted; at 8 or more (4 bytes × 8 = 32 bytes) each thread needs a separate transaction. The Best Practices Guide uses the same kind of example and states that a stride of 2 gives 50% load/store efficiency.

// Not executed (my environment has no NVIDIA GPU and no nvcc).
// The index is computed in size_t (int would overflow once elements × stride exceeds 2^31)
__global__ void stride_copy(float* out, const float* in, int stride, size_t n) {
  const size_t i = (static_cast<size_t>(blockIdx.x) * blockDim.x + threadIdx.x) * stride;
  if (i < n) out[i] = in[i];
}

This kernel, and another that measures warp divergence, have not been run yet, because my environment has no NVIDIA GPU. This article contains no GPU measurements.

Shared memory and bank conflicts

Inside each SM is shared memory, a fast cache the programmer manages explicitly.

According to the CUDA Programming Guide, shared memory is divided into 32 banks, with successive 32-bit words assigned to successive banks; each bank can deliver 32 bits per cycle. When several threads of the same warp read different words in the same bank, the reads are serialized. This is a bank conflict. When they read the same word, one read is broadcast to all of them, so there is no conflict.

bank = (index of the 32-bit word) mod 32

stride 1  (words 0,1,2,...,31)       → one per bank across 32 banks: no conflict
stride 2  (words 0,2,4,...,62)       → two per even-numbered bank: 2-way conflict
stride 32 (a column of a 32x32 array) → everyone in bank 0: 32-way conflict

The Best Practices Guide shows a fix: allocate a 32×32 array with one extra column (padding), like [32][33]. Reading down a column then has a stride of 33, which leaves a remainder of 1 modulo 32, and the conflict disappears.

11. Comparing with the Roofline model: CPU and H100 ridge points (original analysis)

The Roofline model

A tool that pulls all of this into one picture is the Roofline model, proposed by Williams, Waterman and Patterson. It bounds achievable performance PP using the theoretical compute peak PpeakP_{\mathrm{peak}}, memory bandwidth BB and operational intensity II (operations per byte of DRAM traffic):

P≤min⁡(Ppeak, B×I)P \le \min\left(P_{\mathrm{peak}},\ B \times I\right)

The point where the two bounds meet is the ridge point I∗=Ppeak/BI^{*} = P_{\mathrm{peak}} / B; work with lower intensity is bandwidth-bound, work with higher intensity is compute-bound. The definition is covered in detail in Section 6 of the pillar article.

As a worked example, take SAXPY (y←ax+yy \leftarrow ax + y, FP32). Per element it does 2 FLOP and moves 12 bytes (read xx and yy, write yy).

ISAXPY=212≈0.167 FLOP/byteI_{\mathrm{SAXPY}} = \frac{2}{12} \approx 0.167\ \mathrm{FLOP/byte}

The bound is about 51.2 GFLOP/s on one 8480+ socket (307.2 GB/s × 0.167), less than 1% of its FP32 peak. Even on the H100 SXM5 it is about 559 GFLOP/s (3,352 GB/s × 0.167). In both cases, memory bandwidth, not the compute units, sets the speed.

CPU and H100 ridge points side by side

Here are the ridge points I computed from the published specs for the 8480+ (one socket) and the H100 SXM5 (all theoretical). The H100 figures are the whitepaper's non-Tensor-Core values (at the 1,980 MHz peak clock); Tensor Core figures are not mixed in. The CPU is computed under the two clock assumptions from Section 6.

FP32 peakFP64 peakMemory bandwidthRidge point (FP32)Ridge point (FP64)
Xeon 8480+ (1 socket, 2.0 GHz base)7.17 TFLOP/s3.58 TFLOP/s307.2 GB/s23.3 FLOP/byte11.7 FLOP/byte
Xeon 8480+ (1 socket, 3.8 GHz max turbo)13.62 TFLOP/s6.81 TFLOP/s307.2 GB/s44.3 FLOP/byte22.2 FLOP/byte
H100 SXM5 (non-Tensor Core)66.9 TFLOP/s33.5 TFLOP/s3,352 GB/s20.0 FLOP/byte10.0 FLOP/byte

What the table supports is this: the ridge points of the CPU and the H100's non-Tensor-Core math are in the same order of magnitude. Depending on the clock basis, they range from "almost the same" (base frequency) to "the CPU's is about 2x further right" (max turbo). The difference between them is mainly scale: about 10.9x in memory bandwidth, and about 4.9x to 9.3x in peak compute, depending on the CPU clock assumed.

The table rests on other assumptions too:

  • On a two-socket server, the CPU numbers double.
  • Published power differs: the 8480+'s TDP is 350 W, the H100 SXM5's is 700 W.
  • Tensor Cores change the picture dramatically. The ridge point for the H100's dense FP16 Tensor Core math is about 295 FLOP/byte (Section 6 of the pillar article). For work that can be cast as matrix multiplication, the GPU's compute ceiling rises by another order of magnitude. In exchange, reaching it takes about 300 operations per byte.

And an important point: the ratio of CPU cores (56) to CUDA cores (16,896) appears nowhere in this comparison. The roughly 300 you get by dividing 16,896 by 56 is not a ratio of anything's performance. To compare performance, compare theoretical peaks at the same precision and on the same clock basis, plus memory bandwidth, read against the operational intensity of the workload.

12. Amdahl's law and PCIe transfers: does offloading pay? (original analysis)

Amdahl's law

Amdahl's law gives the overall speed-up when only part of a workload is accelerated. The CUDA C++ Best Practices Guide presents it as:

S=1(1−P)+PNS = \frac{1}{(1 - P) + \dfrac{P}{N}}
  • SS: overall speed-up [x]
  • PP: fraction of the original run time that can be parallelized (0 to 1)
  • NN: speed-up of the parallelized part (number of processors, etc.) [x]

As NN goes to infinity, S=1/(1−P)S = 1/(1-P). In the guide's example, if three quarters of the run time can be parallelized, the whole program can be at most 4x faster no matter how fast that part gets. Even at P=0.95P = 0.95, the ceiling is 20x.

Adding PCIe transfers

On a GPU, one more term enters. The CPU and GPU have separate memories, so data must be sent over PCIe and results brought back. The Best Practices Guide says these transfers are costly and should be minimized, and that work with few operations per element transferred, like adding two matrices, doesn't benefit from moving to the GPU.

According to the whitepaper, the H100's PCIe Gen5 x16 link provides 128 GB/s in total, 64 GB/s in each direction. These are theoretical figures; real transfers come in lower. As an older example, the same guide gives about 12 GB/s with page-locked (pinned) memory against a theoretical 16 GB/s for PCIe Gen3 x16 (about 75%).

Expressing the transfer time as a fraction xx of the original run time extends Amdahl's law to:

S=1(1−p)+ps+xS = \frac{1}{(1 - p) + \dfrac{p}{s} + x}

Estimating it in Python

The script below combines all of this to estimate whether moving work to the GPU pays off. Every result is theoretical, under these assumptions:

  • Compute and memory both run at 100% of theoretical peak (in reality both fall short)
  • Two CPU clocks: 2.0 GHz base and 3.8 GHz max turbo
  • Kernel launch and synchronization overheads are ignored
  • Transfers and computation do not overlap (their times are added)
  • Unless stated otherwise, data starts and ends in CPU memory
  • Matrix-multiply DRAM traffic is the lower bound under ideal blocking (each of the three matrices touched once)

It needs Python 3.9 or later (standard library only); run it with python3 offload_model.py.

# offload_model.py — estimate "does moving this to the GPU pay off?" with Roofline + PCIe transfers (all theoretical)
# Run: python3 offload_model.py   (standard library only, Python 3.9+)
from dataclasses import dataclass

G = 1e9
T = 1e12


@dataclass(frozen=True)
class Machine:
    name: str
    peak_fp32: float  # FLOP/s (theoretical peak)
    peak_fp64: float  # FLOP/s (theoretical peak)
    mem_bw: float     # byte/s (theoretical peak)


def xeon_8480(clock_hz: float, label: str) -> Machine:
    """Xeon Platinum 8480+ (1 socket): 56 cores × 2 AVX-512 FMA units.
    FP64 is 8 lanes × 2 FLOP (FMA) per unit; FP32 has twice the lanes."""
    cores, fma_units = 56, 2
    return Machine(
        f"Xeon 8480+ @{label}",
        peak_fp32=cores * clock_hz * fma_units * 16 * 2,
        peak_fp64=cores * clock_hz * fma_units * 8 * 2,
        mem_bw=8 * 4800e6 * 8,  # 8 channels × 4800 MT/s × 8 bytes
    )


# Two clock assumptions: 2.0 GHz base (lower side) and 3.8 GHz max turbo (upper side;
# there is no guarantee all cores run at this clock while using AVX-512)
XEON_BASE = xeon_8480(2.0 * G, "2.0GHz base")
XEON_TURBO = xeon_8480(3.8 * G, "3.8GHz max turbo")
# H100 SXM5: non-Tensor-Core values from whitepaper Table 3 (never mixed with Tensor Core figures).
# These values are computed at the boost (= peak) clock of 1980 MHz
h100 = Machine("H100 SXM5 (non-Tensor)", peak_fp32=66.9 * T, peak_fp64=33.5 * T,
               mem_bw=3352 * G)
PCIE_PER_DIR = 64 * G  # PCIe Gen5 x16, one direction (half of the 128 GB/s bidirectional total)


def roofline_time(flops: float, dram_bytes: float, peak: float, bw: float) -> float:
    """Roofline lower-bound time: set by the slower of compute and bandwidth."""
    return max(flops / peak, dram_bytes / bw)


def report_ridge() -> None:
    print("== Ridge point I* = P_peak / B  [FLOP/byte] ==")
    for m in (XEON_BASE, XEON_TURBO, h100):
        print(f"{m.name:28s} FP32 peak {m.peak_fp32 / T:6.2f} TFLOP/s, "
              f"FP64 peak {m.peak_fp64 / T:6.2f} TFLOP/s, B {m.mem_bw / G:7.1f} GB/s | "
              f"I*(FP32) {m.peak_fp32 / m.mem_bw:5.1f}, I*(FP64) {m.peak_fp64 / m.mem_bw:5.1f}")


def compare(name: str, cpu: Machine, flops: float, dram_bytes: float, h2d: float,
            d2h: float, resident: bool = False, pcie: float = PCIE_PER_DIR) -> None:
    t_cpu = roofline_time(flops, dram_bytes, cpu.peak_fp32, cpu.mem_bw)
    t_kernel = roofline_time(flops, dram_bytes, h100.peak_fp32, h100.mem_bw)
    t_xfer = 0.0 if resident else (h2d + d2h) / pcie
    t_gpu = t_xfer + t_kernel
    print(f"{name:31s} CPU {t_cpu * 1e3:8.3f} ms | GPU kernel {t_kernel * 1e3:7.3f} ms "
          f"+ PCIe {t_xfer * 1e3:7.3f} ms = {t_gpu * 1e3:8.3f} ms | "
          f"speedup {t_cpu / t_gpu:5.2f}x")


def main() -> None:
    report_ridge()
    print("\n== FP32 workloads (theoretical lower bounds; no launch/sync overhead) ==")
    n = 100_000_000  # SAXPY: y = a*x + y, 2 FLOP and 12 bytes per element (bandwidth-bound, so clock-independent)
    compare("SAXPY n=1e8, data on host", XEON_BASE, 2 * n, 12 * n, h2d=8 * n, d2h=4 * n)
    compare("SAXPY n=1e8, data on GPU", XEON_BASE, 2 * n, 12 * n, 0, 0, resident=True)
    m = 4096    # SGEMM: C = A*B, 2m^3 FLOP, DRAM traffic 3*4*m^2 bytes with ideal blocking
    for cpu, tag in ((XEON_BASE, "base"), (XEON_TURBO, "turbo")):
        compare(f"SGEMM m=4096, host, CPU {tag}", cpu, 2 * m**3, 12 * m**2,
                h2d=8 * m**2, d2h=4 * m**2)
        compare(f"SGEMM m=4096, on GPU, CPU {tag}", cpu, 2 * m**3, 12 * m**2, 0, 0,
                resident=True)

    # Break-even 1: from what size does SGEMM pay off including transfers?
    #   solve 2m^3/P_cpu = 12m^2/B_pcie + 2m^3/P_gpu for m
    print()
    for cpu in (XEON_BASE, XEON_TURBO):
        inv = 1 / cpu.peak_fp32 - 1 / h100.peak_fp32
        print(f"SGEMM break-even vs {cpu.name}: m > {6 / (PCIE_PER_DIR * inv):.0f}")

    # Break-even 2: keeping SAXPY data on the GPU for k passes, from which pass does it pay?
    #   solve 12n/B_pcie + k*12n/B_gpu = k*12n/B_cpu for k (n cancels out)
    print("SAXPY iterations needed on resident data (PCIe efficiency sensitivity):")
    for eff in (1.0, 0.75, 0.5):
        k = (1 / (PCIE_PER_DIR * eff)) / (1 / XEON_BASE.mem_bw - 1 / h100.mem_bw)
        print(f"  PCIe at {eff:4.0%} of 64 GB/s -> k > {k:4.1f}")

    # Amdahl: offload only the part that is 90% of the run time; transfer given as a fraction of total time
    print("\nAmdahl with transfer overhead: S = 1 / ((1-p) + p/s + x)")
    p = 0.9
    for s in (2.0, 10.0, 100.0):
        for x in (0.0, 0.05, 0.2):
            print(f"  p={p:.1f} s={s:5.0f}x transfer={x:4.0%} -> S = "
                  f"{1 / ((1 - p) + p / s + x):5.2f}x")


if __name__ == "__main__":
    main()

Output (Python 3.13.7):

== Ridge point I* = P_peak / B  [FLOP/byte] ==
Xeon 8480+ @2.0GHz base      FP32 peak   7.17 TFLOP/s, FP64 peak   3.58 TFLOP/s, B   307.2 GB/s | I*(FP32)  23.3, I*(FP64)  11.7
Xeon 8480+ @3.8GHz max turbo FP32 peak  13.62 TFLOP/s, FP64 peak   6.81 TFLOP/s, B   307.2 GB/s | I*(FP32)  44.3, I*(FP64)  22.2
H100 SXM5 (non-Tensor)       FP32 peak  66.90 TFLOP/s, FP64 peak  33.50 TFLOP/s, B  3352.0 GB/s | I*(FP32)  20.0, I*(FP64)  10.0

== FP32 workloads (theoretical lower bounds; no launch/sync overhead) ==
SAXPY n=1e8, data on host       CPU    3.906 ms | GPU kernel   0.358 ms + PCIe  18.750 ms =   19.108 ms | speedup  0.20x
SAXPY n=1e8, data on GPU        CPU    3.906 ms | GPU kernel   0.358 ms + PCIe   0.000 ms =    0.358 ms | speedup 10.91x
SGEMM m=4096, host, CPU base    CPU   19.174 ms | GPU kernel   2.054 ms + PCIe   3.146 ms =    5.200 ms | speedup  3.69x
SGEMM m=4096, on GPU, CPU base  CPU   19.174 ms | GPU kernel   2.054 ms + PCIe   0.000 ms =    2.054 ms | speedup  9.33x
SGEMM m=4096, host, CPU turbo   CPU   10.092 ms | GPU kernel   2.054 ms + PCIe   3.146 ms =    5.200 ms | speedup  1.94x
SGEMM m=4096, on GPU, CPU turbo CPU   10.092 ms | GPU kernel   2.054 ms + PCIe   0.000 ms =    2.054 ms | speedup  4.91x

SGEMM break-even vs Xeon 8480+ @2.0GHz base: m > 753
SGEMM break-even vs Xeon 8480+ @3.8GHz max turbo: m > 1603
SAXPY iterations needed on resident data (PCIe efficiency sensitivity):
  PCIe at 100% of 64 GB/s -> k >  5.3
  PCIe at  75% of 64 GB/s -> k >  7.0
  PCIe at  50% of 64 GB/s -> k > 10.6

Amdahl with transfer overhead: S = 1 / ((1-p) + p/s + x)
  p=0.9 s=    2x transfer=  0% -> S =  1.82x
  p=0.9 s=    2x transfer=  5% -> S =  1.67x
  p=0.9 s=    2x transfer= 20% -> S =  1.33x
  p=0.9 s=   10x transfer=  0% -> S =  5.26x
  p=0.9 s=   10x transfer=  5% -> S =  4.17x
  p=0.9 s=   10x transfer= 20% -> S =  2.56x
  p=0.9 s=  100x transfer=  0% -> S =  9.17x
  p=0.9 s=  100x transfer=  5% -> S =  6.29x
  p=0.9 s=  100x transfer= 20% -> S =  3.24x

What the estimate shows

With the data in CPU memory, moving SAXPY to the GPU loses. The GPU kernel itself is about 11x faster than the CPU, but the PCIe transfer takes 18.75 ms, and the whole thing takes about 5x as long as on the CPU (a speed-up of 0.20x). The transfer alone takes about 4.8x the CPU's compute time. The fact that PCIe bandwidth (64 GB/s) is narrower than the CPU's memory bandwidth (307.2 GB/s) shows up directly (and because the work is bandwidth-bound, the CPU clock assumption doesn't matter).

If the data stays on the GPU across repeated passes, the story changes. The transfer cost is spread over the passes, so at PCIe's theoretical rate it pays off from the 6th pass, at 75% effective from the 8th, and at 50% from the 11th. This is why the Best Practices Guide says data should stay on the device even when one individual step would be faster on the host.

A matrix multiply pays off even with transfers, but by how much depends heavily on the CPU clock assumed. For SGEMM (m=4096m = 4096), the transfer takes 3.1 ms, yet the whole job is about 3.7x faster against the CPU at base frequency, and about 1.9x faster against max turbo. A matrix multiply does 2m32m^3 operations on 12m212m^2 bytes of transfer, so the larger the matrix, the more computation dominates. Solving for the break-even point:

m∗=6BPCIe(1PCPU−1PGPU)m^{*} = \frac{6}{B_{\mathrm{PCIe}} \left( \dfrac{1}{P_{\mathrm{CPU}}} - \dfrac{1}{P_{\mathrm{GPU}}} \right)}

Under these assumptions, an FP32 matrix multiply with a side larger than about 753 (CPU at base frequency) to about 1,603 (max turbo) is faster on the GPU even including transfers (theoretical).

Transfers make Amdahl's law even harsher. For a workload where 90% can be moved to the GPU, making that part 100x faster yields only 3.24x overall if transfers take 20% of the original time — versus 9.17x without transfers. What sets the ceiling is not how fast the GPU is but the share of work that cannot be moved and the share spent on transfers.

Limits of this estimate

  • Compute and memory are assumed to run at 100% of theoretical peak, so real times are longer on both sides. If the CPU and GPU run at different efficiencies, the break-even point moves.
  • The CPU clock is shown as a range of two assumptions. The all-core AVX-512 frequency is not on the product specification page and could be below the base frequency.
  • Overlapping transfers and computation can hide part of the transfer. This estimate ignores that, so it errs on the side unfavourable to the GPU.
  • The GPU figures are non-Tensor-Core. A matrix multiply that can use Tensor Cores would cut the GPU kernel time further.
  • Kernel launch overhead is ignored. Launching small kernels many times makes it non-negligible.

13. Theoretical versus measured performance: are GPUs really 100x faster?

Lee et al., 2010

The paper that took on the "100x faster than CPUs" reports directly is "Debunking the 100X GPU vs. CPU Myth," presented by Lee et al. at ISCA 2010 (a major international computer architecture conference). Here is what it actually claims, together with its conditions:

  • Products compared: an Intel Core i7-960 (4 cores, 3.2 GHz, 32 GB/s memory bandwidth) and an NVIDIA GeForce GTX 280 (30 SMs, 1.3 GHz, 141 GB/s memory bandwidth), both current products at the time.
  • Workloads: 14 kernels including matrix multiply (SGEMM), Monte Carlo, convolution, FFT, SAXPY, lattice Boltzmann (LBM) and sorting.
  • Conclusion: earlier studies claimed GPUs were 10x to 1000x faster than multi-core CPUs. With optimizations appropriate to both CPU and GPU, the gap narrowed to 2.5x on average.
  • Individual results: only collision detection (GJK), which could use the GPU's texture-sampling hardware, exceeded 10x. Sorting and the constraint solver were actually faster on the CPU. LBM, previously reported at 114x, became memory-bandwidth-bound on both sides once the CPU code was carefully multithreaded, its memory accesses reorganized and SIMD applied, and the gap shrank to 5x.
  • Why big gaps were reported: the pairing of CPU and GPU (a high-end GPU against a mobile CPU, for example) and the level of optimization. Many studies compared optimized GPU code with unoptimized CPU code. On the CPU side, multithreading, cache blocking and reorganizing data for SIMD helped; on the GPU side, minimizing global synchronization and using shared memory.

As a side note, three details of the paper. Computing the GPU/CPU ratio for each kernel from Table 3 (the per-kernel measurements), I get a geometric mean of about 2.55x and an arithmetic mean of about 3.64x; "2.5x on average" matches the geometric mean, though the paper doesn't say which mean it used. The GJK ratio is 14.9x in the text but about 15.2x from Table 3, a small inconsistency inside the paper. And the paper puts the peak memory bandwidth ratio of the GTX 280 to the i7-960 at about 4.7x, while Table 2's 141 GB/s and 32 GB/s give about 4.4x.

Conditions for reading this paper

Citing this paper as the general rule "GPUs are only 2.5x faster" is wrong, because these conditions can't be dropped:

  1. GPU data transfer time is excluded. The paper assumes the data is already in GPU memory and explicitly calls its GPU results an upper bound of what real applications will see.
  2. It compares 2010 products. Today's GPUs have Tensor Cores and HBM, and the shape of the roofline itself has changed. The 2.5x figure cannot be applied to an H100 and a current CPU.
  3. All of the authors were at Intel. The measurements and descriptions of the optimizations can be checked, but it is worth knowing that the evaluation comes from researchers at a CPU vendor.

The paper is still worth reading because it shows how to compare: optimize both sides equally, compare products in comparable positions, and state whether transfers are included.

What lies between theoretical peak and measurement

There is always a gap between theoretical peak and measurement. Lee et al. cite earlier work by Volkov et al. that GPUs reach only about 66% of peak flops even on SGEMM, a kernel known to be compute-bound, and say their own results agree. The Best Practices Guide recommends computing both theoretical and effective bandwidth, and making the gap the first optimization target when effective bandwidth falls far short. Add to that PCIe's effective rate (Section 12) and the clock basis (Section 6). Comparing spec sheets alone won't tell you real speed, so estimate the upper bound with the Roofline model, then measure your own workload at small scale.

14. Limits and counterexamples: where CPUs win and where GPUs win

Turning everything above around gives the conditions under which a GPU is at a disadvantage:

  • Branch-heavy work whose path differs per data element: warp divergence makes the GPU run both paths. On a CPU, a branch is almost free as long as prediction is right, but a GPU has no such mechanism (in Lee et al., the constraint solver and sorting were faster on the CPU).
  • Work with little parallelism: if it can't supply thousands of threads, the GPU can't hide waiting. Sequential work and small data sets suit the CPU.
  • Low-intensity work whose data lives in CPU memory: the transfer alone exceeds the CPU's compute time (SAXPY in Section 12).
  • Work with lots of fine-grained synchronization: Lee et al. point out that CPUs provide fast synchronization while GPUs do not implement it efficiently, and report that synchronization cost dominated in the histogram and constraint-solver kernels.

Conversely, a GPU shines on work with a large amount of independent computation, data that can stay on the GPU for repeated processing, and memory accesses that line up per warp: matrix multiplication, convolution, per-pixel image processing, and the training and inference of neural networks are the typical cases. How to compare CPUs, GPUs and TPUs down to the cost per token of AI inference is covered in CPU vs GPU vs TPU: how to choose.

A counterexample, too. This article contrasted "CPUs for latency, GPUs for throughput," but the boundary blurs every year. CPUs take on wide arithmetic with AVX-512 and AMX, and GPUs adopt CPU-like flexibility such as Independent Thread Scheduling. The contrast is a difference in design emphasis, not a black-and-white split.

15. Decision criteria: CPU or GPU?

To finish, here is a procedure for deciding whether to move work to the GPU:

  1. Check the parallelism. Are there thousands or more independent units that can be processed at once? If not, staying on the CPU is probably better.
  2. Estimate the operational intensity. How many operations per byte of data? Compare it with the ridge point (roughly 10–45 FLOP/byte for non-Tensor-Core math, hundreds for Tensor Cores) to decide whether the work is bandwidth-bound or compute-bound.
  3. Check where the data lives. Where is the data at the start, and where are the results used? If it's in CPU memory, always add the PCIe transfer time, and compute how many passes it takes to recover that cost (Section 12).
  4. Check the shape of branches and memory accesses. Do branches split warps? Do neighbouring threads read neighbouring data? If not, redesign starting from the data layout.
  5. Get the ceiling from Amdahl's law. What share of the whole is work that can't move to the GPU, plus transfers? That sets the maximum speed-up.
  6. Measure small. Everything so far is a theoretical upper bound. Finally, measure the real workload, with the CPU side optimized just as much. As Lee et al. showed, a comparison that optimizes only one side is badly wrong.

My own practice is at the scale of an inference service running vLLM with AWQ-quantized Qwen3-8B on a single Tesla T4 (the GPU pipeline for AI video localization); I have no experience with H100 clusters or with developing CUDA kernels. Even at that scale, the decisions to "keep data on the GPU and keep processing it there" and "leave GPU-unfriendly preprocessing on the CPU" were the starting point of the design.

Applying this procedure to LLM inference: decode, which generates one token at a time, has low operational intensity and is bandwidth-bound, so hardware choice comes down to VRAM capacity and bandwidth. Whether your own GPUs or an API come out cheaper is covered in the article on quantization and VRAM budgets and in the CPU/GPU/TPU cost-per-million-tokens estimate.

Summary

  • CPUs are latency-oriented. Superscalar issue, register renaming, out-of-order execution, the reorder buffer, and branch prediction with speculative execution lower the CPI in T=IC×CPI/fT = IC \times CPI / f and push a single instruction stream forward as fast as possible.
  • GPUs are throughput-oriented. They issue instructions in order, do neither branch prediction nor speculative execution, and hide waiting time by switching among a large number of warps.
  • GPU performance swings widely with warp divergence (a split branch runs both paths), occupancy and the competition for registers, memory coalescing and bank conflicts. Independent Thread Scheduling widened the set of programs that can be written correctly, but it does not remove the cost of warp divergence.
  • Compared with the Roofline model, the ridge points of a Xeon 8480+ (one socket) and the H100 SXM5's non-Tensor-Core math are in the same order of magnitude, ranging from "almost the same" to "the CPU's is about 2x further right" depending on the CPU clock basis. Tensor Cores lift the GPU's ceiling by another order of magnitude (all theoretical).
  • Whether offloading pays is decided by transfers and Amdahl's law. SAXPY with its data in CPU memory loses on the transfer alone, while a large matrix multiply wins even including transfers (theoretical).
  • Much of "GPUs are 100x faster" came from comparisons with unoptimized CPU code. Lee et al.'s 2.5x average is also an upper bound on 2010 products, excluding transfers. When you compare, align the conditions, including clock speed, and measure your own workload.

For the SIMT execution model seen from the side of actually writing kernels, see How CUDA works: a GPU programming primer.

Frequently asked questions

What is the difference between a CPU and a GPU, in one sentence?
A CPU is built to finish a few tasks, one at a time, as quickly as possible; a GPU is built to get through a huge number of similar tasks with the highest total output. The CPU shortens the wait inside a single instruction stream with branch prediction and out-of-order execution. The GPU has neither; it keeps thousands of threads ready and runs other threads while some are waiting, which hides the wait.
Can I tell which is faster by comparing CPU core count with CUDA core count?
No. A CPU core is an independent execution unit with branch prediction and instruction reordering. A CUDA core is closer to one arithmetic lane that executes part of an instruction issued for a whole warp. The two counts measure different things, so their ratio is not a speed ratio. To compare, line up theoretical peaks at the same precision and on the same clock basis, plus memory bandwidth, and read them against the arithmetic intensity of your workload.
How is SIMT different from SIMD?
Both apply one instruction to many data elements. The difference is how you program them. In SIMD the vector width (512 bits for AVX-512) is visible to the program, and you handle branches with masks yourself. In SIMT you write the code for one thread, and the hardware masks threads when a branch splits. The slowdown from a split branch remains, though, so fast code still has to think about branches at warp granularity.
Are GPUs really 100 times faster than CPUs?
Not in general. In 2010 Lee et al. optimized 14 workloads for both a CPU and a GPU and found an average gap of 2.5x. Many earlier claims of 10x to 1000x compared against CPU code that had not been optimized. That measurement is itself an upper bound that excludes data transfer to the GPU, and all of its authors were at Intel. Today's products give different numbers, so the reliable answer is to measure your own workload.
How do I decide whether to move our workload to a GPU?
First check whether there are thousands or more independent units that can be processed at once. Then, if the data starts in CPU memory, always add the PCIe transfer time. In this article's theoretical estimate, the transfer alone for SAXPY (a vector multiply-add) took about 4.8x as long as the CPU's whole computation, so moving it to the GPU made it slower. Offloading pays when there is a lot of computation per byte transferred (large matrix multiplies) or when the data stays on the GPU for many passes. Finally, measure, with the CPU side optimized just as much.

References

友田

友田 陽大

Developer of a METI Minister's Award–winning product. With TypeScript + Python + AWS, I deliver SaaS, industry DX, and production-grade generative AI (RAG) end to end — from requirements to infrastructure and operations — single-handedly.

Do you actually know whether self-hosting beats the API?

Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing

When to use AWQ / GPTQ / FP8 / GGUF, VRAM estimates that include the KV cache, the break-even point against per-token API pricing, and the license terms of Apache-2.0 / Llama / Qwen / Gemma. From hands-on experience adopting commercially usable open quantized models stage by stage against license, quality, and speed trade-offs, I help you assemble the decision criteria before you implement.

Available for both project-based (contract) and advisory engagements. Start with a free 30-minute consult.

Also worth reading