Skip to main content
NVIDIA, GPUs & AI chips
AI
パフォーマンス
インフラ
アーキテクチャ設計
技術選定

How CUDA works: a GPU programming primer — kernels, PTX/SASS, the memory hierarchy and synchronization, with complete code

How CUDA works: host and device, the grid/block/thread/warp hierarchy, NVCC→PTX→SASS, and GPU memory — with complete kernels and T4 vs H100 occupancy math.

Published
Reading time
63 min read
Author
友田 陽大
Share
Contents

"GPU programming with CUDA" can sound like it needs a special language or heavy mathematics. What you actually write is C++ with a handful of extensions. The hard part is not the syntax. It is learning to think in a world where tens of thousands of threads run at once, memory comes in several layers, and synchronization only reaches so far.

The short answer first. CUDA is a programming model in which the CPU (the host) tells the GPU (the device) to "run this one function on a huge number of threads", together with the compiler, driver and libraries that make that work. A function that runs on the GPU (a kernel) is marked __global__ and launched from the host as vecAdd<<<number of blocks, threads per block>>>(a, b, c, n); each thread computes which element it owns from blockIdx.x * blockDim.x + threadIdx.x. Memory comes in three main tiers: fast, small registers private to each thread; shared memory, shared within a block; and global memory (the GPU's DRAM), visible to every thread but far away. NVCC compiles your CUDA C++ into PTX, a virtual instruction set, and then into SASS, the machine code of a particular GPU generation. What decides speed is usually not the arithmetic itself but where the data lives, how it is read, and where threads synchronize.

Sections 1–7 cover the host and device, the thread hierarchy, the compilation pipeline, the memory hierarchy and synchronization. Sections 8–14 write reduction, scan, tiled matrix multiplication and streams as complete programs, compute theoretical occupancy for a T4 and an H100, and place the profilers and PyTorch in the picture. The final section, 15, covers limits and how to decide.

What has been verified. Each program is complete and compiles on its own (which is why every one defines its own CHECK macro), and each compares its result with a CPU reference and prints PASS or FAIL. My machine (Apple M4 Pro) has no NVIDIA GPU, so I checked the logic in an environment that simulates CUDA threads with std::thread, using boundary sizes (1, 31, 32, 33, 1,024, 1,025, 5,000 and 70,001 elements for scan; small matrices of 37 to 64 for the matrix multiply, and so on) and deliberately injected bugs, and I computed the floating-point error and occupancy on the CPU. No GPU timings appear in this post. The target GPU is the T4 (sm_75). My hands-on experience stops at an inference service that runs Qwen3-8B (AWQ-quantized) on vLLM on a single Tesla T4; I have not developed CUDA kernels professionally. The code is study code written against NVIDIA's official documentation.

Tensor Cores, number formats, HBM and NVLink are covered in Why NVIDIA GPUs are so strong at AI, so I won't repeat them here.

1. What CUDA is: not a language, but a model plus a toolbox

"CUDA" names several things at once.

  • A programming model: a way of treating the GPU as a device that runs many threads. It defines the thread hierarchy, the kinds of memory, and the rules for synchronization.
  • CUDA C++: C++ with a few extensions — specifiers such as __global__, the <<< >>> kernel-launch syntax, and built-in variables. NVCC supports the C++03 through C++23 language standards.
  • The CUDA Runtime API and Driver API: the APIs that allocate and copy memory and launch kernels. The Runtime API used in this post (functions starting with cuda) is built on top of the lower-level Driver API (functions starting with cu), and the same functionality can be achieved with the Driver API alone. The Driver API works explicitly with contexts (CUcontext) and modules (CUmodule), loading PTX or cubins as modules before launching kernels. A few features, such as virtual memory management, exist only in the Driver API, but most applications get full performance from the Runtime API alone.
  • NVCC and PTX: the compiler and the virtual instruction set it targets, covered in section 5.
  • The NVIDIA driver: in the CUDA Programming Guide's words, something like "the operating system of the GPU". It ships as a product separate from the CUDA Toolkit.
  • Libraries and tools: cuBLAS (linear algebra), cuDNN (neural networks), Nsight Systems and Nsight Compute (profilers), Compute Sanitizer (correctness checking), and more.

One constraint up front: NVIDIA CUDA does not normally run on Apple Silicon Macs. Those Macs have no NVIDIA GPU. CUDA 10.2's release notes also state that "CUDA 10.2 (Toolkit and NVIDIA driver) is the last release to support macOS for developing and running CUDA applications." PyTorch on a Mac uses Apple's Metal through the mps device, which is a different mechanism from CUDA. To run the code in this post, use Google Colab or an NVIDIA GPU in the cloud.

Do you actually know whether self-hosting beats the API?

Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing

2. Host and device: the CPU gives orders, the GPU runs them on many threads

Two computers, two memories

CUDA assumes a heterogeneous system. The CUDA Programming Guide calls the CPU and the memory attached to it the host and host memory, and the GPU and its memory the device and device memory.

A program always starts on the CPU. Code running there (host code) uses the CUDA APIs to do three things:

  1. copy data between host memory and device memory,
  2. start code running on the GPU (launching a kernel), and
  3. wait for copies or kernels to finish.

A function that runs on the GPU is called a kernel, for historical reasons. Launching a kernel starts many GPU threads executing the same kernel code in parallel.

Think of the host as a site manager and the device as a factory with thousands of workers. The manager brings the materials (data) into the factory, hands out a single instruction sheet (the kernel) saying "everyone, do this", and collects the finished product. Every worker holds the same sheet, but each looks at their own badge number to decide which piece of material is theirs.

Kernel launches are asynchronous

Here is a big difference from ordinary CPU code. A kernel launch is asynchronous with respect to the host thread. The launch returns without waiting for the GPU to finish the kernel — possibly before it has even started. Before using the result you must synchronize, with cudaDeviceSynchronize() (wait for all work issued to the GPU) or with the streams and events of section 12.

The memories are separate by default too. Device memory allocated with cudaMalloc cannot be read directly by CPU code. Unified memory (cudaMallocManaged) is reachable from both CPU and GPU, but the CUDA Programming Guide notes that the best performance still comes from keeping data migration to a minimum and accessing data from the processor it lives next to. This post uses explicit copies, which make it obvious what runs when.

3. The thread hierarchy: grid, block, thread — and warp

Three levels

When you launch a kernel, its threads are organized in three levels:

  • Thread: the smallest unit that executes the kernel code
  • Thread block (block from here on): a group of threads; on current GPUs, up to 1,024 threads per block
  • Grid: a group of blocks; one kernel launch is one grid

Blocks and grids can be laid out in one, two or three dimensions. Each thread learns where it is from built-in variables.

Built-in variableMeaning
threadIdx.x/y/zThe thread's index within its block (from 0)
blockIdx.x/y/zThe block's index within the grid (from 0)
blockDim.x/y/zThe block's size (in threads)
gridDim.x/y/zThe grid's size (in blocks)

In one dimension, the standard way to compute which element a thread owns is:

i=blockIdx.x×blockDim.x+threadIdx.xi = \mathtt{blockIdx.x} \times \mathtt{blockDim.x} + \mathtt{threadIdx.x}

A grid can have up to 231−12^{31}-1 blocks in the x dimension.

Blocks are independent of each other

The most important property of this hierarchy is that blocks are independent. All threads of one block run on the same SM (Streaming Multiprocessor, the GPU's compute building block). Which SM a block lands on, and in what order, is not guaranteed. So, as a rule, a block cannot rely on results from another block of the same grid.

In exchange for that restriction, the same program runs on a GPU with one SM or with hundreds: blocks are simply handed out to whichever SMs are free.

Compute capability (CC, the number that identifies a GPU generation's feature set) 9.0 and later add an optional level called clusters, whose blocks are scheduled together. The code in this post does not use them.

Warps: 32 threads executing the same instruction

Inside a block, threads are executed in groups of 32 called warps, which run the same instruction in SIMT style (Single Instruction, Multiple Threads). A thread's position within its warp (0–31) is its lane. When a branch splits the lanes, the warp runs both paths one after the other (warp divergence), so block sizes should be multiples of 32 and branches within a warp kept to a minimum. SIMT and divergence are covered in detail in section 8 of How CPUs and GPUs differ in architecture.

Do not confuse logical threads with physical units

Now to clear up a common misunderstanding: a CUDA thread is a logical unit of execution, and it does not correspond to an arithmetic unit.

Take a T4 and compute from the published figures (the Turing whitepaper and the CUDA Programming Guide's compute-capability tables):

T4 (CC 7.5)H100 SXM5 (CC 9.0)
SMs40132
FP32 units (CUDA cores) per SM64128
FP32 units on the GPU2,56016,896
Max resident threads per SM1,0242,048
Max resident threads on the GPU (theoretical)40,960270,336

A T4 can keep up to 40,960 threads resident on 2,560 arithmetic units — exactly 16 to 1 (the H100 has the same ratio). Resident threads are not all computing at once. Every cycle, an SM's warp scheduler picks a warp that is ready to run its next instruction and issues it; if a warp is waiting on a memory load, the scheduler switches to another. A warp's execution context (program counters, registers) stays on chip for the warp's whole lifetime, so switching between warps costs nothing.

On top of that, a grid can launch millions of threads. Those that do not fit wait until earlier blocks finish. The point is not "2,560 CUDA cores, so 2,560 threads" but "keep many threads resident and hide waiting time behind other warps' work". That is the basic idea of a GPU.

This is also where the contrast with a CPU shows. The CUDA Programming Guide says: "Unlike CPU cores, SMs issue instructions in order and do not perform branch prediction or speculative execution." A CPU spends chip area on making one thread go fast; a GPU hides latency by switching among many warps.

4. The smallest kernel: vector addition as a complete program

Let's check these ideas with the smallest complete program: one that adds two arrays element by element.

  • Requirements: the CUDA Toolkit (nvcc) and an NVIDIA GPU; target T4 (sm_75)
  • Build: nvcc -arch=sm_75 -O3 -o vector_add vector_add.cu
  • Run: ./vector_add
// vector_add.cu — the smallest useful CUDA program (c[i] = a[i] + b[i])
// Build: nvcc -arch=sm_75 -O3 -o vector_add vector_add.cu
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>

// Check every CUDA API return value; on failure print where and which error, then exit
#define CHECK(call)                                                       \
    do {                                                                  \
        cudaError_t err_ = (call);                                        \
        if (err_ != cudaSuccess) {                                        \
            std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n",          \
                         cudaGetErrorName(err_), __FILE__, __LINE__,      \
                         cudaGetErrorString(err_));                       \
            std::exit(EXIT_FAILURE);                                      \
        }                                                                 \
    } while (0)

// Tiny pseudo-random generator for reproducibility (float in [0, 1))
static float lcg01(unsigned& state) {
    state = state * 1664525u + 1013904223u;
    return (state >> 8) * (1.0f / 16777216.0f);
}

// __global__ = launched from the host; many GPU threads run this same code
__global__ void vecAdd(const float* a, const float* b, float* c, int n) {
    // Index of the element this thread owns (unique across the whole grid)
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {  // Spare threads in the last block do nothing
        c[i] = a[i] + b[i];
    }
}

int main() {
    const int n = (1 << 20) + 3;  // Deliberately not a multiple of the block size
    const size_t bytes = static_cast<size_t>(n) * sizeof(float);

    std::vector<float> ha(n), hb(n), hc(n);
    unsigned seed = 12345u;
    for (int i = 0; i < n; ++i) {
        ha[i] = lcg01(seed);
        hb[i] = lcg01(seed);
    }

    float *da = nullptr, *db = nullptr, *dc = nullptr;
    CHECK(cudaMalloc(&da, bytes));
    CHECK(cudaMalloc(&db, bytes));
    CHECK(cudaMalloc(&dc, bytes));
    CHECK(cudaMemcpy(da, ha.data(), bytes, cudaMemcpyHostToDevice));
    CHECK(cudaMemcpy(db, hb.data(), bytes, cudaMemcpyHostToDevice));

    const int threads = 256;
    const int blocks = (n + threads - 1) / threads;  // Ceiling division
    vecAdd<<<blocks, threads>>>(da, db, dc, n);
    CHECK(cudaGetLastError());       // Catches launch-configuration errors
    CHECK(cudaDeviceSynchronize());  // Catches errors raised while the kernel ran

    CHECK(cudaMemcpy(hc.data(), dc, bytes, cudaMemcpyDeviceToHost));

    // Compare with the same computation on the CPU (one addition, so we expect zero error)
    double maxErr = 0.0;
    for (int i = 0; i < n; ++i) {
        const float ref = ha[i] + hb[i];
        maxErr = std::fmax(maxErr, std::fabs(static_cast<double>(hc[i]) - ref));
    }
    const bool ok = maxErr <= 1e-6;
    std::printf("n=%d blocks=%d threads/block=%d\n", n, blocks, threads);
    std::printf("max abs error = %.3e (tolerance 1e-6)\n", maxErr);
    std::printf("%s\n", ok ? "PASS" : "FAIL");

    CHECK(cudaFree(da));
    CHECK(cudaFree(db));
    CHECK(cudaFree(dc));
    return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}

Expected output (GPU results not published). The program prints the element count, the number of blocks and the threads per block; then a max abs error = ... line with the largest difference between the GPU result and the CPU value; and finally PASS or FAIL. Since each element is a single addition, the difference should be zero.

Key points:

  • __global__ marks a kernel launched from the host. Its return type must be void; the only way to return results is to write them to global (device) memory.
  • __device__ marks a function callable only from GPU code (used by warpReduceSum in section 8). Marking a function __host__ __device__ generates both a CPU and a GPU version.
  • vecAdd<<<blocks, threads>>>(...) launches the kernel: the first argument is the grid size, the second the block size.
  • The bounds check if (i < n): if the element count is not a multiple of the block size, the last block has spare threads. This program deliberately uses 220+32^{20}+3 elements.
  • The CHECK macro: every CUDA API returns a cudaError_t. The CUDA Programming Guide likewise calls checking every return value best practice in production.

Error handling has a pitfall that comes from asynchrony. A <<< >>> launch does not return a cudaError_t, and cudaGetLastError() right after the launch only catches errors in the launch configuration. Errors raised while the kernel runs (an invalid memory access, say) are reported the next time the error state is checked. That is why the program checks both cudaGetLastError() and the return value of cudaDeviceSynchronize().

5. The compilation pipeline: CUDA C++ → NVVM → PTX → SASS

NVCC is a conductor, not a single compiler

nvcc is less a compiler than a driver that calls several tools in turn. The flow is:

  1. split each .cu file into host code and device code;
  2. compile the host code with the host C++ compiler (gcc and the like);
  3. compile the device code to PTX, once per virtual architecture (e.g. compute_75);
  4. have ptxas turn the PTX into a cubin containing machine code for a real GPU (e.g. sm_75);
  5. pack the PTX and cubins into a fatbin container and embed it in the host executable.
vector_add.cu
 ├─ host code ───────── compiled by gcc etc. ────────────────┐
 └─ device code ─ NVVM (LLVM-based) ─→ PTX (compute_75) ─────┤
                                  └─ ptxas ─→ cubin/SASS (sm_75)
                                                             ↓
                     executable = CPU machine code + fatbin (PTX and cubin)

Step 3 uses NVVM. NVIDIA's NVVM IR specification describes NVVM IR as "based on the LLVM IR" and says that "The NVVM compiler (which is based on LLVM) generates PTX code from NVVM IR." NVIDIA's developer page likewise says NVCC is based on the open-source LLVM compiler infrastructure. Languages and compilers other than CUDA can run on NVIDIA GPUs too, as long as they emit PTX. -v shows the tools invoked at each step, and -keep keeps the intermediate files.

PTX versus SASS

PTXSASS (the contents of a cubin)
What it isA virtual instruction set ("a high-level assembly language for NVIDIA GPUs")The machine code a real GPU executes
FormatTextBinary (disassemble with cuobjdump or nvdisasm)
Relation to generationsNamed like compute_75, for the feature generation it requiresNamed like sm_75, for one specific GPU generation
CompatibilityThe driver can JIT-compile it on the same or a newer generationRuns only when the major number matches and the cubin's minor number is not above the GPU's
SpecificationPublished as the PTX ISA (currently version 9.5)The instruction list is published, but not a detailed specification like PTX's

The PTX ISA document lists, among PTX's goals, to "provide a stable ISA that spans multiple GPU generations". In PTX, a thread block is called a CTA (Cooperative Thread Array). Where data lives is expressed as state spaces — .reg (registers), .shared, .global, .local, .const, .param and others — and threadIdx.x becomes the special register %tid.x.

Compute capability and binary compatibility

Here is what -arch=sm_75 means, together with the compatibility rules:

  • cubin compatibility: a cubin runs if the compute capability's major number matches and the GPU's minor number is at least the cubin's. An sm_86 cubin runs on CC 8.6 and 8.9 GPUs but not on 8.0, and not on a GPU with a different major number (such as 9.0). The exceptions: code built for an architecture-specific target such as sm_90a runs only on GPUs of exactly that CC, and CC 10.0 and later add a separate family-level compatibility rule (CUDA Programming Guide, sections 1.3.4 and 5.1.2).
  • PTX compatibility: PTX for compute_80 can be JIT-compiled at run time for any GPU of that CC or higher, so applications run on future GPUs without being rebuilt.
  • -arch=sm_75 generates both compute_75 PTX and an sm_75 cubin. A T4 uses the cubin directly; newer GPUs JIT-compile the PTX.
  • -arch=native builds a cubin for the local GPU only. It contains no PTX, so there is no forward compatibility.

In the NVCC documentation published as of 2026-11-17 (CUDA Toolkit v13.5), the default when -arch is omitted is sm_75, and the list of supported GPUs starts at Turing (sm_75). The T4 is therefore the oldest generation still covered by that CUDA version.

JIT compilation and the compute cache

Run a program that carries only PTX on a new GPU, and the driver compiles the PTX to machine code at load time: JIT (just-in-time) compilation. JIT adds load time but lets programs run on GPUs that did not exist when they were built. The driver stores the generated code in the compute cache, which is invalidated when the driver is upgraded.

On a T4 the sm_75 cubin runs directly, so no JIT happens. The first call still pays for creating a context (the per-GPU execution environment), and since CUDA 12.3 lazy loading — loading a module only when it is first needed — is the default on all platforms, so a kernel's first launch can also include loading time. That is why the programs in this post run a warm-up before timing anything.

Looking at PTX and SASS yourself

These commands show the PTX and the SASS (I have not run them):

# Emit PTX (text)
nvcc -arch=sm_75 -ptx vector_add.cu -o vector_add.ptx
grep -E "ld\.global|add(\.rn)?\.f32|st\.global" vector_add.ptx

# List the cubins and PTX inside the executable's fatbin
cuobjdump -lelf -lptx vector_add

# Disassemble the SASS (T4 machine code)
cuobjdump -sass vector_add | grep -E "LDG|FADD|STG"

# Print registers and shared memory used per kernel
nvcc -arch=sm_75 -O3 -res-usage -c vector_add.cu

The PTX should contain instructions corresponding to ld.global (load from global memory), add.f32 (FP32 add) and st.global (store). In the "Turing Instruction Set (Compute Capability 7.5)" table of NVIDIA's binary utilities documentation, the matching SASS instructions are LDG (Load from Global Memory), FADD (FP32 Add) and STG (Store to Global Memory). The exact output varies with the compiler version.

6. The memory hierarchy: registers, shared, global, local, constant

Five kinds of memory

The memories a kernel uses differ in visibility (scope), lifetime and location. Here is the CUDA Programming Guide's table, with T4 and H100 capacities added (from the compute-capability tables and the Turing whitepaper):

KindVisible toLifetimeLocationT4 (CC 7.5)H100 (CC 9.0)
RegistersThreadKernelSM64K per SM (max 255 per thread)Same
Shared memoryBlockKernelSMUp to 64 KB per SMUp to 228 KB per SM
Local memoryThreadKernelDevice memoryUp to 512 KB per threadSame
Constant memoryGridApplicationDevice memory64 KB64 KB
Global memoryGridApplicationDevice memory16 GB (GDDR6)80 GB (SXM5, HBM3)

A few notes:

  • Shared memory is a programmer-managed scratchpad shared by the threads of a block. Physically it uses the same resource as the L1 cache (the unified data cache). On a T4 the unified data cache is 96 KB, and the split between shared memory and L1 is configurable. It is described as having higher bandwidth and lower latency than global memory, but the CUDA Programming Guide gives no concrete latency figures.
  • Local memory, despite its name, lives off chip in device memory. "Local" describes visibility (that thread only), not location. Arrays indexed by non-constant values, large structures and variables that do not fit in registers (register spills) go there, with the same latency and bandwidth as global memory.
  • Statically allocated shared memory is limited to 48 KB per block. Going beyond that needs dynamic shared memory and an explicit opt-in (cudaFuncSetAttribute).

Coalescing and bank conflicts: what the programmer can do

Global memory is read and written per warp, so access is efficient when consecutive threads read consecutive addresses (coalescing), and when adjacent threads read addresses far apart, most of the data moved is wasted. In the CUDA Programming Guide's example, utilization ranges from 100% at best to 12.5% at worst. Shared memory is split into 32 banks, and when several threads of a warp access different addresses in the same bank, the accesses serialize (a bank conflict). The mechanics are covered in section 10 of How CPUs and GPUs differ in architecture.

The programmer has two standard remedies:

  • Rearrange through a shared-memory tile. For operations whose writes jump along columns, like a matrix transpose, load a tile into shared memory, rearrange it there and then write it out, so that both reads and writes coalesce. The vecAdd of section 4 already has consecutive threads reading consecutive elements, so it coalesces perfectly as it is.
  • Pad to shift the banks. Reading __shared__ float smem[32][32] down a column sends all 32 threads to the same bank. Declaring it [32][33], padding each row by one element, removes the conflict.

7. Synchronization and race conditions: how CPU and GPU differ

The block-level barrier: __syncthreads()

When the threads of a block exchange data through shared memory, they must synchronize so that nobody reads before the writes are done. __syncthreads() waits until every (non-exited) thread of the block reaches the same __syncthreads(), and it also guarantees that writes made before it are visible to the threads after they leave the wait.

The rules are strict. __syncthreads() may appear in conditional code only if the condition evaluates identically across the entire block. Calling it in a branch that only some threads take can hang the kernel or produce unintended effects.

There is no grid-wide barrier

__syncthreads() only reaches within a block. As section 3 showed, block execution order is not guaranteed, so an ordinary kernel has no way to synchronize the whole grid. To combine results across blocks, use one of the following:

  1. Atomics: atomicAdd and friends read, modify and write as one indivisible step. The CUDA Programming Guide says atomics should be used sparingly, because they enforce synchronization between threads that can hurt performance.
  2. Separate kernels: the first kernel writes per-block partial results and the second combines them. Kernels in the same stream run in order, so the kernel boundary acts as a grid-wide synchronization point.

(Cooperative Groups can express synchronization across blocks, with limitations and performance costs. This post does not use them.)

A weak memory model and data races

The CUDA Programming Guide states plainly that the CUDA programming model assumes a weakly ordered memory model. The order in which one thread writes to memory need not be the order in which another thread observes those writes. Reading and writing the same location without fences or synchronization — a data race — is undefined behavior.

Warps do not line up by themselves

Older material contains code built on the assumption that "instructions are SIMD synchronous within a warp, so we don't need __syncthreads()" (for example, the "unrolling the last warp" step in Mark Harris's well-known NVIDIA reduction slides). That assumption does not hold from CC 7.0 on. Independent Thread Scheduling gives each thread its own program counter and lets threads diverge and reconverge at sub-warp granularity. The CUDA Programming Guide asks for such code to be revisited and synchronized explicitly with __syncwarp(). The old warp primitives without a mask argument were deprecated in CUDA 9.

CPU versus GPU synchronization, side by side

Here are the differences so far, in one table:

AspectCPU multithreading (standard C++)CUDA
Thread managementScheduled by the OSTens of thousands resident on the SMs; switching warps costs nothing
Cost of synchronizationIn the CUDA docs' words, standard C++ treats it as "uniform and low"Cheap between nearby threads, expensive between distant ones
Reach of synchronizationAny thread can sync with any other via mutexes or atomicsA hierarchy: warp (__syncwarp, shuffles) → block (__syncthreads) → device (atomics, kernel boundaries) → system
Global barrierstd::barrier can wait for all threadsNo grid-wide barrier in an ordinary kernel
Memory visibilityDefined by std::atomic and memory ordersWeak memory model; cuda::atomic and friends take a scope (block, device, system)
Between CPU and GPU—Kernel launches and copies are asynchronous; wait with streams, events or cudaDeviceSynchronize

CUDA's scopes widen from block (.cta in PTX) to device (.gpu) to system (.sys, including CPUs and other GPUs), and the wider the scope, the more synchronization costs.

8. Reduction: a shared-memory tree versus warp shuffles

Why a sum is hard

Summing an array (a reduction) is a one-line loop on a CPU. On a GPU, tens of thousands of threads have to funnel their results into one value, so every synchronization issue from section 7 shows up.

The approach is a tree. First build partial sums inside each block, then add the per-block partial sums in a second kernel launch. Two ways of building the in-block sum are compared:

  1. A shared-memory tree: put every thread's value in shared memory and add halves repeatedly. Each step needs a __syncthreads().
  2. Warp shuffles: __shfl_down_sync passes values between registers within a warp, with no shared memory; only the eight per-warp partial sums go through shared memory.

__shfl_down_sync(mask, v, offset) returns the v of the lane offset positions above. Adding while halving the offset — 16, 8, 4, 2, 1 — collects the sum of 32 values in lane 0 after five steps. mask is a bitmask of participating lanes; every calling thread must be in it, and all must call with the same mask. CC 8.0 and later also have a dedicated __reduce_add_sync, but it accepts only integers (int / unsigned), so it cannot sum floats (and the T4, at CC 7.5, does not have it at all). So the code uses shuffles.

For comparison, the program also includes a correct but slow atomicAdd version and an unsynchronized data race version. The racy one is an example of undefined behavior and is never used for PASS/FAIL.

  • Build: nvcc -arch=sm_75 -O3 -o reduce reduce.cu
  • Run: ./reduce (the element count can be passed as an argument; the default is 2242^{24})
// reduce.cu — four ways to write a sum (reduction), compared
//   1) tree in shared memory + __syncthreads
//   2) warp shuffles (__shfl_down_sync); shared memory holds only one partial sum per warp
//   3) atomicAdd (correct, but slow, and the summation order changes every run)
//   4) unsynchronized *out += x (a data race: an example of undefined behavior, never judged)
// Build: nvcc -arch=sm_75 -O3 -o reduce reduce.cu
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>

#define CHECK(call)                                                       \
    do {                                                                  \
        cudaError_t err_ = (call);                                        \
        if (err_ != cudaSuccess) {                                        \
            std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n",          \
                         cudaGetErrorName(err_), __FILE__, __LINE__,      \
                         cudaGetErrorString(err_));                       \
            std::exit(EXIT_FAILURE);                                      \
        }                                                                 \
    } while (0)

constexpr int kThreads = 256;           // 1 block = 8 warps (must be a power of two)
constexpr int kWarps = kThreads / 32;

static float lcg01(unsigned& state) {
    state = state * 1664525u + 1013904223u;
    return (state >> 8) * (1.0f / 16777216.0f);
}

// 1) Tree reduction in shared memory
__global__ void reduceSmem(const float* in, float* out, int n) {
    __shared__ float s[kThreads];
    float sum = 0.0f;
    // Grid-stride loop: covers every element even when there are more elements than threads
    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
         i += blockDim.x * gridDim.x) {
        sum += in[i];
    }
    s[threadIdx.x] = sum;
    __syncthreads();  // Wait until everyone has written s[]
    for (unsigned int stride = blockDim.x / 2; stride > 0; stride /= 2) {
        if (threadIdx.x < stride) {
            s[threadIdx.x] += s[threadIdx.x + stride];
        }
#ifndef OMIT_SYNC  // -DOMIT_SYNC removes this barrier and creates a shared-memory race
        __syncthreads();  // Wait until the values read by the next step are written
#endif
    }
    if (threadIdx.x == 0) out[blockIdx.x] = s[0];
}

// Sum the 32 values of a warp register-to-register, without shared memory
__device__ float warpReduceSum(float v) {
    for (int offset = warpSize / 2; offset > 0; offset /= 2) {
        // Receive v from the lane offset positions above (all 32 lanes take part)
        v += __shfl_down_sync(0xffffffffu, v, offset);
    }
    return v;  // Lane 0 now holds the warp total
}

// 2) Warp-shuffle version
__global__ void reduceShfl(const float* in, float* out, int n) {
    __shared__ float warpSums[kWarps];
    float sum = 0.0f;
    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
         i += blockDim.x * gridDim.x) {
        sum += in[i];
    }
    const int lane = threadIdx.x % warpSize;
    const int warp = threadIdx.x / warpSize;
    sum = warpReduceSum(sum);
    if (lane == 0) warpSums[warp] = sum;
    __syncthreads();  // Wait until every warp's partial sum is in place
    if (warp == 0) {  // All 32 threads of warp 0 handle the next step together
        sum = (lane < kWarps) ? warpSums[lane] : 0.0f;
        sum = warpReduceSum(sum);
        if (lane == 0) out[blockIdx.x] = sum;
    }
}

// 3) atomicAdd version: read, add and write as one indivisible step
__global__ void reduceAtomic(const float* in, float* out, int n) {
    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
         i += blockDim.x * gridDim.x) {
        atomicAdd(out, in[i]);
    }
}

// 4) No synchronization: many threads read and write one address (data race = undefined behavior)
__global__ void reduceRacy(const float* in, float* out, int n) {
    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
         i += blockDim.x * gridDim.x) {
        *out += in[i];
    }
}

using Kernel = void (*)(const float*, float*, int);

// Only launches kernels (reading the result is separate). With twoPass=true,
// it computes `blocks` partial sums, then reduces them with one block
static void launch(Kernel k, bool twoPass, const float* d_in, float* d_partial,
                   float* d_out, int n, int blocks) {
    if (twoPass) {
        k<<<blocks, kThreads>>>(d_in, d_partial, n);
        CHECK(cudaGetLastError());
        k<<<1, kThreads>>>(d_partial, d_out, blocks);
        CHECK(cudaGetLastError());
    } else {
        CHECK(cudaMemsetAsync(d_out, 0, sizeof(float)));  // Zero the accumulation target
        k<<<blocks, kThreads>>>(d_in, d_out, n);
        CHECK(cudaGetLastError());
    }
}

template <typename F>
static void printOccupancy(const char* name, F func, int maxThreadsPerSM) {
    cudaFuncAttributes attr{};
    CHECK(cudaFuncGetAttributes(&attr, func));
    int activeBlocks = 0;
    CHECK(cudaOccupancyMaxActiveBlocksPerMultiprocessor(&activeBlocks, func, kThreads, 0));
    std::printf("  %-12s regs/thread=%d static smem=%zu B local=%zu B -> %d blocks/SM, "
                "theoretical occupancy %.0f%%\n",
                name, attr.numRegs, attr.sharedSizeBytes, attr.localSizeBytes, activeBlocks,
                100.0 * activeBlocks * kThreads / maxThreadsPerSM);
}

int main(int argc, char** argv) {
    const int n = (argc > 1) ? std::atoi(argv[1]) : (1 << 24);
    if (n <= 0) {
        std::fprintf(stderr, "n must be a positive integer\n");
        return EXIT_FAILURE;
    }
    const int blocks = 1024;  // Blocks in the first pass (= number of partial sums)
    const size_t bytes = static_cast<size_t>(n) * sizeof(float);

    cudaDeviceProp prop{};
    CHECK(cudaGetDeviceProperties(&prop, 0));
    std::printf("device: %s (CC %d.%d, %d SMs)\n", prop.name, prop.major, prop.minor,
                prop.multiProcessorCount);

    std::vector<float> h(n);
    unsigned seed = 2026u;
    double ref = 0.0;  // Reference: sequential sum in double
    for (int i = 0; i < n; ++i) {
        h[i] = lcg01(seed);
        ref += h[i];
    }

    float *d_in = nullptr, *d_partial = nullptr, *d_out = nullptr;
    CHECK(cudaMalloc(&d_in, bytes));
    CHECK(cudaMalloc(&d_partial, blocks * sizeof(float)));
    CHECK(cudaMalloc(&d_out, sizeof(float)));
    CHECK(cudaMemcpy(d_in, h.data(), bytes, cudaMemcpyHostToDevice));

    cudaEvent_t start, stop;
    CHECK(cudaEventCreate(&start));
    CHECK(cudaEventCreate(&stop));

    struct Case { const char* name; Kernel k; bool twoPassMode; bool judged; };
    const Case cases[] = {
        {"smem-tree", reduceSmem, true, true},
        {"warp-shfl", reduceShfl, true, true},
        {"atomicAdd", reduceAtomic, false, false},
        {"racy", reduceRacy, false, false},
    };

    const double tol = 1e-5;  // Relative-error tolerance
    bool allOk = true;
    std::printf("n=%d, reference (double) = %.6f\n", n, ref);
    for (const Case& c : cases) {
        // First run is a warm-up (keeps first-launch latency out of the timing)
        launch(c.k, c.twoPassMode, d_in, d_partial, d_out, n, blocks);
        CHECK(cudaDeviceSynchronize());

        CHECK(cudaEventRecord(start));
        launch(c.k, c.twoPassMode, d_in, d_partial, d_out, n, blocks);
        CHECK(cudaEventRecord(stop));
        CHECK(cudaEventSynchronize(stop));  // Wait for the GPU to reach `stop`
        float ms = 0.0f;
        CHECK(cudaEventElapsedTime(&ms, start, stop));

        float got = 0.0f;
        CHECK(cudaMemcpy(&got, d_out, sizeof(float), cudaMemcpyDeviceToHost));
        const double relErr = std::fabs(got - ref) / std::fabs(ref);
        const char* verdict = "(reference only)";
        if (c.judged) {
            const bool ok = relErr <= tol;
            allOk = allOk && ok;
            verdict = ok ? "PASS" : "FAIL";
        }
        std::printf("%-10s sum=%.6f rel.err=%.3e time=%.3f ms %s\n", c.name, got, relErr,
                    ms, verdict);
    }

    std::printf("occupancy (block = %d threads):\n", kThreads);
    printOccupancy("reduceSmem", reduceSmem, prop.maxThreadsPerMultiProcessor);
    printOccupancy("reduceShfl", reduceShfl, prop.maxThreadsPerMultiProcessor);

    std::printf("overall: %s (judged: smem-tree, warp-shfl; tolerance %.0e)\n",
                allOk ? "PASS" : "FAIL", tol);

    CHECK(cudaEventDestroy(start));
    CHECK(cudaEventDestroy(stop));
    CHECK(cudaFree(d_in));
    CHECK(cudaFree(d_partial));
    CHECK(cudaFree(d_out));
    return allOk ? EXIT_SUCCESS : EXIT_FAILURE;
}

Expected output (GPU results not published). The program prints the reference value computed in double, then, for each of the four methods, the sum, its relative error against the reference, and the time. The shared-memory tree and the warp-shuffle versions PASS when the relative error is at most 10−510^{-5}; atomicAdd and the racy version print "(reference only)". Finally it prints each kernel's register count, static shared memory, and the theoretical occupancy derived from the blocks per SM that cudaOccupancyMaxActiveBlocksPerMultiprocessor returns.

In the racy version, *out += in[i] makes many threads "read → add → write" the same address, so they overwrite and erase each other's updates. The CUDA Programming Guide has a similar example in which a non-atomic add yields a result smaller than the correct one that can change from run to run. No particular value is guaranteed.

The order of addition changes the error

Floating-point addition gives different results in different orders, and a GPU sum adds in a different order from a simple CPU loop. To justify the tolerance, I reproduced this program's input (2242^{24} values from the same pseudo-random generator) and its order of addition on the CPU with NumPy float32. This is a CPU simulation of the summation order, not a GPU run.

reference (float64) = 8388079.921875
warp-shfl order      sum = 8388080.000000  rel.err = 9.314e-09
smem-tree order      sum = 8388080.000000  rel.err = 9.314e-09
sequential float32   sum = 8388031.000000  rel.err = 5.832e-06

The tree orders give a relative error of about 9×10−99\times10^{-9}, roughly 600 times smaller than adding front to back in float32 (about 5.8×10−65.8\times10^{-6}). Once the running sum is large, adding small values loses their low-order digits to rounding. The 10−510^{-5} tolerance leaves ample margin over the tree-order error.

With atomicAdd the order changes from run to run, so the resulting bit pattern is not guaranteed to be the same each time. Where training must be reproducible, that can matter.

9. Prefix sum (scan): stitching blocks together in three phases

What scan is

A prefix sum, or scan, computes running totals from the start of an array. An inclusive scan puts "the total up to and including me" in each element; an exclusive scan puts "the total of everything before me".

yi=∑j=0ixj(inclusive),yi=∑j=0i−1xj(exclusive)y_i = \sum_{j=0}^{i} x_j \quad (\text{inclusive}), \qquad y_i = \sum_{j=0}^{i-1} x_j \quad (\text{exclusive})

It looks inherently sequential, since each output depends on the previous one, but it can be computed in parallel. It is a building block of many parallel algorithms: stream compaction (packing only the elements that meet a condition), radix sort, histograms and more.

Scan within a warp, and joining the blocks

The GPU Gems 3 chapter by Harris and colleagues presents the Hillis–Steele approach, which at step dd adds elements 2d2^d apart, and points out that it performs O(nlog⁡n)O(n \log n) additions and so is not "work-efficient". This program uses that approach only inside a 32-element warp (five steps of __shfl_up_sync), where fewer steps matter more than the extra work.

The whole computation has three phases:

  1. each block scans its own range and writes its block total;
  2. one block scans the array of block totals;
  3. each block adds the total of all earlier blocks to its elements.

Inside a block, the same two-level shape repeats: scan each warp → scan the warp totals → add the total of earlier warps.

  • Build: nvcc -arch=sm_75 -O3 -o scan scan.cu
  • Run: ./scan (the default element count is 220−52^{20}-5, deliberately not a multiple of the block size)
// scan.cu — inclusive prefix sum (inclusive scan) in three phases
//   Phase 1: each block scans its own range and writes its block total
//   Phase 2: one block scans the array of block totals
//   Phase 3: add "the total of all earlier blocks" to each block
// Build: nvcc -arch=sm_75 -O3 -o scan scan.cu
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>

#define CHECK(call)                                                       \
    do {                                                                  \
        cudaError_t err_ = (call);                                        \
        if (err_ != cudaSuccess) {                                        \
            std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n",          \
                         cudaGetErrorName(err_), __FILE__, __LINE__,      \
                         cudaGetErrorString(err_));                       \
            std::exit(EXIT_FAILURE);                                      \
        }                                                                 \
    } while (0)

constexpr int kBlock = 1024;  // 1 block = 32 warps; this implementation assumes this value

// Inclusive scan within a warp (Hillis-Steele style, 5 steps)
__device__ int warpInclusiveScan(int v) {
    const int lane = threadIdx.x % warpSize;
    for (int offset = 1; offset < warpSize; offset *= 2) {
        const int up = __shfl_up_sync(0xffffffffu, v, offset);  // value from the lane offset positions below
        if (lane >= offset) v += up;
    }
    return v;
}

// Phase 1 (and 2): scan within a block; writes the block total unless blockSums is nullptr
__global__ void scanBlock(const int* in, int* out, int* blockSums, int n) {
    __shared__ int warpTotals[32];
    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    const int lane = threadIdx.x % warpSize;
    const int warp = threadIdx.x / warpSize;

    int v = (i < n) ? in[i] : 0;  // Out-of-range elements count as 0 (no effect on the sums)
    v = warpInclusiveScan(v);
    if (lane == warpSize - 1) warpTotals[warp] = v;  // Each warp's total
    __syncthreads();

    if (warp == 0) {  // Scan the 32 warp totals
        warpTotals[lane] = warpInclusiveScan(warpTotals[lane]);
    }
    __syncthreads();

    if (warp > 0) v += warpTotals[warp - 1];  // Add the total of all earlier warps
    if (i < n) out[i] = v;
    if (blockSums != nullptr && threadIdx.x == blockDim.x - 1) {
        blockSums[blockIdx.x] = v;  // The block's last thread holds the block total
    }
}

// Phase 3: add the total of all earlier blocks
__global__ void addOffsets(int* out, const int* scannedSums, int n) {
    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (blockIdx.x > 0 && i < n) out[i] += scannedSums[blockIdx.x - 1];
}

int main(int argc, char** argv) {
    const int n = (argc > 1) ? std::atoi(argv[1]) : (1 << 20) - 5;
    if (n <= 0) {
        std::fprintf(stderr, "n must be a positive integer\n");
        return EXIT_FAILURE;
    }
    const int numBlocks = (n + kBlock - 1) / kBlock;
    if (numBlocks > kBlock) {
        // Phase 2 runs in a single block, so the block count is capped at kBlock
        std::fprintf(stderr, "n too large for this 3-phase demo (max %d)\n", kBlock * kBlock);
        return EXIT_FAILURE;
    }

    std::vector<int> h(n), got(n);
    unsigned state = 7u;
    for (int i = 0; i < n; ++i) {
        state = state * 1664525u + 1013904223u;
        h[i] = static_cast<int>((state >> 16) % 10u);  // Integers 0-9 (the sums fit in an int)
    }

    int *d_in = nullptr, *d_out = nullptr, *d_sums = nullptr, *d_scannedSums = nullptr;
    CHECK(cudaMalloc(&d_in, n * sizeof(int)));
    CHECK(cudaMalloc(&d_out, n * sizeof(int)));
    CHECK(cudaMalloc(&d_sums, numBlocks * sizeof(int)));
    CHECK(cudaMalloc(&d_scannedSums, numBlocks * sizeof(int)));
    CHECK(cudaMemcpy(d_in, h.data(), n * sizeof(int), cudaMemcpyHostToDevice));

    cudaEvent_t start, stop;
    CHECK(cudaEventCreate(&start));
    CHECK(cudaEventCreate(&stop));

    // Launch the three kernels in order (same stream, so they run in the order queued)
    auto runScan = [&]() {
        scanBlock<<<numBlocks, kBlock>>>(d_in, d_out, d_sums, n);
        CHECK(cudaGetLastError());
        scanBlock<<<1, kBlock>>>(d_sums, d_scannedSums, nullptr, numBlocks);
        CHECK(cudaGetLastError());
        addOffsets<<<numBlocks, kBlock>>>(d_out, d_scannedSums, n);
        CHECK(cudaGetLastError());
    };
    runScan();  // Warm-up (keeps first-launch costs out of the timing)
    CHECK(cudaDeviceSynchronize());

    CHECK(cudaEventRecord(start));
    runScan();  // Same result on every run (phase 1 rewrites all of d_out)
    CHECK(cudaEventRecord(stop));
    CHECK(cudaEventSynchronize(stop));
    float ms = 0.0f;
    CHECK(cudaEventElapsedTime(&ms, start, stop));

    CHECK(cudaMemcpy(got.data(), d_out, n * sizeof(int), cudaMemcpyDeviceToHost));

    // Require an exact element-by-element match with a sequential CPU scan (integers: no tolerance)
    std::vector<long long> ref(n);
    long long running = 0;
    for (int i = 0; i < n; ++i) {
        running += h[i];
        ref[i] = running;
    }
    long long firstBad = -1;
    for (int i = 0; i < n; ++i) {
        if (got[i] != ref[i]) {
            firstBad = i;
            break;
        }
    }
    std::printf("n=%d blocks=%d threads/block=%d\n", n, numBlocks, kBlock);
    std::printf("last element: gpu=%d cpu=%lld\n", got[n - 1], ref[n - 1]);
    if (firstBad >= 0) {
        std::printf("first mismatch at index %lld: gpu=%d cpu=%lld\n", firstBad,
                    got[firstBad], ref[firstBad]);
    }
    std::printf("time (3 kernels) = %.3f ms\n", ms);
    std::printf("%s\n", firstBad < 0 ? "PASS" : "FAIL");

    CHECK(cudaEventDestroy(start));
    CHECK(cudaEventDestroy(stop));
    CHECK(cudaFree(d_in));
    CHECK(cudaFree(d_out));
    CHECK(cudaFree(d_sums));
    CHECK(cudaFree(d_scannedSums));
    return firstBad < 0 ? EXIT_SUCCESS : EXIT_FAILURE;
}

Expected output (GPU results not published). The program prints the element and block counts, the GPU and CPU values of the last element, and the combined time of the three kernels measured after a warm-up, and prints PASS if every element matches a sequential CPU scan exactly. The inputs are integers from 0 to 9, so there is no floating-point error: a single mismatch means FAIL.

In the simulated environment described under "What has been verified" at the top, it passed for all of the element counts above and failed when I shifted an index by one (warpTotals[warp - 1] to warpTotals[warp]). What this confirms is only the logic — the indexing and where the barriers sit. Speed on a GPU, real warp behavior, and whether nvcc accepts the code remain unverified.

10. Tiled matrix multiplication: reusing data through shared memory

The problem with the naive version

An N×NN \times N matrix product C=ABC = AB costs 2N32N^3 FLOPs. In a naive kernel, one thread owns one element of CC and reads a row of AA and a column of BB straight from global memory.

Each element of AA is read by NN threads, and so is each element of BB. If no caching helped at all, every multiply-add (2 FLOPs) would read two 4-byte values — 8 bytes.

Tiling: stage in shared memory and reuse

With tiling, a block loads a small T×TT \times T tile into shared memory and its T2T^2 threads reuse it. Each thread only carries one element, and that value is used TT times inside the block, so the amount read from global memory falls to 1/T1/T of the naive version.

The number of operations per byte exchanged with global memory is the arithmetic intensity (AI). Counting the reads of AA and BB and the writes of CC (a model that ignores reuse from caches):

AI(T)=2N34(2N3T+N2)  [FLOP/byte]\mathrm{AI}(T) = \frac{2N^3}{4\left(\dfrac{2N^3}{T} + N^2\right)} \ \ [\mathrm{FLOP/byte}]

For N=1024N = 1024 that gives about 0.25 FLOP/byte for the naive version (T=1T = 1), about 3.97 for T=16T = 16 and about 7.88 for T=32T = 32 (output of the script in section 11).

  • Build: nvcc -arch=sm_75 -O3 -o matmul matmul.cu -lcublas
  • Run: ./matmul (the size of the timed matrices can be passed as an argument; the default is 1,024)
// matmul.cu — matrix product C = A * B (FP32, row-major), three ways
//   naive : one thread computes one element of C, reading global memory directly
//   tiled : TILE x TILE tiles are staged in shared memory and reused
//   cuBLAS: NVIDIA's library (PyTorch's torch.matmul also calls cuBLAS by default)
// Build: nvcc -arch=sm_75 -O3 -o matmul matmul.cu -lcublas
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cublas_v2.h>
#include <cuda_runtime.h>

#define CHECK(call)                                                       \
    do {                                                                  \
        cudaError_t err_ = (call);                                        \
        if (err_ != cudaSuccess) {                                        \
            std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n",          \
                         cudaGetErrorName(err_), __FILE__, __LINE__,      \
                         cudaGetErrorString(err_));                       \
            std::exit(EXIT_FAILURE);                                      \
        }                                                                 \
    } while (0)

#define CHECK_CUBLAS(call)                                                \
    do {                                                                  \
        cublasStatus_t st_ = (call);                                      \
        if (st_ != CUBLAS_STATUS_SUCCESS) {                               \
            std::fprintf(stderr, "cuBLAS error %d at %s:%d\n",            \
                         static_cast<int>(st_), __FILE__, __LINE__);      \
            std::exit(EXIT_FAILURE);                                      \
        }                                                                 \
    } while (0)

constexpr int TILE = 16;  // 1 block = 16 x 16 = 256 threads

static float lcgSym(unsigned& state) {  // pseudo-random value in [-1, 1)
    state = state * 1664525u + 1013904223u;
    return (state >> 8) * (2.0f / 16777216.0f) - 1.0f;
}

__global__ void matmulNaive(const float* A, const float* B, float* C, int n) {
    const int row = blockIdx.y * blockDim.y + threadIdx.y;
    const int col = blockIdx.x * blockDim.x + threadIdx.x;
    if (row < n && col < n) {
        float acc = 0.0f;
        for (int k = 0; k < n; ++k) {
            acc += A[row * n + k] * B[k * n + col];  // reads global memory every time
        }
        C[row * n + col] = acc;
    }
}

__global__ void matmulTiled(const float* A, const float* B, float* C, int n) {
    __shared__ float As[TILE][TILE];
    __shared__ float Bs[TILE][TILE];
    const int tx = threadIdx.x, ty = threadIdx.y;
    const int row = blockIdx.y * TILE + ty;
    const int col = blockIdx.x * TILE + tx;
    float acc = 0.0f;
    for (int t = 0; t < (n + TILE - 1) / TILE; ++t) {
        const int aCol = t * TILE + tx;
        const int bRow = t * TILE + ty;
        // Each thread brings one element of the A and B tiles (0 outside the matrix)
        As[ty][tx] = (row < n && aCol < n) ? A[row * n + aCol] : 0.0f;
        Bs[ty][tx] = (bRow < n && col < n) ? B[bRow * n + col] : 0.0f;
        __syncthreads();  // Wait until the whole tile is loaded
        for (int k = 0; k < TILE; ++k) {
            acc += As[ty][k] * Bs[k][tx];  // each shared-memory value is reused TILE times
        }
        __syncthreads();  // Do not overwrite with the next tile until everyone has finished reading
    }
    if (row < n && col < n) C[row * n + col] = acc;
}

// CPU reference in double, also accumulating sum|a*b| for the error bound
static void cpuReference(const std::vector<float>& A, const std::vector<float>& B,
                         std::vector<double>& ref, std::vector<double>& absRef, int n) {
    ref.assign(static_cast<size_t>(n) * n, 0.0);
    absRef.assign(static_cast<size_t>(n) * n, 0.0);
    for (int i = 0; i < n; ++i) {
        for (int k = 0; k < n; ++k) {
            const double a = A[static_cast<size_t>(i) * n + k];
            for (int j = 0; j < n; ++j) {
                const double p = a * B[static_cast<size_t>(k) * n + j];
                ref[static_cast<size_t>(i) * n + j] += p;
                absRef[static_cast<size_t>(i) * n + j] += std::fabs(p);
            }
        }
    }
}

// Worst-case FP32 dot-product rounding error is about n * 2^-24 * sum|a*b|; allow twice that
static bool verify(const char* name, const std::vector<float>& got,
                   const std::vector<double>& ref, const std::vector<double>& absRef, int n) {
    const double unit = std::ldexp(1.0, -23);  // 2 * 2^-24
    double maxAbsErr = 0.0, worstRatio = 0.0;
    for (size_t i = 0; i < got.size(); ++i) {
        const double err = std::fabs(got[i] - ref[i]);
        const double bound = n * unit * absRef[i] + 1e-30;
        maxAbsErr = std::fmax(maxAbsErr, err);
        worstRatio = std::fmax(worstRatio, err / bound);
    }
    const bool ok = worstRatio <= 1.0;
    std::printf("  %-7s max abs error = %.3e, worst err/bound = %.3f -> %s\n", name,
                maxAbsErr, worstRatio, ok ? "PASS" : "FAIL");
    return ok;
}

template <typename F>
static void printOccupancy(const char* name, F func, int threadsPerBlock, int maxThreadsPerSM) {
    cudaFuncAttributes attr{};
    CHECK(cudaFuncGetAttributes(&attr, func));
    int activeBlocks = 0;
    CHECK(cudaOccupancyMaxActiveBlocksPerMultiprocessor(&activeBlocks, func, threadsPerBlock, 0));
    std::printf("  %-12s regs/thread=%d static smem=%zu B -> %d blocks/SM, "
                "theoretical occupancy %.0f%%\n",
                name, attr.numRegs, attr.sharedSizeBytes, activeBlocks,
                100.0 * activeBlocks * threadsPerBlock / maxThreadsPerSM);
}

// Run the three methods on n x n matrices; with timed=true, time them with CUDA events
static bool runCase(int n, bool timed, cublasHandle_t handle) {
    const size_t elems = static_cast<size_t>(n) * n;
    const size_t bytes = elems * sizeof(float);
    std::vector<float> hA(elems), hB(elems), hC(elems);
    unsigned seed = 42u + static_cast<unsigned>(n);
    for (size_t i = 0; i < elems; ++i) hA[i] = lcgSym(seed);
    for (size_t i = 0; i < elems; ++i) hB[i] = lcgSym(seed);

    float *dA = nullptr, *dB = nullptr, *dC = nullptr;
    CHECK(cudaMalloc(&dA, bytes));
    CHECK(cudaMalloc(&dB, bytes));
    CHECK(cudaMalloc(&dC, bytes));
    CHECK(cudaMemcpy(dA, hA.data(), bytes, cudaMemcpyHostToDevice));
    CHECK(cudaMemcpy(dB, hB.data(), bytes, cudaMemcpyHostToDevice));

    std::vector<double> ref, absRef;
    cpuReference(hA, hB, ref, absRef, n);

    const dim3 block(TILE, TILE);
    const dim3 grid((n + TILE - 1) / TILE, (n + TILE - 1) / TILE);
    const float alpha = 1.0f, beta = 0.0f;
    const int reps = timed ? 10 : 1;
    const double flops = 2.0 * n * static_cast<double>(n) * n;

    cudaEvent_t start, stop;
    CHECK(cudaEventCreate(&start));
    CHECK(cudaEventCreate(&stop));

    bool allOk = true;
    for (int method = 0; method < 3; ++method) {
        const char* name = (method == 0) ? "naive" : (method == 1) ? "tiled" : "cuBLAS";
        auto run = [&]() {
            if (method == 0) {
                matmulNaive<<<grid, block>>>(dA, dB, dC, n);
                CHECK(cudaGetLastError());
            } else if (method == 1) {
                matmulTiled<<<grid, block>>>(dA, dB, dC, n);
                CHECK(cudaGetLastError());
            } else {
                // cuBLAS is column-major. Row-major C = A*B, viewed column-major, is C^T = B^T * A^T
                CHECK_CUBLAS(cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N, n, n, n, &alpha,
                                         dB, n, dA, n, &beta, dC, n));
            }
        };
        run();  // warm-up, and the result we verify
        CHECK(cudaDeviceSynchronize());
        CHECK(cudaMemcpy(hC.data(), dC, bytes, cudaMemcpyDeviceToHost));
        allOk = verify(name, hC, ref, absRef, n) && allOk;

        if (timed) {
            CHECK(cudaEventRecord(start));
            for (int r = 0; r < reps; ++r) run();
            CHECK(cudaEventRecord(stop));
            CHECK(cudaEventSynchronize(stop));
            float ms = 0.0f;
            CHECK(cudaEventElapsedTime(&ms, start, stop));
            const double perRun = ms / reps;
            std::printf("  %-7s %.3f ms/run, %.1f GFLOP/s\n", name, perRun,
                        flops / (perRun * 1e-3) / 1e9);
        }
    }

    CHECK(cudaEventDestroy(start));
    CHECK(cudaEventDestroy(stop));
    CHECK(cudaFree(dA));
    CHECK(cudaFree(dB));
    CHECK(cudaFree(dC));
    return allOk;
}

int main(int argc, char** argv) {
    const int n = (argc > 1) ? std::atoi(argv[1]) : 1024;
    cudaDeviceProp prop{};
    CHECK(cudaGetDeviceProperties(&prop, 0));
    std::printf("device: %s (CC %d.%d, %d SMs)\n", prop.name, prop.major, prop.minor,
                prop.multiProcessorCount);

    cublasHandle_t handle;
    CHECK_CUBLAS(cublasCreate(&handle));

    // 1) A size that is not a multiple of the tile: checks boundary handling only
    std::printf("[correctness] n=%d (not a multiple of TILE=%d)\n", 1000, TILE);
    bool ok = runCase(1000, false, handle);
    // 2) Timing
    std::printf("[timing] n=%d, 10 runs each (after 1 warm-up)\n", n);
    ok = runCase(n, true, handle) && ok;

    std::printf("occupancy (block = %d threads):\n", TILE * TILE);
    printOccupancy("matmulNaive", matmulNaive, TILE * TILE, prop.maxThreadsPerMultiProcessor);
    printOccupancy("matmulTiled", matmulTiled, TILE * TILE, prop.maxThreadsPerMultiProcessor);

    CHECK_CUBLAS(cublasDestroy(handle));
    std::printf("overall: %s\n", ok ? "PASS" : "FAIL");
    return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}

Expected output (GPU results not published). The program runs in two stages. First, with N=1000N = 1000 — not a multiple of the tile — it compares all three methods with the CPU reference to check boundary handling. Then, with N=1024N = 1024 (the default), it runs each method ten times and prints the time per run and GFLOP/s. For each method it prints max abs error and worst err/bound; a worst err/bound of 1 or less is a PASS. Finally it prints the register count and theoretical occupancy of both kernels.

Choosing the tolerance

Each method adds in a different order, so the results do not match the CPU bit for bit. The program therefore sets a per-element tolerance from the rounding-error bound for an FP32 dot product. With u=2−24u = 2^{-24}, FP32's unit roundoff, the error of an NN-term dot product is bounded by roughly:

∣c^ij−cij∣≲Nu∑k∣aikbkj∣\left| \hat{c}_{ij} - c_{ij} \right| \lesssim N u \sum_{k} \left| a_{ik} b_{kj} \right|

The program allows twice the right-hand side for each element. Any correct implementation lands inside that bound whatever the order of addition, with or without FMA (a multiply and an add with a single rounding), while an implementation with a wrong index lands far outside it. Indeed, changing Bs[k][tx] to Bs[tx][k] in the simulated environment pushed worst err/bound above 150,000 and produced a FAIL.

What the cuBLAS comparison tells you

The third method is cublasSgemm from NVIDIA's cuBLAS library. cuBLAS works in column-major order (the Fortran layout), so the row-major C=ABC = AB is computed as CT=BTATC^T = B^T A^T seen in column-major order.

A hand-written tiled kernel is only the textbook first step. Fast implementations go further: each thread computes several elements and reuses data in registers too (register tiling), and they use Tensor Cores (see Why NVIDIA GPUs are so strong at AI). Which implementation cublasSgemm picks on a T4 is something I have not checked.

11. Original analysis: theoretical occupancy on T4 and H100

Occupancy is the ratio of the warps resident on an SM to the maximum it can hold (the definition, and the caveat that higher is not always faster, are covered in section 9 of How CPUs and GPUs differ in architecture). Here the point is to compute how a T4 and an H100 differ.

The number of blocks resident on one SM is the minimum of four limits:

  1. the maximum resident threads per SM,
  2. the maximum resident blocks per SM,
  3. registers (registers per thread × threads must fit in the SM's 64K), and
  4. shared memory.

According to the CUDA C++ Best Practices Guide, registers are allocated per warp, rounded up to a multiple of 256. On H100, CUDA reserves 1 KB of shared memory per block (Hopper Tuning Guide). With these rules I computed the theoretical occupancy of this post's kernels on a T4 and an H100. Register counts are only known after compiling, so I varied them from 32 to 128.

  • Requirements: Python 3.9+ (standard library only)
  • Run: python3 occupancy.py
"""Theoretical occupancy and GEMM arithmetic intensity from the published per-SM limits.

- Limits: CUDA Programming Guide "Compute Capabilities" Tables 33/34, and the
  Hopper Tuning Guide (CUDA reserves 1 KB of shared memory per thread block)
- Registers are allocated per warp, rounded up to 256 (CUDA C++ Best Practices Guide)
- Everything here is theoretical. Check real values with
  cudaOccupancyMaxActiveBlocksPerMultiprocessor or Nsight Compute.
Python 3.9+, standard library only.
"""
from dataclasses import dataclass
import math


@dataclass(frozen=True)
class SM:
    name: str
    max_threads: int        # max resident threads per SM
    max_blocks: int         # max resident blocks per SM
    regs: int               # 32-bit registers per SM
    smem: int               # shared memory per SM (bytes)
    smem_reserved: int = 0  # shared memory CUDA reserves per block (bytes)


T4 = SM("T4 (CC 7.5)", 1024, 16, 65536, 64 * 1024)
H100 = SM("H100 (CC 9.0)", 2048, 32, 65536, 228 * 1024, smem_reserved=1024)
CC70 = SM("CC 7.0", 2048, 32, 65536, 96 * 1024)  # for checking the Best Practices Guide example


def blocks_per_sm(sm, threads, regs_per_thread, smem_per_block, warp_granularity=1):
    warps = math.ceil(threads / 32)
    regs_per_warp = math.ceil(regs_per_thread * 32 / 256) * 256  # round up to 256
    by_threads = sm.max_threads // (warps * 32)
    # warps allowed by registers (optionally floored to a multiple of warp_granularity)
    warps_by_regs = (sm.regs // regs_per_warp) // warp_granularity * warp_granularity
    by_regs = warps_by_regs // warps
    by_smem = sm.smem // (smem_per_block + sm.smem_reserved) if smem_per_block else sm.max_blocks
    return min(sm.max_blocks, by_threads, by_regs, by_smem)


def occupancy(sm, threads, regs, smem, warp_granularity=1):
    b = blocks_per_sm(sm, threads, regs, smem, warp_granularity)
    return b, b * threads / sm.max_threads


print("== Check against the Best Practices Guide example (CC 7.0, 37 regs/thread) ==")
for threads, doc in ((128, "75%"), (320, "63%")):
    b1, o1 = occupancy(CC70, threads, 37, 0)
    b4, o4 = occupancy(CC70, threads, 37, 0, warp_granularity=4)
    print(f"{threads:4d} threads: 256-rule only -> {b1} blocks ({o1:.1%}), "
          f"warps floored to x4 -> {b4} blocks ({o4:.1%}), guide says {doc}")

kernels = [  # (name, threads/block, static shared memory/block)
    ("reduceShfl 256thr", 256, 8 * 4),
    ("tiled GEMM TILE=16", 256, 2 * 16 * 16 * 4),
    ("tiled GEMM TILE=32", 1024, 2 * 32 * 32 * 4),
    ("smem 48KB 256thr", 256, 48 * 1024),
]
print("\n== Theoretical occupancy (blocks/SM, occupancy) ==")
print(f"{'kernel':20s} {'regs':>4s} | {'T4 (CC 7.5)':>16s} | {'H100 (CC 9.0)':>16s}")
for name, threads, smem in kernels:
    for regs in (32, 64, 96, 128):
        cells = []
        for sm in (T4, H100):
            b, occ = occupancy(sm, threads, regs, smem)
            cells.append("launch fails" if b == 0 else f"{b:2d} blk {occ:6.1%}")
        print(f"{name:20s} {regs:4d} | {cells[0]:>16s} | {cells[1]:>16s}")

print("\n== Max registers/thread for 100% occupancy (256-thread blocks) ==")
for sm in (T4, H100):
    best = max(r for r in range(1, 256) if occupancy(sm, 256, r, 0)[1] == 1.0)
    print(f"{sm.name}: <= {best} regs/thread")

print("\n== N x N FP32 GEMM: arithmetic intensity and bandwidth-bound ceiling (theoretical) ==")
N = 1024
devices = {  # (theoretical FP32 CUDA-core peak FLOP/s, memory bandwidth byte/s)
    "T4 (320 GB/s)": (8.1e12, 320e9),
    "T4 (300 GB/s)": (8.1e12, 300e9),
    "H100 SXM5": (66.9e12, 3352e9),
}
for dev, (peak, bw) in devices.items():
    print(f"{dev}: ridge point = {peak / bw:.1f} FLOP/byte")
for label, tile in (("naive (no reuse)", 1), ("tiled TILE=16", 16), ("tiled TILE=32", 32)):
    flops = 2 * N**3
    traffic = 4 * (2 * N**3 / tile + N**2)  # bytes: reads of A and B + writes of C
    ai = flops / traffic
    bounds = ", ".join(f"{dev} {min(p, b * ai) / 1e12:.2f}" for dev, (p, b) in devices.items())
    print(f"{label:18s} AI = {ai:5.2f} FLOP/byte -> ceiling TFLOP/s: {bounds}")

Output on my machine (Apple M4 Pro, Python 3.13.7):

== Check against the Best Practices Guide example (CC 7.0, 37 regs/thread) ==
 128 threads: 256-rule only -> 12 blocks (75.0%), warps floored to x4 -> 12 blocks (75.0%), guide says 75%
 320 threads: 256-rule only -> 5 blocks (78.1%), warps floored to x4 -> 4 blocks (62.5%), guide says 63%

== Theoretical occupancy (blocks/SM, occupancy) ==
kernel               regs |      T4 (CC 7.5) |    H100 (CC 9.0)
reduceShfl 256thr      32 |     4 blk 100.0% |     8 blk 100.0%
reduceShfl 256thr      64 |     4 blk 100.0% |     4 blk  50.0%
reduceShfl 256thr      96 |     2 blk  50.0% |     2 blk  25.0%
reduceShfl 256thr     128 |     2 blk  50.0% |     2 blk  25.0%
tiled GEMM TILE=16     32 |     4 blk 100.0% |     8 blk 100.0%
tiled GEMM TILE=16     64 |     4 blk 100.0% |     4 blk  50.0%
tiled GEMM TILE=16     96 |     2 blk  50.0% |     2 blk  25.0%
tiled GEMM TILE=16    128 |     2 blk  50.0% |     2 blk  25.0%
tiled GEMM TILE=32     32 |     1 blk 100.0% |     2 blk 100.0%
tiled GEMM TILE=32     64 |     1 blk 100.0% |     1 blk  50.0%
tiled GEMM TILE=32     96 |     launch fails |     launch fails
tiled GEMM TILE=32    128 |     launch fails |     launch fails
smem 48KB 256thr       32 |     1 blk  25.0% |     4 blk  50.0%
smem 48KB 256thr       64 |     1 blk  25.0% |     4 blk  50.0%
smem 48KB 256thr       96 |     1 blk  25.0% |     2 blk  25.0%
smem 48KB 256thr      128 |     1 blk  25.0% |     2 blk  25.0%

== Max registers/thread for 100% occupancy (256-thread blocks) ==
T4 (CC 7.5): <= 64 regs/thread
H100 (CC 9.0): <= 32 regs/thread

== N x N FP32 GEMM: arithmetic intensity and bandwidth-bound ceiling (theoretical) ==
T4 (320 GB/s): ridge point = 25.3 FLOP/byte
T4 (300 GB/s): ridge point = 27.0 FLOP/byte
H100 SXM5: ridge point = 20.0 FLOP/byte
naive (no reuse)   AI =  0.25 FLOP/byte -> ceiling TFLOP/s: T4 (320 GB/s) 0.08, T4 (300 GB/s) 0.07, H100 SXM5 0.84
tiled TILE=16      AI =  3.97 FLOP/byte -> ceiling TFLOP/s: T4 (320 GB/s) 1.27, T4 (300 GB/s) 1.19, H100 SXM5 13.30
tiled TILE=32      AI =  7.88 FLOP/byte -> ceiling TFLOP/s: T4 (320 GB/s) 2.52, T4 (300 GB/s) 2.36, H100 SXM5 26.40

What the numbers show

1. Registers bind earlier on the H100. A 256-thread block keeps 100% theoretical occupancy only at 64 or fewer registers per thread on a T4, but 32 or fewer on an H100. Both SMs have 64K registers, but the H100 can hold twice as many warps per SM (64), so each warp gets half as many registers. The same kernel using 64 registers runs at 100% on a T4 and 50% on an H100.

2. A 1,024-thread block cannot launch at 65 or more registers. With 1,024 threads (32 warps) and 65 registers each, each warp needs 65×32=208065 \times 32 = 2080 registers, rounded up to 2,304, or 73,728 for the block. That exceeds the SM's 65,536, so the kernel cannot be launched. The CUDA Programming Guide warns that unless you tell the compiler the largest block size you will launch with, via __launch_bounds__(maxThreadsPerBlock), you may get "too many resources requested for launch" errors.

3. With heavy shared-memory use, the comparison flips. A kernel using 48 KB of shared memory per block fits only one block on a T4 (64 KB per SM): 25% occupancy. An H100 (228 KB, with 1 KB reserved per block) fits four: 50%.

4. The published rules alone do not reproduce the Best Practices Guide's example. The guide says that on CC 7.0 with 37 registers per thread, 128-thread blocks give 12 blocks (75%) and 320-thread blocks give 4 blocks (63%). Rounding to 256 alone matches the first case but gives 5 blocks (78.1%) for the second. Flooring the register-limited warp count to a multiple of 4 reproduces both. That "multiple of 4", however, is not stated in the guide; it is my inference. Nsight Compute's Python occupancy API exposes a warps_allocation_granularity field, but its value is not documented.

So read the table as theoretical upper bounds (it also ignores shared-memory allocation granularity). The real values can be checked with the cudaOccupancyMaxActiveBlocksPerMultiprocessor results that the programs in sections 8 and 10 print, or with Nsight Compute.

Performance ceilings from arithmetic intensity

The second half of the script compares the arithmetic intensity from section 10 with the ridge point — the ratio of theoretical FP32 peak to memory bandwidth, where a kernel moves from bandwidth-bound to compute-bound (the idea is covered in section 6 of Why NVIDIA GPUs are so strong at AI). The T4 comes out at about 25.3 FLOP/byte (using the whitepaper's 320 GB/s; the datasheet says 300 GB/s, an inconsistency between NVIDIA's own documents, which would give about 27.0), and the H100 SXM5 at about 20.0. Even at T=32T = 32 the arithmetic intensity is only about 7.9, so in this model the tiled multiply is still bandwidth-bound. Keep in mind, though, that the model assumes "every load goes to DRAM" and ignores cache reuse; a real naive kernel may well run faster than this ceiling thanks to the L1 and L2 caches.

12. Streams and events: overlapping transfers with compute

A stream is a queue

A CUDA stream is a queue that orders work sent to the GPU (copies and kernels). Work in one stream runs in the order it was queued, with no overtaking. Work in different streams may run concurrently, so if you split the data and run "copy in → compute → copy out" in a separate stream per chunk, the next chunk can be copied in while the current one is being computed.

Three conditions for overlap

Overlapping transfers with compute requires:

  1. Pinned host memory. Given ordinary memory (malloc, std::vector), cudaMemcpyAsync falls back to synchronous behavior and overlaps with nothing. Allocate pinned (page-locked) memory with cudaMallocHost. It is a scarce resource, though, and overusing it can slow down the whole system.
  2. Separate, non-default streams. According to the Best Practices Guide, operations in the default stream do not begin until all preceding operations in every stream have finished, so overlap needs non-default streams. cudaStreamNonBlocking only matters for implicit synchronization with the legacy default stream; this program never uses the default stream but sets the flag anyway, to be safe.
  3. Enough copy engines. With one copy engine, a GPU can run a transfer in one direction alongside kernels; with two, it can copy in, copy out and run kernels at the same time. The count is in asyncEngineCount. I have not checked the value on a T4 (the program prints it).

Estimating the benefit of overlap

Let's estimate, in theory, how long the chunked version takes. Let the times to copy in, compute and copy out the whole dataset be HH, KK and DD, and split it into nn equal chunks (h=H/nh = H/n and so on). Assuming two copy engines and full concurrency of the three stages, a three-stage pipeline takes:

Toverlap≈h+k+d+(n−1)max⁡(h,k,d)T_{\mathrm{overlap}} \approx h + k + d + (n - 1)\max(h, k, d)

Running serially takes H+K+DH + K + D. If H=K=DH = K = D, then n=4n = 4 gives 1.5H1.5H, half of the serial 3H3H. No nn can go below max⁡(H,K,D)\max(H, K, D), so the ceiling in this case is 3×. This is a theoretical figure that ignores per-chunk launch overhead and contention for PCIe bandwidth. With a single copy engine, copying in and out serialize and the benefit shrinks.

  • Build: nvcc -arch=sm_75 -O3 -o streams streams.cu
  • Run: ./streams [iterations] (the kernel's cost scales with the iteration count; the default is 500)
// streams.cu — overlapping transfers and compute with CUDA streams
//   A) one stream: "copy everything in -> compute everything -> copy everything out"
//   B) split the data into kStreams chunks; each stream runs "copy in -> compute -> copy out"
// Both use pinned (page-locked) host memory, a precondition for asynchronous copies.
// Build: nvcc -arch=sm_75 -O3 -o streams streams.cu
// Run:   ./streams [iterations inside the kernel (default 500)]
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>

#define CHECK(call)                                                       \
    do {                                                                  \
        cudaError_t err_ = (call);                                        \
        if (err_ != cudaSuccess) {                                        \
            std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n",          \
                         cudaGetErrorName(err_), __FILE__, __LINE__,      \
                         cudaGetErrorString(err_));                       \
            std::exit(EXIT_FAILURE);                                      \
        }                                                                 \
    } while (0)

constexpr int kStreams = 4;
constexpr int kThreads = 256;

// Kernel whose cost scales with the iteration count (a contraction that pulls v toward 1.0)
__global__ void heavy(const float* in, float* out, int n, int iters) {
    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        float v = in[i];
        for (int k = 0; k < iters; ++k) v = v * 0.999f + 0.001f;
        out[i] = v;
    }
}

int main(int argc, char** argv) {
    const int iters = (argc > 1) ? std::atoi(argv[1]) : 500;
    const int n = 1 << 24;                 // 16M elements = 64 MiB
    const int chunk = n / kStreams;        // n is divisible by kStreams
    const size_t bytes = static_cast<size_t>(n) * sizeof(float);
    const size_t chunkBytes = static_cast<size_t>(chunk) * sizeof(float);

    cudaDeviceProp prop{};
    CHECK(cudaGetDeviceProperties(&prop, 0));
    std::printf("device: %s, asyncEngineCount=%d (copy engines)\n", prop.name,
                prop.asyncEngineCount);

    float *hIn = nullptr, *hOut = nullptr;
    CHECK(cudaMallocHost(&hIn, bytes));   // pinned memory
    CHECK(cudaMallocHost(&hOut, bytes));
    for (int i = 0; i < n; ++i) hIn[i] = static_cast<float>(i % 1000) / 1000.0f;

    // CPU reference. The input has only 1000 distinct values, so compute those 1000 and look them up
    // (the GPU may fuse the multiply and add into an FMA, so allow a small difference)
    std::vector<float> table(1000);
    for (int r = 0; r < 1000; ++r) {
        float v = static_cast<float>(r) / 1000.0f;
        for (int k = 0; k < iters; ++k) v = v * 0.999f + 0.001f;
        table[r] = v;
    }
    auto check = [&](const char* name) {
        double maxErr = 0.0;
        for (int i = 0; i < n; ++i) {
            const double err = std::fabs(static_cast<double>(hOut[i]) - table[i % 1000]);
            maxErr = std::fmax(maxErr, err);
        }
        const bool ok = maxErr <= 1e-4;
        std::printf("  %-8s max abs error = %.3e (tolerance 1e-4) %s\n", name, maxErr,
                    ok ? "PASS" : "FAIL");
        return ok;
    };

    float *dIn = nullptr, *dOut = nullptr;
    CHECK(cudaMalloc(&dIn, bytes));
    CHECK(cudaMalloc(&dOut, bytes));

    cudaStream_t streams[kStreams];
    cudaEvent_t done[kStreams];
    for (int s = 0; s < kStreams; ++s) {
        CHECK(cudaStreamCreateWithFlags(&streams[s], cudaStreamNonBlocking));
        CHECK(cudaEventCreateWithFlags(&done[s], cudaEventDisableTiming));
    }
    cudaEvent_t start, stop;
    CHECK(cudaEventCreate(&start));
    CHECK(cudaEventCreate(&stop));

    // A) Serial execution in one stream
    auto runSerial = [&]() {
        cudaStream_t s0 = streams[0];
        CHECK(cudaEventRecord(start, s0));
        CHECK(cudaMemcpyAsync(dIn, hIn, bytes, cudaMemcpyHostToDevice, s0));
        heavy<<<(n + kThreads - 1) / kThreads, kThreads, 0, s0>>>(dIn, dOut, n, iters);
        CHECK(cudaGetLastError());
        CHECK(cudaMemcpyAsync(hOut, dOut, bytes, cudaMemcpyDeviceToHost, s0));
        CHECK(cudaEventRecord(stop, s0));
        CHECK(cudaEventSynchronize(stop));
    };

    // B) Split across kStreams streams; dependencies on start/stop are explicit via events
    auto runOverlapped = [&]() {
        CHECK(cudaEventRecord(start, streams[0]));
        for (int s = 1; s < kStreams; ++s) {
            CHECK(cudaStreamWaitEvent(streams[s], start, 0));  // start after `start`
        }
        for (int s = 0; s < kStreams; ++s) {
            const size_t off = static_cast<size_t>(s) * chunk;
            CHECK(cudaMemcpyAsync(dIn + off, hIn + off, chunkBytes, cudaMemcpyHostToDevice,
                                  streams[s]));
            heavy<<<(chunk + kThreads - 1) / kThreads, kThreads, 0, streams[s]>>>(
                dIn + off, dOut + off, chunk, iters);
            CHECK(cudaGetLastError());
            CHECK(cudaMemcpyAsync(hOut + off, dOut + off, chunkBytes, cudaMemcpyDeviceToHost,
                                  streams[s]));
            CHECK(cudaEventRecord(done[s], streams[s]));
        }
        for (int s = 1; s < kStreams; ++s) {
            CHECK(cudaStreamWaitEvent(streams[0], done[s], 0));  // wait for every stream to finish
        }
        CHECK(cudaEventRecord(stop, streams[0]));
        CHECK(cudaEventSynchronize(stop));
    };

    bool ok = true;
    float ms = 0.0f;
    std::printf("n=%d floats (%.0f MiB), iters=%d, streams=%d\n", n, bytes / 1048576.0, iters,
                kStreams);

    runSerial();  // warm-up
    for (int i = 0; i < n; ++i) hOut[i] = 0.0f;  // clear, so that only the timed run's output is verified
    runSerial();
    CHECK(cudaEventElapsedTime(&ms, start, stop));
    std::printf("A) serial (1 stream)     : %.3f ms\n", ms);
    ok = check("serial") && ok;

    runOverlapped();  // warm-up
    for (int i = 0; i < n; ++i) hOut[i] = 0.0f;  // clear, so that only the timed run's output is verified
    runOverlapped();
    CHECK(cudaEventElapsedTime(&ms, start, stop));
    std::printf("B) overlapped (%d streams): %.3f ms\n", kStreams, ms);
    ok = check("overlap") && ok;

    for (int s = 0; s < kStreams; ++s) {
        CHECK(cudaStreamDestroy(streams[s]));
        CHECK(cudaEventDestroy(done[s]));
    }
    CHECK(cudaEventDestroy(start));
    CHECK(cudaEventDestroy(stop));
    CHECK(cudaFree(dIn));
    CHECK(cudaFree(dOut));
    CHECK(cudaFreeHost(hIn));
    CHECK(cudaFreeHost(hOut));
    std::printf("overall: %s\n", ok ? "PASS" : "FAIL");
    return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}

Expected output (GPU results not published). The program prints the GPU name and asyncEngineCount, then the time in milliseconds for A (serial, one stream) and B (split across four streams), and compares each result (from the timed run, with the output cleared after the warm-up) with the CPU reference to print PASS or FAIL. Overlap is easiest to see when kernel time and copy time are similar; if one dominates, adjust the iteration count on the command line.

A few notes:

  • The GPU may fuse v * 0.999f + 0.001f into an FMA. According to the NVCC documentation, --fmad is on by default. The CPU reference is computed without FMA, so the results differ slightly. When I computed the gap between the FMA and non-FMA versions for 1,000 inputs on my Mac, it was at most 6.3×10−66.3\times10^{-6} at 500 iterations and at most 1.3×10−51.3\times10^{-5} even at 100,000 iterations. The 10−410^{-4} tolerance adds margin on top of that.
  • B makes its start and end dependencies explicit with cudaStreamWaitEvent, rather than relying on the default stream's implicit synchronization.
  • According to the CUDA Runtime API documentation, events recorded in a non-default stream may report a longer time than expected, because other work can run in between. Check exactly how the work overlaps on the Nsight Systems timeline (section 13).

Asynchronous copies inside a kernel: cp.async and TMA

Overlap with streams is about transfers between CPU and GPU. Separately, there are mechanisms that make the transfer from global memory to shared memory asynchronous inside a kernel:

  • cp.async (LDGSTS): CC 8.0 (Ampere) and later. A thread starts a copy from global to shared memory and continues computing without waiting for it. The PTX ISA document says it "Requires sm_80 or higher".
  • TMA (Tensor Memory Accelerator): CC 9.0 (Hopper) and later. It moves large blocks, or tensors of up to five dimensions, between global and shared memory asynchronously in one operation.

Neither is available on the T4 (CC 7.5). The tiled multiply in this post keeps the synchronous shape load → __syncthreads() → compute → __syncthreads().

13. Nsight Systems and Nsight Compute: which to reach for first

NVIDIA has two performance tools with completely different jobs.

Nsight SystemsNsight Compute
What it showsA timeline of the whole application (CPU threads, CUDA API calls, kernels, copies, streams)The inside of one kernel (compute and memory utilization, occupancy, traffic at each level of the memory hierarchy)
Official descriptionA system-wide performance analysis toolAn interactive kernel profiler
Question it answers"Where is the GPU idle?" "Do transfers and compute overlap?""Is this kernel compute-bound or bandwidth-bound?" "Why is it slow?"
How it measuresRecords a single run; does not lock clocksReplays the kernel several times to collect metrics; locks SM clocks by default and flushes caches between replay passes
CLI examplensys profile -o report ./streamsncu --section SpeedOfLight --section Occupancy -k regex:matmul ./matmul

The order is Nsight Systems first, then Nsight Compute. Studying one kernel in detail before seeing the whole timeline risks optimizing something that is not the bottleneck; the GPU is often waiting on CPU-side preprocessing or data transfers. Right-clicking a kernel on the Nsight Systems timeline offers to examine it in Nsight Compute.

The Nsight Compute documentation names clock locking and cache flushing as the reasons the two tools can report different durations for the same kernel. For comparisons it recommends locking clocks externally with nvidia-smi and passing --clock-control none to ncu.

For correctness, use Compute Sanitizer: memcheck (out-of-bounds access), racecheck (shared-memory races), initcheck (uninitialized reads) and synccheck (synchronization errors). Building reduce.cu from section 8 with -DOMIT_SYNC removes the __syncthreads() in the shared-memory tree, which compute-sanitizer --tool racecheck should flag as a race (I have not run it). Note that racecheck looks at shared-memory races; the global-memory race version (reduceRacy) is outside its scope.

14. Where PyTorch fits: from torch.matmul down to kernels

If you use PyTorch day to day, where does all this show up?

  • Asynchronous execution: PyTorch's documentation says "By default, GPU operations are asynchronous." Operations are queued on the GPU, not executed immediately. So timing with time.time() alone is inaccurate; you need torch.cuda.synchronize() or torch.cuda.Event(enable_timing=True) — the same reason the programs here use cudaEventRecord. To find where an error occurred, the environment variable CUDA_LAUNCH_BLOCKING=1 forces synchronous execution.
  • Matrix multiplication is cuBLAS: according to the description of torch.backends.cuda.preferred_blas_library, PyTorch defaults to cuBLAS for CUDA BLAS operations even when both cuBLAS and cuBLASLt are available. Comparing against cublasSgemm in section 10 means comparing against what torch.matmul calls under the hood.
  • Convolutions and more are cuDNN: the torch.backends.cudnn settings control the use of cuDNN.
  • Streams and pinned memory: torch.cuda.Stream creates streams. Combining DataLoader(pin_memory=True) with .to(device, non_blocking=True) is exactly section 12's "pinned memory + asynchronous copy".
  • Custom kernels: torch.utils.cpp_extension.load_inline builds and loads C++ and CUDA code given as strings at run time. PyTorch's tutorial shows how to register an operator with TORCH_LIBRARY and its CUDA implementation with TORCH_LIBRARY_IMPL(..., CUDA, m).
  • torch.compile: according to PyTorch's documentation, TorchInductor, the default compiler, uses OpenAI Triton as its key building block on NVIDIA GPUs to generate kernels. It is a way to fuse several operations into one kernel without writing CUDA C++.

In other words, a line of PyTorch ends up running on the GPU as a cuBLAS or cuDNN kernel, one of PyTorch's own CUDA kernels, or a kernel generated by Triton. Which one actually runs can be seen on the Nsight Systems timeline.

15. Should you write CUDA? Limits and how to decide

When CUDA disappoints

Turn the above inside out and the conditions under which CUDA disappoints become clear.

  • Too little parallel work. For an array of a few thousand elements, kernel-launch and transfer costs can dominate. The Best Practices Guide contrasts device-memory bandwidth (898 GB/s in its V100 example) with PCIe x16 Gen3 (16 GB/s) and advises keeping transfers to a minimum.
  • Strong dependencies between blocks. With no grid-wide barrier, a computation where every element depends on all results of the previous step has to be split into many kernels.
  • Heavy branching that splits warps. With a lot of divergence, a warp executes both paths and most of its lanes sit idle.
  • Floating-point reproducibility is required. As section 8 showed, the order of addition changes the result, and a sum built with atomicAdd can produce a different bit pattern on every run.
  • Portability is required. CUDA code runs only on NVIDIA GPUs — and not on Macs.
  • Maintenance cost. A custom kernel must be re-checked for performance whenever the GPU generation changes, whereas NVIDIA updates cuBLAS and cuDNN for each new generation.

The order in which to pick tools

Here is the order in which to pick tools when you need something to run faster on a GPU.

SituationReach for firstWhen to consider a custom kernel
Standard operations: matrix multiply, convolution, attentioncuBLAS, cuDNN, the framework's built-in operationsAlmost never
Many small operations chained togethertorch.compile (fusion via Triton)Memory round trips still dominate after fusion
A custom operation no library providesHigher-level tools such as Triton or CuPyYou need fine control of shared memory or warp-level operations
You do not know what is slowLook at the whole run in Nsight SystemsOnly after the bottleneck is pinned to one specific kernel

When you do write one, proceed in this order: (1) build a CPU reference and a justified tolerance first; (2) confirm the bottleneck with Nsight Systems; (3) estimate what bounds the kernel from its arithmetic intensity (if it is bandwidth-bound, cut reads and writes through tiling or fusion first); (4) use occupancy only to check it is not too low; and (5) confirm both performance and correctness with Nsight Compute and Compute Sanitizer.

The single-T4 inference service I have worked on never needed a hand-written CUDA kernel (GPU pipeline for AI video localization). Still, the facts that kernel launches are asynchronous, and that GPU memory is limited so where data lives decides performance, fed directly into decisions such as vLLM settings and running jobs one at a time. The value of knowing how CUDA works, whether or not you ever write a kernel, is being able to reason about what is happening on the GPU.

In practice, what pays off first is less whether to write a kernel than which GPU to use and how many. Whether the weights and KV cache fit in VRAM, and whether self-hosting beats paying per API call, are worked out in Quantization and the VRAM budget and API or self-hosting.

Summary

  • CUDA is a model and toolbox in which the host (CPU) launches huge numbers of threads on the device (GPU). Launches are asynchronous, and the memories are separate by default. It does not normally run on Apple Silicon Macs.
  • Threads come in three levels — grid, block, thread — and a 32-thread warp executes one instruction at a time. A T4 keeps up to 40,960 threads resident on 2,560 arithmetic units and hides latency by switching warps.
  • NVCC turns CUDA C++ into PTX through NVVM (LLVM-based), and ptxas turns PTX into SASS. PTX is a virtual ISA that spans generations; ship it in the binary and the driver can JIT-compile it. SASS is per-generation machine code.
  • There are five kinds of memory — registers, shared, local, constant, global. Coalescing at the warp level governs global-memory performance, and bank conflicts govern shared memory.
  • Synchronization is hierarchical. __syncthreads() covers a block and shuffles a warp; there is no grid-wide barrier. The memory model is weak, and from CC 7.0 even a warp does not line up automatically.
  • Original analysis: from the published per-SM limits, a 256-thread block keeps 100% theoretical occupancy only at 64 or fewer registers per thread on a T4 and 32 or fewer on an H100. The Best Practices Guide's example cannot be reproduced from the documented rules alone, so treat the table as theoretical upper bounds.
  • In practice, start with cuBLAS, cuDNN and torch.compile, confirm the bottleneck with Nsight Systems, and only then consider a custom kernel.

Frequently asked questions

Can I run CUDA programs on a Mac?
Normally, no. Apple Silicon Macs have no NVIDIA GPU, and according to NVIDIA's release notes CUDA 10.2 was the last CUDA release to support macOS. To try CUDA, use Google Colab or an NVIDIA GPU in the cloud. PyTorch on a Mac uses Apple's Metal through the mps device, but that is a different mechanism from CUDA, and CUDA C++ code will not run on it as is.
What is the difference between PTX and SASS?
PTX is a virtual instruction set for NVIDIA GPUs. It is text, and it is not tied to one GPU generation. SASS is the machine code the GPU actually executes, and it is not compatible across GPUs with different major compute capability numbers. NVCC compiles CUDA C++ to PTX, then ptxas turns it into SASS. If you embed PTX in the binary, the driver can JIT-compile it so the program also runs on future GPUs.
Is the number of CUDA threads the same as the number of CUDA cores?
No. A CUDA thread is a logical unit of execution with its own registers and program position. A CUDA core is a physical FP32 lane. A T4 has 2,560 CUDA cores but can keep up to 1,024 threads resident per SM, or 40,960 across the GPU. A grid can contain millions of threads; those that do not fit run as earlier blocks finish.
Does __syncthreads() synchronize every thread on the GPU?
No. __syncthreads() waits only for the threads of the same thread block. The execution order of blocks is not guaranteed, so in an ordinary kernel one block cannot wait for another block's results. To combine results across blocks, use atomics or split the work into separate kernels and synchronize at the boundary. Calling __syncthreads() inside a branch that only some threads of the block take can hang the kernel or cause undefined behavior.
Can I write CUDA kernels in Python alone?
Yes. There are three common routes. Triton is a language and compiler for writing GPU kernels in Python, and PyTorch's torch.compile uses Triton on NVIDIA GPUs. CuPy's RawKernel takes CUDA C++ source as a string and compiles it at run time, so the kernel body is still CUDA C++. Numba-CUDA turns a Python function into a SIMT kernel with @cuda.jit (Numba's built-in CUDA target is deprecated; development moved to NVIDIA's numba-cuda). All of them need an NVIDIA GPU, and the ideas of synchronization are the same as in this post.

References

友田

友田 陽大

Developer of a METI Minister's Award–winning product. With TypeScript + Python + AWS, I deliver SaaS, industry DX, and production-grade generative AI (RAG) end to end — from requirements to infrastructure and operations — single-handedly.

Do you actually know whether self-hosting beats the API?

Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing

When to use AWQ / GPTQ / FP8 / GGUF, VRAM estimates that include the KV cache, the break-even point against per-token API pricing, and the license terms of Apache-2.0 / Llama / Qwen / Gemma. From hands-on experience adopting commercially usable open quantized models stage by stage against license, quality, and speed trade-offs, I help you assemble the decision criteria before you implement.

Available for both project-based (contract) and advisory engagements. Start with a free 30-minute consult.

Also worth reading