"GPU programming with CUDA" can sound like it needs a special language or heavy mathematics. What you actually write is C++ with a handful of extensions. The hard part is not the syntax. It is learning to think in a world where tens of thousands of threads run at once, memory comes in several layers, and synchronization only reaches so far.
The short answer first. CUDA is a programming model in which the CPU (the host) tells the GPU (the device) to "run this one function on a huge number of threads", together with the compiler, driver and libraries that make that work. A function that runs on the GPU (a kernel) is marked __global__ and launched from the host as vecAdd<<<number of blocks, threads per block>>>(a, b, c, n); each thread computes which element it owns from blockIdx.x * blockDim.x + threadIdx.x. Memory comes in three main tiers: fast, small registers private to each thread; shared memory, shared within a block; and global memory (the GPU's DRAM), visible to every thread but far away. NVCC compiles your CUDA C++ into PTX, a virtual instruction set, and then into SASS, the machine code of a particular GPU generation. What decides speed is usually not the arithmetic itself but where the data lives, how it is read, and where threads synchronize.
Sections 1–7 cover the host and device, the thread hierarchy, the compilation pipeline, the memory hierarchy and synchronization. Sections 8–14 write reduction, scan, tiled matrix multiplication and streams as complete programs, compute theoretical occupancy for a T4 and an H100, and place the profilers and PyTorch in the picture. The final section, 15, covers limits and how to decide.
What has been verified. Each program is complete and compiles on its own (which is why every one defines its own CHECK macro), and each compares its result with a CPU reference and prints PASS or FAIL. My machine (Apple M4 Pro) has no NVIDIA GPU, so I checked the logic in an environment that simulates CUDA threads with std::thread, using boundary sizes (1, 31, 32, 33, 1,024, 1,025, 5,000 and 70,001 elements for scan; small matrices of 37 to 64 for the matrix multiply, and so on) and deliberately injected bugs, and I computed the floating-point error and occupancy on the CPU. No GPU timings appear in this post. The target GPU is the T4 (sm_75). My hands-on experience stops at an inference service that runs Qwen3-8B (AWQ-quantized) on vLLM on a single Tesla T4; I have not developed CUDA kernels professionally. The code is study code written against NVIDIA's official documentation.
Tensor Cores, number formats, HBM and NVLink are covered in Why NVIDIA GPUs are so strong at AI, so I won't repeat them here.
1. What CUDA is: not a language, but a model plus a toolbox
"CUDA" names several things at once.
- A programming model: a way of treating the GPU as a device that runs many threads. It defines the thread hierarchy, the kinds of memory, and the rules for synchronization.
- CUDA C++: C++ with a few extensions — specifiers such as
__global__, the<<< >>>kernel-launch syntax, and built-in variables. NVCC supports the C++03 through C++23 language standards. - The CUDA Runtime API and Driver API: the APIs that allocate and copy memory and launch kernels. The Runtime API used in this post (functions starting with
cuda) is built on top of the lower-level Driver API (functions starting withcu), and the same functionality can be achieved with the Driver API alone. The Driver API works explicitly with contexts (CUcontext) and modules (CUmodule), loading PTX or cubins as modules before launching kernels. A few features, such as virtual memory management, exist only in the Driver API, but most applications get full performance from the Runtime API alone. - NVCC and PTX: the compiler and the virtual instruction set it targets, covered in section 5.
- The NVIDIA driver: in the CUDA Programming Guide's words, something like "the operating system of the GPU". It ships as a product separate from the CUDA Toolkit.
- Libraries and tools: cuBLAS (linear algebra), cuDNN (neural networks), Nsight Systems and Nsight Compute (profilers), Compute Sanitizer (correctness checking), and more.
One constraint up front: NVIDIA CUDA does not normally run on Apple Silicon Macs. Those Macs have no NVIDIA GPU. CUDA 10.2's release notes also state that "CUDA 10.2 (Toolkit and NVIDIA driver) is the last release to support macOS for developing and running CUDA applications." PyTorch on a Mac uses Apple's Metal through the mps device, which is a different mechanism from CUDA. To run the code in this post, use Google Colab or an NVIDIA GPU in the cloud.
Do you actually know whether self-hosting beats the API?
Self-hosted LLM technology selection — quantization, GPU budget, commercial licensing
2. Host and device: the CPU gives orders, the GPU runs them on many threads
Two computers, two memories
CUDA assumes a heterogeneous system. The CUDA Programming Guide calls the CPU and the memory attached to it the host and host memory, and the GPU and its memory the device and device memory.
A program always starts on the CPU. Code running there (host code) uses the CUDA APIs to do three things:
- copy data between host memory and device memory,
- start code running on the GPU (launching a kernel), and
- wait for copies or kernels to finish.
A function that runs on the GPU is called a kernel, for historical reasons. Launching a kernel starts many GPU threads executing the same kernel code in parallel.
Think of the host as a site manager and the device as a factory with thousands of workers. The manager brings the materials (data) into the factory, hands out a single instruction sheet (the kernel) saying "everyone, do this", and collects the finished product. Every worker holds the same sheet, but each looks at their own badge number to decide which piece of material is theirs.
Kernel launches are asynchronous
Here is a big difference from ordinary CPU code. A kernel launch is asynchronous with respect to the host thread. The launch returns without waiting for the GPU to finish the kernel — possibly before it has even started. Before using the result you must synchronize, with cudaDeviceSynchronize() (wait for all work issued to the GPU) or with the streams and events of section 12.
The memories are separate by default too. Device memory allocated with cudaMalloc cannot be read directly by CPU code. Unified memory (cudaMallocManaged) is reachable from both CPU and GPU, but the CUDA Programming Guide notes that the best performance still comes from keeping data migration to a minimum and accessing data from the processor it lives next to. This post uses explicit copies, which make it obvious what runs when.
3. The thread hierarchy: grid, block, thread — and warp
Three levels
When you launch a kernel, its threads are organized in three levels:
- Thread: the smallest unit that executes the kernel code
- Thread block (block from here on): a group of threads; on current GPUs, up to 1,024 threads per block
- Grid: a group of blocks; one kernel launch is one grid
Blocks and grids can be laid out in one, two or three dimensions. Each thread learns where it is from built-in variables.
| Built-in variable | Meaning |
|---|---|
threadIdx.x/y/z | The thread's index within its block (from 0) |
blockIdx.x/y/z | The block's index within the grid (from 0) |
blockDim.x/y/z | The block's size (in threads) |
gridDim.x/y/z | The grid's size (in blocks) |
In one dimension, the standard way to compute which element a thread owns is:
A grid can have up to blocks in the x dimension.
Blocks are independent of each other
The most important property of this hierarchy is that blocks are independent. All threads of one block run on the same SM (Streaming Multiprocessor, the GPU's compute building block). Which SM a block lands on, and in what order, is not guaranteed. So, as a rule, a block cannot rely on results from another block of the same grid.
In exchange for that restriction, the same program runs on a GPU with one SM or with hundreds: blocks are simply handed out to whichever SMs are free.
Compute capability (CC, the number that identifies a GPU generation's feature set) 9.0 and later add an optional level called clusters, whose blocks are scheduled together. The code in this post does not use them.
Warps: 32 threads executing the same instruction
Inside a block, threads are executed in groups of 32 called warps, which run the same instruction in SIMT style (Single Instruction, Multiple Threads). A thread's position within its warp (0–31) is its lane. When a branch splits the lanes, the warp runs both paths one after the other (warp divergence), so block sizes should be multiples of 32 and branches within a warp kept to a minimum. SIMT and divergence are covered in detail in section 8 of How CPUs and GPUs differ in architecture.
Do not confuse logical threads with physical units
Now to clear up a common misunderstanding: a CUDA thread is a logical unit of execution, and it does not correspond to an arithmetic unit.
Take a T4 and compute from the published figures (the Turing whitepaper and the CUDA Programming Guide's compute-capability tables):
| T4 (CC 7.5) | H100 SXM5 (CC 9.0) | |
|---|---|---|
| SMs | 40 | 132 |
| FP32 units (CUDA cores) per SM | 64 | 128 |
| FP32 units on the GPU | 2,560 | 16,896 |
| Max resident threads per SM | 1,024 | 2,048 |
| Max resident threads on the GPU (theoretical) | 40,960 | 270,336 |
A T4 can keep up to 40,960 threads resident on 2,560 arithmetic units — exactly 16 to 1 (the H100 has the same ratio). Resident threads are not all computing at once. Every cycle, an SM's warp scheduler picks a warp that is ready to run its next instruction and issues it; if a warp is waiting on a memory load, the scheduler switches to another. A warp's execution context (program counters, registers) stays on chip for the warp's whole lifetime, so switching between warps costs nothing.
On top of that, a grid can launch millions of threads. Those that do not fit wait until earlier blocks finish. The point is not "2,560 CUDA cores, so 2,560 threads" but "keep many threads resident and hide waiting time behind other warps' work". That is the basic idea of a GPU.
This is also where the contrast with a CPU shows. The CUDA Programming Guide says: "Unlike CPU cores, SMs issue instructions in order and do not perform branch prediction or speculative execution." A CPU spends chip area on making one thread go fast; a GPU hides latency by switching among many warps.
4. The smallest kernel: vector addition as a complete program
Let's check these ideas with the smallest complete program: one that adds two arrays element by element.
- Requirements: the CUDA Toolkit (
nvcc) and an NVIDIA GPU; target T4 (sm_75) - Build:
nvcc -arch=sm_75 -O3 -o vector_add vector_add.cu - Run:
./vector_add
// vector_add.cu — the smallest useful CUDA program (c[i] = a[i] + b[i])
// Build: nvcc -arch=sm_75 -O3 -o vector_add vector_add.cu
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>
// Check every CUDA API return value; on failure print where and which error, then exit
#define CHECK(call) \
do { \
cudaError_t err_ = (call); \
if (err_ != cudaSuccess) { \
std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n", \
cudaGetErrorName(err_), __FILE__, __LINE__, \
cudaGetErrorString(err_)); \
std::exit(EXIT_FAILURE); \
} \
} while (0)
// Tiny pseudo-random generator for reproducibility (float in [0, 1))
static float lcg01(unsigned& state) {
state = state * 1664525u + 1013904223u;
return (state >> 8) * (1.0f / 16777216.0f);
}
// __global__ = launched from the host; many GPU threads run this same code
__global__ void vecAdd(const float* a, const float* b, float* c, int n) {
// Index of the element this thread owns (unique across the whole grid)
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) { // Spare threads in the last block do nothing
c[i] = a[i] + b[i];
}
}
int main() {
const int n = (1 << 20) + 3; // Deliberately not a multiple of the block size
const size_t bytes = static_cast<size_t>(n) * sizeof(float);
std::vector<float> ha(n), hb(n), hc(n);
unsigned seed = 12345u;
for (int i = 0; i < n; ++i) {
ha[i] = lcg01(seed);
hb[i] = lcg01(seed);
}
float *da = nullptr, *db = nullptr, *dc = nullptr;
CHECK(cudaMalloc(&da, bytes));
CHECK(cudaMalloc(&db, bytes));
CHECK(cudaMalloc(&dc, bytes));
CHECK(cudaMemcpy(da, ha.data(), bytes, cudaMemcpyHostToDevice));
CHECK(cudaMemcpy(db, hb.data(), bytes, cudaMemcpyHostToDevice));
const int threads = 256;
const int blocks = (n + threads - 1) / threads; // Ceiling division
vecAdd<<<blocks, threads>>>(da, db, dc, n);
CHECK(cudaGetLastError()); // Catches launch-configuration errors
CHECK(cudaDeviceSynchronize()); // Catches errors raised while the kernel ran
CHECK(cudaMemcpy(hc.data(), dc, bytes, cudaMemcpyDeviceToHost));
// Compare with the same computation on the CPU (one addition, so we expect zero error)
double maxErr = 0.0;
for (int i = 0; i < n; ++i) {
const float ref = ha[i] + hb[i];
maxErr = std::fmax(maxErr, std::fabs(static_cast<double>(hc[i]) - ref));
}
const bool ok = maxErr <= 1e-6;
std::printf("n=%d blocks=%d threads/block=%d\n", n, blocks, threads);
std::printf("max abs error = %.3e (tolerance 1e-6)\n", maxErr);
std::printf("%s\n", ok ? "PASS" : "FAIL");
CHECK(cudaFree(da));
CHECK(cudaFree(db));
CHECK(cudaFree(dc));
return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}
Expected output (GPU results not published). The program prints the element count, the number of blocks and the threads per block; then a max abs error = ... line with the largest difference between the GPU result and the CPU value; and finally PASS or FAIL. Since each element is a single addition, the difference should be zero.
Key points:
__global__marks a kernel launched from the host. Its return type must bevoid; the only way to return results is to write them to global (device) memory.__device__marks a function callable only from GPU code (used bywarpReduceSumin section 8). Marking a function__host__ __device__generates both a CPU and a GPU version.vecAdd<<<blocks, threads>>>(...)launches the kernel: the first argument is the grid size, the second the block size.- The bounds check
if (i < n): if the element count is not a multiple of the block size, the last block has spare threads. This program deliberately uses elements. - The
CHECKmacro: every CUDA API returns acudaError_t. The CUDA Programming Guide likewise calls checking every return value best practice in production.
Error handling has a pitfall that comes from asynchrony. A <<< >>> launch does not return a cudaError_t, and cudaGetLastError() right after the launch only catches errors in the launch configuration. Errors raised while the kernel runs (an invalid memory access, say) are reported the next time the error state is checked. That is why the program checks both cudaGetLastError() and the return value of cudaDeviceSynchronize().
5. The compilation pipeline: CUDA C++ → NVVM → PTX → SASS
NVCC is a conductor, not a single compiler
nvcc is less a compiler than a driver that calls several tools in turn. The flow is:
- split each
.cufile into host code and device code; - compile the host code with the host C++ compiler (gcc and the like);
- compile the device code to PTX, once per virtual architecture (e.g.
compute_75); - have
ptxasturn the PTX into a cubin containing machine code for a real GPU (e.g.sm_75); - pack the PTX and cubins into a fatbin container and embed it in the host executable.
vector_add.cu
├─ host code ───────── compiled by gcc etc. ────────────────┐
└─ device code ─ NVVM (LLVM-based) ─→ PTX (compute_75) ─────┤
└─ ptxas ─→ cubin/SASS (sm_75)
↓
executable = CPU machine code + fatbin (PTX and cubin)
Step 3 uses NVVM. NVIDIA's NVVM IR specification describes NVVM IR as "based on the LLVM IR" and says that "The NVVM compiler (which is based on LLVM) generates PTX code from NVVM IR." NVIDIA's developer page likewise says NVCC is based on the open-source LLVM compiler infrastructure. Languages and compilers other than CUDA can run on NVIDIA GPUs too, as long as they emit PTX. -v shows the tools invoked at each step, and -keep keeps the intermediate files.
PTX versus SASS
| PTX | SASS (the contents of a cubin) | |
|---|---|---|
| What it is | A virtual instruction set ("a high-level assembly language for NVIDIA GPUs") | The machine code a real GPU executes |
| Format | Text | Binary (disassemble with cuobjdump or nvdisasm) |
| Relation to generations | Named like compute_75, for the feature generation it requires | Named like sm_75, for one specific GPU generation |
| Compatibility | The driver can JIT-compile it on the same or a newer generation | Runs only when the major number matches and the cubin's minor number is not above the GPU's |
| Specification | Published as the PTX ISA (currently version 9.5) | The instruction list is published, but not a detailed specification like PTX's |
The PTX ISA document lists, among PTX's goals, to "provide a stable ISA that spans multiple GPU generations". In PTX, a thread block is called a CTA (Cooperative Thread Array). Where data lives is expressed as state spaces — .reg (registers), .shared, .global, .local, .const, .param and others — and threadIdx.x becomes the special register %tid.x.
Compute capability and binary compatibility
Here is what -arch=sm_75 means, together with the compatibility rules:
- cubin compatibility: a cubin runs if the compute capability's major number matches and the GPU's minor number is at least the cubin's. An
sm_86cubin runs on CC 8.6 and 8.9 GPUs but not on 8.0, and not on a GPU with a different major number (such as 9.0). The exceptions: code built for an architecture-specific target such assm_90aruns only on GPUs of exactly that CC, and CC 10.0 and later add a separate family-level compatibility rule (CUDA Programming Guide, sections 1.3.4 and 5.1.2). - PTX compatibility: PTX for
compute_80can be JIT-compiled at run time for any GPU of that CC or higher, so applications run on future GPUs without being rebuilt. -arch=sm_75generates bothcompute_75PTX and ansm_75cubin. A T4 uses the cubin directly; newer GPUs JIT-compile the PTX.-arch=nativebuilds a cubin for the local GPU only. It contains no PTX, so there is no forward compatibility.
In the NVCC documentation published as of 2026-11-17 (CUDA Toolkit v13.5), the default when -arch is omitted is sm_75, and the list of supported GPUs starts at Turing (sm_75). The T4 is therefore the oldest generation still covered by that CUDA version.
JIT compilation and the compute cache
Run a program that carries only PTX on a new GPU, and the driver compiles the PTX to machine code at load time: JIT (just-in-time) compilation. JIT adds load time but lets programs run on GPUs that did not exist when they were built. The driver stores the generated code in the compute cache, which is invalidated when the driver is upgraded.
On a T4 the sm_75 cubin runs directly, so no JIT happens. The first call still pays for creating a context (the per-GPU execution environment), and since CUDA 12.3 lazy loading — loading a module only when it is first needed — is the default on all platforms, so a kernel's first launch can also include loading time. That is why the programs in this post run a warm-up before timing anything.
Looking at PTX and SASS yourself
These commands show the PTX and the SASS (I have not run them):
# Emit PTX (text)
nvcc -arch=sm_75 -ptx vector_add.cu -o vector_add.ptx
grep -E "ld\.global|add(\.rn)?\.f32|st\.global" vector_add.ptx
# List the cubins and PTX inside the executable's fatbin
cuobjdump -lelf -lptx vector_add
# Disassemble the SASS (T4 machine code)
cuobjdump -sass vector_add | grep -E "LDG|FADD|STG"
# Print registers and shared memory used per kernel
nvcc -arch=sm_75 -O3 -res-usage -c vector_add.cu
The PTX should contain instructions corresponding to ld.global (load from global memory), add.f32 (FP32 add) and st.global (store). In the "Turing Instruction Set (Compute Capability 7.5)" table of NVIDIA's binary utilities documentation, the matching SASS instructions are LDG (Load from Global Memory), FADD (FP32 Add) and STG (Store to Global Memory). The exact output varies with the compiler version.
6. The memory hierarchy: registers, shared, global, local, constant
Five kinds of memory
The memories a kernel uses differ in visibility (scope), lifetime and location. Here is the CUDA Programming Guide's table, with T4 and H100 capacities added (from the compute-capability tables and the Turing whitepaper):
| Kind | Visible to | Lifetime | Location | T4 (CC 7.5) | H100 (CC 9.0) |
|---|---|---|---|---|---|
| Registers | Thread | Kernel | SM | 64K per SM (max 255 per thread) | Same |
| Shared memory | Block | Kernel | SM | Up to 64 KB per SM | Up to 228 KB per SM |
| Local memory | Thread | Kernel | Device memory | Up to 512 KB per thread | Same |
| Constant memory | Grid | Application | Device memory | 64 KB | 64 KB |
| Global memory | Grid | Application | Device memory | 16 GB (GDDR6) | 80 GB (SXM5, HBM3) |
A few notes:
- Shared memory is a programmer-managed scratchpad shared by the threads of a block. Physically it uses the same resource as the L1 cache (the unified data cache). On a T4 the unified data cache is 96 KB, and the split between shared memory and L1 is configurable. It is described as having higher bandwidth and lower latency than global memory, but the CUDA Programming Guide gives no concrete latency figures.
- Local memory, despite its name, lives off chip in device memory. "Local" describes visibility (that thread only), not location. Arrays indexed by non-constant values, large structures and variables that do not fit in registers (register spills) go there, with the same latency and bandwidth as global memory.
- Statically allocated shared memory is limited to 48 KB per block. Going beyond that needs dynamic shared memory and an explicit opt-in (
cudaFuncSetAttribute).
Coalescing and bank conflicts: what the programmer can do
Global memory is read and written per warp, so access is efficient when consecutive threads read consecutive addresses (coalescing), and when adjacent threads read addresses far apart, most of the data moved is wasted. In the CUDA Programming Guide's example, utilization ranges from 100% at best to 12.5% at worst. Shared memory is split into 32 banks, and when several threads of a warp access different addresses in the same bank, the accesses serialize (a bank conflict). The mechanics are covered in section 10 of How CPUs and GPUs differ in architecture.
The programmer has two standard remedies:
- Rearrange through a shared-memory tile. For operations whose writes jump along columns, like a matrix transpose, load a tile into shared memory, rearrange it there and then write it out, so that both reads and writes coalesce. The
vecAddof section 4 already has consecutive threads reading consecutive elements, so it coalesces perfectly as it is. - Pad to shift the banks. Reading
__shared__ float smem[32][32]down a column sends all 32 threads to the same bank. Declaring it[32][33], padding each row by one element, removes the conflict.
7. Synchronization and race conditions: how CPU and GPU differ
The block-level barrier: __syncthreads()
When the threads of a block exchange data through shared memory, they must synchronize so that nobody reads before the writes are done. __syncthreads() waits until every (non-exited) thread of the block reaches the same __syncthreads(), and it also guarantees that writes made before it are visible to the threads after they leave the wait.
The rules are strict. __syncthreads() may appear in conditional code only if the condition evaluates identically across the entire block. Calling it in a branch that only some threads take can hang the kernel or produce unintended effects.
There is no grid-wide barrier
__syncthreads() only reaches within a block. As section 3 showed, block execution order is not guaranteed, so an ordinary kernel has no way to synchronize the whole grid. To combine results across blocks, use one of the following:
- Atomics:
atomicAddand friends read, modify and write as one indivisible step. The CUDA Programming Guide says atomics should be used sparingly, because they enforce synchronization between threads that can hurt performance. - Separate kernels: the first kernel writes per-block partial results and the second combines them. Kernels in the same stream run in order, so the kernel boundary acts as a grid-wide synchronization point.
(Cooperative Groups can express synchronization across blocks, with limitations and performance costs. This post does not use them.)
A weak memory model and data races
The CUDA Programming Guide states plainly that the CUDA programming model assumes a weakly ordered memory model. The order in which one thread writes to memory need not be the order in which another thread observes those writes. Reading and writing the same location without fences or synchronization — a data race — is undefined behavior.
Warps do not line up by themselves
Older material contains code built on the assumption that "instructions are SIMD synchronous within a warp, so we don't need __syncthreads()" (for example, the "unrolling the last warp" step in Mark Harris's well-known NVIDIA reduction slides). That assumption does not hold from CC 7.0 on. Independent Thread Scheduling gives each thread its own program counter and lets threads diverge and reconverge at sub-warp granularity. The CUDA Programming Guide asks for such code to be revisited and synchronized explicitly with __syncwarp(). The old warp primitives without a mask argument were deprecated in CUDA 9.
CPU versus GPU synchronization, side by side
Here are the differences so far, in one table:
| Aspect | CPU multithreading (standard C++) | CUDA |
|---|---|---|
| Thread management | Scheduled by the OS | Tens of thousands resident on the SMs; switching warps costs nothing |
| Cost of synchronization | In the CUDA docs' words, standard C++ treats it as "uniform and low" | Cheap between nearby threads, expensive between distant ones |
| Reach of synchronization | Any thread can sync with any other via mutexes or atomics | A hierarchy: warp (__syncwarp, shuffles) → block (__syncthreads) → device (atomics, kernel boundaries) → system |
| Global barrier | std::barrier can wait for all threads | No grid-wide barrier in an ordinary kernel |
| Memory visibility | Defined by std::atomic and memory orders | Weak memory model; cuda::atomic and friends take a scope (block, device, system) |
| Between CPU and GPU | — | Kernel launches and copies are asynchronous; wait with streams, events or cudaDeviceSynchronize |
CUDA's scopes widen from block (.cta in PTX) to device (.gpu) to system (.sys, including CPUs and other GPUs), and the wider the scope, the more synchronization costs.
8. Reduction: a shared-memory tree versus warp shuffles
Why a sum is hard
Summing an array (a reduction) is a one-line loop on a CPU. On a GPU, tens of thousands of threads have to funnel their results into one value, so every synchronization issue from section 7 shows up.
The approach is a tree. First build partial sums inside each block, then add the per-block partial sums in a second kernel launch. Two ways of building the in-block sum are compared:
- A shared-memory tree: put every thread's value in shared memory and add halves repeatedly. Each step needs a
__syncthreads(). - Warp shuffles:
__shfl_down_syncpasses values between registers within a warp, with no shared memory; only the eight per-warp partial sums go through shared memory.
__shfl_down_sync(mask, v, offset) returns the v of the lane offset positions above. Adding while halving the offset — 16, 8, 4, 2, 1 — collects the sum of 32 values in lane 0 after five steps. mask is a bitmask of participating lanes; every calling thread must be in it, and all must call with the same mask. CC 8.0 and later also have a dedicated __reduce_add_sync, but it accepts only integers (int / unsigned), so it cannot sum floats (and the T4, at CC 7.5, does not have it at all). So the code uses shuffles.
For comparison, the program also includes a correct but slow atomicAdd version and an unsynchronized data race version. The racy one is an example of undefined behavior and is never used for PASS/FAIL.
- Build:
nvcc -arch=sm_75 -O3 -o reduce reduce.cu - Run:
./reduce(the element count can be passed as an argument; the default is )
// reduce.cu — four ways to write a sum (reduction), compared
// 1) tree in shared memory + __syncthreads
// 2) warp shuffles (__shfl_down_sync); shared memory holds only one partial sum per warp
// 3) atomicAdd (correct, but slow, and the summation order changes every run)
// 4) unsynchronized *out += x (a data race: an example of undefined behavior, never judged)
// Build: nvcc -arch=sm_75 -O3 -o reduce reduce.cu
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>
#define CHECK(call) \
do { \
cudaError_t err_ = (call); \
if (err_ != cudaSuccess) { \
std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n", \
cudaGetErrorName(err_), __FILE__, __LINE__, \
cudaGetErrorString(err_)); \
std::exit(EXIT_FAILURE); \
} \
} while (0)
constexpr int kThreads = 256; // 1 block = 8 warps (must be a power of two)
constexpr int kWarps = kThreads / 32;
static float lcg01(unsigned& state) {
state = state * 1664525u + 1013904223u;
return (state >> 8) * (1.0f / 16777216.0f);
}
// 1) Tree reduction in shared memory
__global__ void reduceSmem(const float* in, float* out, int n) {
__shared__ float s[kThreads];
float sum = 0.0f;
// Grid-stride loop: covers every element even when there are more elements than threads
for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
i += blockDim.x * gridDim.x) {
sum += in[i];
}
s[threadIdx.x] = sum;
__syncthreads(); // Wait until everyone has written s[]
for (unsigned int stride = blockDim.x / 2; stride > 0; stride /= 2) {
if (threadIdx.x < stride) {
s[threadIdx.x] += s[threadIdx.x + stride];
}
#ifndef OMIT_SYNC // -DOMIT_SYNC removes this barrier and creates a shared-memory race
__syncthreads(); // Wait until the values read by the next step are written
#endif
}
if (threadIdx.x == 0) out[blockIdx.x] = s[0];
}
// Sum the 32 values of a warp register-to-register, without shared memory
__device__ float warpReduceSum(float v) {
for (int offset = warpSize / 2; offset > 0; offset /= 2) {
// Receive v from the lane offset positions above (all 32 lanes take part)
v += __shfl_down_sync(0xffffffffu, v, offset);
}
return v; // Lane 0 now holds the warp total
}
// 2) Warp-shuffle version
__global__ void reduceShfl(const float* in, float* out, int n) {
__shared__ float warpSums[kWarps];
float sum = 0.0f;
for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
i += blockDim.x * gridDim.x) {
sum += in[i];
}
const int lane = threadIdx.x % warpSize;
const int warp = threadIdx.x / warpSize;
sum = warpReduceSum(sum);
if (lane == 0) warpSums[warp] = sum;
__syncthreads(); // Wait until every warp's partial sum is in place
if (warp == 0) { // All 32 threads of warp 0 handle the next step together
sum = (lane < kWarps) ? warpSums[lane] : 0.0f;
sum = warpReduceSum(sum);
if (lane == 0) out[blockIdx.x] = sum;
}
}
// 3) atomicAdd version: read, add and write as one indivisible step
__global__ void reduceAtomic(const float* in, float* out, int n) {
for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
i += blockDim.x * gridDim.x) {
atomicAdd(out, in[i]);
}
}
// 4) No synchronization: many threads read and write one address (data race = undefined behavior)
__global__ void reduceRacy(const float* in, float* out, int n) {
for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
i += blockDim.x * gridDim.x) {
*out += in[i];
}
}
using Kernel = void (*)(const float*, float*, int);
// Only launches kernels (reading the result is separate). With twoPass=true,
// it computes `blocks` partial sums, then reduces them with one block
static void launch(Kernel k, bool twoPass, const float* d_in, float* d_partial,
float* d_out, int n, int blocks) {
if (twoPass) {
k<<<blocks, kThreads>>>(d_in, d_partial, n);
CHECK(cudaGetLastError());
k<<<1, kThreads>>>(d_partial, d_out, blocks);
CHECK(cudaGetLastError());
} else {
CHECK(cudaMemsetAsync(d_out, 0, sizeof(float))); // Zero the accumulation target
k<<<blocks, kThreads>>>(d_in, d_out, n);
CHECK(cudaGetLastError());
}
}
template <typename F>
static void printOccupancy(const char* name, F func, int maxThreadsPerSM) {
cudaFuncAttributes attr{};
CHECK(cudaFuncGetAttributes(&attr, func));
int activeBlocks = 0;
CHECK(cudaOccupancyMaxActiveBlocksPerMultiprocessor(&activeBlocks, func, kThreads, 0));
std::printf(" %-12s regs/thread=%d static smem=%zu B local=%zu B -> %d blocks/SM, "
"theoretical occupancy %.0f%%\n",
name, attr.numRegs, attr.sharedSizeBytes, attr.localSizeBytes, activeBlocks,
100.0 * activeBlocks * kThreads / maxThreadsPerSM);
}
int main(int argc, char** argv) {
const int n = (argc > 1) ? std::atoi(argv[1]) : (1 << 24);
if (n <= 0) {
std::fprintf(stderr, "n must be a positive integer\n");
return EXIT_FAILURE;
}
const int blocks = 1024; // Blocks in the first pass (= number of partial sums)
const size_t bytes = static_cast<size_t>(n) * sizeof(float);
cudaDeviceProp prop{};
CHECK(cudaGetDeviceProperties(&prop, 0));
std::printf("device: %s (CC %d.%d, %d SMs)\n", prop.name, prop.major, prop.minor,
prop.multiProcessorCount);
std::vector<float> h(n);
unsigned seed = 2026u;
double ref = 0.0; // Reference: sequential sum in double
for (int i = 0; i < n; ++i) {
h[i] = lcg01(seed);
ref += h[i];
}
float *d_in = nullptr, *d_partial = nullptr, *d_out = nullptr;
CHECK(cudaMalloc(&d_in, bytes));
CHECK(cudaMalloc(&d_partial, blocks * sizeof(float)));
CHECK(cudaMalloc(&d_out, sizeof(float)));
CHECK(cudaMemcpy(d_in, h.data(), bytes, cudaMemcpyHostToDevice));
cudaEvent_t start, stop;
CHECK(cudaEventCreate(&start));
CHECK(cudaEventCreate(&stop));
struct Case { const char* name; Kernel k; bool twoPassMode; bool judged; };
const Case cases[] = {
{"smem-tree", reduceSmem, true, true},
{"warp-shfl", reduceShfl, true, true},
{"atomicAdd", reduceAtomic, false, false},
{"racy", reduceRacy, false, false},
};
const double tol = 1e-5; // Relative-error tolerance
bool allOk = true;
std::printf("n=%d, reference (double) = %.6f\n", n, ref);
for (const Case& c : cases) {
// First run is a warm-up (keeps first-launch latency out of the timing)
launch(c.k, c.twoPassMode, d_in, d_partial, d_out, n, blocks);
CHECK(cudaDeviceSynchronize());
CHECK(cudaEventRecord(start));
launch(c.k, c.twoPassMode, d_in, d_partial, d_out, n, blocks);
CHECK(cudaEventRecord(stop));
CHECK(cudaEventSynchronize(stop)); // Wait for the GPU to reach `stop`
float ms = 0.0f;
CHECK(cudaEventElapsedTime(&ms, start, stop));
float got = 0.0f;
CHECK(cudaMemcpy(&got, d_out, sizeof(float), cudaMemcpyDeviceToHost));
const double relErr = std::fabs(got - ref) / std::fabs(ref);
const char* verdict = "(reference only)";
if (c.judged) {
const bool ok = relErr <= tol;
allOk = allOk && ok;
verdict = ok ? "PASS" : "FAIL";
}
std::printf("%-10s sum=%.6f rel.err=%.3e time=%.3f ms %s\n", c.name, got, relErr,
ms, verdict);
}
std::printf("occupancy (block = %d threads):\n", kThreads);
printOccupancy("reduceSmem", reduceSmem, prop.maxThreadsPerMultiProcessor);
printOccupancy("reduceShfl", reduceShfl, prop.maxThreadsPerMultiProcessor);
std::printf("overall: %s (judged: smem-tree, warp-shfl; tolerance %.0e)\n",
allOk ? "PASS" : "FAIL", tol);
CHECK(cudaEventDestroy(start));
CHECK(cudaEventDestroy(stop));
CHECK(cudaFree(d_in));
CHECK(cudaFree(d_partial));
CHECK(cudaFree(d_out));
return allOk ? EXIT_SUCCESS : EXIT_FAILURE;
}
Expected output (GPU results not published). The program prints the reference value computed in double, then, for each of the four methods, the sum, its relative error against the reference, and the time. The shared-memory tree and the warp-shuffle versions PASS when the relative error is at most ; atomicAdd and the racy version print "(reference only)". Finally it prints each kernel's register count, static shared memory, and the theoretical occupancy derived from the blocks per SM that cudaOccupancyMaxActiveBlocksPerMultiprocessor returns.
In the racy version, *out += in[i] makes many threads "read → add → write" the same address, so they overwrite and erase each other's updates. The CUDA Programming Guide has a similar example in which a non-atomic add yields a result smaller than the correct one that can change from run to run. No particular value is guaranteed.
The order of addition changes the error
Floating-point addition gives different results in different orders, and a GPU sum adds in a different order from a simple CPU loop. To justify the tolerance, I reproduced this program's input ( values from the same pseudo-random generator) and its order of addition on the CPU with NumPy float32. This is a CPU simulation of the summation order, not a GPU run.
reference (float64) = 8388079.921875
warp-shfl order sum = 8388080.000000 rel.err = 9.314e-09
smem-tree order sum = 8388080.000000 rel.err = 9.314e-09
sequential float32 sum = 8388031.000000 rel.err = 5.832e-06
The tree orders give a relative error of about , roughly 600 times smaller than adding front to back in float32 (about ). Once the running sum is large, adding small values loses their low-order digits to rounding. The tolerance leaves ample margin over the tree-order error.
With atomicAdd the order changes from run to run, so the resulting bit pattern is not guaranteed to be the same each time. Where training must be reproducible, that can matter.
9. Prefix sum (scan): stitching blocks together in three phases
What scan is
A prefix sum, or scan, computes running totals from the start of an array. An inclusive scan puts "the total up to and including me" in each element; an exclusive scan puts "the total of everything before me".
It looks inherently sequential, since each output depends on the previous one, but it can be computed in parallel. It is a building block of many parallel algorithms: stream compaction (packing only the elements that meet a condition), radix sort, histograms and more.
Scan within a warp, and joining the blocks
The GPU Gems 3 chapter by Harris and colleagues presents the Hillis–Steele approach, which at step adds elements apart, and points out that it performs additions and so is not "work-efficient". This program uses that approach only inside a 32-element warp (five steps of __shfl_up_sync), where fewer steps matter more than the extra work.
The whole computation has three phases:
- each block scans its own range and writes its block total;
- one block scans the array of block totals;
- each block adds the total of all earlier blocks to its elements.
Inside a block, the same two-level shape repeats: scan each warp → scan the warp totals → add the total of earlier warps.
- Build:
nvcc -arch=sm_75 -O3 -o scan scan.cu - Run:
./scan(the default element count is , deliberately not a multiple of the block size)
// scan.cu — inclusive prefix sum (inclusive scan) in three phases
// Phase 1: each block scans its own range and writes its block total
// Phase 2: one block scans the array of block totals
// Phase 3: add "the total of all earlier blocks" to each block
// Build: nvcc -arch=sm_75 -O3 -o scan scan.cu
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>
#define CHECK(call) \
do { \
cudaError_t err_ = (call); \
if (err_ != cudaSuccess) { \
std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n", \
cudaGetErrorName(err_), __FILE__, __LINE__, \
cudaGetErrorString(err_)); \
std::exit(EXIT_FAILURE); \
} \
} while (0)
constexpr int kBlock = 1024; // 1 block = 32 warps; this implementation assumes this value
// Inclusive scan within a warp (Hillis-Steele style, 5 steps)
__device__ int warpInclusiveScan(int v) {
const int lane = threadIdx.x % warpSize;
for (int offset = 1; offset < warpSize; offset *= 2) {
const int up = __shfl_up_sync(0xffffffffu, v, offset); // value from the lane offset positions below
if (lane >= offset) v += up;
}
return v;
}
// Phase 1 (and 2): scan within a block; writes the block total unless blockSums is nullptr
__global__ void scanBlock(const int* in, int* out, int* blockSums, int n) {
__shared__ int warpTotals[32];
const int i = blockIdx.x * blockDim.x + threadIdx.x;
const int lane = threadIdx.x % warpSize;
const int warp = threadIdx.x / warpSize;
int v = (i < n) ? in[i] : 0; // Out-of-range elements count as 0 (no effect on the sums)
v = warpInclusiveScan(v);
if (lane == warpSize - 1) warpTotals[warp] = v; // Each warp's total
__syncthreads();
if (warp == 0) { // Scan the 32 warp totals
warpTotals[lane] = warpInclusiveScan(warpTotals[lane]);
}
__syncthreads();
if (warp > 0) v += warpTotals[warp - 1]; // Add the total of all earlier warps
if (i < n) out[i] = v;
if (blockSums != nullptr && threadIdx.x == blockDim.x - 1) {
blockSums[blockIdx.x] = v; // The block's last thread holds the block total
}
}
// Phase 3: add the total of all earlier blocks
__global__ void addOffsets(int* out, const int* scannedSums, int n) {
const int i = blockIdx.x * blockDim.x + threadIdx.x;
if (blockIdx.x > 0 && i < n) out[i] += scannedSums[blockIdx.x - 1];
}
int main(int argc, char** argv) {
const int n = (argc > 1) ? std::atoi(argv[1]) : (1 << 20) - 5;
if (n <= 0) {
std::fprintf(stderr, "n must be a positive integer\n");
return EXIT_FAILURE;
}
const int numBlocks = (n + kBlock - 1) / kBlock;
if (numBlocks > kBlock) {
// Phase 2 runs in a single block, so the block count is capped at kBlock
std::fprintf(stderr, "n too large for this 3-phase demo (max %d)\n", kBlock * kBlock);
return EXIT_FAILURE;
}
std::vector<int> h(n), got(n);
unsigned state = 7u;
for (int i = 0; i < n; ++i) {
state = state * 1664525u + 1013904223u;
h[i] = static_cast<int>((state >> 16) % 10u); // Integers 0-9 (the sums fit in an int)
}
int *d_in = nullptr, *d_out = nullptr, *d_sums = nullptr, *d_scannedSums = nullptr;
CHECK(cudaMalloc(&d_in, n * sizeof(int)));
CHECK(cudaMalloc(&d_out, n * sizeof(int)));
CHECK(cudaMalloc(&d_sums, numBlocks * sizeof(int)));
CHECK(cudaMalloc(&d_scannedSums, numBlocks * sizeof(int)));
CHECK(cudaMemcpy(d_in, h.data(), n * sizeof(int), cudaMemcpyHostToDevice));
cudaEvent_t start, stop;
CHECK(cudaEventCreate(&start));
CHECK(cudaEventCreate(&stop));
// Launch the three kernels in order (same stream, so they run in the order queued)
auto runScan = [&]() {
scanBlock<<<numBlocks, kBlock>>>(d_in, d_out, d_sums, n);
CHECK(cudaGetLastError());
scanBlock<<<1, kBlock>>>(d_sums, d_scannedSums, nullptr, numBlocks);
CHECK(cudaGetLastError());
addOffsets<<<numBlocks, kBlock>>>(d_out, d_scannedSums, n);
CHECK(cudaGetLastError());
};
runScan(); // Warm-up (keeps first-launch costs out of the timing)
CHECK(cudaDeviceSynchronize());
CHECK(cudaEventRecord(start));
runScan(); // Same result on every run (phase 1 rewrites all of d_out)
CHECK(cudaEventRecord(stop));
CHECK(cudaEventSynchronize(stop));
float ms = 0.0f;
CHECK(cudaEventElapsedTime(&ms, start, stop));
CHECK(cudaMemcpy(got.data(), d_out, n * sizeof(int), cudaMemcpyDeviceToHost));
// Require an exact element-by-element match with a sequential CPU scan (integers: no tolerance)
std::vector<long long> ref(n);
long long running = 0;
for (int i = 0; i < n; ++i) {
running += h[i];
ref[i] = running;
}
long long firstBad = -1;
for (int i = 0; i < n; ++i) {
if (got[i] != ref[i]) {
firstBad = i;
break;
}
}
std::printf("n=%d blocks=%d threads/block=%d\n", n, numBlocks, kBlock);
std::printf("last element: gpu=%d cpu=%lld\n", got[n - 1], ref[n - 1]);
if (firstBad >= 0) {
std::printf("first mismatch at index %lld: gpu=%d cpu=%lld\n", firstBad,
got[firstBad], ref[firstBad]);
}
std::printf("time (3 kernels) = %.3f ms\n", ms);
std::printf("%s\n", firstBad < 0 ? "PASS" : "FAIL");
CHECK(cudaEventDestroy(start));
CHECK(cudaEventDestroy(stop));
CHECK(cudaFree(d_in));
CHECK(cudaFree(d_out));
CHECK(cudaFree(d_sums));
CHECK(cudaFree(d_scannedSums));
return firstBad < 0 ? EXIT_SUCCESS : EXIT_FAILURE;
}
Expected output (GPU results not published). The program prints the element and block counts, the GPU and CPU values of the last element, and the combined time of the three kernels measured after a warm-up, and prints PASS if every element matches a sequential CPU scan exactly. The inputs are integers from 0 to 9, so there is no floating-point error: a single mismatch means FAIL.
In the simulated environment described under "What has been verified" at the top, it passed for all of the element counts above and failed when I shifted an index by one (warpTotals[warp - 1] to warpTotals[warp]). What this confirms is only the logic — the indexing and where the barriers sit. Speed on a GPU, real warp behavior, and whether nvcc accepts the code remain unverified.
10. Tiled matrix multiplication: reusing data through shared memory
The problem with the naive version
An matrix product costs FLOPs. In a naive kernel, one thread owns one element of and reads a row of and a column of straight from global memory.
Each element of is read by threads, and so is each element of . If no caching helped at all, every multiply-add (2 FLOPs) would read two 4-byte values — 8 bytes.
Tiling: stage in shared memory and reuse
With tiling, a block loads a small tile into shared memory and its threads reuse it. Each thread only carries one element, and that value is used times inside the block, so the amount read from global memory falls to of the naive version.
The number of operations per byte exchanged with global memory is the arithmetic intensity (AI). Counting the reads of and and the writes of (a model that ignores reuse from caches):
For that gives about 0.25 FLOP/byte for the naive version (), about 3.97 for and about 7.88 for (output of the script in section 11).
- Build:
nvcc -arch=sm_75 -O3 -o matmul matmul.cu -lcublas - Run:
./matmul(the size of the timed matrices can be passed as an argument; the default is 1,024)
// matmul.cu — matrix product C = A * B (FP32, row-major), three ways
// naive : one thread computes one element of C, reading global memory directly
// tiled : TILE x TILE tiles are staged in shared memory and reused
// cuBLAS: NVIDIA's library (PyTorch's torch.matmul also calls cuBLAS by default)
// Build: nvcc -arch=sm_75 -O3 -o matmul matmul.cu -lcublas
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cublas_v2.h>
#include <cuda_runtime.h>
#define CHECK(call) \
do { \
cudaError_t err_ = (call); \
if (err_ != cudaSuccess) { \
std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n", \
cudaGetErrorName(err_), __FILE__, __LINE__, \
cudaGetErrorString(err_)); \
std::exit(EXIT_FAILURE); \
} \
} while (0)
#define CHECK_CUBLAS(call) \
do { \
cublasStatus_t st_ = (call); \
if (st_ != CUBLAS_STATUS_SUCCESS) { \
std::fprintf(stderr, "cuBLAS error %d at %s:%d\n", \
static_cast<int>(st_), __FILE__, __LINE__); \
std::exit(EXIT_FAILURE); \
} \
} while (0)
constexpr int TILE = 16; // 1 block = 16 x 16 = 256 threads
static float lcgSym(unsigned& state) { // pseudo-random value in [-1, 1)
state = state * 1664525u + 1013904223u;
return (state >> 8) * (2.0f / 16777216.0f) - 1.0f;
}
__global__ void matmulNaive(const float* A, const float* B, float* C, int n) {
const int row = blockIdx.y * blockDim.y + threadIdx.y;
const int col = blockIdx.x * blockDim.x + threadIdx.x;
if (row < n && col < n) {
float acc = 0.0f;
for (int k = 0; k < n; ++k) {
acc += A[row * n + k] * B[k * n + col]; // reads global memory every time
}
C[row * n + col] = acc;
}
}
__global__ void matmulTiled(const float* A, const float* B, float* C, int n) {
__shared__ float As[TILE][TILE];
__shared__ float Bs[TILE][TILE];
const int tx = threadIdx.x, ty = threadIdx.y;
const int row = blockIdx.y * TILE + ty;
const int col = blockIdx.x * TILE + tx;
float acc = 0.0f;
for (int t = 0; t < (n + TILE - 1) / TILE; ++t) {
const int aCol = t * TILE + tx;
const int bRow = t * TILE + ty;
// Each thread brings one element of the A and B tiles (0 outside the matrix)
As[ty][tx] = (row < n && aCol < n) ? A[row * n + aCol] : 0.0f;
Bs[ty][tx] = (bRow < n && col < n) ? B[bRow * n + col] : 0.0f;
__syncthreads(); // Wait until the whole tile is loaded
for (int k = 0; k < TILE; ++k) {
acc += As[ty][k] * Bs[k][tx]; // each shared-memory value is reused TILE times
}
__syncthreads(); // Do not overwrite with the next tile until everyone has finished reading
}
if (row < n && col < n) C[row * n + col] = acc;
}
// CPU reference in double, also accumulating sum|a*b| for the error bound
static void cpuReference(const std::vector<float>& A, const std::vector<float>& B,
std::vector<double>& ref, std::vector<double>& absRef, int n) {
ref.assign(static_cast<size_t>(n) * n, 0.0);
absRef.assign(static_cast<size_t>(n) * n, 0.0);
for (int i = 0; i < n; ++i) {
for (int k = 0; k < n; ++k) {
const double a = A[static_cast<size_t>(i) * n + k];
for (int j = 0; j < n; ++j) {
const double p = a * B[static_cast<size_t>(k) * n + j];
ref[static_cast<size_t>(i) * n + j] += p;
absRef[static_cast<size_t>(i) * n + j] += std::fabs(p);
}
}
}
}
// Worst-case FP32 dot-product rounding error is about n * 2^-24 * sum|a*b|; allow twice that
static bool verify(const char* name, const std::vector<float>& got,
const std::vector<double>& ref, const std::vector<double>& absRef, int n) {
const double unit = std::ldexp(1.0, -23); // 2 * 2^-24
double maxAbsErr = 0.0, worstRatio = 0.0;
for (size_t i = 0; i < got.size(); ++i) {
const double err = std::fabs(got[i] - ref[i]);
const double bound = n * unit * absRef[i] + 1e-30;
maxAbsErr = std::fmax(maxAbsErr, err);
worstRatio = std::fmax(worstRatio, err / bound);
}
const bool ok = worstRatio <= 1.0;
std::printf(" %-7s max abs error = %.3e, worst err/bound = %.3f -> %s\n", name,
maxAbsErr, worstRatio, ok ? "PASS" : "FAIL");
return ok;
}
template <typename F>
static void printOccupancy(const char* name, F func, int threadsPerBlock, int maxThreadsPerSM) {
cudaFuncAttributes attr{};
CHECK(cudaFuncGetAttributes(&attr, func));
int activeBlocks = 0;
CHECK(cudaOccupancyMaxActiveBlocksPerMultiprocessor(&activeBlocks, func, threadsPerBlock, 0));
std::printf(" %-12s regs/thread=%d static smem=%zu B -> %d blocks/SM, "
"theoretical occupancy %.0f%%\n",
name, attr.numRegs, attr.sharedSizeBytes, activeBlocks,
100.0 * activeBlocks * threadsPerBlock / maxThreadsPerSM);
}
// Run the three methods on n x n matrices; with timed=true, time them with CUDA events
static bool runCase(int n, bool timed, cublasHandle_t handle) {
const size_t elems = static_cast<size_t>(n) * n;
const size_t bytes = elems * sizeof(float);
std::vector<float> hA(elems), hB(elems), hC(elems);
unsigned seed = 42u + static_cast<unsigned>(n);
for (size_t i = 0; i < elems; ++i) hA[i] = lcgSym(seed);
for (size_t i = 0; i < elems; ++i) hB[i] = lcgSym(seed);
float *dA = nullptr, *dB = nullptr, *dC = nullptr;
CHECK(cudaMalloc(&dA, bytes));
CHECK(cudaMalloc(&dB, bytes));
CHECK(cudaMalloc(&dC, bytes));
CHECK(cudaMemcpy(dA, hA.data(), bytes, cudaMemcpyHostToDevice));
CHECK(cudaMemcpy(dB, hB.data(), bytes, cudaMemcpyHostToDevice));
std::vector<double> ref, absRef;
cpuReference(hA, hB, ref, absRef, n);
const dim3 block(TILE, TILE);
const dim3 grid((n + TILE - 1) / TILE, (n + TILE - 1) / TILE);
const float alpha = 1.0f, beta = 0.0f;
const int reps = timed ? 10 : 1;
const double flops = 2.0 * n * static_cast<double>(n) * n;
cudaEvent_t start, stop;
CHECK(cudaEventCreate(&start));
CHECK(cudaEventCreate(&stop));
bool allOk = true;
for (int method = 0; method < 3; ++method) {
const char* name = (method == 0) ? "naive" : (method == 1) ? "tiled" : "cuBLAS";
auto run = [&]() {
if (method == 0) {
matmulNaive<<<grid, block>>>(dA, dB, dC, n);
CHECK(cudaGetLastError());
} else if (method == 1) {
matmulTiled<<<grid, block>>>(dA, dB, dC, n);
CHECK(cudaGetLastError());
} else {
// cuBLAS is column-major. Row-major C = A*B, viewed column-major, is C^T = B^T * A^T
CHECK_CUBLAS(cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N, n, n, n, &alpha,
dB, n, dA, n, &beta, dC, n));
}
};
run(); // warm-up, and the result we verify
CHECK(cudaDeviceSynchronize());
CHECK(cudaMemcpy(hC.data(), dC, bytes, cudaMemcpyDeviceToHost));
allOk = verify(name, hC, ref, absRef, n) && allOk;
if (timed) {
CHECK(cudaEventRecord(start));
for (int r = 0; r < reps; ++r) run();
CHECK(cudaEventRecord(stop));
CHECK(cudaEventSynchronize(stop));
float ms = 0.0f;
CHECK(cudaEventElapsedTime(&ms, start, stop));
const double perRun = ms / reps;
std::printf(" %-7s %.3f ms/run, %.1f GFLOP/s\n", name, perRun,
flops / (perRun * 1e-3) / 1e9);
}
}
CHECK(cudaEventDestroy(start));
CHECK(cudaEventDestroy(stop));
CHECK(cudaFree(dA));
CHECK(cudaFree(dB));
CHECK(cudaFree(dC));
return allOk;
}
int main(int argc, char** argv) {
const int n = (argc > 1) ? std::atoi(argv[1]) : 1024;
cudaDeviceProp prop{};
CHECK(cudaGetDeviceProperties(&prop, 0));
std::printf("device: %s (CC %d.%d, %d SMs)\n", prop.name, prop.major, prop.minor,
prop.multiProcessorCount);
cublasHandle_t handle;
CHECK_CUBLAS(cublasCreate(&handle));
// 1) A size that is not a multiple of the tile: checks boundary handling only
std::printf("[correctness] n=%d (not a multiple of TILE=%d)\n", 1000, TILE);
bool ok = runCase(1000, false, handle);
// 2) Timing
std::printf("[timing] n=%d, 10 runs each (after 1 warm-up)\n", n);
ok = runCase(n, true, handle) && ok;
std::printf("occupancy (block = %d threads):\n", TILE * TILE);
printOccupancy("matmulNaive", matmulNaive, TILE * TILE, prop.maxThreadsPerMultiProcessor);
printOccupancy("matmulTiled", matmulTiled, TILE * TILE, prop.maxThreadsPerMultiProcessor);
CHECK_CUBLAS(cublasDestroy(handle));
std::printf("overall: %s\n", ok ? "PASS" : "FAIL");
return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}
Expected output (GPU results not published). The program runs in two stages. First, with — not a multiple of the tile — it compares all three methods with the CPU reference to check boundary handling. Then, with (the default), it runs each method ten times and prints the time per run and GFLOP/s. For each method it prints max abs error and worst err/bound; a worst err/bound of 1 or less is a PASS. Finally it prints the register count and theoretical occupancy of both kernels.
Choosing the tolerance
Each method adds in a different order, so the results do not match the CPU bit for bit. The program therefore sets a per-element tolerance from the rounding-error bound for an FP32 dot product. With , FP32's unit roundoff, the error of an -term dot product is bounded by roughly:
The program allows twice the right-hand side for each element. Any correct implementation lands inside that bound whatever the order of addition, with or without FMA (a multiply and an add with a single rounding), while an implementation with a wrong index lands far outside it. Indeed, changing Bs[k][tx] to Bs[tx][k] in the simulated environment pushed worst err/bound above 150,000 and produced a FAIL.
What the cuBLAS comparison tells you
The third method is cublasSgemm from NVIDIA's cuBLAS library. cuBLAS works in column-major order (the Fortran layout), so the row-major is computed as seen in column-major order.
A hand-written tiled kernel is only the textbook first step. Fast implementations go further: each thread computes several elements and reuses data in registers too (register tiling), and they use Tensor Cores (see Why NVIDIA GPUs are so strong at AI). Which implementation cublasSgemm picks on a T4 is something I have not checked.
11. Original analysis: theoretical occupancy on T4 and H100
Occupancy is the ratio of the warps resident on an SM to the maximum it can hold (the definition, and the caveat that higher is not always faster, are covered in section 9 of How CPUs and GPUs differ in architecture). Here the point is to compute how a T4 and an H100 differ.
The number of blocks resident on one SM is the minimum of four limits:
- the maximum resident threads per SM,
- the maximum resident blocks per SM,
- registers (registers per thread × threads must fit in the SM's 64K), and
- shared memory.
According to the CUDA C++ Best Practices Guide, registers are allocated per warp, rounded up to a multiple of 256. On H100, CUDA reserves 1 KB of shared memory per block (Hopper Tuning Guide). With these rules I computed the theoretical occupancy of this post's kernels on a T4 and an H100. Register counts are only known after compiling, so I varied them from 32 to 128.
- Requirements: Python 3.9+ (standard library only)
- Run:
python3 occupancy.py
"""Theoretical occupancy and GEMM arithmetic intensity from the published per-SM limits.
- Limits: CUDA Programming Guide "Compute Capabilities" Tables 33/34, and the
Hopper Tuning Guide (CUDA reserves 1 KB of shared memory per thread block)
- Registers are allocated per warp, rounded up to 256 (CUDA C++ Best Practices Guide)
- Everything here is theoretical. Check real values with
cudaOccupancyMaxActiveBlocksPerMultiprocessor or Nsight Compute.
Python 3.9+, standard library only.
"""
from dataclasses import dataclass
import math
@dataclass(frozen=True)
class SM:
name: str
max_threads: int # max resident threads per SM
max_blocks: int # max resident blocks per SM
regs: int # 32-bit registers per SM
smem: int # shared memory per SM (bytes)
smem_reserved: int = 0 # shared memory CUDA reserves per block (bytes)
T4 = SM("T4 (CC 7.5)", 1024, 16, 65536, 64 * 1024)
H100 = SM("H100 (CC 9.0)", 2048, 32, 65536, 228 * 1024, smem_reserved=1024)
CC70 = SM("CC 7.0", 2048, 32, 65536, 96 * 1024) # for checking the Best Practices Guide example
def blocks_per_sm(sm, threads, regs_per_thread, smem_per_block, warp_granularity=1):
warps = math.ceil(threads / 32)
regs_per_warp = math.ceil(regs_per_thread * 32 / 256) * 256 # round up to 256
by_threads = sm.max_threads // (warps * 32)
# warps allowed by registers (optionally floored to a multiple of warp_granularity)
warps_by_regs = (sm.regs // regs_per_warp) // warp_granularity * warp_granularity
by_regs = warps_by_regs // warps
by_smem = sm.smem // (smem_per_block + sm.smem_reserved) if smem_per_block else sm.max_blocks
return min(sm.max_blocks, by_threads, by_regs, by_smem)
def occupancy(sm, threads, regs, smem, warp_granularity=1):
b = blocks_per_sm(sm, threads, regs, smem, warp_granularity)
return b, b * threads / sm.max_threads
print("== Check against the Best Practices Guide example (CC 7.0, 37 regs/thread) ==")
for threads, doc in ((128, "75%"), (320, "63%")):
b1, o1 = occupancy(CC70, threads, 37, 0)
b4, o4 = occupancy(CC70, threads, 37, 0, warp_granularity=4)
print(f"{threads:4d} threads: 256-rule only -> {b1} blocks ({o1:.1%}), "
f"warps floored to x4 -> {b4} blocks ({o4:.1%}), guide says {doc}")
kernels = [ # (name, threads/block, static shared memory/block)
("reduceShfl 256thr", 256, 8 * 4),
("tiled GEMM TILE=16", 256, 2 * 16 * 16 * 4),
("tiled GEMM TILE=32", 1024, 2 * 32 * 32 * 4),
("smem 48KB 256thr", 256, 48 * 1024),
]
print("\n== Theoretical occupancy (blocks/SM, occupancy) ==")
print(f"{'kernel':20s} {'regs':>4s} | {'T4 (CC 7.5)':>16s} | {'H100 (CC 9.0)':>16s}")
for name, threads, smem in kernels:
for regs in (32, 64, 96, 128):
cells = []
for sm in (T4, H100):
b, occ = occupancy(sm, threads, regs, smem)
cells.append("launch fails" if b == 0 else f"{b:2d} blk {occ:6.1%}")
print(f"{name:20s} {regs:4d} | {cells[0]:>16s} | {cells[1]:>16s}")
print("\n== Max registers/thread for 100% occupancy (256-thread blocks) ==")
for sm in (T4, H100):
best = max(r for r in range(1, 256) if occupancy(sm, 256, r, 0)[1] == 1.0)
print(f"{sm.name}: <= {best} regs/thread")
print("\n== N x N FP32 GEMM: arithmetic intensity and bandwidth-bound ceiling (theoretical) ==")
N = 1024
devices = { # (theoretical FP32 CUDA-core peak FLOP/s, memory bandwidth byte/s)
"T4 (320 GB/s)": (8.1e12, 320e9),
"T4 (300 GB/s)": (8.1e12, 300e9),
"H100 SXM5": (66.9e12, 3352e9),
}
for dev, (peak, bw) in devices.items():
print(f"{dev}: ridge point = {peak / bw:.1f} FLOP/byte")
for label, tile in (("naive (no reuse)", 1), ("tiled TILE=16", 16), ("tiled TILE=32", 32)):
flops = 2 * N**3
traffic = 4 * (2 * N**3 / tile + N**2) # bytes: reads of A and B + writes of C
ai = flops / traffic
bounds = ", ".join(f"{dev} {min(p, b * ai) / 1e12:.2f}" for dev, (p, b) in devices.items())
print(f"{label:18s} AI = {ai:5.2f} FLOP/byte -> ceiling TFLOP/s: {bounds}")
Output on my machine (Apple M4 Pro, Python 3.13.7):
== Check against the Best Practices Guide example (CC 7.0, 37 regs/thread) ==
128 threads: 256-rule only -> 12 blocks (75.0%), warps floored to x4 -> 12 blocks (75.0%), guide says 75%
320 threads: 256-rule only -> 5 blocks (78.1%), warps floored to x4 -> 4 blocks (62.5%), guide says 63%
== Theoretical occupancy (blocks/SM, occupancy) ==
kernel regs | T4 (CC 7.5) | H100 (CC 9.0)
reduceShfl 256thr 32 | 4 blk 100.0% | 8 blk 100.0%
reduceShfl 256thr 64 | 4 blk 100.0% | 4 blk 50.0%
reduceShfl 256thr 96 | 2 blk 50.0% | 2 blk 25.0%
reduceShfl 256thr 128 | 2 blk 50.0% | 2 blk 25.0%
tiled GEMM TILE=16 32 | 4 blk 100.0% | 8 blk 100.0%
tiled GEMM TILE=16 64 | 4 blk 100.0% | 4 blk 50.0%
tiled GEMM TILE=16 96 | 2 blk 50.0% | 2 blk 25.0%
tiled GEMM TILE=16 128 | 2 blk 50.0% | 2 blk 25.0%
tiled GEMM TILE=32 32 | 1 blk 100.0% | 2 blk 100.0%
tiled GEMM TILE=32 64 | 1 blk 100.0% | 1 blk 50.0%
tiled GEMM TILE=32 96 | launch fails | launch fails
tiled GEMM TILE=32 128 | launch fails | launch fails
smem 48KB 256thr 32 | 1 blk 25.0% | 4 blk 50.0%
smem 48KB 256thr 64 | 1 blk 25.0% | 4 blk 50.0%
smem 48KB 256thr 96 | 1 blk 25.0% | 2 blk 25.0%
smem 48KB 256thr 128 | 1 blk 25.0% | 2 blk 25.0%
== Max registers/thread for 100% occupancy (256-thread blocks) ==
T4 (CC 7.5): <= 64 regs/thread
H100 (CC 9.0): <= 32 regs/thread
== N x N FP32 GEMM: arithmetic intensity and bandwidth-bound ceiling (theoretical) ==
T4 (320 GB/s): ridge point = 25.3 FLOP/byte
T4 (300 GB/s): ridge point = 27.0 FLOP/byte
H100 SXM5: ridge point = 20.0 FLOP/byte
naive (no reuse) AI = 0.25 FLOP/byte -> ceiling TFLOP/s: T4 (320 GB/s) 0.08, T4 (300 GB/s) 0.07, H100 SXM5 0.84
tiled TILE=16 AI = 3.97 FLOP/byte -> ceiling TFLOP/s: T4 (320 GB/s) 1.27, T4 (300 GB/s) 1.19, H100 SXM5 13.30
tiled TILE=32 AI = 7.88 FLOP/byte -> ceiling TFLOP/s: T4 (320 GB/s) 2.52, T4 (300 GB/s) 2.36, H100 SXM5 26.40
What the numbers show
1. Registers bind earlier on the H100. A 256-thread block keeps 100% theoretical occupancy only at 64 or fewer registers per thread on a T4, but 32 or fewer on an H100. Both SMs have 64K registers, but the H100 can hold twice as many warps per SM (64), so each warp gets half as many registers. The same kernel using 64 registers runs at 100% on a T4 and 50% on an H100.
2. A 1,024-thread block cannot launch at 65 or more registers. With 1,024 threads (32 warps) and 65 registers each, each warp needs registers, rounded up to 2,304, or 73,728 for the block. That exceeds the SM's 65,536, so the kernel cannot be launched. The CUDA Programming Guide warns that unless you tell the compiler the largest block size you will launch with, via __launch_bounds__(maxThreadsPerBlock), you may get "too many resources requested for launch" errors.
3. With heavy shared-memory use, the comparison flips. A kernel using 48 KB of shared memory per block fits only one block on a T4 (64 KB per SM): 25% occupancy. An H100 (228 KB, with 1 KB reserved per block) fits four: 50%.
4. The published rules alone do not reproduce the Best Practices Guide's example. The guide says that on CC 7.0 with 37 registers per thread, 128-thread blocks give 12 blocks (75%) and 320-thread blocks give 4 blocks (63%). Rounding to 256 alone matches the first case but gives 5 blocks (78.1%) for the second. Flooring the register-limited warp count to a multiple of 4 reproduces both. That "multiple of 4", however, is not stated in the guide; it is my inference. Nsight Compute's Python occupancy API exposes a warps_allocation_granularity field, but its value is not documented.
So read the table as theoretical upper bounds (it also ignores shared-memory allocation granularity). The real values can be checked with the cudaOccupancyMaxActiveBlocksPerMultiprocessor results that the programs in sections 8 and 10 print, or with Nsight Compute.
Performance ceilings from arithmetic intensity
The second half of the script compares the arithmetic intensity from section 10 with the ridge point — the ratio of theoretical FP32 peak to memory bandwidth, where a kernel moves from bandwidth-bound to compute-bound (the idea is covered in section 6 of Why NVIDIA GPUs are so strong at AI). The T4 comes out at about 25.3 FLOP/byte (using the whitepaper's 320 GB/s; the datasheet says 300 GB/s, an inconsistency between NVIDIA's own documents, which would give about 27.0), and the H100 SXM5 at about 20.0. Even at the arithmetic intensity is only about 7.9, so in this model the tiled multiply is still bandwidth-bound. Keep in mind, though, that the model assumes "every load goes to DRAM" and ignores cache reuse; a real naive kernel may well run faster than this ceiling thanks to the L1 and L2 caches.
12. Streams and events: overlapping transfers with compute
A stream is a queue
A CUDA stream is a queue that orders work sent to the GPU (copies and kernels). Work in one stream runs in the order it was queued, with no overtaking. Work in different streams may run concurrently, so if you split the data and run "copy in → compute → copy out" in a separate stream per chunk, the next chunk can be copied in while the current one is being computed.
Three conditions for overlap
Overlapping transfers with compute requires:
- Pinned host memory. Given ordinary memory (
malloc,std::vector),cudaMemcpyAsyncfalls back to synchronous behavior and overlaps with nothing. Allocate pinned (page-locked) memory withcudaMallocHost. It is a scarce resource, though, and overusing it can slow down the whole system. - Separate, non-default streams. According to the Best Practices Guide, operations in the default stream do not begin until all preceding operations in every stream have finished, so overlap needs non-default streams.
cudaStreamNonBlockingonly matters for implicit synchronization with the legacy default stream; this program never uses the default stream but sets the flag anyway, to be safe. - Enough copy engines. With one copy engine, a GPU can run a transfer in one direction alongside kernels; with two, it can copy in, copy out and run kernels at the same time. The count is in
asyncEngineCount. I have not checked the value on a T4 (the program prints it).
Estimating the benefit of overlap
Let's estimate, in theory, how long the chunked version takes. Let the times to copy in, compute and copy out the whole dataset be , and , and split it into equal chunks ( and so on). Assuming two copy engines and full concurrency of the three stages, a three-stage pipeline takes:
Running serially takes . If , then gives , half of the serial . No can go below , so the ceiling in this case is 3×. This is a theoretical figure that ignores per-chunk launch overhead and contention for PCIe bandwidth. With a single copy engine, copying in and out serialize and the benefit shrinks.
- Build:
nvcc -arch=sm_75 -O3 -o streams streams.cu - Run:
./streams [iterations](the kernel's cost scales with the iteration count; the default is 500)
// streams.cu — overlapping transfers and compute with CUDA streams
// A) one stream: "copy everything in -> compute everything -> copy everything out"
// B) split the data into kStreams chunks; each stream runs "copy in -> compute -> copy out"
// Both use pinned (page-locked) host memory, a precondition for asynchronous copies.
// Build: nvcc -arch=sm_75 -O3 -o streams streams.cu
// Run: ./streams [iterations inside the kernel (default 500)]
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>
#include <cuda_runtime.h>
#define CHECK(call) \
do { \
cudaError_t err_ = (call); \
if (err_ != cudaSuccess) { \
std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n", \
cudaGetErrorName(err_), __FILE__, __LINE__, \
cudaGetErrorString(err_)); \
std::exit(EXIT_FAILURE); \
} \
} while (0)
constexpr int kStreams = 4;
constexpr int kThreads = 256;
// Kernel whose cost scales with the iteration count (a contraction that pulls v toward 1.0)
__global__ void heavy(const float* in, float* out, int n, int iters) {
const int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
float v = in[i];
for (int k = 0; k < iters; ++k) v = v * 0.999f + 0.001f;
out[i] = v;
}
}
int main(int argc, char** argv) {
const int iters = (argc > 1) ? std::atoi(argv[1]) : 500;
const int n = 1 << 24; // 16M elements = 64 MiB
const int chunk = n / kStreams; // n is divisible by kStreams
const size_t bytes = static_cast<size_t>(n) * sizeof(float);
const size_t chunkBytes = static_cast<size_t>(chunk) * sizeof(float);
cudaDeviceProp prop{};
CHECK(cudaGetDeviceProperties(&prop, 0));
std::printf("device: %s, asyncEngineCount=%d (copy engines)\n", prop.name,
prop.asyncEngineCount);
float *hIn = nullptr, *hOut = nullptr;
CHECK(cudaMallocHost(&hIn, bytes)); // pinned memory
CHECK(cudaMallocHost(&hOut, bytes));
for (int i = 0; i < n; ++i) hIn[i] = static_cast<float>(i % 1000) / 1000.0f;
// CPU reference. The input has only 1000 distinct values, so compute those 1000 and look them up
// (the GPU may fuse the multiply and add into an FMA, so allow a small difference)
std::vector<float> table(1000);
for (int r = 0; r < 1000; ++r) {
float v = static_cast<float>(r) / 1000.0f;
for (int k = 0; k < iters; ++k) v = v * 0.999f + 0.001f;
table[r] = v;
}
auto check = [&](const char* name) {
double maxErr = 0.0;
for (int i = 0; i < n; ++i) {
const double err = std::fabs(static_cast<double>(hOut[i]) - table[i % 1000]);
maxErr = std::fmax(maxErr, err);
}
const bool ok = maxErr <= 1e-4;
std::printf(" %-8s max abs error = %.3e (tolerance 1e-4) %s\n", name, maxErr,
ok ? "PASS" : "FAIL");
return ok;
};
float *dIn = nullptr, *dOut = nullptr;
CHECK(cudaMalloc(&dIn, bytes));
CHECK(cudaMalloc(&dOut, bytes));
cudaStream_t streams[kStreams];
cudaEvent_t done[kStreams];
for (int s = 0; s < kStreams; ++s) {
CHECK(cudaStreamCreateWithFlags(&streams[s], cudaStreamNonBlocking));
CHECK(cudaEventCreateWithFlags(&done[s], cudaEventDisableTiming));
}
cudaEvent_t start, stop;
CHECK(cudaEventCreate(&start));
CHECK(cudaEventCreate(&stop));
// A) Serial execution in one stream
auto runSerial = [&]() {
cudaStream_t s0 = streams[0];
CHECK(cudaEventRecord(start, s0));
CHECK(cudaMemcpyAsync(dIn, hIn, bytes, cudaMemcpyHostToDevice, s0));
heavy<<<(n + kThreads - 1) / kThreads, kThreads, 0, s0>>>(dIn, dOut, n, iters);
CHECK(cudaGetLastError());
CHECK(cudaMemcpyAsync(hOut, dOut, bytes, cudaMemcpyDeviceToHost, s0));
CHECK(cudaEventRecord(stop, s0));
CHECK(cudaEventSynchronize(stop));
};
// B) Split across kStreams streams; dependencies on start/stop are explicit via events
auto runOverlapped = [&]() {
CHECK(cudaEventRecord(start, streams[0]));
for (int s = 1; s < kStreams; ++s) {
CHECK(cudaStreamWaitEvent(streams[s], start, 0)); // start after `start`
}
for (int s = 0; s < kStreams; ++s) {
const size_t off = static_cast<size_t>(s) * chunk;
CHECK(cudaMemcpyAsync(dIn + off, hIn + off, chunkBytes, cudaMemcpyHostToDevice,
streams[s]));
heavy<<<(chunk + kThreads - 1) / kThreads, kThreads, 0, streams[s]>>>(
dIn + off, dOut + off, chunk, iters);
CHECK(cudaGetLastError());
CHECK(cudaMemcpyAsync(hOut + off, dOut + off, chunkBytes, cudaMemcpyDeviceToHost,
streams[s]));
CHECK(cudaEventRecord(done[s], streams[s]));
}
for (int s = 1; s < kStreams; ++s) {
CHECK(cudaStreamWaitEvent(streams[0], done[s], 0)); // wait for every stream to finish
}
CHECK(cudaEventRecord(stop, streams[0]));
CHECK(cudaEventSynchronize(stop));
};
bool ok = true;
float ms = 0.0f;
std::printf("n=%d floats (%.0f MiB), iters=%d, streams=%d\n", n, bytes / 1048576.0, iters,
kStreams);
runSerial(); // warm-up
for (int i = 0; i < n; ++i) hOut[i] = 0.0f; // clear, so that only the timed run's output is verified
runSerial();
CHECK(cudaEventElapsedTime(&ms, start, stop));
std::printf("A) serial (1 stream) : %.3f ms\n", ms);
ok = check("serial") && ok;
runOverlapped(); // warm-up
for (int i = 0; i < n; ++i) hOut[i] = 0.0f; // clear, so that only the timed run's output is verified
runOverlapped();
CHECK(cudaEventElapsedTime(&ms, start, stop));
std::printf("B) overlapped (%d streams): %.3f ms\n", kStreams, ms);
ok = check("overlap") && ok;
for (int s = 0; s < kStreams; ++s) {
CHECK(cudaStreamDestroy(streams[s]));
CHECK(cudaEventDestroy(done[s]));
}
CHECK(cudaEventDestroy(start));
CHECK(cudaEventDestroy(stop));
CHECK(cudaFree(dIn));
CHECK(cudaFree(dOut));
CHECK(cudaFreeHost(hIn));
CHECK(cudaFreeHost(hOut));
std::printf("overall: %s\n", ok ? "PASS" : "FAIL");
return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}
Expected output (GPU results not published). The program prints the GPU name and asyncEngineCount, then the time in milliseconds for A (serial, one stream) and B (split across four streams), and compares each result (from the timed run, with the output cleared after the warm-up) with the CPU reference to print PASS or FAIL. Overlap is easiest to see when kernel time and copy time are similar; if one dominates, adjust the iteration count on the command line.
A few notes:
- The GPU may fuse
v * 0.999f + 0.001finto an FMA. According to the NVCC documentation,--fmadis on by default. The CPU reference is computed without FMA, so the results differ slightly. When I computed the gap between the FMA and non-FMA versions for 1,000 inputs on my Mac, it was at most at 500 iterations and at most even at 100,000 iterations. The tolerance adds margin on top of that. - B makes its start and end dependencies explicit with
cudaStreamWaitEvent, rather than relying on the default stream's implicit synchronization. - According to the CUDA Runtime API documentation, events recorded in a non-default stream may report a longer time than expected, because other work can run in between. Check exactly how the work overlaps on the Nsight Systems timeline (section 13).
Asynchronous copies inside a kernel: cp.async and TMA
Overlap with streams is about transfers between CPU and GPU. Separately, there are mechanisms that make the transfer from global memory to shared memory asynchronous inside a kernel:
- cp.async (LDGSTS): CC 8.0 (Ampere) and later. A thread starts a copy from global to shared memory and continues computing without waiting for it. The PTX ISA document says it "Requires sm_80 or higher".
- TMA (Tensor Memory Accelerator): CC 9.0 (Hopper) and later. It moves large blocks, or tensors of up to five dimensions, between global and shared memory asynchronously in one operation.
Neither is available on the T4 (CC 7.5). The tiled multiply in this post keeps the synchronous shape load → __syncthreads() → compute → __syncthreads().
13. Nsight Systems and Nsight Compute: which to reach for first
NVIDIA has two performance tools with completely different jobs.
| Nsight Systems | Nsight Compute | |
|---|---|---|
| What it shows | A timeline of the whole application (CPU threads, CUDA API calls, kernels, copies, streams) | The inside of one kernel (compute and memory utilization, occupancy, traffic at each level of the memory hierarchy) |
| Official description | A system-wide performance analysis tool | An interactive kernel profiler |
| Question it answers | "Where is the GPU idle?" "Do transfers and compute overlap?" | "Is this kernel compute-bound or bandwidth-bound?" "Why is it slow?" |
| How it measures | Records a single run; does not lock clocks | Replays the kernel several times to collect metrics; locks SM clocks by default and flushes caches between replay passes |
| CLI example | nsys profile -o report ./streams | ncu --section SpeedOfLight --section Occupancy -k regex:matmul ./matmul |
The order is Nsight Systems first, then Nsight Compute. Studying one kernel in detail before seeing the whole timeline risks optimizing something that is not the bottleneck; the GPU is often waiting on CPU-side preprocessing or data transfers. Right-clicking a kernel on the Nsight Systems timeline offers to examine it in Nsight Compute.
The Nsight Compute documentation names clock locking and cache flushing as the reasons the two tools can report different durations for the same kernel. For comparisons it recommends locking clocks externally with nvidia-smi and passing --clock-control none to ncu.
For correctness, use Compute Sanitizer: memcheck (out-of-bounds access), racecheck (shared-memory races), initcheck (uninitialized reads) and synccheck (synchronization errors). Building reduce.cu from section 8 with -DOMIT_SYNC removes the __syncthreads() in the shared-memory tree, which compute-sanitizer --tool racecheck should flag as a race (I have not run it). Note that racecheck looks at shared-memory races; the global-memory race version (reduceRacy) is outside its scope.
14. Where PyTorch fits: from torch.matmul down to kernels
If you use PyTorch day to day, where does all this show up?
- Asynchronous execution: PyTorch's documentation says "By default, GPU operations are asynchronous." Operations are queued on the GPU, not executed immediately. So timing with
time.time()alone is inaccurate; you needtorch.cuda.synchronize()ortorch.cuda.Event(enable_timing=True)— the same reason the programs here usecudaEventRecord. To find where an error occurred, the environment variableCUDA_LAUNCH_BLOCKING=1forces synchronous execution. - Matrix multiplication is cuBLAS: according to the description of
torch.backends.cuda.preferred_blas_library, PyTorch defaults to cuBLAS for CUDA BLAS operations even when both cuBLAS and cuBLASLt are available. Comparing againstcublasSgemmin section 10 means comparing against whattorch.matmulcalls under the hood. - Convolutions and more are cuDNN: the
torch.backends.cudnnsettings control the use of cuDNN. - Streams and pinned memory:
torch.cuda.Streamcreates streams. CombiningDataLoader(pin_memory=True)with.to(device, non_blocking=True)is exactly section 12's "pinned memory + asynchronous copy". - Custom kernels:
torch.utils.cpp_extension.load_inlinebuilds and loads C++ and CUDA code given as strings at run time. PyTorch's tutorial shows how to register an operator withTORCH_LIBRARYand its CUDA implementation withTORCH_LIBRARY_IMPL(..., CUDA, m). - torch.compile: according to PyTorch's documentation, TorchInductor, the default compiler, uses OpenAI Triton as its key building block on NVIDIA GPUs to generate kernels. It is a way to fuse several operations into one kernel without writing CUDA C++.
In other words, a line of PyTorch ends up running on the GPU as a cuBLAS or cuDNN kernel, one of PyTorch's own CUDA kernels, or a kernel generated by Triton. Which one actually runs can be seen on the Nsight Systems timeline.
15. Should you write CUDA? Limits and how to decide
When CUDA disappoints
Turn the above inside out and the conditions under which CUDA disappoints become clear.
- Too little parallel work. For an array of a few thousand elements, kernel-launch and transfer costs can dominate. The Best Practices Guide contrasts device-memory bandwidth (898 GB/s in its V100 example) with PCIe x16 Gen3 (16 GB/s) and advises keeping transfers to a minimum.
- Strong dependencies between blocks. With no grid-wide barrier, a computation where every element depends on all results of the previous step has to be split into many kernels.
- Heavy branching that splits warps. With a lot of divergence, a warp executes both paths and most of its lanes sit idle.
- Floating-point reproducibility is required. As section 8 showed, the order of addition changes the result, and a sum built with atomicAdd can produce a different bit pattern on every run.
- Portability is required. CUDA code runs only on NVIDIA GPUs — and not on Macs.
- Maintenance cost. A custom kernel must be re-checked for performance whenever the GPU generation changes, whereas NVIDIA updates cuBLAS and cuDNN for each new generation.
The order in which to pick tools
Here is the order in which to pick tools when you need something to run faster on a GPU.
| Situation | Reach for first | When to consider a custom kernel |
|---|---|---|
| Standard operations: matrix multiply, convolution, attention | cuBLAS, cuDNN, the framework's built-in operations | Almost never |
| Many small operations chained together | torch.compile (fusion via Triton) | Memory round trips still dominate after fusion |
| A custom operation no library provides | Higher-level tools such as Triton or CuPy | You need fine control of shared memory or warp-level operations |
| You do not know what is slow | Look at the whole run in Nsight Systems | Only after the bottleneck is pinned to one specific kernel |
When you do write one, proceed in this order: (1) build a CPU reference and a justified tolerance first; (2) confirm the bottleneck with Nsight Systems; (3) estimate what bounds the kernel from its arithmetic intensity (if it is bandwidth-bound, cut reads and writes through tiling or fusion first); (4) use occupancy only to check it is not too low; and (5) confirm both performance and correctness with Nsight Compute and Compute Sanitizer.
The single-T4 inference service I have worked on never needed a hand-written CUDA kernel (GPU pipeline for AI video localization). Still, the facts that kernel launches are asynchronous, and that GPU memory is limited so where data lives decides performance, fed directly into decisions such as vLLM settings and running jobs one at a time. The value of knowing how CUDA works, whether or not you ever write a kernel, is being able to reason about what is happening on the GPU.
In practice, what pays off first is less whether to write a kernel than which GPU to use and how many. Whether the weights and KV cache fit in VRAM, and whether self-hosting beats paying per API call, are worked out in Quantization and the VRAM budget and API or self-hosting.
Summary
- CUDA is a model and toolbox in which the host (CPU) launches huge numbers of threads on the device (GPU). Launches are asynchronous, and the memories are separate by default. It does not normally run on Apple Silicon Macs.
- Threads come in three levels — grid, block, thread — and a 32-thread warp executes one instruction at a time. A T4 keeps up to 40,960 threads resident on 2,560 arithmetic units and hides latency by switching warps.
- NVCC turns CUDA C++ into PTX through NVVM (LLVM-based), and ptxas turns PTX into SASS. PTX is a virtual ISA that spans generations; ship it in the binary and the driver can JIT-compile it. SASS is per-generation machine code.
- There are five kinds of memory — registers, shared, local, constant, global. Coalescing at the warp level governs global-memory performance, and bank conflicts govern shared memory.
- Synchronization is hierarchical.
__syncthreads()covers a block and shuffles a warp; there is no grid-wide barrier. The memory model is weak, and from CC 7.0 even a warp does not line up automatically. - Original analysis: from the published per-SM limits, a 256-thread block keeps 100% theoretical occupancy only at 64 or fewer registers per thread on a T4 and 32 or fewer on an H100. The Best Practices Guide's example cannot be reproduced from the documented rules alone, so treat the table as theoretical upper bounds.
- In practice, start with cuBLAS, cuDNN and
torch.compile, confirm the bottleneck with Nsight Systems, and only then consider a custom kernel.