← All resources

Inside NVIDIA GPUs: Anatomy of high performance matmul kernels

Auto-discovered

Blog Advanced

Aleksa Gordić

Visit resource ↗

Description

Inside NVIDIA GPUs: Anatomy of high performance matmul kernels is a long self-contained blog post by Aleksa Gordić that teaches how near-peak NVIDIA matrix-multiply kernels are built, with Hopper H100 as the main hardware target.

Coverage moves from HBM, L2, shared memory, registers, tensor cores, and TMA, through PTX and SASS, then warp-tiled synchronous matmul, and finally asynchronous Hopper pipelines that overlap tensor-core compute with loads and stores.

It is for advanced GPU programmers and ML systems engineers who write or read CUDA kernels. Comfort with CUDA grids, warps, and basic linear algebra is assumed.

The article is free on aleksagordic.com as the first part of a planned series, with figures and staged explanations rather than a short tip list.

It is useful when you need one narrative that connects hardware hierarchy to SOTA matmul kernel structure.