How to Think About GPUs
Auto-discovered
Visit resource ↗Description
How to Think About GPUs is a free online chapter from How To Scale Your Model by Jacob Austin, Swapnil Patil, Adam Paszke, and Reiner Pope. Published August 18, 2025, it compares modern NVIDIA ML GPUs with Google TPUs for large-model workloads.
It explains Streaming Multiprocessors, Tensor Cores, CUDA cores and SIMT warp scheduling, shared memory and L2 hierarchy, HBM capacity and bandwidth, and spec tables from V100 through B200. Later sections cover multi-GPU networking, LLM communication patterns, and where GPU versus TPU design trades flexibility for matrix-multiply efficiency, with diagrams mapping units such as MXU and VMEM.
It is for advanced ML systems engineers and kernel authors who already know basic accelerator rooflines and want a GPU mental model aligned with the book's TPU chapters.
Access is a free static page on the jax-ml scaling-book site with figures, tables, and notes. No login is required.
It is useful when you need concrete H100 and Blackwell vocabulary while reasoning about LLM scaling bottlenecks relative to TPUs.