mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-16 23:24:55 -05:00
266 lines
9.6 KiB
Plaintext
266 lines
9.6 KiB
Plaintext
---
|
||
title: "The Memory Wall"
|
||
subtitle: "Why 3.2× more FLOPS gives only 1.7× speedup — and how to know in advance."
|
||
description: "Compare A100 and H100 GPUs to discover that for memory-bound workloads, bandwidth — not compute — determines performance. The most important fallacy in ML systems."
|
||
categories: ["node", "beginner"]
|
||
---
|
||
|
||
## The Question
|
||
|
||
NVIDIA's H100 has **3.2× more FLOP/s** than the A100. So upgrading should give you a 3.2×
|
||
speedup, right?
|
||
|
||
**Wrong.** For the workloads that matter most in production — LLM inference, recommendation
|
||
models, any memory-bound task — you get closer to **1.7×**. This tutorial shows you exactly
|
||
why, and teaches you to predict the actual speedup before spending a dollar on hardware.
|
||
|
||
::: {.callout-note}
|
||
## Prerequisites
|
||
Complete [Tutorial 0: Hello, Roofline](00_hello_roofline.qmd). You should understand
|
||
memory-bound vs. compute-bound and the ridge point concept.
|
||
:::
|
||
|
||
::: {.callout-note}
|
||
## What You Will Learn
|
||
|
||
- **Calculate** the actual speedup between two GPUs for a given workload
|
||
- **Explain** why the binding constraint determines which spec matters
|
||
- **Predict** whether a hardware upgrade will help a specific model
|
||
- **Apply** the roofline model to hardware procurement decisions
|
||
:::
|
||
|
||
::: {.callout-tip}
|
||
## Background: The Two Specs That Matter
|
||
|
||
GPU vendors advertise peak FLOP/s prominently. But every GPU also has a memory bandwidth
|
||
spec (in TB/s) that is equally important. Which spec determines your actual performance
|
||
depends entirely on which **regime** your workload is in:
|
||
|
||
| Regime | Binding Constraint | Speedup Scales With |
|
||
|:-------|:-------------------|:--------------------|
|
||
| Memory-bound | HBM bandwidth (TB/s) | Bandwidth ratio between GPUs |
|
||
| Compute-bound | Peak arithmetic (FLOP/s) | FLOP/s ratio between GPUs |
|
||
|
||
The key numbers for this tutorial:
|
||
|
||
| Spec | A100 | H100 | Ratio |
|
||
|:-----|:-----|:-----|:------|
|
||
| Peak FP16 | 312 TFLOP/s | 989 TFLOP/s | **3.2×** |
|
||
| HBM Bandwidth | 2.0 TB/s | 3.35 TB/s | **1.7×** |
|
||
|
||
If your workload is memory-bound, the speedup ceiling is 1.7×, regardless of the 3.2× compute improvement.
|
||
:::
|
||
|
||
---
|
||
|
||
## 1. Setup
|
||
|
||
```{python}
|
||
#| echo: false
|
||
#| output: false
|
||
import mlsysim # installed via `pip install mlsysim` (see workflow)
|
||
Engine = mlsysim.Engine
|
||
```
|
||
|
||
```python
|
||
import mlsysim
|
||
from mlsysim import Engine
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Side-by-Side Hardware Comparison
|
||
|
||
Let's load both GPUs from the Silicon Zoo and confirm the specs:
|
||
|
||
```{python}
|
||
from mlsysim.show import table
|
||
|
||
a100 = mlsysim.Hardware.Cloud.A100
|
||
h100 = mlsysim.Hardware.Cloud.H100
|
||
|
||
flops_a = a100.compute.peak_flops.to("TFLOPs/s").magnitude
|
||
flops_h = h100.compute.peak_flops.to("TFLOPs/s").magnitude
|
||
bw_a = a100.memory.bandwidth.to("TB/s").magnitude
|
||
bw_h = h100.memory.bandwidth.to("TB/s").magnitude
|
||
|
||
table(
|
||
["Spec", "A100", "H100", "Ratio"],
|
||
[
|
||
["Peak FP16 (TFLOP/s)", flops_a, flops_h, f"{flops_h/flops_a:.1f}x"],
|
||
["HBM BW (TB/s)", bw_a, bw_h, f"{bw_h/bw_a:.1f}x"],
|
||
["Ridge (FLOP/byte)", flops_a*1e12/(bw_a*1e12), flops_h*1e12/(bw_h*1e12), ""]
|
||
]
|
||
)
|
||
```
|
||
|
||
The FLOP/s ratio is 3.2× but the bandwidth ratio is only 1.7×. The ridge point also
|
||
shifts: the H100 has a *higher* ridge, meaning more workloads fall into the memory-bound
|
||
regime on the H100 than on the A100.
|
||
|
||
---
|
||
|
||
## 3. The Fallacy: LLM Inference Speedup
|
||
|
||
Let's test the "3.2× speedup" claim with a workload that dominates production
|
||
today — Llama-3 8B inference at batch size 1:
|
||
|
||
```{python}
|
||
model = mlsysim.Models.Language.Llama3_8B
|
||
|
||
# Solve on both GPUs
|
||
prof_a100 = Engine.solve(model=model, hardware=a100, batch_size=1, precision="fp16")
|
||
prof_h100 = Engine.solve(model=model, hardware=h100, batch_size=1, precision="fp16")
|
||
|
||
lat_a = prof_a100.latency.to("ms").magnitude
|
||
lat_h = prof_h100.latency.to("ms").magnitude
|
||
speedup = lat_a / lat_h
|
||
|
||
table(
|
||
["", "A100", "H100", "Speedup"],
|
||
[
|
||
["Bottleneck", prof_a100.bottleneck, prof_h100.bottleneck, ""],
|
||
["Latency", prof_a100.latency.to("ms"), prof_h100.latency.to("ms"), f"{speedup:.1f}x"],
|
||
["Throughput", prof_a100.throughput, prof_h100.throughput, ""]
|
||
]
|
||
)
|
||
```
|
||
|
||
Both GPUs report **memory-bound**. The actual speedup is approximately **1.7×** — matching
|
||
the bandwidth ratio, not the FLOP/s ratio. The extra 1.5× compute power of the H100 is
|
||
entirely wasted for this workload.
|
||
|
||
**Sanity check:** Llama-3 8B at FP16 = 8B params × 2 bytes = 16 GB of weights. On the A100
|
||
(2.0 TB/s), minimum decode latency ≈ 16 GB ÷ 2.0 TB/s = **8.0 ms**. On the H100
|
||
(3.35 TB/s), it is 16 GB ÷ 3.35 TB/s = **4.8 ms**. Speedup = 8.0 / 4.8 = **1.67×** —
|
||
matching the bandwidth ratio, as expected for a memory-bound workload.
|
||
|
||
---
|
||
|
||
## 4. When DOES 3.2× Matter? The Batch Size Crossover
|
||
|
||
The FLOP/s advantage only kicks in when you cross into the compute-bound regime.
|
||
Let's sweep batch size on both GPUs and find the crossover:
|
||
|
||
```{python}
|
||
rows = []
|
||
for batch in [1, 4, 16, 32, 64, 128, 256]:
|
||
pa = Engine.solve(model=model, hardware=a100, batch_size=batch, precision="fp16")
|
||
ph = Engine.solve(model=model, hardware=h100, batch_size=batch, precision="fp16")
|
||
|
||
la = pa.latency.to("ms").magnitude
|
||
lh = ph.latency.to("ms").magnitude
|
||
sp = la / lh if lh > 0 else 0
|
||
|
||
rows.append([batch, pa.bottleneck, ph.bottleneck, f"{sp:.1f}x"])
|
||
|
||
table(["Batch", "A100 Bottleneck", "H100 Bottleneck", "Speedup"], rows)
|
||
```
|
||
|
||
We can visualize where Llama-3 8B sits on the H100's Roofline model. Note the high ridge point:
|
||
|
||
```{python}
|
||
#| fig-width: 7
|
||
#| fig-height: 4.5
|
||
#| warning: false
|
||
from mlsysim.viz.plots import plot_roofline
|
||
|
||
# Plot the H100 roofline and see where Llama-3 8B (batch 1) falls
|
||
fig, ax = plot_roofline(h100, workloads=[model])
|
||
```
|
||
|
||
::: {.callout-important}
|
||
## Key Insight
|
||
|
||
**The binding constraint determines which hardware spec matters.** When you are memory-bound,
|
||
speedup scales with the bandwidth ratio (1.7×). When you are compute-bound, speedup scales
|
||
with the FLOP/s ratio (up to 3.2×). The transition happens at different batch sizes on each
|
||
GPU because the H100's higher ridge point means it *stays memory-bound longer*. If you are
|
||
making a procurement decision, the first question is not "how many FLOP/s?" but "which regime
|
||
will my production workload operate in?"
|
||
:::
|
||
|
||
---
|
||
|
||
## 5. The Procurement Table: Three Generations
|
||
|
||
Let's extend the analysis across three GPU generations to see the trend:
|
||
|
||
```{python}
|
||
gpus = [
|
||
("V100", mlsysim.Hardware.Cloud.V100),
|
||
("A100", mlsysim.Hardware.Cloud.A100),
|
||
("H100", mlsysim.Hardware.Cloud.H100),
|
||
]
|
||
|
||
rows = []
|
||
for name, hw in gpus:
|
||
p = Engine.solve(model=model, hardware=hw, batch_size=1, precision="fp16")
|
||
flops = hw.compute.peak_flops.to("TFLOPs/s").magnitude
|
||
bw = hw.memory.bandwidth.to("TB/s").magnitude
|
||
ridge = flops * 1e12 / (bw * 1e12)
|
||
lat = p.latency.to("ms").magnitude
|
||
rows.append([name, flops, bw, ridge, p.latency, p.bottleneck])
|
||
|
||
table(["GPU", "TFLOP/s", "BW (TB/s)", "Ridge", "Latency", "Bottleneck"], rows)
|
||
```
|
||
|
||
Across three generations, compute has grown faster than bandwidth. The ridge point keeps
|
||
rising, which means **more workloads are memory-bound on newer hardware**. This is the
|
||
memory wall — and it is getting worse, not better.
|
||
|
||
---
|
||
|
||
## Your Turn
|
||
|
||
::: {.callout-caution}
|
||
## Exercises
|
||
|
||
**Exercise 1: Predict before you compute.**
|
||
The B200 has ~8 TB/s HBM3e bandwidth and ~2250 TFLOP/s (FP16 dense). Before running any
|
||
code, predict: *write the speedup as a ratio (e.g., 2.3×)* for Llama-3 8B going from
|
||
H100 → B200 at batch size 1. Record your reasoning in one sentence. Then verify with
|
||
`mlsysim.Hardware.Cloud.B200`. How close were you?
|
||
|
||
**Exercise 2: Find the crossover batch size.**
|
||
For the A100, at what exact batch size does Llama-3 8B transition from memory-bound to
|
||
compute-bound? Write a loop that sweeps batch sizes from 1 to 512 in steps of 1 and
|
||
prints the first compute-bound batch size. Do the same for the H100. Why is the crossover
|
||
different?
|
||
|
||
**Exercise 3: Validate against published benchmarks.**
|
||
Look up the MLPerf Inference results for LLM workloads on A100 vs. H100 (available at
|
||
mlcommons.org). Compare the measured throughput ratio to our analytical prediction from
|
||
Section 3. What accounts for the difference? (Hint: real systems include software
|
||
optimizations like FlashAttention and continuous batching that our first-order model
|
||
does not capture. The gap between analytical prediction and measured performance is
|
||
itself informative.)
|
||
|
||
**Self-check:** If GPU-A has 500 TFLOP/s and 2 TB/s bandwidth, and GPU-B has 1000 TFLOP/s
|
||
and 4 TB/s bandwidth, what speedup do you expect for a memory-bound workload? For a
|
||
compute-bound workload? *(Write each answer as a ratio.)*
|
||
:::
|
||
|
||
---
|
||
|
||
## Key Takeaways
|
||
|
||
::: {.callout-tip}
|
||
## Summary
|
||
|
||
- **The memory wall is real**: HBM bandwidth has grown slower than compute across GPU generations
|
||
- **Speedup depends on regime**: memory-bound workloads scale with bandwidth ratio, not FLOP/s ratio
|
||
- **The ridge point rises each generation**: more production workloads are memory-bound on newer GPUs
|
||
- **Procurement decisions require regime analysis**: always check which wall binds before comparing specs
|
||
- **The roofline model predicts this**: `Engine.solve` tells you the regime before you spend a dollar
|
||
:::
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
- **[Two Phases, One Request](02_two_phases.qmd)** — Discover that LLM serving hits *both* ceilings in a single request
|
||
- **[Quantization: Not a Free Lunch](05_quantization.qmd)** — Learn when reducing precision helps (memory-bound) vs. when it doesn't (compute-bound)
|
||
- **[Where to Invest](09_sensitivity.qmd)** — Use sensitivity analysis to quantify exactly how much each spec matters
|
||
- **[Silicon Zoo](../zoo/hardware.qmd)** — Browse all GPU specs including V100, A100, H100, H200, B200, MI300X, and TPUs
|