mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-08-03 12:14:39 -05:00
285 lines
10 KiB
Plaintext
285 lines
10 KiB
Plaintext
---
|
||
title: "Starving the GPU"
|
||
subtitle: "Your GPU can process 5,300 images per second. Your CPU decodes 850."
|
||
description: "Discover that the data pipeline — not the GPU — is often the binding constraint in training. Use DataModel and TransformationModel to find the crossover where CPU preprocessing stalls the accelerator."
|
||
categories: ["data", "intermediate"]
|
||
---
|
||
|
||
## The Question
|
||
|
||
You launch ResNet-50 training on an A100 and watch `nvidia-smi`. GPU utilization reads 40%.
|
||
You expected 95%. The model is compute-bound. The hardware is top-tier. **Why is your GPU
|
||
sitting idle 60% of the time?**
|
||
|
||
The answer is almost never the model or the GPU. It is the invisible pipeline upstream:
|
||
JPEG decoding, random cropping, color jitter, and normalization — all running on the CPU.
|
||
When the CPU cannot prepare batches fast enough, the GPU starves.
|
||
|
||
::: {.callout-note}
|
||
## Prerequisites
|
||
Complete [Tutorial 0: Hello, Roofline](00_hello_roofline.qmd) and
|
||
[Tutorial 1: The Memory Wall](01_memory_wall.qmd). You should understand memory-bound
|
||
vs. compute-bound regimes and how `Engine.solve` reports bottlenecks.
|
||
:::
|
||
|
||
::: {.callout-note}
|
||
## What You Will Learn
|
||
|
||
- **Measure** the GPU's step time in isolation using `SingleNodeModel`
|
||
- **Calculate** the data pipeline's throughput using `DataModel` and `TransformationModel`
|
||
- **Identify** the batch size crossover where the CPU becomes the binding constraint
|
||
- **Predict** how many CPU workers are needed to eliminate the data bottleneck
|
||
:::
|
||
|
||
::: {.callout-tip}
|
||
## Background: The Three Stages of a Training Step
|
||
|
||
Every training step has three sequential stages. The slowest one determines your actual
|
||
throughput — not the GPU alone:
|
||
|
||
1. **Storage I/O** (Wall 8) — Read raw data from disk into CPU memory
|
||
2. **CPU Preprocessing** (Wall 9) — Decode, resize, augment, normalize
|
||
3. **Accelerator Compute** (Wall 1) — Forward pass, backward pass, weight update
|
||
|
||
The GPU cannot start until stages 1 and 2 finish. If either is slower than the GPU, the
|
||
accelerator utilization drops below 100%. This is the data pipeline bottleneck.
|
||
:::
|
||
|
||
---
|
||
|
||
## 1. Setup
|
||
|
||
```{python}
|
||
#| echo: false
|
||
#| output: false
|
||
import mlsysim # installed via `pip install mlsysim` (see workflow)
|
||
Engine = mlsysim.Engine
|
||
```
|
||
|
||
```python
|
||
import mlsysim
|
||
from mlsysim.solvers import SingleNodeModel, DataModel
|
||
from mlsysim.solvers import TransformationModel
|
||
```
|
||
|
||
---
|
||
|
||
## 2. GPU Compute Time: The Ceiling You Think You Have
|
||
|
||
We switch from LLM serving (Tutorials 2–3) to **CNN training** because the data pipeline
|
||
bottleneck is most visible here. LLM training on tokenized text has a tiny data footprint
|
||
(~8 MB/s as we will see in [Tutorial 12](12_full_stack_audit.qmd)). Image training with
|
||
JPEG decoding, resizing, and augmentation can demand 10–100× more CPU work per sample —
|
||
this is where the GPU actually starves.
|
||
|
||
First, establish how fast the A100 processes a ResNet-50 training step in isolation — no
|
||
data loading, no preprocessing, just pure compute:
|
||
|
||
```{python}
|
||
from mlsysim.solvers import SingleNodeModel
|
||
from mlsysim.core.units import Q_
|
||
from mlsysim.show import table, info
|
||
|
||
model = mlsysim.Models.Vision.ResNet50
|
||
hardware = mlsysim.Hardware.Cloud.A100
|
||
solver = SingleNodeModel()
|
||
|
||
# Baseline: ResNet-50 on A100, batch 256, FP16
|
||
profile = solver.solve(model=model, hardware=hardware, batch_size=256, precision="fp16")
|
||
|
||
info("GPU Compute Baseline",
|
||
Model=model.name,
|
||
Hardware=hardware.name,
|
||
Batch_size=256,
|
||
Step_latency=profile.latency.to('ms'),
|
||
Throughput=f"{profile.throughput:.0f} img/s",
|
||
Bottleneck=profile.bottleneck)
|
||
```
|
||
|
||
The GPU can process this batch in tens of milliseconds. That is the ceiling. Now let's
|
||
check whether the data pipeline can keep up.
|
||
|
||
---
|
||
|
||
## 3. Storage I/O Check: Can the Disk Deliver?
|
||
|
||
ImageNet images average ~500 KB each (JPEG compressed). At batch 256, the GPU demands a
|
||
burst of data every step. Can the storage subsystem supply it?
|
||
|
||
```{python}
|
||
from mlsysim.solvers import DataModel
|
||
|
||
sample_size = Q_("500 KB") # Average ImageNet JPEG
|
||
batch_size = 256
|
||
|
||
# Data demand = batch_size x sample_size / step_time
|
||
step_time_s = profile.latency.to("s").magnitude
|
||
data_per_step = (batch_size * sample_size.to("GB")).magnitude
|
||
demand_rate = Q_(data_per_step / step_time_s, "GB/s")
|
||
|
||
data_solver = DataModel()
|
||
data_result = data_solver.solve(workload_data_rate=demand_rate, hardware=hardware)
|
||
|
||
info("Storage I/O Check",
|
||
Data_demand=f"{demand_rate:.3f}",
|
||
Storage_supply=f"{data_result.supply_bw:.2f}",
|
||
Utilization=f"{data_result.utilization:.1%}",
|
||
Is_stalled=data_result.is_stalled)
|
||
```
|
||
|
||
Storage I/O is fine — modern NVMe SSDs can deliver multi-GB/s easily. The bottleneck is
|
||
not reading the bytes. It is *transforming* them.
|
||
|
||
---
|
||
|
||
## 4. The Reveal: CPU Preprocessing Is the Wall
|
||
|
||
Even with fast storage, the CPU must decode JPEGs, apply random crops, color jitter, and
|
||
normalization. A typical CPU worker processes ImageNet images at ~250 MB/s. With 8 workers,
|
||
total CPU throughput is ~2 GB/s:
|
||
|
||
```{python}
|
||
from mlsysim.solvers import TransformationModel
|
||
|
||
transform_solver = TransformationModel()
|
||
cpu_throughput = Q_("2 GB/s") # 8 workers x 250 MB/s each
|
||
|
||
t = transform_solver.solve(
|
||
batch_size=256,
|
||
sample_size_bytes=sample_size,
|
||
cpu_throughput=cpu_throughput,
|
||
accelerator_step_time=profile.latency
|
||
)
|
||
|
||
info("CPU vs GPU Pipeline",
|
||
CPU_transform_time=t.transform_time,
|
||
GPU_step_time=t.accelerator_step_time,
|
||
CPU_is_bottleneck=t.is_bottleneck,
|
||
GPU_utilization=f"{t.accelerator_utilization:.1%}",
|
||
Slowdown_factor=f"{t.slowdown_factor:.2f}x")
|
||
```
|
||
|
||
::: {.callout-important}
|
||
## Key Insight
|
||
|
||
**The binding constraint is not silicon — it is JPEG decoding on the CPU.** The data
|
||
pipeline (Wall 9: Transformation) becomes the bottleneck before the GPU (Wall 1: Compute).
|
||
Your GPU can process 5,300+ images per second, but your 8 CPU workers can only prepare
|
||
~850. The GPU sits idle waiting for data. This is why production training pipelines use
|
||
GPU-accelerated preprocessing (NVIDIA DALI), pre-decoded datasets, or aggressive
|
||
prefetching.
|
||
:::
|
||
|
||
---
|
||
|
||
## 5. Batch Size Sweep: Finding the Crossover
|
||
|
||
Let's sweep batch sizes to find exactly where the CPU becomes the binding constraint. At
|
||
small batches, the GPU is slower and data arrives in time. At large batches, the GPU
|
||
becomes more efficient but the CPU falls behind:
|
||
|
||
```{python}
|
||
rows = []
|
||
for bs in [32, 64, 128, 256, 512, 1024]:
|
||
p = solver.solve(model=model, hardware=hardware, batch_size=bs, precision="fp16")
|
||
|
||
t = transform_solver.solve(
|
||
batch_size=bs,
|
||
sample_size_bytes=sample_size,
|
||
cpu_throughput=cpu_throughput,
|
||
accelerator_step_time=p.latency
|
||
)
|
||
|
||
binding = "Transformation" if t.is_bottleneck else p.bottleneck
|
||
rows.append([
|
||
bs,
|
||
f"{p.latency.to('ms').magnitude:.2f} ms",
|
||
f"{t.transform_time.to('ms').magnitude:.2f} ms",
|
||
binding,
|
||
f"{t.accelerator_utilization:.1%}"
|
||
])
|
||
|
||
table(["Batch", "GPU Step", "CPU Xform", "Binding", "GPU Util"], rows)
|
||
```
|
||
|
||
Watch the crossover: at small batch sizes the GPU is the bottleneck (100% utilization).
|
||
As batch size grows, CPU preprocessing time grows linearly while GPU step time grows
|
||
sub-linearly. Eventually Wall 9 becomes the binding constraint and GPU utilization drops.
|
||
|
||
---
|
||
|
||
## 6. The Fix: Adding CPU Workers
|
||
|
||
The simplest fix for a CPU bottleneck is more workers. Let's compare 8 vs. 16 vs. 32:
|
||
|
||
```{python}
|
||
rows = []
|
||
for n_workers in [8, 16, 32]:
|
||
cpu_tp = Q_(f"{n_workers * 250} MB/s")
|
||
|
||
p = solver.solve(model=model, hardware=hardware, batch_size=512, precision="fp16")
|
||
|
||
t = transform_solver.solve(
|
||
batch_size=512,
|
||
sample_size_bytes=sample_size,
|
||
cpu_throughput=cpu_tp,
|
||
accelerator_step_time=p.latency
|
||
)
|
||
|
||
rows.append([n_workers, cpu_tp.to('GB/s'), f"{t.accelerator_utilization:.1%}"])
|
||
|
||
table(["Workers", "Throughput", "GPU Util @ bs=512"], rows)
|
||
```
|
||
|
||
Doubling workers doubles throughput — but you eventually hit either storage I/O limits
|
||
(Wall 8) or PCIe bandwidth. The takeaway: always check *all three stages* of the pipeline.
|
||
|
||
---
|
||
|
||
## Your Turn
|
||
|
||
::: {.callout-caution}
|
||
## Exercises
|
||
|
||
**Exercise 1: Predict before you compute.**
|
||
At batch size 64 with 8 CPU workers (2 GB/s total), will ResNet-50 training on the A100
|
||
be GPU-bound or CPU-bound? Write your prediction, then run the code. What determines the
|
||
answer? (Hint: compare `transform_time` vs. `accelerator_step_time`.)
|
||
|
||
**Exercise 2: Medical imaging — larger samples.**
|
||
Medical imaging uses images 10x larger than ImageNet (~5 MB per sample). Change
|
||
`sample_size` to `Q_("5 MB")` and re-run the batch size sweep. At what batch size does
|
||
the CPU stall the GPU now? How many workers would you need to keep up at batch 256?
|
||
|
||
**Exercise 3: GPU-accelerated preprocessing.**
|
||
If you use NVIDIA DALI to move preprocessing to the GPU, the CPU bottleneck effectively
|
||
disappears. Model this by setting `cpu_throughput = Q_("50 GB/s")`. Run the sweep again.
|
||
Does the bottleneck shift back to compute? What is the new GPU utilization at batch 512?
|
||
|
||
**Self-check:** If the GPU step takes 20 ms and CPU preprocessing takes 35 ms, what is the
|
||
accelerator utilization? (Answer: 20/35 = 57%.)
|
||
:::
|
||
|
||
---
|
||
|
||
## Key Takeaways
|
||
|
||
::: {.callout-tip}
|
||
## Summary
|
||
|
||
- **Data pipelines have three stages**: storage I/O, CPU preprocessing, and GPU compute — the slowest determines throughput
|
||
- **CPU preprocessing (Wall 9)** is the most common bottleneck: JPEG decode, augmentation, and normalization are all CPU-bound
|
||
- **Batch size shifts the binding constraint**: small batches are GPU-bound; large batches often become CPU-bound
|
||
- **Adding CPU workers** helps linearly but has diminishing returns when storage I/O becomes the limit
|
||
- **Always check all three stages** before concluding that the GPU is the bottleneck
|
||
:::
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
- **[Quantization: Not a Free Lunch](05_quantization.qmd)** — When reducing precision helps (and when it doesn't)
|
||
- **[KV-Cache: The Hidden Tax](03_kv_cache.qmd)** — Another hidden memory consumer: the KV-cache in LLM serving
|
||
- **[Where to Invest](09_sensitivity.qmd)** — Use sensitivity analysis to decide whether more CPU workers or a faster GPU is the better investment
|
||
- **[Silicon Zoo](../zoo/hardware.qmd)** — Compare storage and interconnect specs across GPU platforms
|