mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-18 08:54:25 -05:00
fix(math): wrap multi-letter math subscripts in \text{} (5 chapters)
Five chapters had multi-letter descriptive subscripts rendered as bare
italic letter sequences instead of upright text. Per book-prose.md $2,
multi-letter quantity-name subscripts (io, tx, op, etc.) must wrap in
\text{} so they render as words rather than the product of italic
variables.
Sites:
- vol1/data_engineering: T_{io} -> T_{\text{io}}
(asymmetric within same equation as T_{\text{compute}})
- vol1/ml_systems: E_{tx}, E_{op} -> E_{\text{tx}}, E_{\text{op}}
- vol1/optimizations/model_compression: E_{op} -> E_{\text{op}}
- vol2/data_storage: N_{GPUs\_per\_node} -> N_{\text{GPUs per node}}
- vol2/collective_communication: T_{first\_layer\_comm},
T_{backward\_per\_layer}, T_{AllReduce\_per\_layer} all wrapped
This commit is contained in:
@@ -2137,7 +2137,7 @@ Retry logic, dead letter queues, and circuit breakers are the runtime error hand
|
||||
|
||||
\index{Data Ingestion!pipeline boundary}\index{I/O Bottleneck!data loading}Continuing the compilation analogy, ingestion is the *lexer*: it reads raw source (data streams) and tokenizes them into well-formed records that the rest of the pipeline can process.
|
||||
|
||||
A critical and often overlooked constraint in ingestion design is the **Input/Output (IO) Bottleneck**. We invest heavily in expensive GPUs, but their utilization depends entirely on whether data arrives fast enough to keep them busy. Training speed is governed by a simple inequality: $T_{\text{step}} = \max(T_{\text{compute}}, T_{io})$.
|
||||
A critical and often overlooked constraint in ingestion design is the **Input/Output (IO) Bottleneck**. We invest heavily in expensive GPUs, but their utilization depends entirely on whether data arrives fast enough to keep them busy. Training speed is governed by a simple inequality: $T_{\text{step}} = \max(T_{\text{compute}}, T_{\text{io}})$.
|
||||
If the data pipeline cannot decode images fast enough to keep the GPU busy, the expensive accelerator sits idle. This phenomenon creates a "Choke Point" where adding more GPUs yields zero speedup—a counterintuitive result that frustrates teams who expect linear scaling from hardware investments. This bottleneck frequently occurs in computer vision, where decoding high-resolution JPEG images on the CPU consumes more time than the GPU requires to perform the forward and backward passes of a lightweight model like ResNet-18. Typically, training ResNet-50 on an A100 requires at least `{python} DataloaderStats.resnet_worker_count_str` CPU workers just to keep the GPU from starving.
|
||||
|
||||
Examine @fig-dataloader-choke-point to see this **Dataloader Choke Point**\index{Dataloader!dataloader choke point} in action. Notice the "Starvation Region" on the left where the CPU limits performance: no matter how powerful the GPU, training throughput is capped by data loading speed until enough workers are allocated to saturate the accelerator. Throughput levels are representative and vary by model and hardware.
|
||||
|
||||
@@ -604,15 +604,15 @@ class EnergyTransmission:
|
||||
\vspace*{-3pt}
|
||||
```
|
||||
* **Data ($D_{\text{vol}}$)**: `{python} EnergyTransmission.data_mb_str` MB (for example, one second of audio).
|
||||
* **Transmission Energy ($E_{tx}$)**: `{python} EnergyTransmission.tx_energy_str` mJ/MB (Wi-Fi/LTE).
|
||||
* **Compute Energy ($E_{op}$)**: `{python} EnergyTransmission.compute_energy_str` mJ/inference (MobileNet on NPU).
|
||||
* **Transmission Energy ($E_{\text{tx}}$)**: `{python} EnergyTransmission.tx_energy_str` mJ/MB (Wi-Fi/LTE).
|
||||
* **Compute Energy ($E_{\text{op}}$)**: `{python} EnergyTransmission.compute_energy_str` mJ/inference (MobileNet on NPU).
|
||||
|
||||
**The Calculation**:
|
||||
|
||||
```{=latex}
|
||||
\vspace*{-3pt}
|
||||
```
|
||||
1. **Cloud Approach**: $E_{\text{cloud}} \approx D_{\text{vol}} \times E_{tx}$ = `{python} EnergyTransmission.data_mb_str` MB$\times$ `{python} EnergyTransmission.tx_energy_str` mJ/MB = **`{python} EnergyTransmission.cloud_total_str` mJ**.
|
||||
1. **Cloud Approach**: $E_{\text{cloud}} \approx D_{\text{vol}} \times E_{\text{tx}}$ = `{python} EnergyTransmission.data_mb_str` MB$\times$ `{python} EnergyTransmission.tx_energy_str` mJ/MB = **`{python} EnergyTransmission.cloud_total_str` mJ**.
|
||||
2. **Local Approach**: $E_{\text{local}} \approx$ Inference = **`{python} EnergyTransmission.local_total_str` mJ**.
|
||||
|
||||
**The Systems Conclusion**: Transmitting raw data is **`{python} EnergyTransmission.ratio_str`$\times$ more expensive** than processing it locally. Even if the cloud had infinite speed ($\text{Time} \approx 0$), the **Energy Wall** makes cloud offloading physically impossible for always-on battery devices. The "Machine" constraint (Battery) dictates the "Algorithm" choice (TinyML).
|
||||
|
||||
@@ -3082,7 +3082,7 @@ class EnergyDividend:
|
||||
|
||||
The memory reduction of quantization is often the primary motivation, but the **Energy Dividend**\index{Energy Dividend!quantization efficiency} is the most significant systems consequence. While moving from FP32 to INT8 precision reduces the model's footprint by exactly 4$\times$, the energy required to perform the math drops much more precipitously.
|
||||
|
||||
In the tradition of quantitative architecture, we compare the energy per operation ($E_{op}$) across numerical formats. Consider an addition operation—the backbone of the accumulators in matrix multiplication:
|
||||
In the tradition of quantitative architecture, we compare the energy per operation ($E_{\text{op}}$) across numerical formats. Consider an addition operation—the backbone of the accumulators in matrix multiplication:
|
||||
|
||||
* **FP32 Addition**: Requires `{python} EnergyDividend.energy_add_fp32_str` pJ.
|
||||
* **INT8 Addition**: Requires `{python} EnergyDividend.energy_add_int8_str` pJ.
|
||||
|
||||
@@ -1978,9 +1978,9 @@ $T_{\text{sequential}} = T_{\text{backward}} + T_{\text{comm}}$ = `{python} Over
|
||||
|
||||
Each layer's AllReduce (`{python} OverlapBudgetCalc.allreduce_per_layer_ms_str` ms) runs in parallel with the next layer's backward pass (15 ms). Since `{python} OverlapBudgetCalc.allreduce_per_layer_ms_str` ms > 15 ms, there is `{python} OverlapBudgetCalc.exposed_per_layer_ms_str` ms of exposed communication per layer that cannot be hidden.
|
||||
|
||||
$T_{\text{pipelined}} = T_{\text{backward}} + T_{first\_layer\_comm} + (N_{\text{layers}} - 1) \times T_{\text{exposed}}$ = `{python} OverlapBudgetCalc.with_overlap_ms_str` ms.
|
||||
$T_{\text{pipelined}} = T_{\text{backward}} + T_{\text{first layer comm}} + (N_{\text{layers}} - 1) \times T_{\text{exposed}}$ = `{python} OverlapBudgetCalc.with_overlap_ms_str` ms.
|
||||
|
||||
**The Systems Insight**: Overlap reduces total step time by `{python} OverlapBudgetCalc.overlap_savings_pct_str` percent. The remaining exposed communication comes from the AllReduce being slower than the per-layer backward pass. To eliminate this residual, either increase the backward pass computation (larger batch size) or reduce AllReduce time (more aggressive compression, better topology). The overlap is most effective when $T_{backward\_per\_layer} > T_{AllReduce\_per\_layer}$.
|
||||
**The Systems Insight**: Overlap reduces total step time by `{python} OverlapBudgetCalc.overlap_savings_pct_str` percent. The remaining exposed communication comes from the AllReduce being slower than the per-layer backward pass. To eliminate this residual, either increase the backward pass computation (larger batch size) or reduce AllReduce time (more aggressive compression, better topology). The overlap is most effective when $T_{\text{backward per layer}} > T_{\text{AllReduce per layer}}$.
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -1269,7 +1269,7 @@ A single CPU core cannot keep a modern accelerator fed, even when the storage ha
|
||||
|
||||
At 1,000 images per second per GPU and 8 GPUs per node, a single core would need to decode and augment 8,000 images per second, a throughput 10$\times$ to 40$\times$ beyond what a single core can sustain. The solution is **multi-worker data loading**\index{Multi-Worker Data Loading}, where $W$ worker processes each read from storage, decode, augment, and enqueue batches independently. The effective I/O throughput scales approximately linearly with $W$ until one of three bottlenecks is reached: the storage device's bandwidth saturates, the PCIe bus between host and device saturates, or the workers exhaust host CPU cycles.
|
||||
|
||||
In PyTorch's `DataLoader`, each worker is a separate process with its own file descriptors and memory space. The `num_workers` parameter controls $W$. Setting $W$ too low leaves the GPU starved; setting $W$ too high wastes CPU resources on context switching and contention. A good heuristic is to start with $W = 4 \times N_{GPUs\_per\_node}$ and profile the data stall ratio, adjusting until stalls are below 2 percent. For a node with 8 GPUs, this yields 32 workers, which on a system with 64 CPU cores leaves 32 cores for the PyTorch runtime, NCCL communication threads, and operating system overhead. The division of CPU resources between data loading and training orchestration is itself a capacity planning exercise.
|
||||
In PyTorch's `DataLoader`, each worker is a separate process with its own file descriptors and memory space. The `num_workers` parameter controls $W$. Setting $W$ too low leaves the GPU starved; setting $W$ too high wastes CPU resources on context switching and contention. A good heuristic is to start with $W = 4 \times N_{\text{GPUs per node}}$ and profile the data stall ratio, adjusting until stalls are below 2 percent. For a node with 8 GPUs, this yields 32 workers, which on a system with 64 CPU cores leaves 32 cores for the PyTorch runtime, NCCL communication threads, and operating system overhead. The division of CPU resources between data loading and training orchestration is itself a capacity planning exercise.
|
||||
|
||||
The interaction between multi-worker loading and the storage hierarchy matters. When reading from local NVMe, workers can issue concurrent reads to the same RAID array without contention, because NVMe's deep command queues (up to 64K outstanding commands) handle parallelism in hardware. When reading from a parallel file system, workers distribute their reads across different OSS nodes, naturally aggregating bandwidth. When reading from object storage, workers issue concurrent GET requests, each to a different shard, achieving parallelism at the HTTP level.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user