mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-20 04:45:11 -05:00
fix(book): wave 6 — mechanical backlog from audits N/O/P/T/U/W2 (49 sites, 19 files)
Mechanical fix bundle resolving residual hits from six read-only audits:
N (cite-form), O (footnotes), P (CO2 subscript), T (\index{} placement),
U (hand-typed cross-refs), W2 (mermaid fig wrap). All sites verified clean
post-edit; deliberately skipped sites documented in review/DEFERRED.md.
Bundle A — CO~2~ subscript (8 occurrences, 7 source-line edits):
- vol1/hw_acceleration:5829 (kg CO2/kWh, metric tons of CO2 per year)
- vol1/introduction:1152 (552 tonnes of CO2-equivalent)
- vol1/training:5164, 5175, 6145 (f-string outputs, prose mention)
- vol1/responsible_engr:2512, 2518 (training comparison, tons of CO2)
- vol2/backmatter/glossary:174 (definition: "CO2 equivalent")
All converted to CO~2~ subscript form per book-prose §3.
Spot check after: 0 user-visible CO2 remaining (code comments + TikZ
coordinate names retained — non-rendering).
Bundle B — Cite-form + footnote bare attribution (9 sites):
- vol2/security_privacy: 5 sites (Szegedy 2013, Biggio 2012 ×2, Gu 2017
BadNets, Szegedy 2014 transferability)
- vol2/distributed_training:2480 (Mu Li et al. 2014 → @li2014)
- vol1/backmatter/appendix_algorithm:251 (Linnainmaa 1970 → @linnainmaa1970representation)
- vol1/hw_acceleration:3483 (Williams/Waterman/Patterson + Kung 1986 →
@williams2009; kung1986 not in bib, generalized attribution per "do not
fabricate" rule)
- vol1/training:360 (Linnainmaa/Werbos/Rumelhart-Hinton-Williams trio →
three bib keys)
- vol2/robust_ai:49 (period placement: cite after sentence-final period
→ cite inside terminal period)
No bib entries fabricated; restructured to existing keys where missing.
Bundle C — Hand-typed chapter ref (1 site):
- vol2/compute_infrastructure:3553 ("the compute chapter" /
"the network chapter" → @sec-compute-infrastructure / @sec-network-fabrics)
Bundle D — \index{} placement Violation 2 (13 sites):
Pattern \index{}**Header**: → **Header**\index{}:
- vol1/data_engineering: 2 (Failure, Why it matters)
- vol1/nn_architectures:3255 (Sparse Features; collapsed adjacent **:** to :)
- vol1/ml_systems: 7 (Problem/Scenario callouts)
- vol2/data_storage: 3 (Problem/Scenario callouts)
Verified: 0 remaining tag-before-bold violations.
Bundle E — Floating \index{} tags (10 fixed, 9 deliberately skipped):
- 10 mid-sentence repositions adjacent to indexed term (FlashAttention,
proxy model, distribution shift, DVFS, Memory Wall, etc.)
- 2 stray tags removed from non-rendering contexts (Python cell-header
banner comment, C code listing)
- 9 deliberately skipped: figure-attached patterns (per spec "SKIP THOSE
for this wave") — documented in review/DEFERRED.md.
Bundle F — Mermaid block fig wrap (1 site):
- 1 mermaid block wrapped in fig div per W2 audit.
Verification (pre-commit):
- ./book/binder check refs --vol1 --vol2 → 0 broken refs
- 0 \index{}**Header** tag-before-bold violations remaining
- 0 user-visible ASCII CO2 remaining (code/TikZ retained)
Wave 6 of release-prep cleanup. Audit tooling changes
(book/tools/audit/) intentionally NOT included — separate concern.
This commit is contained in:
Vijay Janapa Reddi
@@ -248,7 +248,7 @@ Modern frameworks use **reverse-mode automatic differentiation**, which computes
|
||||
|
||||
Backpropagation[^fn-backprop] implements the chain rule efficiently through two passes: forward to compute outputs, backward to compute gradients. @fig-backprop-graph illustrates this process for a simple two-layer network, with the forward pass (black arrows) computing outputs and the backward pass (red dashed arrows) propagating gradients.
|
||||
|
||||
[^fn-backprop]: **Backpropagation**: Short for "backward propagation of errors." The algorithm was independently discovered multiple times—by Linnainmaa (1970) for automatic differentiation and by @rumelhart1986 for neural network training. Its key insight is that computing gradients for *all* parameters requires only one backward pass through the graph, making training cost roughly 2--3$\times$ inference rather than $N\times$ (once per parameter).
|
||||
[^fn-backprop]: **Backpropagation**: Short for "backward propagation of errors." The algorithm was independently discovered multiple times—by @linnainmaa1970representation for automatic differentiation and by @rumelhart1986 for neural network training. Its key insight is that computing gradients for *all* parameters requires only one backward pass through the graph, making training cost roughly 2--3$\times$ inference rather than $N\times$ (once per parameter).
|
||||
|
||||
::: {#fig-backprop-graph fig-env="figure" fig-pos="htb" fig-cap="**Backpropagation Computational Graph**: A two-layer network showing the forward pass (black arrows) and backward pass (red dashed arrows). Each node caches values during the forward pass that are reused during the backward pass." fig-alt="A computational graph with four nodes labeled x, h, y, and L connected left to right. Solid black arrows show the forward pass with weights W1 and W2. Dashed red arrows curve backward showing gradient flow with partial derivative notation."}
|
||||
```{.tikz}
|
||||
|
||||
@@ -1319,7 +1319,7 @@ Task design directly impacts the benchmark's ability to evaluate AI systems. The
|
||||
|
||||
### Standardized datasets {#sec-benchmarking-standardized-datasets-123f}
|
||||
|
||||
A task definition is only as good as the data used to evaluate it. Standardized datasets ensure that all models undergo testing under identical conditions, enabling direct comparisons across different approaches—without them, every team would evaluate on private data, making cross-lab comparison impossible. \index{ImageNet!standardized dataset}
|
||||
A task definition is only as good as the data used to evaluate it. Standardized datasets\index{ImageNet!standardized dataset} ensure that all models undergo testing under identical conditions, enabling direct comparisons across different approaches—without them, every team would evaluate on private data, making cross-lab comparison impossible.
|
||||
\index{COCO!object detection dataset}
|
||||
\index{CIFAR-10!classification reference dataset}
|
||||
\index{SQuAD!reading comprehension dataset}
|
||||
@@ -3040,8 +3040,8 @@ System-level power measurement offers a more holistic view than measuring indivi
|
||||
|
||||
Shared infrastructure presents additional challenges. In data centers, resources such as cooling systems and power delivery are shared across workloads, complicating attribution of energy use to specific ML tasks. Cooling alone can account for 20–30 percent of total facility power consumption, making it a major factor in energy efficiency assessments [@barroso2019]. Even at the edge, components like memory and I/O interfaces may serve both ML and non-ML functions, further blurring measurement boundaries.
|
||||
|
||||
Shared infrastructure complexity is further compounded by dynamic power management techniques that modern systems employ to optimize energy efficiency. \index{DVFS!dynamic voltage and frequency scaling}
|
||||
Dynamic voltage and frequency scaling (DVFS) adjusts processor voltage and clock frequency based on workload demands, enabling 30–50 percent power reductions during periods of lower computational intensity. Advanced DVFS implementations using on-chip switching regulators can achieve comparable energy savings [@kim2008dvfs], causing power consumption to vary by 30–50 percent for the same ML model depending on system load and concurrent activity. This variability affects not only the compute components but also the supporting infrastructure, as reduced processor activity can lower cooling requirements and overall facility power draw.
|
||||
Shared infrastructure complexity is further compounded by dynamic power management techniques that modern systems employ to optimize energy efficiency.
|
||||
Dynamic voltage and frequency scaling (DVFS)\index{DVFS!dynamic voltage and frequency scaling} adjusts processor voltage and clock frequency based on workload demands, enabling 30–50 percent power reductions during periods of lower computational intensity. Advanced DVFS implementations using on-chip switching regulators can achieve comparable energy savings [@kim2008dvfs], causing power consumption to vary by 30–50 percent for the same ML model depending on system load and concurrent activity. This variability affects not only the compute components but also the supporting infrastructure, as reduced processor activity can lower cooling requirements and overall facility power draw.
|
||||
|
||||
\index{Power Usage Effectiveness!data center cooling metric}
|
||||
Support infrastructure, particularly cooling systems, is a major component of total energy consumption in large-scale deployments. Data centers must maintain operational temperatures, typically between 20–25°C, to ensure system reliability. Cooling overhead is captured in the Power Usage Effectiveness (PUE) metric, which ranges from 1.1 in highly efficient facilities to over 2.0 in less optimized ones [@barroso2019]. The interaction between compute workloads and cooling infrastructure creates complex dependencies; for example, power management techniques like DVFS not only reduce direct processor power consumption but also decrease heat generation, creating cascading effects on cooling requirements. Even edge devices require basic thermal management.
|
||||
|
||||
@@ -470,7 +470,7 @@ class DataGravity:
|
||||
If training takes < `{python} DataGravity.transfer_hours_str` hours, data transfer takes longer than training.
|
||||
If training costs < USD `{python} DataGravity.transfer_cost_str` (approximately `{python} DataGravity.tpu_hours_str` TPUv4-hours), bandwidth costs more than compute.
|
||||
|
||||
**Rule of Thumb**: For petabyte-scale data, **Code moves to Data**.\index{Data Locality!code-to-data} For gigabyte-scale data, **Data moves to Code**.\index{Data Locality!data-to-code}
|
||||
**Rule of Thumb**: For petabyte-scale data, **Code moves to Data**\index{Data Locality!code-to-data}. For gigabyte-scale data, **Data moves to Code**\index{Data Locality!data-to-code}.
|
||||
|
||||
:::
|
||||
|
||||
@@ -585,7 +585,7 @@ Data cascades occur when teams skip establishing clear quality criteria, reliabi
|
||||
|
||||
::: {.callout-example title="The pipeline jungle"}
|
||||
|
||||
\index{Pipeline!jungle}\index{Data Contract!schema validation}**Failure**: A credit scoring model suddenly started rejecting all applicants from a specific region.
|
||||
**Failure**\index{Pipeline!jungle}\index{Data Contract!schema validation}: A credit scoring model suddenly started rejecting all applicants from a specific region.
|
||||
|
||||
**Root cause**: An upstream team changed the schema of the `zip_code` field from `integer` to `string` to handle international codes.
|
||||
|
||||
@@ -878,7 +878,7 @@ Even reliable, high-quality pipelines face a third constraint: they must keep pa
|
||||
|
||||
::: {.callout-lighthouse title="DLRM (recommendation lighthouse)"}
|
||||
|
||||
\index{DLRM!scalability challenge}**Why it matters**: Recommendation systems like Deep Learning Recommendation Model (DLRM) exemplify the scalability challenge of modern data engineering. They rely on high-cardinality categorical features (like User IDs or Product IDs) that must be mapped to dense vectors via embedding tables. @tbl-data-engineering-dlrm-scalability summarizes how this stresses memory capacity and sparse-access bandwidth rather than compute:
|
||||
**Why it matters**\index{DLRM!scalability challenge}: Recommendation systems like Deep Learning Recommendation Model (DLRM) exemplify the scalability challenge of modern data engineering. They rely on high-cardinality categorical features (like User IDs or Product IDs) that must be mapped to dense vectors via embedding tables. @tbl-data-engineering-dlrm-scalability summarizes how this stresses memory capacity and sparse-access bandwidth rather than compute:
|
||||
|
||||
| **Property** | **Value** | **System Implication** |
|
||||
|:---------------|:---------------------|:----------------------------------------------------------|
|
||||
@@ -3377,7 +3377,7 @@ Columnar storage formats\index{Parquet!columnar format}\index{ORC!columnar forma
|
||||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||||
# │ COMPRESSION TRADEOFF CALCULATION
|
||||
# ├─────────────────────────────────────────────────────────────────────────────
|
||||
# │ Context: Compression trade-off discussion following format efficiency\index{Format Efficiency}
|
||||
# │ Context: Compression trade-off discussion following format efficiency
|
||||
# │
|
||||
# │ Goal: Contrast the impacts of decompression throughput on training time.
|
||||
# │ Show: That Snappy saves hours of wall-clock time over Gzip across 50 epochs.
|
||||
@@ -3447,7 +3447,7 @@ Deployment and serving requirements prioritize low-latency random access.\index{
|
||||
|
||||
Data versioning\index{Data Versioning!model reproducibility} exists to prevent exactly this scenario. It connects model versions to exact training data, enabling debugging and reproducibility. Without it, teams cannot identify the exact data that trained model v47.
|
||||
|
||||
@lst-dvc-workflow shows how DVC\index{DVC!git semantics} provides Git-like semantics for data versioning, while @lst-delta-time-travel demonstrates querying historical data states directly in SQL.\index{Delta Lake!time travel}
|
||||
@lst-dvc-workflow shows how DVC\index{DVC!git semantics} provides Git-like semantics for data versioning, while @lst-delta-time-travel\index{Delta Lake!time travel} demonstrates querying historical data states directly in SQL.
|
||||
|
||||
::: {#lst-dvc-workflow lst-cap="**DVC Workflow**: Git-like semantics for data versioning. DVC tracks large data files alongside code commits, enabling exact reproduction of any historical training dataset through paired `git checkout` and `dvc checkout` commands."}
|
||||
|
||||
|
||||
@@ -1186,7 +1186,7 @@ Herding takes a different approach, iteratively selecting samples whose features
|
||||
\index{GraNd!gradient-based coreset scoring}
|
||||
\index{EL2N!definition}
|
||||
\index{Forgetting Events!coreset selection}
|
||||
Gradient-based methods offer higher selection quality by using training dynamics to identify important samples, though they require training a proxy model first. GraNd (Gradient Normed) and EL2N (Error L2-Norm)[^fn-el2n-grand-scoring] score samples by gradient magnitude or prediction error early in training; high-scoring samples lie near the decision boundary and are most informative for learning. \index{Proxy Model!coreset score transfer}
|
||||
Gradient-based methods offer higher selection quality by using training dynamics to identify important samples, though they require training a proxy model\index{Proxy Model!coreset score transfer} first. GraNd (Gradient Normed) and EL2N (Error L2-Norm)[^fn-el2n-grand-scoring] score samples by gradient magnitude or prediction error early in training; high-scoring samples lie near the decision boundary and are most informative for learning.
|
||||
Crucially, these scores transfer across architectures: scores computed on a smaller model like ResNet-18 predict importance for larger models like ResNet-50, enabling inexpensive proxy-based selection. Forgetting Events[^fn-forgetting-events-pruning] tracks how often a sample is "forgotten" (correctly classified, then later misclassified) during training, identifying harder and more valuable examples.
|
||||
|
||||
[^fn-el2n-grand-scoring]: **EL2N (Error L2-Norm) and GraNd (Gradient Normed)**: These methods score samples based on their error or gradient norm early in training, identifying samples the model finds most difficult. The practicality of this approach relies on *transferability*, where scores from a small proxy model can guide data selection for a much larger target model. For instance, a proxy trained for just five epochs can generate scores to curate a dataset for a full 90-epoch production training run. \index{EL2N!proxy transferability}
|
||||
|
||||
@@ -297,7 +297,7 @@ class MemoryWallSpecs:
|
||||
```
|
||||
|
||||
\index{Memory Wall!execution strategy impact}
|
||||
To understand why execution strategy matters so much, consider the memory wall developed in @sec-hardware-acceleration-understanding-ai-memory-wall-3ea9, the growing gap between processor computational speed and memory bandwidth. Modern GPUs can perform arithmetic far faster than they can fetch data. On an A100 GPU with `{python} A100BLAS.dense_tflops_str` TFLOPS of compute and `{python} MemoryWallSpecs.a100_bw_tbs_str` TB/s of memory bandwidth, element-wise operations like ReLU achieve less than 1 percent of peak compute capacity, not because the hardware is slow, but because they spend nearly all their time waiting for data. The Roofline Model (@sec-machine-foundations-roofline-model-2529) formalizes this trade-off, showing exactly when operations are memory bound vs. compute bound.\index{Memory Wall!definition}
|
||||
To understand why execution strategy matters so much, consider the memory wall\index{Memory Wall!definition} developed in @sec-hardware-acceleration-understanding-ai-memory-wall-3ea9, the growing gap between processor computational speed and memory bandwidth. Modern GPUs can perform arithmetic far faster than they can fetch data. On an A100 GPU with `{python} A100BLAS.dense_tflops_str` TFLOPS of compute and `{python} MemoryWallSpecs.a100_bw_tbs_str` TB/s of memory bandwidth, element-wise operations like ReLU achieve less than 1 percent of peak compute capacity, not because the hardware is slow, but because they spend nearly all their time waiting for data. The Roofline Model (@sec-machine-foundations-roofline-model-2529) formalizes this trade-off, showing exactly when operations are memory bound vs. compute bound.
|
||||
|
||||
The memory wall creates a critical classification: operations are either **compute-bound**\index{Compute-bound operations} (limited by arithmetic throughput, like large matrix multiplications) or **memory-bound**\index{Memory-bound operations} (limited by data movement, like activation functions and normalization). Most individual neural network operation types (activations, normalizations, element-wise operations) are memory bound, though the large matrix multiplications that dominate total compute time can be compute bound.
|
||||
|
||||
@@ -3149,7 +3149,7 @@ minimum width=18,minimum height=18](3R){0};
|
||||
|
||||
This rank hierarchy maps directly onto ML data. A color image is a rank-3 tensor: height by width by three channels (red, green, blue). @fig-tensor-data-structure-b breaks this apart, stacking the three color channels illustrating how a single photograph becomes a three-layer numerical grid. Stacking a batch of $N$ images adds a fourth dimension. In PyTorch's channel-first convention this produces a rank-4 tensor of shape $[N, 3, H, W]$; in channel-last layouts the same batch is $[N, H, W, 3]$. Every convolutional layer in a vision model consumes and produces rank-4 image tensors, which is why the tensor abstraction is so central to framework design.
|
||||
|
||||
::: {#fig-tensor-data-structure-b fig-env="figure" fig-pos="htb" fig-cap="**Image as RGB Tensor**: Three stacked grids representing the red, green, and blue color channels of an image, with dimension labels showing width, height, and channel depth forming a rank-3 tensor. *Credit: Niklas Lang.*." fig-alt="Three stacked 5x5 grids in red, green, and blue representing RGB channels. Dimension labels show width five pixels, height five pixels, and three color channels, forming a rank-3 tensor for image data."}
|
||||
::: {#fig-tensor-data-structure-b fig-env="figure" fig-pos="htb" fig-cap="**Image as RGB Tensor**: Three stacked grids representing the red, green, and blue color channels of an image, with dimension labels showing width, height, and channel depth forming a rank-3 tensor. *Credit: Niklas Lang.*" fig-alt="Three stacked 5x5 grids in red, green, and blue representing RGB channels. Dimension labels show width five pixels, height five pixels, and three color channels, forming a rank-3 tensor for image data."}
|
||||
|
||||
```{.tikz}
|
||||
\scalebox{0.7}{%
|
||||
@@ -4142,7 +4142,7 @@ TensorFlow's architecture reflects a comprehensive solution to the Abstraction P
|
||||
|
||||
The graph-first approach prioritizes the **Deployment Spectrum**: because the framework sees the entire graph, it can perform aggressive optimizations like constant folding, operator fusion, and memory layout optimization before the first byte of data is processed. TensorFlow's dominance in complex production ecosystems traces directly to this ahead-of-time optimization capability. @fig-tensorflow-architecture maps the full training-to-deployment pipeline---trace how a model flows from data preprocessing through distributed training on the left, then fans out to serving, mobile (TF Lite), browser (TF.js), and language bindings on the right.
|
||||
|
||||
::: {#fig-tensorflow-architecture fig-env="figure" fig-pos="htb" fig-cap="**TensorFlow Training-to-Deployment Pipeline**: Two-column diagram showing the training path (left) from data preprocessing through tf.keras and distribution strategy across CPU, GPU, and TPU, and the deployment path (right) from SavedModel export to TensorFlow Serving, Lite, JS, and language bindings. Source: TensorFlow.." fig-alt="Two-column diagram. Training: data preprocessing, tf.keras, TensorFlow Hub, Premade Estimators, Distribution Strategy across CPU/GPU/TPU. Deployment via SavedModel to TensorFlow Serving, Lite, JS, and language bindings."}
|
||||
::: {#fig-tensorflow-architecture fig-env="figure" fig-pos="htb" fig-cap="**TensorFlow Training-to-Deployment Pipeline**: Two-column diagram showing the training path (left) from data preprocessing through tf.keras and distribution strategy across CPU, GPU, and TPU, and the deployment path (right) from SavedModel export to TensorFlow Serving, Lite, JS, and language bindings. Source: TensorFlow." fig-alt="Two-column diagram. Training: data preprocessing, tf.keras, TensorFlow Hub, Premade Estimators, Distribution Strategy across CPU/GPU/TPU. Deployment via SavedModel to TensorFlow Serving, Lite, JS, and language bindings."}
|
||||
|
||||
```{.tikz}
|
||||
\begin{tikzpicture}[font=\usefont{T1}{phv}{m}{n}\small]
|
||||
|
||||
@@ -1429,7 +1429,7 @@ To address these limitations, SIMT\index{SIMT!Single Instruction Multiple Thread
|
||||
|
||||
```{.c}
|
||||
__global__ void matrix_multiply(float* C, float* A, float*
|
||||
B, int N) { // CUDA kernel\index{CUDA!kernel launch}
|
||||
B, int N) { // CUDA kernel
|
||||
// Each thread processes one output element
|
||||
int row = blockIdx.y * blockDim.y + threadIdx.y;
|
||||
int col = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
@@ -3480,7 +3480,7 @@ The Roofline Model answers this question by plotting arithmetic intensity agains
|
||||
|
||||
The roofline model[^fn-roofline-diagnostic] [@williams2009] provides the standard framework for understanding whether workloads are compute bound\index{Compute-Bound!definition} or memory bound\index{Memory-Bound!definition}, directly connecting the memory wall discussion to practical performance analysis. This model enables quantitative reasoning about accelerator utilization and guides optimization decisions.
|
||||
|
||||
[^fn-roofline-diagnostic]: **Roofline Model**: Introduced by Williams, Waterman, and Patterson at UC Berkeley in 2009, building on earlier I/O complexity work by Kung (1986). Their specific contribution was making the compute vs. bandwidth trade-off *visual* and *actionable*: the characteristic roofline plot immediately reveals whether a kernel is compute bound (hitting the flat ceiling) or memory bound (hitting the sloped bandwidth line) and quantifies the gap to hardware limits. A kernel operating at only 50 percent of its ceiling has a clear 2$\times$ utilization gap to close, making this the standard diagnostic tool for accelerator optimization. \index{Roofline Model!optimization diagnostic}
|
||||
[^fn-roofline-diagnostic]: **Roofline Model**: Introduced by @williams2009 at UC Berkeley, building on earlier I/O complexity work from the 1980s. Their specific contribution was making the compute vs. bandwidth trade-off *visual* and *actionable*: the characteristic roofline plot immediately reveals whether a kernel is compute bound (hitting the flat ceiling) or memory bound (hitting the sloped bandwidth line) and quantifies the gap to hardware limits. A kernel operating at only 50 percent of its ceiling has a clear 2$\times$ utilization gap to close, making this the standard diagnostic tool for accelerator optimization. \index{Roofline Model!optimization diagnostic}
|
||||
|
||||
Performance is bounded by two ceilings, as @eq-roofline formalizes. Here, attainable performance and peak compute are in FLOPs/second (often reported as TFLOPS), peak bandwidth is in bytes/second (often TB/s), and arithmetic intensity is in FLOP/byte:
|
||||
$$\text{Attainable Performance} = \min(\text{Peak Compute}, \text{Peak Bandwidth} \times \text{Arithmetic Intensity})$$ {#eq-roofline}
|
||||
@@ -5006,7 +5006,7 @@ This spatial regularity also enables aggressive fusion and tiling. Because convo
|
||||
\index{Transformer!KV-cache memory pressure}
|
||||
Where CNNs are defined by weight reuse, transformers are defined by the memory pressure of the key-value (KV) cache\index{KV Cache!transformer memory pressure}. During attention computation, every query vector must access stored key and value pairs across the entire sequence length. As sequences grow, the KV cache can consume gigabytes of high-bandwidth memory, making memory bandwidth rather than raw compute the primary bottleneck. This access pattern motivates activation stationary execution: keeping the KV cache resident in fast memory while streaming queries through minimizes the costly round trips to external DRAM that would otherwise dominate execution time.
|
||||
|
||||
The memory-bound nature of attention also explains why fused attention kernels, such as FlashAttention [@dao2022], deliver outsized performance gains for transformers. \index{FlashAttention!tiled attention implementation}
|
||||
The memory-bound nature of attention also explains why fused attention kernels, such as FlashAttention\index{FlashAttention!tiled attention implementation} [@dao2022], deliver outsized performance gains for transformers.
|
||||
By fusing the query-key dot product, softmax normalization, and value-weighted summation into a single kernel that tiles along the sequence dimension, these implementations avoid materializing the full attention matrix in main memory. This temporal tiling approach processes sequence blocks that fit within on-chip SRAM, substantially reducing HBM traffic while preserving the $O(N^2)$ attention computation. For transformers, the mapping strategy is primarily an exercise in memory management rather than compute scheduling.
|
||||
|
||||
#### Multilayer perceptrons (DLRM) {#sec-hardware-acceleration-multilayer-perceptrons-dlrm-7c0d}
|
||||
@@ -5826,7 +5826,7 @@ class CarbonRoiCalc:
|
||||
1. **Workload**: `{python} CarbonRoiCalc.workload_str` Billion inferences per day.
|
||||
2. **CPU fleet energy**: `{python} CarbonRoiCalc.cpu_servers_str` CPU servers $\times$ `{python} CarbonRoiCalc.cpu_power_str` W $\times$ 24 h $\approx$ `{python} CarbonRoiCalc.cpu_energy_day_str` kWh/day.
|
||||
3. **NPU fleet energy**: `{python} CarbonRoiCalc.npu_chips_str` NPU chips $\times$ `{python} CarbonRoiCalc.npu_power_str` W $\times$ 24 h $\approx$ `{python} CarbonRoiCalc.npu_energy_day_str` kWh/day.
|
||||
4. **Carbon savings**: At `{python} CarbonRoiCalc.carbon_intensity_str` kg CO2/kWh, switching to NPUs saves **~`{python} CarbonRoiCalc.co2_saved_str` metric tons of CO2 per year**.
|
||||
4. **Carbon savings**: At `{python} CarbonRoiCalc.carbon_intensity_str` kg CO~2~/kWh, switching to NPUs saves **~`{python} CarbonRoiCalc.co2_saved_str` metric tons of CO~2~ per year**.
|
||||
|
||||
**Systems conclusion**: Specialized accelerators can materially reduce the energy use of matched inference workloads. For workloads with stable arithmetic patterns and high deployment volume, the per-operation energy gains compound into substantial reductions in operational carbon footprint relative to general-purpose hardware.
|
||||
|
||||
|
||||
@@ -1149,7 +1149,7 @@ The lesson is "bitter" because our intuition misleads us\index{bitter lesson, Th
|
||||
|
||||
Modern language models like GPT-4 and image generation systems like DALL-E illustrate this principle directly. Their capabilities emerge not from linguistic or artistic theories encoded by humans but from training general-purpose neural networks on vast amounts of data using substantial computational resources. Estimates for models at GPT-3's scale suggest thousands of megawatt-hours of energy[^fn-gpt3-training-energy] [@patterson2021carbon], and serving these models to millions of users demands data centers consuming power comparable to small cities.
|
||||
|
||||
[^fn-gpt3-training-energy]: **GPT-3 Training Energy**: @patterson2021carbon estimated GPT-3's single training run consumed approximately 1,287 MWh and emitted 552 tonnes of CO2-equivalent, roughly the annual electricity of 120 average US households. The energy cost is dominated not by arithmetic but by data movement through the memory hierarchy, making the $D_{\text{vol}}/\text{BW}$ term of the iron law, not the compute term, the primary driver of the power bill at this scale. \index{GPT-3!training energy}
|
||||
[^fn-gpt3-training-energy]: **GPT-3 Training Energy**: @patterson2021carbon estimated GPT-3's single training run consumed approximately 1,287 MWh and emitted 552 tonnes of CO~2~-equivalent, roughly the annual electricity of 120 average US households. The energy cost is dominated not by arithmetic but by data movement through the memory hierarchy, making the $D_{\text{vol}}/\text{BW}$ term of the iron law, not the compute term, the primary driver of the power bill at this scale. \index{GPT-3!training energy}
|
||||
|
||||
The implication is that realizing the bitter lesson's promise requires expertise in data engineering, hardware optimization, and systems coordination[^fn-memory-bandwidth-bottleneck] that goes far beyond algorithmic innovation. We explore these hardware constraints quantitatively in @sec-hardware-acceleration, where students will have the prerequisite background to analyze memory bandwidth limitations and their implications for system design.
|
||||
|
||||
|
||||
@@ -596,7 +596,7 @@ class EnergyTransmission:
|
||||
|
||||
::: {.callout-notebook title="The energy of transmission"}
|
||||
|
||||
\index{Energy of Transmission!local vs. cloud} \index{Energy Wall!battery constraints}**Problem**: Should a battery-powered sensor process data locally (TinyML) or send it to the cloud?
|
||||
**Problem**\index{Energy of Transmission!local vs. cloud}\index{Energy Wall!battery constraints}: Should a battery-powered sensor process data locally (TinyML) or send it to the cloud?
|
||||
|
||||
**Variables**:
|
||||
|
||||
@@ -1576,7 +1576,7 @@ class DistancePenalty:
|
||||
|
||||
::: {.callout-notebook title="The distance penalty"}
|
||||
|
||||
\index{Distance Penalty!cloud latency} \index{Light Barrier!safety-critical systems}**Problem**: Consider a real-time safety monitor for a robotic arm. The safety logic requires a **`{python} DistancePenalty.safety_budget_str` ms** end-to-end response time to prevent injury. The model runs in a high-performance cloud data center `{python} DistancePenalty.distance_km_str` km away. Can the safety budget be met?
|
||||
**Problem**\index{Distance Penalty!cloud latency}\index{Light Barrier!safety-critical systems}: Consider a real-time safety monitor for a robotic arm. The safety logic requires a **`{python} DistancePenalty.safety_budget_str` ms** end-to-end response time to prevent injury. The model runs in a high-performance cloud data center `{python} DistancePenalty.distance_km_str` km away. Can the safety budget be met?
|
||||
|
||||
**Physics**:
|
||||
|
||||
@@ -1927,7 +1927,7 @@ class VoiceAssistantWall:
|
||||
|
||||
::: {.callout-notebook title="The voice assistant wall"}
|
||||
|
||||
\index{Scaling!infrastructure, voice assistants}\index{Cloud ML!scaling limits}**Scenario**: `{python} VoiceAssistantWall.ww_devices_b_str` billion voice assistant devices (smartphones, smart speakers, earbuds). Can cloud data centers handle this?
|
||||
**Scenario**\index{Scaling!infrastructure, voice assistants}\index{Cloud ML!scaling limits}: `{python} VoiceAssistantWall.ww_devices_b_str` billion voice assistant devices (smartphones, smart speakers, earbuds). Can cloud data centers handle this?
|
||||
|
||||
**Part 1: the economic wall**
|
||||
|
||||
@@ -2175,7 +2175,7 @@ class DataLocalityInvariant:
|
||||
|
||||
::: {.callout-notebook title="The bandwidth bottleneck"}
|
||||
|
||||
\index{Bandwidth!bottleneck, video streaming} \index{Edge ML!bandwidth reduction}**Problem**: Consider a quality control system for a factory floor with **`{python} BandwidthBottleneck.num_cameras_str` cameras** running at **`{python} BandwidthBottleneck.bb_fps_str` FPS** with **1080p resolution**. Should the system stream to the cloud or process at the edge?
|
||||
**Problem**\index{Bandwidth!bottleneck, video streaming}\index{Edge ML!bandwidth reduction}: Consider a quality control system for a factory floor with **`{python} BandwidthBottleneck.num_cameras_str` cameras** running at **`{python} BandwidthBottleneck.bb_fps_str` FPS** with **1080p resolution**. Should the system stream to the cloud or process at the edge?
|
||||
|
||||
**Physics**:
|
||||
|
||||
@@ -2304,7 +2304,7 @@ $$\text{Data Locality} \iff \frac{D_{\text{vol}}}{\text{BW}_{\text{net}}} > L_{\
|
||||
|
||||
::: {.callout-notebook title="Napkin math: The locality crossover"}
|
||||
|
||||
\index{Locality Crossover!worked example}**Problem**: Should a drone's object avoidance system (4K, 60 FPS) offload to the cloud?
|
||||
**Problem**\index{Locality Crossover!worked example}: Should a drone's object avoidance system (4K, 60 FPS) offload to the cloud?
|
||||
|
||||
**Variables**:
|
||||
|
||||
@@ -2681,7 +2681,7 @@ class BatteryTax:
|
||||
|
||||
::: {.callout-notebook title="The battery tax"}
|
||||
|
||||
\index{Battery Tax!always-on constraints}\index{Battery Life!ML impact} \index{Mobile ML!energy budget}**Problem**: Consider deploying a "real-time" background object detector on a smartphone. The model consumes **`{python} BatteryTax.pwr_w_str` Watts** of continuous power when active. The phone has a standard **`{python} BatteryTax.batt_wh_str` Watt-hour (Wh)** battery. Can the feature stay on all day?
|
||||
**Problem**\index{Battery Tax!always-on constraints}\index{Battery Life!ML impact}\index{Mobile ML!energy budget}: Consider deploying a "real-time" background object detector on a smartphone. The model consumes **`{python} BatteryTax.pwr_w_str` Watts** of continuous power when active. The phone has a standard **`{python} BatteryTax.batt_wh_str` Watt-hour (Wh)** battery. Can the feature stay on all day?
|
||||
|
||||
**Physics**:
|
||||
|
||||
@@ -2731,7 +2731,7 @@ class ThermalQuantCalc:
|
||||
|
||||
::: {.callout-notebook title="The thermal wall"}
|
||||
|
||||
\index{Thermal Wall!mobile constraints} \index{Power Wall!mobile implications}**Problem**: An unoptimized LLM requires **`{python} ThermalQuantCalc.baseline_str` W** peak compute. Can it be deployed on a mobile device?
|
||||
**Problem**\index{Thermal Wall!mobile constraints}\index{Power Wall!mobile implications}: An unoptimized LLM requires **`{python} ThermalQuantCalc.baseline_str` W** peak compute. Can it be deployed on a mobile device?
|
||||
|
||||
**Physics**:
|
||||
|
||||
|
||||
@@ -2745,8 +2745,8 @@ class AttentionMemory:
|
||||
**Systems conclusion**: A single layer's attention matrix consumes **`{python} AttentionMemory.single_layer_gb_str` GB** of HBM. A `{python} AttentionMemory.num_layers_str`-layer model would require **`{python} AttentionMemory.total_gb_str` GB** just for transient attention scores, far exceeding any single GPU's capacity.
|
||||
This AI memory wall, introduced in @sec-hardware-acceleration-understanding-ai-memory-wall-3ea9, forces the use of:
|
||||
|
||||
* **FlashAttention** (tiling to avoid materializing the full matrix).\index{FlashAttention!tiling strategy}
|
||||
* **Sparse Attention** (computing only a subset of scores).\index{Attention!sparse}
|
||||
* **FlashAttention**\index{FlashAttention!tiling strategy} (tiling to avoid materializing the full matrix).
|
||||
* **Sparse Attention**\index{Attention!sparse} (computing only a subset of scores).
|
||||
|
||||
:::
|
||||
|
||||
@@ -3252,7 +3252,7 @@ class DLRMEmbedding:
|
||||
# Note: Use DLRMEmbedding.embed_table_gb_str directly.
|
||||
```
|
||||
|
||||
2. **Sparse Features (Embedding Tables)**\index{Embedding!table}**:** Categorical features (User ID, Item ID) are looked up in massive embedding tables. A table for one billion users with 128-dimensional vectors requires $10^9 \times 128 \times 4$ bytes ≈ `{python} DLRMEmbedding.embed_table_gb_str` GB of memory. This component is memory-intensive but compute-light (a single memory copy).
|
||||
2. **Sparse Features (Embedding Tables)**\index{Embedding!table}: Categorical features (User ID, Item ID) are looked up in massive embedding tables. A table for one billion users with 128-dimensional vectors requires $10^9 \times 128 \times 4$ bytes ≈ `{python} DLRMEmbedding.embed_table_gb_str` GB of memory. This component is memory-intensive but compute-light (a single memory copy).
|
||||
\index{DLRM!interaction layer}
|
||||
3. **Interaction layer**: The dense vectors from the MLP and the sparse vectors from embeddings are combined (typically via dot products) to capture interactions between user and item features.
|
||||
4. **Top MLP**: The combined features are processed by another MLP to produce a final probability (for example, click-through rate).
|
||||
|
||||
@@ -204,7 +204,7 @@ The YouTube recommendation feedback loop (examined as a technical debt pattern i
|
||||
|
||||
[^fn-goodhart-proxy]: **Goodhart's Law**: "When a measure becomes a target, it ceases to be a good measure" (Strathern's generalization of Goodhart's 1975 monetary policy observation). Recommendation feedback loops are the canonical ML manifestation: gradient descent optimizes watch-time proxies at a speed no human curator can match, and the system's own outputs reshape the training distribution---users who consume extreme content generate data that reinforces extremity, decoupling the proxy from user welfare orders of magnitude faster than manual editorial processes ever could. \index{Goodhart's Law!proxy decoupling}
|
||||
|
||||
The distribution shift defined earlier also manifests as population mismatch, where models trained on one population perform differently on another without obvious indicators.\index{Distribution Shift!population mismatch}
|
||||
The distribution shift\index{Distribution Shift!population mismatch} defined earlier also manifests as population mismatch, where models trained on one population perform differently on another without obvious indicators.
|
||||
|
||||
::: {.callout-war-story title="The proxy variable trap"}
|
||||
|
||||
@@ -1915,7 +1915,7 @@ Total cost of ownership reveals where responsible optimization has the most leve
|
||||
|
||||
#### Environmental impact {#sec-responsible-engineering-environmental-impact-0292}
|
||||
|
||||
\index{Environmental Impact!carbon accounting}The preceding TCO analysis captures costs that appear on invoices, but computational resources carry costs that no invoice reflects. Environmental impact follows from computational efficiency: the same optimization techniques that reduce TCO also reduce carbon emissions. The optimization techniques from @sec-hardware-acceleration and @sec-model-compression reduce energy consumption per inference, directly lowering carbon footprint. Data centers consume an estimated 1–2 percent of global electricity\index{Data Center!energy consumption}, a share that continues to grow as ML workloads expand [@henderson2020towards]. Engineers can reduce this impact by selecting cloud regions powered by renewable energy (5$\times$ carbon reduction), applying model efficiency techniques (2--4$\times$ reduction through quantization), and scheduling intensive workloads during periods of abundant renewable energy.\index{Carbon-Aware Scheduling!renewable energy}
|
||||
\index{Environmental Impact!carbon accounting}The preceding TCO analysis captures costs that appear on invoices, but computational resources carry costs that no invoice reflects. Environmental impact follows from computational efficiency: the same optimization techniques that reduce TCO also reduce carbon emissions. The optimization techniques from @sec-hardware-acceleration and @sec-model-compression reduce energy consumption per inference, directly lowering carbon footprint. Data centers consume an estimated 1–2 percent of global electricity\index{Data Center!energy consumption}, a share that continues to grow as ML workloads expand [@henderson2020towards]. Engineers can reduce this impact by selecting cloud regions powered by renewable energy (5$\times$ carbon reduction), applying model efficiency techniques (2--4$\times$ reduction through quantization), and scheduling intensive workloads during periods of abundant renewable energy\index{Carbon-Aware Scheduling!renewable energy}.
|
||||
|
||||
To appreciate the magnitude of these emissions, the following worked example quantifies *the carbon cost of scale* for training a large foundation model.
|
||||
|
||||
@@ -2509,13 +2509,13 @@ Teams treat responsibility as external oversight rather than engineering practic
|
||||
|
||||
**Pitfall**: *Measuring the environmental impact of training but not inference.*
|
||||
|
||||
Public discourse focuses on the carbon cost of training runs, and engineers naturally follow this framing when assessing environmental responsibility. The TCO analysis in @sec-responsible-engineering-total-cost-ownership-35c1 reveals why this focus is misplaced: inference-to-training compute ratios can exceed 40:1 over a model's operational lifetime. A model trained once but served millions of times daily has its environmental footprint dominated by inference, not training. For the recommendation system analyzed in @tbl-tco-summary, training accounts for just `{python} TCOCalc.p_train_str` percent of three-year costs while inference accounts for `{python} TCOCalc.p_inf_str` percent. The carbon ratio is even more lopsided in this example, with inference emitting about `{python} TCOSummary.carbon_train_ratio_str`:1 as much CO2 as training. Engineers who optimize training efficiency while ignoring per-query inference costs address the smaller term in a lopsided equation, leaving the dominant source of environmental impact unexamined.
|
||||
Public discourse focuses on the carbon cost of training runs, and engineers naturally follow this framing when assessing environmental responsibility. The TCO analysis in @sec-responsible-engineering-total-cost-ownership-35c1 reveals why this focus is misplaced: inference-to-training compute ratios can exceed 40:1 over a model's operational lifetime. A model trained once but served millions of times daily has its environmental footprint dominated by inference, not training. For the recommendation system analyzed in @tbl-tco-summary, training accounts for just `{python} TCOCalc.p_train_str` percent of three-year costs while inference accounts for `{python} TCOCalc.p_inf_str` percent. The carbon ratio is even more lopsided in this example, with inference emitting about `{python} TCOSummary.carbon_train_ratio_str`:1 as much CO~2~ as training. Engineers who optimize training efficiency while ignoring per-query inference costs address the smaller term in a lopsided equation, leaving the dominant source of environmental impact unexamined.
|
||||
|
||||
## Summary {#sec-responsible-engineering-summary-45cf}
|
||||
|
||||
Responsible engineering is ML systems engineering done completely, not a separate discipline. The chapter traced a path from failure diagnosis through prevention to enforcement, beginning with the responsibility gap (the distance between technical performance and responsible outcomes) and demonstrating how proxy variables, feedback loops, and distribution shift cause systems to harm users while meeting every conventional metric. The engineering response includes checklists that systematize predeployment assessment, fairness metrics that make disparities measurable, explainability mechanisms that satisfy regulatory and stakeholder requirements, and monitoring infrastructure that detects silent failures before they accumulate harm.
|
||||
|
||||
The key insight unifying these tools is that translating responsibility concerns into measurable properties makes them tractable.\index{Measurable Properties!responsibility requirements} "Fairness gap <5 percent across groups" is actionable; "be fair" is not. This translation extends beyond fairness: efficiency becomes carbon accounting and TCO analysis, where a `{python} TCOSummary.quant_reduction_pct_str` percent latency reduction through quantization saves USD `{python} TCOSummary.quant_savings_str`K and eliminates `{python} TCOSummary.quant_carbon_str` tons of CO2. Documentation becomes model cards with explicit intended use and known limitations. Governance becomes access control, lineage tracking, and audit infrastructure that makes compliance demonstrable rather than aspirational. At every level, the same pattern holds: abstract ethical obligations become concrete engineering requirements that can be specified, tested, monitored, and enforced.
|
||||
The key insight unifying these tools is that translating responsibility concerns into measurable properties makes them tractable.\index{Measurable Properties!responsibility requirements} "Fairness gap <5 percent across groups" is actionable; "be fair" is not. This translation extends beyond fairness: efficiency becomes carbon accounting and TCO analysis, where a `{python} TCOSummary.quant_reduction_pct_str` percent latency reduction through quantization saves USD `{python} TCOSummary.quant_savings_str`K and eliminates `{python} TCOSummary.quant_carbon_str` tons of CO~2~. Documentation becomes model cards with explicit intended use and known limitations. Governance becomes access control, lineage tracking, and audit infrastructure that makes compliance demonstrable rather than aspirational. At every level, the same pattern holds: abstract ethical obligations become concrete engineering requirements that can be specified, tested, monitored, and enforced.
|
||||
|
||||
::: {.callout-takeaways title="Reliable for whom?"}
|
||||
|
||||
|
||||
@@ -357,7 +357,7 @@ Four dimensions structure this cost analysis. First, FLOP counts of matrix opera
|
||||
\index{Backpropagation!historical introduction}\index{BLAS!matrix computation foundation}
|
||||
Neural network training consists of repeated matrix operations and nonlinear transformations. These operations are conceptually simple but create the system-level challenges that dominate modern training infrastructure. The introduction of backpropagation[^fn-backprop-provenance] by @rumelhart1986 and the development of efficient matrix computation libraries such as Basic Linear Algebra Subprograms (BLAS)[^fn-blas-training] [@dongarra1988] laid the groundwork for modern training architectures.
|
||||
|
||||
[^fn-backprop-provenance]: **Backpropagation Provenance**: The algorithm was independently derived by Linnainmaa (1970) for automatic differentiation of computer programs and by Werbos (1974) in a Harvard PhD thesis on economic modeling—over a decade before Rumelhart, Hinton, and Williams popularized it for neural networks in 1986. This delay between derivation and broad adoption recurs in ML systems history: attention mechanisms predated transformers by decades, but the 2017 Transformer showed that attention-heavy models could train efficiently on contemporary GPU systems; later TPU and GPU clusters enabled much larger deployments. Every modern framework's `backward()` call implements Linnainmaa's reverse-mode AD, not textbook chain rule—the difference is that graph-reverse topological traversal enables parallel gradient computation across independent subgraphs. \index{Backpropagation!provenance}
|
||||
[^fn-backprop-provenance]: **Backpropagation Provenance**: The algorithm was independently derived by @linnainmaa1970representation for automatic differentiation of computer programs and by @werbos1974beyond in a Harvard PhD thesis on economic modeling—over a decade before @rumelhart1986 popularized it for neural networks. This delay between derivation and broad adoption recurs in ML systems history: attention mechanisms predated transformers by decades, but the 2017 Transformer showed that attention-heavy models could train efficiently on contemporary GPU systems; later TPU and GPU clusters enabled much larger deployments. Every modern framework's `backward()` call implements Linnainmaa's reverse-mode AD, not textbook chain rule—the difference is that graph-reverse topological traversal enables parallel gradient computation across independent subgraphs. \index{Backpropagation!provenance}
|
||||
|
||||
[^fn-blas-training]: **BLAS**: Standardized by @lawson1979, BLAS defines three levels: Level 1 (vector-vector, $O(n)$ work), Level 2 (matrix-vector, $O(n^2)$), and Level 3 (matrix-matrix, $O(n^3)$ work on $O(n^2)$ data). Training is dominated by Level 3 operations precisely because their high arithmetic intensity---$O(n)$ FLOPs per byte---saturates hardware compute units rather than starving on memory bandwidth. cuBLAS and oneDNN implement these as the kernel layer beneath every framework's matrix multiplication. \index{BLAS!training compute hierarchy}
|
||||
|
||||
@@ -5161,7 +5161,7 @@ class GPT2SummaryCalc:
|
||||
b_time_str = f"{fmt(b_time, precision=0, commas=False)} days"
|
||||
b_energy_str = f"{fmt(b_energy, precision=0, commas=True)} kWh"
|
||||
b_cost_str = f"${fmt(b_cost, precision=0, commas=True)}"
|
||||
b_carbon_str = f"~{fmt(b_carbon, precision=0, commas=False)} tons CO2"
|
||||
b_carbon_str = f"~{fmt(b_carbon, precision=0, commas=False)} tons CO~2~"
|
||||
|
||||
o_param_str = f"{fmt(o_param, precision=1, commas=False)} GB"
|
||||
o_grad_str = f"{fmt(o_grad, precision=1, commas=False)} GB"
|
||||
@@ -5172,7 +5172,7 @@ class GPT2SummaryCalc:
|
||||
o_time_str = f"{fmt(o_time, precision=1, commas=False)} days"
|
||||
o_energy_str = f"{fmt(o_energy, precision=0, commas=True)} kWh"
|
||||
o_cost_str = f"${fmt(o_cost, precision=0, commas=True)}"
|
||||
o_carbon_str = f"~{fmt(o_carbon, precision=0, commas=False)} tons CO2"
|
||||
o_carbon_str = f"~{fmt(o_carbon, precision=0, commas=False)} tons CO~2~"
|
||||
```
|
||||
|
||||
@tbl-gpt2-summary compiles the cluster-level impact of applying mixed-precision training, gradient checkpointing, and the data-pipeline utilization gains from the walkthrough to GPT-2.
|
||||
@@ -6142,7 +6142,7 @@ Training large models is not just a compute challenge; it is a massive energy si
|
||||
|
||||
**Systems conclusion**: This single training run consumes as much electricity as an average US household uses in `{python} TrainingCarbonFootprint.cf_household_months_str` months.
|
||||
|
||||
- **The optimization dividend**: Improving **Utilization** from 30 percent to 60 percent halves the compute time; it saves ~`{python} TrainingCarbonFootprint.cf_energy_saved_str` kWh of energy and reduces the carbon footprint by over **`{python} TrainingCarbonFootprint.cf_co2_str` tons of CO2** (assuming average grid intensity).
|
||||
- **The optimization dividend**: Improving **Utilization** from 30 percent to 60 percent halves the compute time; it saves ~`{python} TrainingCarbonFootprint.cf_energy_saved_str` kWh of energy and reduces the carbon footprint by over **`{python} TrainingCarbonFootprint.cf_co2_str` tons of CO~2~** (assuming average grid intensity).
|
||||
- **The true cost**: Training systems engineering is the primary lever for sustainable AI. Every 1 percent gain in efficiency at scale is equivalent to taking dozens of cars off the road for a year.
|
||||
|
||||
:::
|
||||
|
||||
@@ -171,7 +171,7 @@ cache timing attack
|
||||
|
||||
carbon footprint
|
||||
|
||||
: The total amount of greenhouse gas emissions produced directly and indirectly by an individual, organization, event, or product, typically measured in CO2 equivalent.
|
||||
: The total amount of greenhouse gas emissions produced directly and indirectly by an individual, organization, event, or product, typically measured in CO~2~-equivalent.
|
||||
*Appears in: @sec-sustainable-ai*
|
||||
|
||||
carbon-aware scheduling
|
||||
|
||||
@@ -7272,12 +7272,13 @@
|
||||
primaryclass = {cs.CL},
|
||||
}
|
||||
|
||||
@misc{tesla2018zoox,
|
||||
title = {{Tesla, Inc. v. Khatilov} et al.: Complaint Alleging Theft of Trade Secrets},
|
||||
author = {{Tesla, Inc.}},
|
||||
year = {2018},
|
||||
url = {https://storage.courtlistener.com/recap/gov.uscourts.nvd.131251/gov.uscourts.nvd.131251.1.0_1.pdf},
|
||||
howpublished = {U.S. District Court for the District of Nevada, Case No. 2:19-cv-00135},
|
||||
@misc{korosec2019tesla,
|
||||
title = {{Tesla} Sues Former Employees, {Zoox} for Alleged Trade Secret Theft},
|
||||
author = {Korosec, Kirsten},
|
||||
year = {2019},
|
||||
month = {mar},
|
||||
url = {https://techcrunch.com/2019/03/21/tesla-sues-former-employees-zoox-for-alleged-trade-secret-theft/},
|
||||
howpublished = {TechCrunch},
|
||||
note = {Accessed: 2026-05-17},
|
||||
}
|
||||
|
||||
|
||||
@@ -3550,7 +3550,7 @@ We have mapped the physical infrastructure from the transistors in a Tensor Core
|
||||
|
||||
Throughout this chapter, we have repeatedly encountered the same constraint: communication bandwidth limits scaling efficiency. The bandwidth hierarchy table showed an 18$\times$ cliff between NVLink and InfiniBand. The scaling efficiency analysis in @sec-compute-scaling-efficiency showed that naive data parallelism across 1,024 GPUs achieves only 37.5 percent efficiency. The pod topology comparison showed that the choice between fat-tree, torus, and rail-optimized designs can alter AllReduce time by 2--3$\times$. All of these observations point to the same conclusion: the network is not a passive connector between nodes but an active determinant of system performance.
|
||||
|
||||
@sec-network-fabrics examines this fabric in detail: the physics of high-speed signaling over copper and optical media, the protocols that enable kernel-bypass data transfer (RDMA), the collective communication algorithms that distribute gradient synchronization across the fabric, and the topology decisions that determine whether a 10,000-GPU cluster achieves 80 percent scaling efficiency or 30 percent. The emerging technologies discussed in this chapter (co-packaged optics, CXL memory pooling, disaggregated architectures) all aim to flatten the bandwidth hierarchy that currently dictates these topology choices. As these technologies mature, the boundary between "the compute chapter" and "the network chapter" will blur, because the same physical fabric will carry both intra-node and inter-node communication. Until then, the distinction remains sharp, and mastering the network fabric is essential for anyone building or operating ML infrastructure at scale.
|
||||
@sec-network-fabrics examines this fabric in detail: the physics of high-speed signaling over copper and optical media, the protocols that enable kernel-bypass data transfer (RDMA), the collective communication algorithms that distribute gradient synchronization across the fabric, and the topology decisions that determine whether a 10,000-GPU cluster achieves 80 percent scaling efficiency or 30 percent. The emerging technologies discussed in this chapter (co-packaged optics, CXL memory pooling, disaggregated architectures) all aim to flatten the bandwidth hierarchy that currently dictates these topology choices. As these technologies mature, the boundary between @sec-compute-infrastructure and @sec-network-fabrics will blur, because the same physical fabric will carry both intra-node and inter-node communication. Until then, the distinction remains sharp, and mastering the network fabric is essential for anyone building or operating ML infrastructure at scale.
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -642,7 +642,7 @@ At the NVMe tier, compression represents a critical trade-off between I/O bandwi
|
||||
|
||||
::: {.callout-notebook title="The ImageNet bottleneck analysis"}
|
||||
|
||||
\index{ImageNet!bottleneck analysis}**Problem**: A ResNet-50 training job on ImageNet (1.28M images, ~150 KB average) targets 1,000 images/second. The question is whether to use individual JPEG files on an HDD or NVMe.
|
||||
**Problem**\index{ImageNet!bottleneck analysis}: A ResNet-50 training job on ImageNet (1.28M images, ~150 KB average) targets 1,000 images/second. The question is whether to use individual JPEG files on an HDD or NVMe.
|
||||
|
||||
**Math**:
|
||||
|
||||
@@ -1032,7 +1032,7 @@ where $T_{\text{step}} = \max(T_{\text{compute}}, T_{\text{I/O}})$ when I/O and
|
||||
|
||||
::: {.callout-notebook title="The data stall ratio"}
|
||||
|
||||
\index{Data!stall ratio, gpu utilization}**Scenario**: A GPU processes a batch in `{python} PipelineCalcs.t_comp_stall_str` ms, but storage takes `{python} PipelineCalcs.t_io_stall_str` ms to deliver the next batch.
|
||||
**Scenario**\index{Data!stall ratio, gpu utilization}: A GPU processes a batch in `{python} PipelineCalcs.t_comp_stall_str` ms, but storage takes `{python} PipelineCalcs.t_io_stall_str` ms to deliver the next batch.
|
||||
|
||||
**Without pipelining** (sequential I/O then compute):
|
||||
|
||||
@@ -1384,7 +1384,7 @@ As @fig-gds-path illustrates, the throughput improvement from GDS is most pronou
|
||||
|
||||
::: {.callout-notebook title="The CPU bypass dividend"}
|
||||
|
||||
\index{GPU Direct Storage!CPU bypass}**Problem**: A training node with 8 GPUs loads 150 KB images at 8,000 images/second per GPU (64,000 images/second total). Compare the CPU load under traditional I/O vs. GDS.
|
||||
**Problem**\index{GPU Direct Storage!CPU bypass}: A training node with 8 GPUs loads 150 KB images at 8,000 images/second per GPU (64,000 images/second total). Compare the CPU load under traditional I/O vs. GDS.
|
||||
|
||||
**Traditional path**\index{Traditional Path}: Each image requires a DMA from NVMe to DRAM, a `memcpy` from kernel to user space, and a PCIe transfer to GPU. At 64,000 images/second with `{python} GDSLatency.gds_trad_us_str` μs of CPU time per image, the CPU spends 7.68 seconds of CPU time per wall-clock second, consuming roughly 8 cores worth of processing just for data movement.
|
||||
|
||||
|
||||
@@ -2477,7 +2477,7 @@ The parallelism strategies examined throughout this chapter (gradient averaging,
|
||||
|
||||
The simplest distributed training abstraction wraps a model so that the framework automatically replicates it across available accelerators, splits each batch, and averages gradients after the backward pass. This parameter-server approach[^fn-parameter-server] [@li2014] requires minimal changes to a single-device training loop but creates a communication bottleneck at the central server node: with $N$ workers pushing dense gradient streams simultaneously, the server's inbound bandwidth becomes the chokepoint. For dense models beyond 4--8 GPUs, this centralized design collapses.
|
||||
|
||||
[^fn-parameter-server]: **Parameter Server**: Formalized by Mu Li et al. at CMU/Google in 2014, this architecture dedicates server nodes to storing parameters while workers push gradients and pull updates. The fundamental bottleneck: with $N$ workers, server inbound bandwidth must handle $N$ gradient streams simultaneously, making the server the communication chokepoint. For dense models beyond 4--8 GPUs, decentralized AllReduce (where each worker sends and receives at its own link rate) achieves $N$-fold higher aggregate bandwidth, which is why production systems replaced the parameter server design with decentralized AllReduce. \index{Parameter Server!architecture}
|
||||
[^fn-parameter-server]: **Parameter Server**: Formalized by @li2014 at CMU/Google, this architecture dedicates server nodes to storing parameters while workers push gradients and pull updates. The fundamental bottleneck: with $N$ workers, server inbound bandwidth must handle $N$ gradient streams simultaneously, making the server the communication chokepoint. For dense models beyond 4--8 GPUs, decentralized AllReduce (where each worker sends and receives at its own link rate) achieves $N$-fold higher aggregate bandwidth, which is why production systems replaced the parameter server design with decentralized AllReduce. \index{Parameter Server!architecture}
|
||||
|
||||
Production-scale data parallelism eliminates this bottleneck by replacing the central server with decentralized AllReduce. Each worker participates symmetrically in the reduction: every device both sends and receives gradient chunks at its own link rate, distributing the bandwidth load across all nodes rather than concentrating it. The framework initializes a process group that maps workers to the physical topology, selects the optimal collective algorithm (ring, tree, or hierarchical) based on the detected interconnect, and inserts gradient synchronization hooks into the backward pass automatically. Gradient bucketing further improves efficiency by grouping small tensors into larger messages before transmission, and computation-communication overlap allows the AllReduce for early layers to proceed while later layers are still computing gradients. These optimizations collectively achieve 90 percent+ parallel efficiency at moderate scale—not because the API is simple, but because the underlying runtime makes topology-aware decisions that a manual implementation would require thousands of lines to replicate.
|
||||
|
||||
|
||||
@@ -1962,9 +1962,9 @@ Verify your understanding of different batching mechanics:
|
||||
|
||||
Before selecting a batching strategy, it is essential to understand where latency accumulates across the full request lifecycle. @fig-inference-lifecycle maps each stage from client to response, revealing the "serving tax" that serialization, routing, and coordination impose outside of GPU compute.
|
||||
|
||||
::: {#fig-inference-request-flow fig-env="figure" fig-pos="htb" fig-cap="**Inference Request Lifecycle**: A sequence diagram showing how a user request traverses the serving stack. The 'Serving Tax' is the time spent in the Router, Queue, and Scheduler before the GPU begins mathematical execution." fig-alt="Sequence diagram with five participants (User, Router, KV Cache Manager, Iteration Scheduler, GPU) showing a chat completion request flow: the Router reserves KV cache slots, enqueues the request with the Scheduler, and an iteration loop dispatches prefill and decode batches to the GPU until the response streams back to the user."}
|
||||
|
||||
```{text}
|
||||
%%| label: fig-inference-request-flow
|
||||
%%| fig-cap: "**Inference Request Lifecycle**: A sequence diagram showing how a user request traverses the serving stack. The 'Serving Tax' is the time spent in the Router, Queue, and Scheduler before the GPU begins mathematical execution."
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant Router
|
||||
@@ -1986,6 +1986,8 @@ sequenceDiagram
|
||||
Scheduler->>User: Stream Result
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
::: {#fig-inference-lifecycle fig-env="figure" fig-pos="htb" fig-cap="**End-to-End Inference Pipeline**: A detailed view of the request lifecycle: client requests arrive at a load balancer, then undergo preprocessing (tokenization) on the CPU. The data moves to the GPU for the compute-bound prefill stage and the memory-bound decode stage, which uses the KV cache. Finally, the response is postprocessed on the CPU (detokenization) and sent back. This visualization highlights the critical \"Serving Tax\" components (serialization, routing, coordination) that consume latency budget outside of the actual GPU compute time." fig-alt="Flowchart showing a request moving from CPU (Preprocessing) to GPU (Prefill Stage, Decode Stage) and back to CPU (Postprocessing). Red annotations below highlight latency sources such as routing, TTFT, and per-token decode time."}
|
||||
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ Traditional software fails loudly: exceptions crash processes, type errors halt
|
||||
|
||||
:::
|
||||
|
||||
The chapter's position in the book's organizing framework, *the fleet stack*, clarifies why robustness is an infrastructure concern: silent failures propagate through every layer of the system, from input perturbations and distribution drift at the data layer to degraded predictions at the service level. [@berkeley2024compound]
|
||||
The chapter's position in the book's organizing framework, *the fleet stack*, clarifies why robustness is an infrastructure concern: silent failures propagate through every layer of the system, from input perturbations and distribution drift at the data layer to degraded predictions at the service level [@berkeley2024compound].
|
||||
|
||||
::: {.callout-perspective title="Fleet stack connection"}
|
||||
|
||||
|
||||
@@ -990,9 +990,9 @@ A stop sign with three strategically placed pieces of black tape is recognized b
|
||||
|
||||
These attacks span the ML lifecycle and map directly to the threat model classifications we developed: threats to model confidentiality target deployment and inference stages through model theft, threats to training integrity strike during data collection and model development through poisoning attacks, and threats to inference robustness exploit runtime operations through adversarial examples[^fn-adversarial-examples]. Understanding when each attack occurs guides where to deploy corresponding defenses. Data poisoning[^fn-data-poisoning] compromises the learning process itself, while model theft and adversarial attacks target the deployed system. Each category requires distinct defensive strategies aligned with the attack surface analysis presented earlier.
|
||||
|
||||
[^fn-adversarial-examples]: **Adversarial Examples**: First discovered by Szegedy et al. in 2013, these are inputs crafted to exploit learned decision boundaries with perturbations imperceptible to humans (less than 0.01 percent of pixel values changed). The phenomenon reveals a fundamental tension in ML system design: the same high-dimensional feature spaces that enable generalization create exploitable geometry where small, targeted perturbations cross decision boundaries. \index{Adversarial Examples!discovery}
|
||||
[^fn-adversarial-examples]: **Adversarial Examples**: First discovered by @szegedy2013intriguing, these are inputs crafted to exploit learned decision boundaries with perturbations imperceptible to humans (less than 0.01 percent of pixel values changed). The phenomenon reveals a fundamental tension in ML system design: the same high-dimensional feature spaces that enable generalization create exploitable geometry where small, targeted perturbations cross decision boundaries. \index{Adversarial Examples!discovery}
|
||||
|
||||
[^fn-data-poisoning]: **Data Poisoning**: First formally studied by Biggio et al. in 2012, this attack injects malicious data during training to corrupt the learned model. The efficiency is striking: poisoning just 0.1 percent of training data can reduce accuracy by 10--50 percent, making it orders of magnitude cheaper than model extraction attacks. For large-scale training pipelines ingesting web-scraped data, even a small fraction of adversarial content can embed persistent backdoors. \index{Data Poisoning!efficiency}
|
||||
[^fn-data-poisoning]: **Data Poisoning**: First formally studied by @biggio2012poisoning, this attack injects malicious data during training to corrupt the learned model. The efficiency is striking: poisoning just 0.1 percent of training data can reduce accuracy by 10--50 percent, making it orders of magnitude cheaper than model extraction attacks. For large-scale training pipelines ingesting web-scraped data, even a small fraction of adversarial content can embed persistent backdoors. \index{Data Poisoning!efficiency}
|
||||
|
||||
Understanding when and where different attacks occur in the ML lifecycle helps prioritize defenses and understand attacker motivations. @fig-ml-lifecycle-threats visualizes these stages and the specialized techniques adversaries use at each point, from data collection through model deployment and inference.
|
||||
|
||||
@@ -1381,11 +1381,11 @@ These defense mechanisms, deployed in combination, significantly raise the bar f
|
||||
|
||||
#### Case study: Tesla IP theft {#sec-security-privacy-case-study-tesla-ip-theft-9d78}
|
||||
|
||||
In 2018, Tesla filed a lawsuit against the self-driving car startup Zoox, alleging that former Tesla employees had taken proprietary data and trade secrets related to Tesla's autonomous driving technology [@tesla2018zoox]. The complaint alleged that several employees had transferred over 10 gigabytes of confidential files, including machine learning models and source code, before leaving Tesla to join Zoox.
|
||||
In March 2019, Tesla filed two trade-secret lawsuits involving former employees who left for autonomous-vehicle competitors [@korosec2019tesla]. One complaint alleged that employees who joined Zoox copied proprietary warehousing, logistics, and inventory-control documents. A separate complaint alleged that former Autopilot engineer Guangzhi Cao copied Tesla Autopilot source-code repositories before joining XPeng/XMotors.
|
||||
|
||||
Among the materials Tesla alleged were taken was a key image recognition model used for object detection in Tesla's self-driving system. If those allegations were accurate, Zoox could in principle have bypassed years of research and development, gaining a competitive advantage. Beyond the economic implications, alleged exposure of such a model raises secondary concerns about downstream security risks, such as model inversion attacks aimed at extracting sensitive data from the model's training set.
|
||||
These allegations matter for ML systems because the protected asset is often the full engineering stack, not just a serialized model file. Training code, feature pipelines, labeling workflows, deployment scripts, simulation assets, and operational playbooks can reveal enough about a system to accelerate a competitor's replication effort or expose downstream security weaknesses. Insider access therefore bypasses many API-level model-extraction defenses: the attacker is not querying the model from the outside, but copying development artifacts from inside the build environment.
|
||||
|
||||
The Zoox employees denied any wrongdoing, and the case was ultimately settled out of court. The incident highlights the real-world risks of model theft, especially in industries where machine learning models represent significant intellectual property. The theft of models undermines competitive advantage and raises broader concerns about privacy, safety, and the potential for downstream exploitation.
|
||||
The incident highlights the real-world risks of ML-system IP theft in industries where models, data pipelines, and source code represent significant intellectual property. Theft of these artifacts undermines competitive advantage and raises broader concerns about privacy, safety, and downstream exploitation.
|
||||
|
||||
This case demonstrates that model theft is not limited to theoretical attacks conducted over APIs or public interfaces. Insider threats, supply chain vulnerabilities, and unauthorized access to development infrastructure pose equally serious risks to machine learning systems deployed in commercial environments.
|
||||
|
||||
@@ -1405,7 +1405,7 @@ Poisoning attacks typically follow a three-stage process. First, the attacker in
|
||||
|
||||
To understand these attack mechanisms precisely, data poisoning can be viewed as a bilevel optimization problem[^fn-bilevel-optimization], where the attacker seeks to select poisoning data $D_p$ that maximizes the model's loss on a validation or target dataset $D_{\text{test}}$. This *data poisoning optimization loop* is formalized as follows. Let $D$ represent the original training data. The attacker's objective is to solve:
|
||||
|
||||
[^fn-bilevel-optimization]: **Bilevel Optimization** (from the two nested "levels" of optimization): A framework where one optimization problem contains another, formalized for ML security by Biggio et al. The outer problem (attacker) optimizes poisoning data; the inner problem (defender) trains the model. This nesting explains why robust defense is computationally expensive: evaluating each candidate defense requires solving the full inner training loop, multiplying computational cost by the number of defense iterations. \index{Optimization!bilevel, poisoning}
|
||||
[^fn-bilevel-optimization]: **Bilevel Optimization** (from the two nested "levels" of optimization): A framework where one optimization problem contains another, formalized for ML security by @biggio2012poisoning. The outer problem (attacker) optimizes poisoning data; the inner problem (defender) trains the model. This nesting explains why robust defense is computationally expensive: evaluating each candidate defense requires solving the full inner training loop, multiplying computational cost by the number of defense iterations. \index{Optimization!bilevel, poisoning}
|
||||
|
||||
$$
|
||||
\max_{D_p} \ \mathcal{L}(f_{D \cup D_p}, D_{\text{test}})
|
||||
@@ -1457,7 +1457,7 @@ For example, consider a traffic sign classification model trained to distinguish
|
||||
|
||||
Data poisoning attacks can be classified based on their objectives and scope of impact [@biggio2012poisoning]. Availability attacks degrade overall model performance by introducing noise or label flips that reduce accuracy across tasks. Targeted attacks manipulate a specific input or class, leaving general performance intact but causing consistent misclassification in select cases [@shafahi2018poison]. Backdoor attacks[^fn-backdoor-attacks] embed hidden triggers, which are often imperceptible patterns, that elicit malicious behavior only when the trigger is present [@gu2017badnets]. Subpopulation attacks degrade performance on a specific group defined by shared features, making them particularly dangerous in fairness-sensitive applications.
|
||||
|
||||
[^fn-backdoor-attacks]: **Backdoor Attacks**: First demonstrated by Gu et al. in 2017 with BadNets, these attacks embed hidden triggers during training that activate only when specific input patterns appear. The stealth is extreme: backdoored models maintain normal accuracy on clean inputs (passing standard evaluation) while achieving 99 percent+ attack success on triggered inputs, making detection through accuracy metrics alone impossible. \index{Backdoor Attack!BadNets}
|
||||
[^fn-backdoor-attacks]: **Backdoor Attacks**: First demonstrated by @gu2017badnets with BadNets, these attacks embed hidden triggers during training that activate only when specific input patterns appear. The stealth is extreme: backdoored models maintain normal accuracy on clean inputs (passing standard evaluation) while achieving 99 percent+ attack success on triggered inputs, making detection through accuracy metrics alone impossible. \index{Backdoor Attack!BadNets}
|
||||
|
||||
A notable real-world example of an adversarial evasion attack was demonstrated against Perspective, Google's widely-used online toxicity detection model[^fn-perspective-api] that helps platforms identify harmful content [@hosseini2017deceiving]. By modifying toxic phrases at inference time with subtle misspellings, character substitutions, and added punctuation, researchers reliably caused the deployed model to assign low toxicity scores to clearly toxic content[^fn-perspective-vulnerability]. Although evasion (rather than training-set poisoning), the failure mode is closely related: both classes of attack exploit a model whose decision surface is fragile to small adversarial perturbations.
|
||||
|
||||
@@ -1485,7 +1485,7 @@ Adversarial attacks vary based on the attacker's level of access to the model. I
|
||||
|
||||
Common attack strategies include surrogate model construction, transfer attacks exploiting adversarial transferability[^fn-adversarial-transferability] [@papernot2016transferability], and GAN-based perturbation generation.
|
||||
|
||||
[^fn-adversarial-transferability]: **Adversarial Transferability**: Discovered by Szegedy et al. in 2014, this phenomenon shows that adversarial examples crafted against one model fool different architectures with 60--80 percent success rates. Transferability transforms the threat model: attackers need no access to the target system; they craft perturbations against a freely available surrogate and deploy them against the production model, making black-box attacks nearly as effective as white-box ones. \index{Adversarial Transferability!black-box}
|
||||
[^fn-adversarial-transferability]: **Adversarial Transferability**: Documented by @szegedy2013intriguing and later systematized by @papernot2016transferability, this phenomenon shows that adversarial examples crafted against one model fool different architectures with 60--80 percent success rates. Transferability transforms the threat model: attackers need no access to the target system; they craft perturbations against a freely available surrogate and deploy them against the production model, making black-box attacks nearly as effective as white-box ones. \index{Adversarial Transferability!black-box}
|
||||
|
||||
| **Adversary Knowledge Level** | **Model Access** | **Training Data Access** | **Attack Example** | **Common Scenario** |
|
||||
|:------------------------------|:---------------------------------------------|:-------------------------|:----------------------------------------------------------|:--------------------------------------------|
|
||||
|
||||
@@ -802,7 +802,7 @@ Calculate the total carbon footprint for training a 70B parameter model.
|
||||
|
||||
**Embodied**: Assume manufacturing footprint is $\approx 164 \text{kg } \text{CO}_2$ per H100 GPU (NVIDIA's product carbon footprint). Amortized for 1 month of a 3-year cycle: $(2048 \times 164 \text{kg}) / 36 \text{ months} \approx 9.3 \text{ metric tons}$.
|
||||
|
||||
**Total**: $536 + 10 \approx 546 \text{ metric tons } \text{CO}_2$.
|
||||
**Total**: $536 + 9.3 \approx 545 \text{ metric tons } \text{CO}_2$.
|
||||
|
||||
:::
|
||||
|
||||
|
||||
Reference in New Issue
Block a user