github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-03-08 23:03:55 -05:00
Code Issues Packages Projects Releases Wiki Activity

Standardize LaTeX subscripts to \text{} across both volumes.

Browse Source 
Replace D_{vol}, R_{peak}, L_{lat} with D_{\text{vol}},
R_{\text{peak}}, L_{\text{lat}} in all QMD files and notation.qmd
to match the canonical notation convention. Also escape bare
FLOPs/$ to FLOPs/\$ in vol1 introduction. 288 replacements
across 24 files.
This commit is contained in:
Vijay Janapa Reddi
2026-03-05 11:04:34 -05:00
parent 048492d0e1
commit 2100099efb
25 changed files with 611 additions and 299 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
book/quarto/contents/shared/notation.qmd
View File

@@ -39,7 +39,7 @@ Does this mean:
With our notation, we can write precisely:
> *"Reducing $D_{vol}$ through INT8 quantization cuts memory traffic to one quarter while $D$ (training data) remains unchanged."*
> *"Reducing $D_{\text{vol}}$ through INT8 quantization cuts memory traffic to one quarter while $D$ (training data) remains unchanged."*
Our notation makes such ambiguity explicit: *"$\text{BW}$ limits throughput"* is unambiguous.

42
book/quarto/contents/vol1/backmatter/appendix_dam.qmd
View File

@@ -120,11 +120,11 @@ The **Data · Algorithm · Machine (D·A·M) taxonomy** is the primary diagnosti
The taxonomy maps directly to the **Iron Law of ML Systems**, as established in @sec-introduction. @tbl-dam-components-ref summarizes the role, primary physical constraint, and core optimization pathway for each axis.
| **Axis** | **Role** | **Physical Constraint** | **High-Leverage Optimization** |
|:------------------|:---------------------------|:------------------------|:---------------------------------------------------|
| **Data (D)** | **Information** (The Fuel) | Bandwidth ($BW$) | Data Selection (@sec-data-selection) |
| **Algorithm (A)** | **Logic** (The Blueprint) | Operations ($O$) | Model Compression (@sec-model-compression) |
| **Machine (M)** | **Physics** (The Engine) | Throughput ($R_{peak}$) | Hardware Acceleration (@sec-hardware-acceleration) |
| **Axis** | **Role** | **Physical Constraint** | **High-Leverage Optimization** |
|:------------------|:---------------------------|:-------------------------------|:---------------------------------------------------|
| **Data (D)** | **Information** (The Fuel) | Bandwidth ($BW$) | Data Selection (@sec-data-selection) |
| **Algorithm (A)** | **Logic** (The Blueprint) | Operations ($O$) | Model Compression (@sec-model-compression) |
| **Machine (M)** | **Physics** (The Engine) | Throughput ($R_{\text{peak}}$) | Hardware Acceleration (@sec-hardware-acceleration) |
: **D·A·M Axis Reference.** Each axis maps to a distinct physical constraint and a high-leverage optimization strategy. Start diagnosis here: identify which constraint is binding, then follow the optimization pointer to the relevant chapter. {#tbl-dam-components-ref}
@@ -164,23 +164,23 @@ Understanding the landscape tells you *where* a technique lives. The next step i
The performance of any ML task is governed by the distribution of work across the D·A·M axes. The Iron Law Mapping reveals which component's variables dominate the execution time:
$$ T = \underbrace{ \frac{D_{vol}}{BW} }_{\text{Data (D)}} + \underbrace{ \frac{O}{R_{peak} \cdot \eta} }_{\text{Algorithm (A) / Machine (M)}} + \underbrace{ L_{lat} }_{\text{Overhead}} $$
$$ T = \underbrace{ \frac{D_{\text{vol}}}{BW} }_{\text{Data (D)}} + \underbrace{ \frac{O}{R_{\text{peak}} \cdot \eta} }_{\text{Algorithm (A) / Machine (M)}} + \underbrace{ L_{\text{lat}} }_{\text{Overhead}} $$
Note that Algorithm and Machine share the compute term; they are separated by which variable you control. Reducing the total operations ($O$) is an **Algorithm** lever, while improving the hardware's peak throughput ($R_{peak}$) or utilization ($\eta$) is a **Machine** lever.
Note that Algorithm and Machine share the compute term; they are separated by which variable you control. Reducing the total operations ($O$) is an **Algorithm** lever, while improving the hardware's peak throughput ($R_{\text{peak}}$) or utilization ($\eta$) is a **Machine** lever.
This equation transforms performance debugging from a qualitative guessing game into a quantitative engineering problem. Every bottleneck hides in one of these terms. If your system is slow, it is because you are moving too much data ($D_{vol}$), lacking bandwidth ($BW$), executing too many operations ($O$), or failing to use your hardware's peak capability ($\eta$). The levers below map specific optimizations to the variable they improve.
This equation transforms performance debugging from a qualitative guessing game into a quantitative engineering problem. Every bottleneck hides in one of these terms. If your system is slow, it is because you are moving too much data ($D_{\text{vol}}$), lacking bandwidth ($BW$), executing too many operations ($O$), or failing to use your hardware's peak capability ($\eta$). The levers below map specific optimizations to the variable they improve.
### Component Levers {#sec-dam-taxonomy-component-levers-d441}
* **Data Lever**: Reducing the volume of data ($D_{vol}$) through deduplication or curriculum learning, or increasing I/O bandwidth ($BW$).
* **Data Lever**: Reducing the volume of data ($D_{\text{vol}}$) through deduplication or curriculum learning, or increasing I/O bandwidth ($BW$).
* **Algorithm Lever**: Reducing total arithmetic operations ($O$) through pruning, quantization, or architectural refinement.
* **Machine Lever**: Increasing the denominator of the compute term by improving peak throughput ($R_{peak}$) or increasing the utilization factor ($\eta$) via kernel fusion.
* **Machine Lever**: Increasing the denominator of the compute term by improving peak throughput ($R_{\text{peak}}$) or increasing the utilization factor ($\eta$) via kernel fusion.
### D·A·M Coordination: From Sum to Max {#sec-dam-taxonomy-dam-coordination-sum-max-92bb}
The additive Iron Law represents **sequential execution**—the worst case where Data, Algorithm, and Machine take turns. But skilled systems engineering transforms the sum into a max:
$$ T_{sequential} = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat} \quad \xrightarrow{\text{overlap}} \quad T_{pipelined} = \max\left(\frac{D_{vol}}{BW}, \frac{O}{R_{peak} \cdot \eta}\right) + L_{lat} $$
$$ T_{sequential} = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}} \quad \xrightarrow{\text{overlap}} \quad T_{pipelined} = \max\left(\frac{D_{\text{vol}}}{BW}, \frac{O}{R_{\text{peak}} \cdot \eta}\right) + L_{\text{lat}} $$
The systems engineer's job is to make these components run in parallel, not in series. @tbl-dam-overlap summarizes key D·A·M Coordination techniques:
@@ -193,10 +193,10 @@ The systems engineer's job is to make these components run in parallel, not in s
: **D·A·M Overlap Techniques.** Each technique allows one D·A·M axis to execute while another is in flight, converting the Iron Law's additive terms into overlapped terms. The payoff is transforming $T = a + b$ into $T = \max(a, b)$, which can cut latency nearly in half when the terms are balanced. {#tbl-dam-overlap}
Overlap only helps when the D·A·M axes are reasonably balanced. If one term dominates (e.g., severely memory-bound), overlapping the smaller term with the larger yields negligible gain—the max is still dominated by the same bottleneck. Overlap provides the greatest benefit when $D_{vol}/BW \approx O/(R_{peak} \cdot \eta)$.
Overlap only helps when the D·A·M axes are reasonably balanced. If one term dominates (e.g., severely memory-bound), overlapping the smaller term with the larger yields negligible gain—the max is still dominated by the same bottleneck. Overlap provides the greatest benefit when $D_{\text{vol}}/BW \approx O/(R_{\text{peak}} \cdot \eta)$.
::: {.callout-warning title="The Overhead That Cannot Hide"}
The latency term $L_{lat}$ (kernel launch, synchronization barriers, Python dispatch) typically cannot be overlapped—it represents serialization points where all components must wait. This is why **kernel fusion** is so powerful: it eliminates $L_{lat}$ by combining operations, not just by speeding up any single component.
The latency term $L_{\text{lat}}$ (kernel launch, synchronization barriers, Python dispatch) typically cannot be overlapped—it represents serialization points where all components must wait. This is why **kernel fusion** is so powerful: it eliminates $L_{\text{lat}}$ by combining operations, not just by speeding up any single component.
:::
The Iron Law tells you *how much* time each axis consumes. But it leaves one critical question unanswered: when the bottleneck sits at the boundary between Data and Machine, how do you tell which side you're on? The answer lies in a single ratio.
@@ -205,7 +205,7 @@ The Iron Law tells you *how much* time each axis consumes. But it leaves one cri
The boundary between **Data** (Memory-Bound) and **Machine** (Compute-Bound) is not arbitrary; it is defined mathematically by the **Arithmetic Intensity**[^fn-arith-intensity] ($I$) of the workload.
[^fn-arith-intensity]: **Arithmetic Intensity**: The ratio of floating-point operations to bytes transferred (FLOPs/byte), introduced by Williams, Waterman, and Patterson (2009) as the key parameter in the Roofline Model. It determines whether a workload is memory-bound or compute-bound by comparison against the hardware's *ridge point* ($R_{peak}/BW$). @sec-machine-foundations-roofline-model-2529 provides a complete derivation.
[^fn-arith-intensity]: **Arithmetic Intensity**: The ratio of floating-point operations to bytes transferred (FLOPs/byte), introduced by Williams, Waterman, and Patterson (2009) as the key parameter in the Roofline Model. It determines whether a workload is memory-bound or compute-bound by comparison against the hardware's *ridge point* ($R_{\text{peak}}/BW$). @sec-machine-foundations-roofline-model-2529 provides a complete derivation.
@sec-machine-foundations-roofline-model-2529 provides rigorous definitions of Arithmetic Intensity and the Roofline Model. Use that model to quantitatively distinguish between Data and Machine bottlenecks before applying the optimizations below.
@@ -218,7 +218,7 @@ In the heat of a production outage, you rarely have time to solve the full Iron
* **If Accelerator Utilization $<$ 80%**: You are likely **Data Bound** (or CPU bound). The accelerator is starving.
* **If Accelerator Utilization $>$ 95%**: You are likely **Machine Bound**. The accelerator is fully saturated.
* **If Batch Size is 1**: You are likely **Latency Bound** (Algorithm overhead dominates).
* **If Arithmetic Intensity $<$ 100 FLOPs/byte**: You are likely **Memory Bound** (Data/Machine boundary). This threshold is approximate for current-generation accelerators; compute your hardware's specific ridge point ($R_{peak}/BW$) for a precise boundary.
* **If Arithmetic Intensity $<$ 100 FLOPs/byte**: You are likely **Memory Bound** (Data/Machine boundary). This threshold is approximate for current-generation accelerators; compute your hardware's specific ridge point ($R_{\text{peak}}/BW$) for a precise boundary.
* **If System works in Dev but fails in Prod**: Suspect **Data Drift** (Data component).
Note that common industry labels map to DAM components as follows: **Memory Bound** typically indicates a **Data** bottleneck (information cannot reach the accelerator fast enough), **Compute Bound** indicates a **Machine** bottleneck (the accelerator is fully saturated), and **Latency Bound** indicates an **Algorithm** bottleneck (serial operation depth or overhead dominates).
@@ -227,11 +227,11 @@ Note that common industry labels map to DAM components as follows: **Memory Boun
Once you identify the bottleneck, @tbl-bottleneck-actions tells you what to do—and what NOT to do:
| **If You're...** | **Dominant Term** | **Optimization That Works** | **Optimization That is Wasted** |
|:------------------|:--------------------------|:------------------------------------------------------------------|:---------------------------------------------------|
| **Memory-Bound** | $D_{vol}/BW$ | Quantization, pruning, batching, kernel fusion | Faster accelerator (more FLOP/s will not help) |
| **Compute-Bound** | $O/(R_{peak} \cdot \eta)$ | Better kernels, Tensor Cores, faster accelerator, lower precision | More memory bandwidth (already saturated) |
| **Latency-Bound** | $L_{lat}$ | Batching requests, kernel fusion, async dispatch | Neither compute nor bandwidth (overhead dominates) |
| **If You're...** | **Dominant Term** | **Optimization That Works** | **Optimization That is Wasted** |
|:------------------|:---------------------------------|:------------------------------------------------------------------|:---------------------------------------------------|
| **Memory-Bound** | $D_{\text{vol}}/BW$ | Quantization, pruning, batching, kernel fusion | Faster accelerator (more FLOP/s will not help) |
| **Compute-Bound** | $O/(R_{\text{peak}} \cdot \eta)$ | Better kernels, Tensor Cores, faster accelerator, lower precision | More memory bandwidth (already saturated) |
| **Latency-Bound** | $L_{\text{lat}}$ | Batching requests, kernel fusion, async dispatch | Neither compute nor bandwidth (overhead dominates) |
: **What Works vs. What is Wasted.** Optimizing the wrong term yields exactly zero improvement. A memory-bound LLM will not speed up from a faster accelerator; the accelerator will simply idle faster while waiting for memory. {#tbl-bottleneck-actions}
@@ -416,7 +416,7 @@ Computed values: Achieved = `{python} DAMTaxonomy.ex2_achieved_str` TFLOP/s, Uti
This system is **Memory-bound** (a **Data** bottleneck). At batch size 1, each layer performs a matrix-vector multiply (GEMV) rather than a matrix-matrix multiply (GEMM). The model's `{python} DAMTaxonomy.ex2_params_str` parameters (~`{python} DAMTaxonomy.ex2_model_size_gb_str` GB in FP16) must be loaded from HBM for every forward pass, but each loaded weight is used for only a single input vector—yielding very low arithmetic intensity. The GPU's compute units sit idle waiting for memory transfers, which is why utilization is only `{python} DAMTaxonomy.ex2_util_str`%.
The fix targets the **Data/Algorithm boundary**: increasing the batch size transforms GEMV into GEMM, dramatically raising arithmetic intensity and pushing the workload toward compute-bound. Other effective strategies include quantization (INT8 halves the bytes moved per parameter, directly reducing the $D_{vol}/BW$ term) or speculative decoding to amortize weight loads across multiple tokens.
The fix targets the **Data/Algorithm boundary**: increasing the batch size transforms GEMV into GEMM, dramatically raising arithmetic intensity and pushing the workload toward compute-bound. Other effective strategies include quantization (INT8 halves the bytes moved per parameter, directly reducing the $D_{\text{vol}}/BW$ term) or speculative decoding to amortize weight loads across multiple tokens.
##### Exercise 3: *Scaling Law vs. Information Roofline* {.unnumbered}

2
book/quarto/contents/vol1/backmatter/appendix_machine.qmd
View File

@@ -848,7 +848,7 @@ While latency tells us how long we wait for the *first* byte, bandwidth tells us
| **Spec** | **NVIDIA H100 (SXM)** | **Google TPU v5p** | **System Impact** |
|:---------------------|----------------------------------------------------------:|-----------------------------------------------------:|:----------------------------------------|
| **FP16/BF16 Peak** | `{python} AppendixMachineSetup.h100_flops` TFLOPS | `{python} AppendixMachineSetup.tpuv5_flops` TFLOPS | The "Speed Limit" ($R_{peak}$) |
| **FP16/BF16 Peak** | `{python} AppendixMachineSetup.h100_flops` TFLOPS | `{python} AppendixMachineSetup.tpuv5_flops` TFLOPS | The "Speed Limit" ($R_{\text{peak}}$) |
| **Memory Bandwidth** | `{python} AppendixMachineSetup.h100_bw` TB/s | `{python} AppendixMachineSetup.tpuv5_bw` TB/s | The "Width of the Pipe" ($BW$) |
| **HBM Capacity** | `{python} AppendixMachineSetup.h100_cap` GB | `{python} AppendixMachineSetup.tpuv5_cap` GB | Max Model Size ($P$) / Batch Size ($B$) |
| **L2/SRAM Cache** | `{python} AppendixMachineSetup.h100_l2_mb` MB | ~`{python} AppendixMachineSetup.tpuv5_l2_mb` MB | Critical for Operator Fusion |

14
book/quarto/contents/vol1/benchmarking/benchmarking.qmd
View File

@@ -400,10 +400,10 @@ System benchmarks serve two functions. For practitioners, they enable informed h
::: {.callout-definition title="Machine Learning System Benchmarks"}
***Machine Learning System Benchmarks***\index{ML System Benchmarks!definition} are standardized evaluation protocols that hold the workload and quality target constant while varying the hardware-software stack, measuring $\eta = R_{\text{sustained}} / R_{peak}$ and $L_{lat}$ to isolate infrastructure efficiency from algorithmic improvements.
***Machine Learning System Benchmarks***\index{ML System Benchmarks!definition} are standardized evaluation protocols that hold the workload and quality target constant while varying the hardware-software stack, measuring $\eta = R_{\text{sustained}} / R_{\text{peak}}$ and $L_{\text{lat}}$ to isolate infrastructure efficiency from algorithmic improvements.
1. **Significance (Quantitative):** The same ResNet-50 model can deliver 10$\times$ different throughput across hardware stacks and compiler configurations (from ~300 images/second on a CPU to ~3,000 images/second on an A100 with INT8 quantization), yet both implementations achieve identical ImageNet Top-1 accuracy. System benchmarks capture this 10$\times$ gap, which is entirely invisible to algorithmic benchmarks that only report accuracy.
2. **Distinction (Durable):** Unlike algorithmic benchmarks (which vary model architectures and training procedures to improve convergence accuracy), system benchmarks hold the algorithm fixed and vary the implementation (kernel libraries, quantization formats, batch sizes, and hardware generations) to measure how efficiently the hardware-software stack executes the Iron Law's $O/(R_{peak} \cdot \eta)$ term.
2. **Distinction (Durable):** Unlike algorithmic benchmarks (which vary model architectures and training procedures to improve convergence accuracy), system benchmarks hold the algorithm fixed and vary the implementation (kernel libraries, quantization formats, batch sizes, and hardware generations) to measure how efficiently the hardware-software stack executes the Iron Law's $O/(R_{\text{peak}} \cdot \eta)$ term.
3. **Common Pitfall:** A frequent misconception is that a system benchmark result generalizes across workloads. An A100 that achieves 90% utilization on ResNet-50 (a compute-bound workload) may achieve only 40% utilization on a recommendation system (a memory-bandwidth-bound workload). System benchmarks are workload-specific; no single metric characterizes a hardware platform.
:::
@@ -984,7 +984,7 @@ With these measurement principles established, we can now examine how to diagnos
**From Theory to Trace**: How to map the **Iron Law** equation (from @sec-hardware-acceleration) to a profiler timeline (like Nsight Systems or PyTorch Profiler).
**1. Measuring the Data Term ($\frac{D_{vol}}{BW}$)**
**1. Measuring the Data Term ($\frac{D_{\text{vol}}}{BW}$)**
* **Signal:** Look for the **"Memory Throughput"** or **"DRAM Bandwidth"** line.
* **Calculation:** $\text{Effective BW} = \frac{\text{Total Bytes Transferred}}{\text{Kernel Duration}}$.
@@ -996,7 +996,7 @@ With these measurement principles established, we can now examine how to diagnos
* **Calculation:** $\text{Achieved TFLOPS} = \frac{\text{FLOP Count}}{\text{Kernel Duration}}$.
* **Diagnosis:** If $\text{Achieved TFLOPS} \ll \text{Peak TFLOPS}$ AND $\text{Memory BW} \ll \text{Peak BW}$, the system is in the **"Utilization Trap"**: likely Latency Bound (kernels too small) or Grid Bound (not enough threads).
**3. Measuring the Latency Term ($L_{lat}$)**
**3. Measuring the Latency Term ($L_{\text{lat}}$)**
* **Signal:** Look for **Gaps** (empty space) between colored kernel bars on the timeline.
* **Calculation:** $\text{Overhead Ratio} = \frac{\text{Gap Duration}}{\text{Kernel Duration} + \text{Gap Duration}}$.
@@ -1731,7 +1731,7 @@ For instance, large-scale models like OpenAI's GPT-3[^fn-bench-gpt3] [@brown2020
***ML Training Benchmarks***\index{ML Training Benchmarks!definition} measure the **Rate of Convergence** per unit of resource (time, energy, cost).
1. **Significance (Quantitative):** They validate the system's ability to sustain high **Arithmetic Intensity** across distributed accelerators while managing the **Communication Overhead** ($L_{lat}$) of gradient synchronization.
1. **Significance (Quantitative):** They validate the system's ability to sustain high **Arithmetic Intensity** across distributed accelerators while managing the **Communication Overhead** ($L_{\text{lat}}$) of gradient synchronization.
2. **Distinction (Durable):** Unlike **Inference Benchmarks**, which focus on **Input-Output Latency**, Training Benchmarks focus on **Throughput ($\eta$)** and **Total Training Time ($T_{train}$)**.
3. **Common Pitfall:** A frequent misconception is that training benchmarks only measure "how fast the GPU runs." In reality, for large models, the **Interconnect Bandwidth ($BW$)** and the **Fault Tolerance Overhead** are often more critical to the benchmark result than the raw FLOPs.
@@ -2302,7 +2302,7 @@ This is where the optimization chapters converge: the accelerated hardware from
::: {.callout-definition title="ML Inference Benchmarks"}
***ML Inference Benchmarks***\index{ML Inference Benchmarks!definition} quantify the system's ability to meet **Latency Constraints** ($L_{lat}$) under load.
***ML Inference Benchmarks***\index{ML Inference Benchmarks!definition} quantify the system's ability to meet **Latency Constraints** ($L_{\text{lat}}$) under load.
1. **Significance (Quantitative):** They measure the **Tail Latency** (p99) and **Jitter** of the serving stack, validating its suitability for interactive applications.
2. **Distinction (Durable):** Unlike **Training Benchmarks**, which prioritize **Throughput ($\eta$)**, Inference Benchmarks prioritize **Response Time** and **Determinism**.
@@ -2393,7 +2393,7 @@ These measurements form the basis for Service Level Objectives (SLOs) and Servic
***SLOs and SLAs***\index{SLO!definition}\index{SLA!definition} are performance commitment specifications: a Service Level Objective (SLO) is the internal engineering target that the team optimizes toward, while a Service Level Agreement (SLA) is the external contractual threshold whose breach triggers financial penalties.
1. **Significance (Quantitative):** SLOs directly constrain the $L_{lat}$ term in the Iron Law by setting a hard latency ceiling that the serving system must satisfy at a given percentile. A typical production setup sets the SLO at p99 $\leq$ 100 ms and the SLA at p99 $\leq$ 200 ms — the 100 ms headroom constitutes the error budget, allowing the system to absorb transient spikes, maintenance windows, and cascading failures without breaching the customer-facing commitment.
1. **Significance (Quantitative):** SLOs directly constrain the $L_{\text{lat}}$ term in the Iron Law by setting a hard latency ceiling that the serving system must satisfy at a given percentile. A typical production setup sets the SLO at p99 $\leq$ 100 ms and the SLA at p99 $\leq$ 200 ms — the 100 ms headroom constitutes the error budget, allowing the system to absorb transient spikes, maintenance windows, and cascading failures without breaching the customer-facing commitment.
2. **Distinction (Durable):** An SLO is violated internally (triggering a paging alert and an engineering response), while an SLA breach is a contract violation (triggering customer credits or penalties). The SLO must be tighter than the SLA; setting them equal leaves no headroom for measurement variance, deploy windows, or incident response time.
3. **Common Pitfall:** A frequent misconception is that meeting average latency satisfies an SLO. SLOs are defined at tail percentiles (p99, p99.9), not means. A system with 50 ms average latency but 500 ms p99 tail latency violates a 200 ms SLO for 1% of all requests — which at 10,000 requests per second means 100 users per second experiencing unacceptable response times.

34
book/quarto/contents/vol1/conclusion/conclusion.qmd
View File

@@ -186,20 +186,20 @@ The table reveals a pattern: every row's decisions constrain the next row's opti
\index{Twelve Invariants!quantitative framework}
Throughout this book, each Part introduced quantitative principles that govern ML system behavior. The principles are not rules of thumb or best practices that evolve with fashion. They are invariants: constraints rooted in physics, information theory, and statistics. @tbl-twelve-principles collects all twelve in one place, organized by the four Parts that revealed them. The first two columns identify each principle, the third locates where it was introduced, and the final two columns capture its mathematical essence and predictive power.
| **#** | **Principle** | **Part** | **Core Equation / Statement** | **What It Predicts** |
|:--------------------------------|:----------------------------|:---------------|:--------------------------------------------------------------------------------|:-------------------------------------------------------------------------------|
| \ref{pri-data-as-code} | Data as Code Invariant | I: Foundations | System Behavior $\approx f(\text{Data})$ | Changing data changes the program |
| \ref{pri-data-gravity} | Data Gravity Invariant | I: Foundations | $C_{move}(D) \gg C_{move}(\text{Compute})$ | Move compute to data, not data to compute |
| \ref{pri-iron-law} | Iron Law of ML Systems | II: Build | $T = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat}$ | Every optimization pulls one of three levers; reducing one may inflate another |
| \ref{pri-silicon-contract} | Silicon Contract | II: Build | Every architecture bets on which hardware resource it saturates | Mismatched hardware wastes money; matched hardware achieves peak throughput |
| \ref{pri-pareto-frontier} | Pareto Frontier | III: Optimize | Multi-objective optimization; no free improvements | There is no universal optimum; every gain trades against another metric |
| \ref{pri-arithmetic-intensity} | Arithmetic Intensity Law | III: Optimize | $R = \min(R_{peak},\; I \times BW)$ | Adding compute to a memory-bound model yields zero gain |
| \ref{pri-energy-movement} | Energy-Movement Invariant | III: Optimize | $E_{move} \gg E_{compute}$ (100--1,000$\times$) | Data locality, not raw FLOPS, drives efficiency |
| \ref{pri-amdahl} | Amdahl's Law | III: Optimize | $\text{Speedup} = \frac{1}{(1-p) + \frac{p}{s}}$ | The serial fraction caps all parallelism gains |
| \ref{pri-verification-gap} | Verification Gap | IV: Deploy | $P(f(X) \approx Y) > 1 - \epsilon$ | ML testing is statistical; you bound error, not prove correctness |
| \ref{pri-statistical-drift} | Statistical Drift Invariant | IV: Deploy | $\text{Acc}(t) \approx \text{Acc}_0 - \lambda \cdot D(P_t \Vert P_0)$ | Models decay without code changes; the world drifts away from training data |
| \ref{pri-training-serving-skew} | Training-Serving Skew Law | IV: Deploy | $\Delta\text{Acc} \approx \mathbb{E}[\lvert f_{serve}(x) - f_{train}(x)\rvert]$ | Even subtle preprocessing differences silently degrade accuracy |
| \ref{pri-latency-budget} | Latency Budget Invariant | IV: Deploy | P99 is the hard constraint; throughput is optimized within it | Throughput is optimized within the latency envelope, never at its expense |
| **#** | **Principle** | **Part** | **Core Equation / Statement** | **What It Predicts** |
|:--------------------------------|:----------------------------|:---------------|:----------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------|
| \ref{pri-data-as-code} | Data as Code Invariant | I: Foundations | System Behavior $\approx f(\text{Data})$ | Changing data changes the program |
| \ref{pri-data-gravity} | Data Gravity Invariant | I: Foundations | $C_{move}(D) \gg C_{move}(\text{Compute})$ | Move compute to data, not data to compute |
| \ref{pri-iron-law} | Iron Law of ML Systems | II: Build | $T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$ | Every optimization pulls one of three levers; reducing one may inflate another |
| \ref{pri-silicon-contract} | Silicon Contract | II: Build | Every architecture bets on which hardware resource it saturates | Mismatched hardware wastes money; matched hardware achieves peak throughput |
| \ref{pri-pareto-frontier} | Pareto Frontier | III: Optimize | Multi-objective optimization; no free improvements | There is no universal optimum; every gain trades against another metric |
| \ref{pri-arithmetic-intensity} | Arithmetic Intensity Law | III: Optimize | $R = \min(R_{\text{peak}},\; I \times BW)$ | Adding compute to a memory-bound model yields zero gain |
| \ref{pri-energy-movement} | Energy-Movement Invariant | III: Optimize | $E_{move} \gg E_{compute}$ (100--1,000$\times$) | Data locality, not raw FLOPS, drives efficiency |
| \ref{pri-amdahl} | Amdahl's Law | III: Optimize | $\text{Speedup} = \frac{1}{(1-p) + \frac{p}{s}}$ | The serial fraction caps all parallelism gains |
| \ref{pri-verification-gap} | Verification Gap | IV: Deploy | $P(f(X) \approx Y) > 1 - \epsilon$ | ML testing is statistical; you bound error, not prove correctness |
| \ref{pri-statistical-drift} | Statistical Drift Invariant | IV: Deploy | $\text{Acc}(t) \approx \text{Acc}_0 - \lambda \cdot D(P_t \Vert P_0)$ | Models decay without code changes; the world drifts away from training data |
| \ref{pri-training-serving-skew} | Training-Serving Skew Law | IV: Deploy | $\Delta\text{Acc} \approx \mathbb{E}[\lvert f_{serve}(x) - f_{train}(x)\rvert]$ | Even subtle preprocessing differences silently degrade accuracy |
| \ref{pri-latency-budget} | Latency Budget Invariant | IV: Deploy | P99 is the hard constraint; throughput is optimized within it | Throughput is optimized within the latency envelope, never at its expense |
: **The Twelve Quantitative Invariants of ML Systems Engineering.** Each invariant was introduced in the Part where its governing constraint first becomes visible. Together, they form the complete analytical framework for reasoning about ML system design, optimization, and deployment. The meta-principle that unifies them all is the Conservation of Complexity: you cannot destroy complexity, only move it between Data, Algorithm, and Machine. {#tbl-twelve-principles tbl-colwidths="[5,18,12,33,32]"}
@@ -568,9 +568,9 @@ We can apply the Iron Law (Principle \ref{pri-iron-law}) and Arithmetic Intensit
**The Physics:**
- **Model Size** ($D_{vol}$): `{python} ConclusionRoofline.llama_params_str` params$\times$ 2 bytes (FP16) = `{python} ConclusionRoofline.llama_dvol_gb_str` GB.
- **Model Size** ($D_{\text{vol}}$): `{python} ConclusionRoofline.llama_params_str` params$\times$ 2 bytes (FP16) = `{python} ConclusionRoofline.llama_dvol_gb_str` GB.
- **Compute** ($O$): ≈ 2$\times$ P per token = `{python} ConclusionRoofline.llama_compute_gflops_str` GFLOPs.
- **Hardware:** H100 with $BW$ = `{python} ConclusionRoofline.h100_bw_tb_str` TB/s, $R_{peak} \approx$ `{python} ConclusionRoofline.h100_peak_tflops_str` TFLOPS FP16.
- **Hardware:** H100 with $BW$ = `{python} ConclusionRoofline.h100_bw_tb_str` TB/s, $R_{\text{peak}} \approx$ `{python} ConclusionRoofline.h100_peak_tflops_str` TFLOPS FP16.
**The Calculation:**
@@ -579,7 +579,7 @@ We can apply the Iron Law (Principle \ref{pri-iron-law}) and Arithmetic Intensit
**The Systems Insight:**
The memory time $T_{mem}$ is `{python} ConclusionRoofline.ratio_str`$\times$ larger than compute time $T_{comp}$. The system is heavily memory-bound (arithmetic intensity $\approx$ 1). To honor the Silicon Contract, we must either increase Arithmetic Intensity (via batching users to reuse $D_{vol}$) or reduce Data Volume (via quantization to INT4). A systems engineer who optimizes compute kernels ($T_{comp}$) without addressing memory ($T_{mem}$) wastes 100% of their effort.
The memory time $T_{mem}$ is `{python} ConclusionRoofline.ratio_str`$\times$ larger than compute time $T_{comp}$. The system is heavily memory-bound (arithmetic intensity $\approx$ 1). To honor the Silicon Contract, we must either increase Arithmetic Intensity (via batching users to reuse $D_{\text{vol}}$) or reduce Data Volume (via quantization to INT4). A systems engineer who optimizes compute kernels ($T_{comp}$) without addressing memory ($T_{mem}$) wastes 100% of their effort.
:::

10
book/quarto/contents/vol1/data_engineering/data_engineering.qmd
View File

@@ -267,7 +267,7 @@ The workflow stages from @sec-ml-workflow establish the *when* and *why* of data
***Data Engineering***\index{Data Engineering!definition} is the infrastructure layer that manages the lifecycle of data from source to model, encompassing acquisition, transformation, storage, and governance.
1. **Significance (Quantitative):** Its critical function is ensuring **Training-Serving Consistency**, preventing **Silent Degradation** by decoupling the model from the volatility of raw data. Within the **Iron Law**, it governs the **Data Volume ($D_{vol}$)** and ensures that it remains representative of the target distribution.
1. **Significance (Quantitative):** Its critical function is ensuring **Training-Serving Consistency**, preventing **Silent Degradation** by decoupling the model from the volatility of raw data. Within the **Iron Law**, it governs the **Data Volume ($D_{\text{vol}}$)** and ensures that it remains representative of the target distribution.
2. **Distinction (Durable):** Unlike **Data Science**, which focuses on **Inference and Insight**, Data Engineering addresses the **Scalability and Reliability** of the data pipeline.
3. **Common Pitfall:** A frequent misconception is that Data Engineering is "data cleaning." In reality, it is **Dataset Compilation**: transforming raw, noisy observations into an optimized binary that the model consumes.
@@ -287,9 +287,9 @@ Before any of these pipeline stages can be designed well, however, we need to un
### Data Gravity {#sec-data-engineering-data-gravity-adcb}
\index{Data Gravity!physics of} Data gravity is the cost of movement. It is a function of volume ($D_{vol}$) and network bandwidth ($BW$). The time to move a petabyte dataset across a 10 Gbps link is fixed by physics (`{python} DataEngineeringSetup.transfer_time_10g_md`); the "Physics of Data Gravity" callout below quantifies these transfer times for a 100 Gbps link. This gravity dictates architecture: because moving 1PB to the compute is slow and expensive, we must move the compute to the data. This explains the rise of "Data Lakehouse" architectures[^fn-lakehouse-gravity]\index{Data Lakehouse!architecture} [@armbrust2021lakehouse] where processing engines (Spark, Presto) run directly on storage nodes. In contrast, **Data Mesh**\index{Data Mesh!decentralized ownership} [@dehghani2022data] proposes decentralizing ownership to manage this scale organizationally, treating data as a product owned by domain teams.
\index{Data Gravity!physics of} Data gravity is the cost of movement. It is a function of volume ($D_{\text{vol}}$) and network bandwidth ($BW$). The time to move a petabyte dataset across a 10 Gbps link is fixed by physics (`{python} DataEngineeringSetup.transfer_time_10g_md`); the "Physics of Data Gravity" callout below quantifies these transfer times for a 100 Gbps link. This gravity dictates architecture: because moving 1PB to the compute is slow and expensive, we must move the compute to the data. This explains the rise of "Data Lakehouse" architectures[^fn-lakehouse-gravity]\index{Data Lakehouse!architecture} [@armbrust2021lakehouse] where processing engines (Spark, Presto) run directly on storage nodes. In contrast, **Data Mesh**\index{Data Mesh!decentralized ownership} [@dehghani2022data] proposes decentralizing ownership to manage this scale organizationally, treating data as a product owned by domain teams.
[^fn-lakehouse-gravity]: **Data Lakehouse**: Combines data lake storage (cheap, schema-less) with warehouse query semantics (ACID transactions, schema enforcement) using open table formats like Delta Lake and Apache Iceberg. For ML workloads, the lakehouse eliminates the ETL copy between lake and warehouse, enabling direct feature computation on the storage layer where data already resides -- a direct response to data gravity, since moving petabytes to a separate warehouse doubles the $D_{vol}/BW$ cost. \index{Data Lakehouse!data gravity}
[^fn-lakehouse-gravity]: **Data Lakehouse**: Combines data lake storage (cheap, schema-less) with warehouse query semantics (ACID transactions, schema enforcement) using open table formats like Delta Lake and Apache Iceberg. For ML workloads, the lakehouse eliminates the ETL copy between lake and warehouse, enabling direct feature computation on the storage layer where data already resides -- a direct response to data gravity, since moving petabytes to a separate warehouse doubles the $D_{\text{vol}}/BW$ cost. \index{Data Lakehouse!data gravity}
### Information Entropy {#sec-data-engineering-information-entropy-5622}
@@ -1179,7 +1179,7 @@ The contrast matters: **weeks** for human labeling, **hours** for GPU training,
All cost figures reflect approximate 2024 cloud provider rates and are intended to convey relative magnitudes rather than exact pricing.[^fn-pricing-cost-ratios]
[^fn-pricing-cost-ratios]: **Pricing Ratios**: The absolute dollar amounts in this chapter will shift with provider pricing, but the ratios between tiers are remarkably stable because they reflect physical constraints, not business decisions. S3 Glacier retrieval illustrates the pattern: standard (\$0.01/GB, 3--5 hours), expedited (\$0.03/GB, 1--5 minutes), and bulk (free, 5--12 hours) span a 30$\times$ cost range that maps directly to the $D_{vol}/BW$ trade-off in the Iron Law. Engineers who memorize ratios rather than prices make storage decisions that survive the next pricing revision. \index{Cloud Pricing!cost-latency trade-off}
[^fn-pricing-cost-ratios]: **Pricing Ratios**: The absolute dollar amounts in this chapter will shift with provider pricing, but the ratios between tiers are remarkably stable because they reflect physical constraints, not business decisions. S3 Glacier retrieval illustrates the pattern: standard (\$0.01/GB, 3--5 hours), expedited (\$0.03/GB, 1--5 minutes), and bulk (free, 5--12 hours) span a 30$\times$ cost range that maps directly to the $D_{\text{vol}}/BW$ trade-off in the Iron Law. Engineers who memorize ratios rather than prices make storage decisions that survive the next pricing revision. \index{Cloud Pricing!cost-latency trade-off}
::: {.callout-checkpoint title="Four Pillars Framework"}
@@ -3086,7 +3086,7 @@ $$\text{Data Supply Rate} = \text{Storage Bandwidth} \times (1 - \text{Overhead}
When storage bandwidth becomes the limiting factor, teams must either improve storage performance through faster media, parallelization, or caching, or reduce data movement requirements through compression, quantization, or architectural changes. Large language model training may require processing hundreds of gigabytes of text per hour, while computer vision models processing high-resolution imagery can demand sustained data rates exceeding 50 gigabytes per second across distributed clusters. These requirements explain the rise of specialized ML storage systems optimizing data loading pipelines: PyTorch DataLoader with multiple worker processes parallelizing I/O, TensorFlow tf.data API with prefetching and caching, and frameworks like NVIDIA DALI (Data Loading Library) that offload data augmentation to GPUs rather than loading pre-augmented data from storage.
File format selection dramatically impacts the **Data Term** ($\frac{D_{vol}}{BW}$) of the Iron Law. We can quantify this impact as *format efficiency* ($\eta_{format}$), which acts as a multiplier on effective bandwidth.
File format selection dramatically impacts the **Data Term** ($\frac{D_{\text{vol}}}{BW}$) of the Iron Law. We can quantify this impact as *format efficiency* ($\eta_{format}$), which acts as a multiplier on effective bandwidth.
```{python}
#| label: format-efficiency-calc

8
book/quarto/contents/vol1/data_selection/data_selection.qmd
View File

@@ -186,7 +186,7 @@ Each stage increases the *information density* of the data that reaches the mode
***Data Selection***\index{Data Selection!definition} is the process of maximizing the **Information-Compute Ratio** of a training dataset.
1. **Significance (Quantitative):** It identifies the smallest subset of data sufficient to define the decision boundary, reducing the **Total Operations ($O$)** of the Iron Law by eliminating redundant or noisy samples ($D_{vol}$) before they consume GPU cycles.
1. **Significance (Quantitative):** It identifies the smallest subset of data sufficient to define the decision boundary, reducing the **Total Operations ($O$)** of the Iron Law by eliminating redundant or noisy samples ($D_{\text{vol}}$) before they consume GPU cycles.
2. **Distinction (Durable):** Unlike **Data Engineering**, which focuses on the **Cleanliness** and **Consistency** of data, Data Selection focuses on the **Informativeness** and **Diversity** of the samples.
3. **Common Pitfall:** A frequent misconception is that more data is always better. In reality, it is the **Quality of the Samples** that matters: adding 10$\times$ more low-quality data may yield less accuracy than 1.1$\times$ carefully selected, high-quality data.
@@ -324,7 +324,7 @@ class IronLawSavings:
In the **Iron Law of ML Systems** ($T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$), data selection is the only technique that reduces the *Total Operations* term at its source. Model compression reduces operations per sample; hardware acceleration increases throughput per operation. Data selection, by contrast, reduces the number of samples processed entirely.
- **Model compression**: Reduces $O$ per forward/backward pass
- **Hardware acceleration**: Increases $R_{peak}$ (peak throughput) and $\eta$ (utilization)
- **Hardware acceleration**: Increases $R_{\text{peak}}$ (peak throughput) and $\eta$ (utilization)
- **Data selection**: Reduces the number of passes through the entire equation
\index{Iron Law!multiplicative savings from data selection}
@@ -607,7 +607,7 @@ $$\text{ICR}(D) = \frac{\frac{d}{dD} I(D)}{\frac{d}{dD} C(D)} \approx \frac{1/D}
The $1/(O \cdot D)$ decay creates what we call **The Data Wall**\index{Data Wall!zero learning signal}. Beyond the frontier, adding more data yields near-zero learning but still costs linear compute. In this regime, data is no longer an asset; it is a **Data Tax**\index{Data Tax!redundant compute cost} that inflates the $O$ term of the **Iron Law** without improving the accuracy numerator of the **RoC** (Return on Compute, see @sec-introduction-roc-invariant). A systems engineer's goal is to keep the system operating at the "Knee" of the ICR curve, where the learning signal per FLOP is maximized. The static and dynamic selection techniques that follow are designed to achieve exactly that.
\index{Information-Compute Ratio!equivalence to hardware speedup}
As detailed in the "Data Selection and the Iron Law" callout above, data selection turns the Total Operations ($O$) term from a fixed constant into a variable. By maximizing ICR, we reduce the total FLOPs required to reach a target performance level. A 2$\times$ improvement in ICR is mathematically equivalent to a 2$\times$ improvement in hardware Peak Throughput ($R_{peak}$), but often much cheaper to achieve. ICR focuses specifically on the compute component of the broader Selection Efficiency metric defined earlier, which also accounts for acquisition, labeling, and storage costs.
As detailed in the "Data Selection and the Iron Law" callout above, data selection turns the Total Operations ($O$) term from a fixed constant into a variable. By maximizing ICR, we reduce the total FLOPs required to reach a target performance level. A 2$\times$ improvement in ICR is mathematically equivalent to a 2$\times$ improvement in hardware Peak Throughput ($R_{\text{peak}}$), but often much cheaper to achieve. ICR focuses specifically on the compute component of the broader Selection Efficiency metric defined earlier, which also accounts for acquisition, labeling, and storage costs.
A random batch of raw data often has low ICR: it contains redundant examples, noisy samples, or "easy" examples the model has already mastered, wasting GPU cycles on zero-information updates. High-efficiency data pipelines (@fig-data-selection-pipeline) filter, order, and synthesize data to maximize ICR, ensuring that every FLOP contributes to learning. To illustrate, consider *computing ICR* on a concrete coreset selection task. Later in this chapter, @sec-data-selection-measurement-framework-733b provides the complete measurement framework for evaluating these efficiency gains, including the compute-optimal frontier diagnostic that determines whether training is data-starved or compute-starved.
@@ -2490,7 +2490,7 @@ The techniques in this chapter are not mutually exclusive; in practice, the most
*Each stage compounds the efficiency gains of previous stages, turning individual percentage improvements into multiplicative savings.*
The decision framework above answers the *what* of data selection: which samples to prune, when to select dynamically, and how to synthesize new data. Understanding these algorithmic choices is essential, but algorithms alone do not translate into faster training. A perfectly designed coreset algorithm that takes 10 hours to select samples for a 2-hour training run yields no practical benefit. Similarly, a curriculum learning strategy that requires scanning the entire dataset to determine difficulty rankings may idle GPUs while CPUs compute scores. The *how* of implementation matters as much as the *what* of algorithm choice. Concretely, a 2$\times$ improvement in the Information-Compute Ratio (ICR) is mathematically equivalent to doubling the hardware's peak throughput ($R_{peak}$) for that training run.
The decision framework above answers the *what* of data selection: which samples to prune, when to select dynamically, and how to synthesize new data. Understanding these algorithmic choices is essential, but algorithms alone do not translate into faster training. A perfectly designed coreset algorithm that takes 10 hours to select samples for a 2-hour training run yields no practical benefit. Similarly, a curriculum learning strategy that requires scanning the entire dataset to determine difficulty rankings may idle GPUs while CPUs compute scores. The *how* of implementation matters as much as the *what* of algorithm choice. Concretely, a 2$\times$ improvement in the Information-Compute Ratio (ICR) is mathematically equivalent to doubling the hardware's peak throughput ($R_{\text{peak}}$) for that training run.
The gap between algorithmic elegance and practical value raises several systems challenges: preventing selection overhead from negating theoretical gains, handling non-sequential I/O patterns that confuse prefetching logic, and coordinating selection decisions across distributed workers without introducing synchronization bottlenecks. The engineering patterns that follow bridge the gap between data selection theory and production reality.

66
book/quarto/contents/vol1/frameworks/frameworks.qmd
View File

@@ -130,9 +130,9 @@ In the context of the **Iron Law** (@sec-introduction-iron-law-ml-systems-c32a),
Your "Source Code" is the model architecture (the **$O$** term). The framework's job is to take this high-level math and compile it into a series of hardware-specific kernel launches that:
1. Minimize **Data Movement ($D_{vol}$)** through techniques like kernel fusion.
1. Minimize **Data Movement ($D_{\text{vol}}$)** through techniques like kernel fusion.
2. Maximize **Utilization ($\eta$)** by matching operations to specialized hardware units like Tensor Cores.
3. Minimize **Overhead ($L_{lat}$)** through efficient asynchronous dispatch and graph capture.
3. Minimize **Overhead ($L_{\text{lat}}$)** through efficient asynchronous dispatch and graph capture.
Choosing a framework means choosing the compiler that determines *how* efficiently a model uses hardware.
@@ -144,7 +144,7 @@ With these three problems in mind, we can now define *what* a machine learning f
***Machine Learning Frameworks***\index{ML Framework!definition} are software systems that translate high-level mathematical model definitions into hardware-optimized execution plans by managing the computational graph, automatic differentiation, kernel dispatch, and memory allocation across the hardware hierarchy.
1. **Significance (Quantitative):** Frameworks directly determine the system efficiency ($\eta$) term in the Iron Law. XLA's operator fusion, for example, eliminates intermediate memory writes between consecutive elementwise operations: fusing a matrix multiplication, bias add, and ReLU into a single kernel reduces the total data movement ($D_{vol}$) by 2–3$\times$ versus three separate kernel launches, yielding observed end-to-end speedups of 1.5–2$\times$ on Transformer training without any model changes.
1. **Significance (Quantitative):** Frameworks directly determine the system efficiency ($\eta$) term in the Iron Law. XLA's operator fusion, for example, eliminates intermediate memory writes between consecutive elementwise operations: fusing a matrix multiplication, bias add, and ReLU into a single kernel reduces the total data movement ($D_{\text{vol}}$) by 2–3$\times$ versus three separate kernel launches, yielding observed end-to-end speedups of 1.5–2$\times$ on Transformer training without any model changes.
2. **Distinction (Durable):** Unlike a numerical library such as NumPy, which executes each operation immediately (eager evaluation), an ML framework can defer execution to analyze the full computational graph and apply global optimizations: operator fusion, memory layout transformations, and parallel scheduling. These optimizations are impossible when operations are evaluated one at a time.
3. **Common Pitfall:** A frequent misconception is that frameworks are interchangeable API wrappers. Framework choice determines which hardware optimizations are available: a PyTorch model using the default eager execution mode cannot benefit from XLA's graph-level fusion until explicitly compiled with `torch.compile()`, and the resulting throughput difference can exceed 2$\times$ on the same hardware.
@@ -164,7 +164,7 @@ In 1979, writing a matrix multiplication in Fortran that saturated the hardware
1. **Solving Performance (1979–1992)**: The **Basic Linear Algebra Subprograms (BLAS)**\index{BLAS!historical foundation}[^fn-blas-performance] and **LAPACK**[^fn-lapack-algebra] solved the problem of *Hardware Primitives*. They provided standardized, highly optimized implementations of matrix operations (like GEMM[^fn-gemm-utilization]). This layer ensures that `C = A @ B` runs at near-peak silicon speed, regardless of the language calling it.
[^fn-gemm-utilization]: **GEMM (General Matrix Multiply)**: The single operation that the "near-peak silicon speed" claim rests on. Hardware vendors hand-tune GEMM for their specific chips because every layer in a neural network reduces to matrix multiplication, making this one routine the performance floor for all frameworks above it on the ladder. The catch: GEMM achieves peak throughput only when matrix dimensions satisfy strict alignment constraints (multiples of 8 for NVIDIA Tensor Cores), and violating these rules drops a framework from over 90% to roughly 30% of $R_{peak}$. \index{GEMM!hardware utilization}
[^fn-gemm-utilization]: **GEMM (General Matrix Multiply)**: The single operation that the "near-peak silicon speed" claim rests on. Hardware vendors hand-tune GEMM for their specific chips because every layer in a neural network reduces to matrix multiplication, making this one routine the performance floor for all frameworks above it on the ladder. The catch: GEMM achieves peak throughput only when matrix dimensions satisfy strict alignment constraints (multiples of 8 for NVIDIA Tensor Cores), and violating these rules drops a framework from over 90% to roughly 30% of $R_{\text{peak}}$. \index{GEMM!hardware utilization}
[^fn-lapack-algebra]: **LAPACK (Linear Algebra PACKage)**: Extends BLAS by providing a standard API for higher-level routines (SVD, eigendecomposition, least-squares) that vendors implement with chip-specific code layered on top of fast GEMM kernels. This layered design is the architectural pattern every ML framework inherits: high-level operations delegate downward to hand-tuned primitives, so a vendor-optimized LAPACK call can execute over 10$\times$ faster than a naive implementation without the framework author writing a single line of hardware-specific code. \index{LAPACK!ML initialization}
@@ -304,7 +304,7 @@ The memory wall creates a critical classification: operations are either **compu
The key optimization for memory-bound operations is **kernel fusion**\index{Kernel Fusion!optimizing memory-bound ops}, combining multiple operations into a single GPU function (called a *kernel*)[^fn-kernel-gpu-dispatch] to avoid intermediate memory traffic. Fusing a sequence of LayerNorm, Dropout, and ReLU into one kernel can yield 5$\times$ speedup by eliminating intermediate writes between operations. FlashAttention[^fn-flashattention-fusion-fw] fuses the entire attention computation, reducing HBM traffic by 10--20$\times$ and achieving 2--4$\times$ wall-clock speedup.
[^fn-kernel-gpu-dispatch]: **Kernel (GPU)**: In GPU programming, a kernel is the function dispatched to execute in parallel across thousands of threads. Each kernel launch incurs 5--20 $\mu$s of CPU-side overhead for parameter assembly and GPU signaling, which means that small, unfused operations spend more time on launch overhead ($L_{lat}$) than on useful arithmetic. Reducing kernel count through fusion is therefore a direct attack on the overhead term of the Iron Law. \index{Kernel!GPU dispatch overhead}
[^fn-kernel-gpu-dispatch]: **Kernel (GPU)**: In GPU programming, a kernel is the function dispatched to execute in parallel across thousands of threads. Each kernel launch incurs 5--20 $\mu$s of CPU-side overhead for parameter assembly and GPU signaling, which means that small, unfused operations spend more time on launch overhead ($L_{\text{lat}}$) than on useful arithmetic. Reducing kernel count through fusion is therefore a direct attack on the overhead term of the Iron Law. \index{Kernel!GPU dispatch overhead}
[^fn-flashattention-fusion-fw]: **FlashAttention**: Kernel fusion taken to its logical extreme, fusing the entire attention computation (Q, K, V projections, softmax, output) into a single kernel that tiles data to fit in SRAM (introduced in @sec-network-architectures). By reducing HBM traffic 10--20$\times$, FlashAttention transforms a memory-bound operation into a compute-bound one, demonstrating that framework-level fusion can shift an operation's position on the Roofline Model from bandwidth-limited to throughput-limited. \index{FlashAttention!kernel fusion}
@@ -573,7 +573,7 @@ The dynamic autograd tape enables capabilities impossible with static graphs. Co
##### Systems Implications: Overhead {.unnumbered}
This flexibility comes with performance costs that map directly to the Iron Law (@sec-introduction-iron-law-ml-systems-c32a). Each forward pass rebuilds the autograd tape from scratch, adding Python object creation, reference counting, and node linking overhead to $L_{lat}$ on every iteration. Every operation goes through Python dispatch---function lookup, argument parsing, type checking---costing ~10μs per operation, which becomes significant for models with thousands of operations. Because the graph is built during execution, the framework cannot see across operations to fuse kernels, so each operation launches its own GPU kernel, inflating both $O$ and $D_{vol}$. The autograd tape itself stores references to all intermediate tensors and `Function` nodes, increasing memory consumption by 2--3$\times$ compared to forward-only execution and adding pressure to $D_{vol}$. Together, these costs create a performance ceiling that becomes visible as models grow smaller and dispatch overhead dominates computation.
This flexibility comes with performance costs that map directly to the Iron Law (@sec-introduction-iron-law-ml-systems-c32a). Each forward pass rebuilds the autograd tape from scratch, adding Python object creation, reference counting, and node linking overhead to $L_{\text{lat}}$ on every iteration. Every operation goes through Python dispatch---function lookup, argument parsing, type checking---costing ~10μs per operation, which becomes significant for models with thousands of operations. Because the graph is built during execution, the framework cannot see across operations to fuse kernels, so each operation launches its own GPU kernel, inflating both $O$ and $D_{\text{vol}}$. The autograd tape itself stores references to all intermediate tensors and `Function` nodes, increasing memory consumption by 2--3$\times$ compared to forward-only execution and adding pressure to $D_{\text{vol}}$. Together, these costs create a performance ceiling that becomes visible as models grow smaller and dispatch overhead dominates computation.
For a typical ResNet-50 forward pass, eager execution overhead adds approximately 5--10 ms compared to an optimized compiled version, with the majority spent in Python dispatch and tape construction rather than actual computation.
@@ -722,9 +722,9 @@ The key difference from eager execution is that during construction, `x`, `y`, a
\index{Ahead-of-Time (AOT) Optimization}
Because the framework has the complete graph before execution, it can perform optimizations impossible in eager mode. The kernel fusion\index{Kernel Fusion!static graph optimization} opportunity introduced in @sec-ml-frameworks-execution-strategy-matters-memory-wall-1ce8 becomes actionable here: because the framework sees `y = x * 2` and `z = y + 1` together in the graph, it can fuse them into `z = x * 2 + 1`, eliminating the intermediate `y` and halving memory traffic. With the full graph visible, the compiler can also calculate exact memory requirements for all tensors before execution, pre-allocating memory in a single pass and reusing buffers where lifetimes do not overlap. Tensor layouts can be transformed globally (e.g., NCHW to NHWC) to match hardware preferences without runtime copying. Dead code elimination[^fn-dce-graph-optimization]\index{Dead Code Elimination} removes operations whose results are never consumed, and constant folding\index{Constant Folding} pre-computes operations on constant values at graph construction time, so the cost is paid once rather than on every forward pass.
[^fn-dce-graph-optimization]: **Dead Code Elimination (DCE)**: Removes graph nodes whose results are never consumed by any downstream operation. In ML graphs, dead code arises from debugging operations left in production (print nodes, assertions), unused conditional branches, and gradient computations for frozen layers. For large transformer models, DCE eliminates 5--15% of graph nodes, reducing both $O$ (fewer operations) and $L_{lat}$ (fewer kernel launches). The DAG structure makes this safe: the framework verifies no downstream node depends on a candidate before removing it. \index{Dead Code Elimination!graph optimization}
[^fn-dce-graph-optimization]: **Dead Code Elimination (DCE)**: Removes graph nodes whose results are never consumed by any downstream operation. In ML graphs, dead code arises from debugging operations left in production (print nodes, assertions), unused conditional branches, and gradient computations for frozen layers. For large transformer models, DCE eliminates 5--15% of graph nodes, reducing both $O$ (fewer operations) and $L_{\text{lat}}$ (fewer kernel launches). The DAG structure makes this safe: the framework verifies no downstream node depends on a candidate before removing it. \index{Dead Code Elimination!graph optimization}
These optimizations map directly to **Iron Law** terms: kernel fusion reduces $D_{vol}$ by eliminating intermediate memory writes, constant folding reduces $O$ by computing values once, memory pre-allocation reduces $L_{lat}$ by avoiding runtime allocation overhead, and dead code elimination reduces both $O$ and $D_{vol}$. Concretely, in large Transformer models, constant folding and dead code elimination can reduce total FLOPs by `{python} GraphOptimizationStats.flop_reduction_range_str` before the first batch even arrives.
These optimizations map directly to **Iron Law** terms: kernel fusion reduces $D_{\text{vol}}$ by eliminating intermediate memory writes, constant folding reduces $O$ by computing values once, memory pre-allocation reduces $L_{\text{lat}}$ by avoiding runtime allocation overhead, and dead code elimination reduces both $O$ and $D_{\text{vol}}$. Concretely, in large Transformer models, constant folding and dead code elimination can reduce total FLOPs by `{python} GraphOptimizationStats.flop_reduction_range_str` before the first batch even arrives.
\index{XLA (Accelerated Linear Algebra)!definition}
Compilation frameworks like XLA (Accelerated Linear Algebra)\index{XLA (Accelerated Linear Algebra)!graph compilation}[^fn-xla-compiler] [@GoogleXLA] take this further, compiling the TensorFlow graph to optimized machine code for specific hardware. For a transformer encoder block, XLA can achieve 1.5--2$\times$ speedup over unoptimized execution through aggressive fusion and hardware-specific code generation.
@@ -743,7 +743,7 @@ Can we have both eager debugging and graph optimization? JIT compilation attempt
[^fn-ir-compilation]: **Intermediate Representation (IR)**: The "intermediate" captures this format's architectural role: a language-independent layer that decouples the frontend (Python capture) from the backend (hardware code generation), exactly as LLVM IR decouples C/Rust/Swift frontends from x86/ARM backends. ML frameworks adopted this compiler pattern because it reduces the $O(M \times N)$ cost of supporting $M$ frontends and $N$ backends to $O(M + N)$: a single graph capture mechanism (TorchDynamo, tf2xla) can target multiple hardware backends without rewriting the capture logic. \index{Intermediate Representation!compiler pattern}
The eager-versus-compiled trade-off has a direct **Iron Law** consequence. JIT compilation amortizes the $L_{lat}$ (dispatch overhead) across the compiled region. Longer compiled regions mean more overhead amortized per operation, which explains why graph breaks are performance-critical: each break forces a return to eager dispatch, resetting the amortization.
The eager-versus-compiled trade-off has a direct **Iron Law** consequence. JIT compilation amortizes the $L_{\text{lat}}$ (dispatch overhead) across the compiled region. Longer compiled regions mean more overhead amortized per operation, which explains why graph breaks are performance-critical: each break forces a return to eager dispatch, resetting the amortization.
PyTorch's TorchScript exemplifies both strategies. Tracing\index{JIT Compilation!tracing} executes a function once with example inputs and records every tensor operation into a static computation graph. @lst-torchscript-trace demonstrates the approach: the traced module becomes a compiled artifact that can be serialized, optimized, and executed independently of the Python interpreter:
@@ -1077,7 +1077,7 @@ The natural question is: can this fusion happen automatically? PyTorch 2.0's `to
[^fn-torch-compile-hybrid]: **`torch.compile`**: It enables this automatic fusion by intercepting Python bytecode (via TorchDynamo) to extract a computational graph from unmodified eager code. This graph is then compiled into optimized kernels, trading a one-time compilation delay for a permanent 1.3--$2\times$ throughput gain on transformer models by reducing kernel launch overhead. \index{torch.compile!hybrid execution}
[^fn-cuda-dispatch-overhead]: **CUDA (Compute Unified Device Architecture)**: NVIDIA's parallel computing platform (2007) serving as the foundational layer between high-level Python operations and GPU silicon. When PyTorch executes `torch.matmul(A, B)`, the call traverses the framework's dispatcher, selects a cuBLAS kernel, and launches it on the GPU. Each launch incurs 5--20 $\mu$s of CPU-side overhead. For small operations, this dispatch overhead ($L_{lat}$) exceeds the useful compute time, which is why compilation (fusing $N$ operations into one kernel launch) yields speedups proportional to the reduction in launch count rather than the reduction in arithmetic. \index{CUDA!dispatch overhead}
[^fn-cuda-dispatch-overhead]: **CUDA (Compute Unified Device Architecture)**: NVIDIA's parallel computing platform (2007) serving as the foundational layer between high-level Python operations and GPU silicon. When PyTorch executes `torch.matmul(A, B)`, the call traverses the framework's dispatcher, selects a cuBLAS kernel, and launches it on the GPU. Each launch incurs 5--20 $\mu$s of CPU-side overhead. For small operations, this dispatch overhead ($L_{\text{lat}}$) exceeds the useful compute time, which is why compilation (fusing $N$ operations into one kernel launch) yields speedups proportional to the reduction in launch count rather than the reduction in arithmetic. \index{CUDA!dispatch overhead}
##### Architecture: Three-Stage Compilation Pipeline {.unnumbered}
@@ -1319,16 +1319,16 @@ These throughput differences across execution modes raise a practical question
The optimal framework execution strategy depends on which **Iron Law** term dominates the workload. @tbl-framework-archetype-strategy aligns each archetype to its recommended execution strategy:
| **Archetype** | **Dominant Iron Law Term** | **Optimal Framework Strategy** | **Rationale** |
|:----------------------|:------------------------------------------|:-----------------------------------|:-------------------------------------------|
| **ResNet-50** | $\frac{O}{R_{peak} \cdot \eta}$ (Compute) | **TensorRT** (inference) | Kernel fusion maximizes MFU; compute-bound |
| **(Compute Beast)** | | **torch.compile** (training) | workloads benefit most from optimization |
| **GPT-2** | $\frac{D_{vol}}{BW}$ (Memory Bandwidth) | **torch.compile** | Kernel fusion reduces HBM round-trips; |
| **(Bandwidth Hog)** | | | keeps data in cache to mitigate bandwidth |
| **DLRM** | $\frac{D_{vol}}{BW}$ (Random Access) + | **Eager** with specialized kernels | Embedding lookups are inherently irregular |
| **(Sparse Scatter)** | $T_{network}$ | (FBGEMM) | and dynamic; compilation gains are small |
| **DS-CNN** | $L_{lat}$ (Overhead) | **AOT compilation** (TFLite, ONNX) | Sub-ms inference; every microsecond of |
| **(Tiny Constraint)** | | | Python overhead is unacceptable |
| **Archetype** | **Dominant Iron Law Term** | **Optimal Framework Strategy** | **Rationale** |
|:----------------------|:-------------------------------------------------|:-----------------------------------|:-------------------------------------------|
| **ResNet-50** | $\frac{O}{R_{\text{peak}} \cdot \eta}$ (Compute) | **TensorRT** (inference) | Kernel fusion maximizes MFU; compute-bound |
| **(Compute Beast)** | | **torch.compile** (training) | workloads benefit most from optimization |
| **GPT-2** | $\frac{D_{\text{vol}}}{BW}$ (Memory Bandwidth) | **torch.compile** | Kernel fusion reduces HBM round-trips; |
| **(Bandwidth Hog)** | | | keeps data in cache to mitigate bandwidth |
| **DLRM** | $\frac{D_{\text{vol}}}{BW}$ (Random Access) + | **Eager** with specialized kernels | Embedding lookups are inherently irregular |
| **(Sparse Scatter)** | $T_{network}$ | (FBGEMM) | and dynamic; compilation gains are small |
| **DS-CNN** | $L_{\text{lat}}$ (Overhead) | **AOT compilation** (TFLite, ONNX) | Sub-ms inference; every microsecond of |
| **(Tiny Constraint)** | | | Python overhead is unacceptable |
: **Framework Execution Strategy by Workload.** Recommended execution strategy for each workload archetype, aligned to the dominant Iron Law term. Compute-bound workloads benefit most from compilation, while irregular access patterns favor eager execution. {#tbl-framework-archetype-strategy}
@@ -1929,7 +1929,7 @@ Every activation saved for the backward pass persists in memory until consumed b
# │
# │ Goal: Quantify ResNet-50 training vs. inference memory (25.6 M params,
# │ ~102 MB FP32 weights, 10–15 GB training footprint) to show the ~100×
# │ ratio driving the $D_{vol}$ term in the Iron Law.
# │ ratio driving the $D_{\text{vol}}$ term in the Iron Law.
# │ Show: "~102 MB inference vs. 10–15 GB training" — inline in Principle 2 prose
# │ and in "The Administrative Tax" callout (@sec-ml-frameworks-tensor-structure-dimensions-4a14).
# │ How: model.size_in_bytes() helpers; m_as(MB/Mparam) for unit extraction.
@@ -1980,9 +1980,9 @@ class ResNetMemory:
For a network with $N_L$ layers, the system must save approximately $N_L$ activation tensors, one per layer, for the entire batch. Consider a concrete example: ResNet-50 has `{python} ResNetMemory.resnet_params_m_str` M parameters (~`{python} ResNetMemory.resnet_fp32_mb_str` MB in FP32) and processes batch size 64 with $224\times224$ images. The memory breakdown reveals the scale of this trade-off. Forward activations alone consume approximately 8--12 GB (varying by implementation and checkpointing strategy). Parameter gradients add another ~`{python} ResNetMemory.resnet_fp32_mb_str` MB (the same size as the parameters themselves), and Adam optimizer state contributes ~`{python} ResNetMemory.resnet_adam_mb_str` MB for its two momentum buffers per parameter. The total training footprint reaches `{python} ResNetMemory.resnet_training_min_gb_str`--`{python} ResNetMemory.resnet_training_max_gb_str` GB, compared to just ~`{python} ResNetMemory.resnet_fp32_mb_str` MB for inference alone.
This `{python} ResNetMemory.resnet_training_ratio_str`$\times$ ratio between training and inference memory quantifies why the Data Movement ($D_{vol}$) term dominates training latency in the **Iron Law**. During training, the framework must write all activations to memory during the forward pass and read them back during the backward pass, doubling the memory traffic compared to inference alone. For a complete derivation of the four-component training memory equation ($M_{total} = M_{weights} + M_{gradients} + M_{optimizer} + M_{activations}$) and worked examples at larger model scales, see @sec-algorithm-foundations-true-cost-training-memory-e54e.
This `{python} ResNetMemory.resnet_training_ratio_str`$\times$ ratio between training and inference memory quantifies why the Data Movement ($D_{\text{vol}}$) term dominates training latency in the **Iron Law**. During training, the framework must write all activations to memory during the forward pass and read them back during the backward pass, doubling the memory traffic compared to inference alone. For a complete derivation of the four-component training memory equation ($M_{total} = M_{weights} + M_{gradients} + M_{optimizer} + M_{activations}$) and worked examples at larger model scales, see @sec-algorithm-foundations-true-cost-training-memory-e54e.
Frameworks provide two primary mechanisms to manage this trade-off. **Gradient checkpointing**\index{Gradient Checkpointing!recomputation strategy} [@chen2016training] trades recomputation for memory: instead of saving all activations, the framework saves only a subset and recomputes the rest during the backward pass. This typically reduces activation memory by 50--90% at the cost of 20--33% additional compute (with optimal $\sqrt{n}$ checkpoint placement). In Iron Law terms, checkpointing increases the $O$ term (recomputation) to reduce the $D_{vol}$ term (memory traffic). **Tensor detachment** provides a complementary mechanism: calling `.detach()` on a tensor removes it from the computation graph entirely, preventing the framework from saving activations through that path. This is essential for transfer learning, where pretrained layers should not accumulate gradients, and reduces the $D_{vol}$ term by eliminating unnecessary activation storage.
Frameworks provide two primary mechanisms to manage this trade-off. **Gradient checkpointing**\index{Gradient Checkpointing!recomputation strategy} [@chen2016training] trades recomputation for memory: instead of saving all activations, the framework saves only a subset and recomputes the rest during the backward pass. This typically reduces activation memory by 50--90% at the cost of 20--33% additional compute (with optimal $\sqrt{n}$ checkpoint placement). In Iron Law terms, checkpointing increases the $O$ term (recomputation) to reduce the $D_{\text{vol}}$ term (memory traffic). **Tensor detachment** provides a complementary mechanism: calling `.detach()` on a tensor removes it from the computation graph entirely, preventing the framework from saving activations through that path. This is essential for transfer learning, where pretrained layers should not accumulate gradients, and reduces the $D_{\text{vol}}$ term by eliminating unnecessary activation storage.
Mixed-precision training offers a third approach, reducing activation memory by storing values in lower precision formats. The detailed trade-offs of mixed precision are examined later in this chapter.
@@ -2145,12 +2145,12 @@ with torch.no_grad():
:::
These three principles connect directly to the framework's role as a compiler for the **Silicon Contract**. The reverse-linked graph determines which operations the backward pass must execute (the $O$ term). The memory-compute trade-off governs how much data the framework must move through the memory hierarchy (the $D_{vol}$ term). And the extensibility mechanisms allow engineers to tune both terms for their specific workload. The interaction between autograd memory management and numerical precision leads naturally to mixed-precision training, which further reduces the $D_{vol}$ term.
These three principles connect directly to the framework's role as a compiler for the **Silicon Contract**. The reverse-linked graph determines which operations the backward pass must execute (the $O$ term). The memory-compute trade-off governs how much data the framework must move through the memory hierarchy (the $D_{\text{vol}}$ term). And the extensibility mechanisms allow engineers to tune both terms for their specific workload. The interaction between autograd memory management and numerical precision leads naturally to mixed-precision training, which further reduces the $D_{\text{vol}}$ term.
#### Mixed-Precision Training Support {#sec-ml-frameworks-mixedprecision-training-support-d31d}
\index{Mixed Precision!FP16 vs. FP32 trade-offs}
Mixed precision exploits a hardware asymmetry to improve two Iron Law terms simultaneously: Tensor Cores execute FP16 matrix multiplications at 2$\times$ the throughput of FP32 (increasing effective $O/R_{peak}$), while FP16 activations halve the memory footprint (reducing $D_{vol}$). Improving both terms simultaneously is rare; most optimizations improve one at the expense of the other.
Mixed precision exploits a hardware asymmetry to improve two Iron Law terms simultaneously: Tensor Cores execute FP16 matrix multiplications at 2$\times$ the throughput of FP32 (increasing effective $O/R_{\text{peak}}$), while FP16 activations halve the memory footprint (reducing $D_{\text{vol}}$). Improving both terms simultaneously is rare; most optimizations improve one at the expense of the other.
Frameworks exploit this through automatic mixed-precision APIs that select reduced precision for compute-intensive operations while maintaining FP32 where numerical stability demands it. Inside these APIs, frameworks automatically apply precision rules: matrix multiplications and convolutions use FP16 for bandwidth efficiency, while numerically sensitive operations like softmax and layer normalization remain in FP32. This selective precision maintains accuracy while achieving speedups on modern GPUs with specialized hardware units. Because FP16 has a narrower dynamic range than FP32, gradients can underflow to zero during backpropagation. Loss scaling addresses this by multiplying the loss by a large factor before the backward pass, then dividing gradients by the same factor afterward.
@@ -2255,7 +2255,7 @@ class Model7B:
# Note: Use Model7B.model_7b_total_gb_str directly.
```
Resuming training after interruption requires restoring model weights and optimizer state together: momentum buffers, adaptive learning rates, and gradient statistics. For Adam, optimizer state typically quintuples the memory footprint beyond weights alone (since two FP32 states are stored for each FP16 parameter), meaning a 7B-parameter model requires approximately `{python} Model7B.model_7b_total_gb_str` GB total (`{python} Model7B.model_7b_fp16_gb_str` GB weights + `{python} Model7B.model_7b_adam_gb_str` GB optimizer state). Checkpoint size therefore bounds recovery speed after failure, connecting fault tolerance directly to the Iron Law's $D_{vol}$ term.
Resuming training after interruption requires restoring model weights and optimizer state together: momentum buffers, adaptive learning rates, and gradient statistics. For Adam, optimizer state typically quintuples the memory footprint beyond weights alone (since two FP32 states are stored for each FP16 parameter), meaning a 7B-parameter model requires approximately `{python} Model7B.model_7b_total_gb_str` GB total (`{python} Model7B.model_7b_fp16_gb_str` GB weights + `{python} Model7B.model_7b_adam_gb_str` GB optimizer state). Checkpoint size therefore bounds recovery speed after failure, connecting fault tolerance directly to the Iron Law's $D_{\text{vol}}$ term.
@sec-model-training covers optimizer memory requirements and optimization strategies for large-scale training, where checkpoint size becomes a binding constraint. Frameworks provide the `state_dict()` interface to access optimizer state for serialization (@lst-state-dict-interface), and resuming training requires loading both model parameters and optimizer state (@lst-checkpoint-save-load).
@@ -2433,7 +2433,7 @@ At the foundation of every framework's data representation lies a single abstrac
***Tensors***\index{Tensor!definition} are $n$-dimensional arrays with explicit shape, data type, and memory layout metadata that allow ML frameworks to map mathematical operations directly onto hardware vector units without intermediate data transformation.
1. **Significance (Quantitative):** Tensor memory footprint is fully deterministic from its metadata: a contiguous FP32 tensor of shape $[1024, 1024]$ occupies exactly $1024 \times 1024 \times 4 = 4$ MB. Non-contiguous layouts (e.g., from a transpose operation) require explicit `.contiguous()` calls before certain CUDA kernels can execute, adding a memory-copy overhead that can dominate the $L_{lat}$ term for tensors under 1 MB.
1. **Significance (Quantitative):** Tensor memory footprint is fully deterministic from its metadata: a contiguous FP32 tensor of shape $[1024, 1024]$ occupies exactly $1024 \times 1024 \times 4 = 4$ MB. Non-contiguous layouts (e.g., from a transpose operation) require explicit `.contiguous()` calls before certain CUDA kernels can execute, adding a memory-copy overhead that can dominate the $L_{\text{lat}}$ term for tensors under 1 MB.
2. **Distinction (Durable):** Unlike a Python list or generic NumPy array, a framework tensor carries device placement metadata (CPU vs. GPU), dtype (FP32, BF16, INT8), and stride information that enables zero-copy view operations and CUDA kernel dispatch without any runtime type checking or data movement.
3. **Common Pitfall:** A frequent misconception is that tensor operations are always in-place. Framework tensor operations return new tensors by default, allocating fresh GPU memory for each intermediate result. In a long computation graph, these intermediate allocations accumulate and can exhaust GPU memory before any weights are updated.
@@ -2793,7 +2793,7 @@ Three systems principles govern effective device and memory management: understa
The cost of moving data between devices varies by orders of magnitude depending on the interconnect.[^fn-nvlink-bandwidth-hierarchy] Before examining optimization strategies, we need to understand these costs quantitatively. @tbl-device-transfer-overhead shows transfer times for a $1000\times1000$ float32 tensor (4 MB)---roughly the size of a typical activation tensor in a moderately sized model. The numbers reveal why careless device placement can erase any speedup from GPU acceleration:
[^fn-nvlink-bandwidth-hierarchy]: **NVLink**: NVIDIA's high-bandwidth GPU-to-GPU interconnect (see @sec-hardware-acceleration), providing `{python} DeviceBandwidthHierarchy.nvlink_a100_gbs_str` GB/s bidirectional bandwidth (NVLink 3.0 on A100) compared to `{python} DeviceBandwidthHierarchy.pcie4_bidir_gbs_str` GB/s for PCIe 4.0 x16. This ~10$\times$ bandwidth advantage determines whether tensor parallelism is practical for a given model size: splitting a model across GPUs connected by PCIe can make the $D_{vol}/BW$ communication term dominate total training time, erasing the benefit of additional compute. \index{NVLink!bandwidth hierarchy}
[^fn-nvlink-bandwidth-hierarchy]: **NVLink**: NVIDIA's high-bandwidth GPU-to-GPU interconnect (see @sec-hardware-acceleration), providing `{python} DeviceBandwidthHierarchy.nvlink_a100_gbs_str` GB/s bidirectional bandwidth (NVLink 3.0 on A100) compared to `{python} DeviceBandwidthHierarchy.pcie4_bidir_gbs_str` GB/s for PCIe 4.0 x16. This ~10$\times$ bandwidth advantage determines whether tensor parallelism is practical for a given model size: splitting a model across GPUs connected by PCIe can make the $D_{\text{vol}}/BW$ communication term dominate total training time, erasing the benefit of additional compute. \index{NVLink!bandwidth hierarchy}
| **Interconnect** | **Bandwidth** | **Transfer Time** | **Relative to Compute** |
|:-----------------|---------------------------------------------------------------------------:|---------------------------------------------------------:|:-----------------------------------|
@@ -2804,7 +2804,7 @@ The cost of moving data between devices varies by orders of magnitude depending
: **Device Transfer Overhead.** Transfer time for a 4 MB tensor across different interconnects. PCIe bandwidth shown is unidirectional (typical for GPU transfers), with full-duplex operation providing 2$\times$ total bandwidth. NVLink bandwidth is bidirectional (300 GB/s per direction). Transfer times dominate for small operations, making device placement critical for performance. {#tbl-device-transfer-overhead}
These numbers connect directly to the **Iron Law** of performance. Every cross-device transfer inflates the data movement term ($D_{vol}/BW$) at a fraction of the available on-device bandwidth. A PCIe 4.0 transfer at `{python} DeviceBandwidthHierarchy.pcie4_gbs_str` GB/s means moving a 1 GB activation tensor adds approximately `{python} DeviceBandwidthHierarchy.pcie4_1gb_ms_str` ms to the data movement cost, equivalent to roughly `{python} DeviceBandwidthHierarchy.pcie4_1gb_equiv_ops_str` trillion operations on a GPU delivering `{python} A100BLAS.dense_tflops_str` TFLOPS. For a model forward pass taking 0.5 ms on GPU, transferring inputs and outputs over PCIe 3.0 doubles the total latency. When batches are small or models are lightweight, transfer overhead can exceed computation time entirely.
These numbers connect directly to the **Iron Law** of performance. Every cross-device transfer inflates the data movement term ($D_{\text{vol}}/BW$) at a fraction of the available on-device bandwidth. A PCIe 4.0 transfer at `{python} DeviceBandwidthHierarchy.pcie4_gbs_str` GB/s means moving a 1 GB activation tensor adds approximately `{python} DeviceBandwidthHierarchy.pcie4_1gb_ms_str` ms to the data movement cost, equivalent to roughly `{python} DeviceBandwidthHierarchy.pcie4_1gb_equiv_ops_str` trillion operations on a GPU delivering `{python} A100BLAS.dense_tflops_str` TFLOPS. For a model forward pass taking 0.5 ms on GPU, transferring inputs and outputs over PCIe 3.0 doubles the total latency. When batches are small or models are lightweight, transfer overhead can exceed computation time entirely.
The systems implication is clear: every tensor should reside on the device where it will be consumed, and transfers should occur only when unavoidable. Frameworks track device placement for every tensor and raise errors when operations attempt to combine tensors from different devices, enforcing this discipline at the API level.
@@ -2814,7 +2814,7 @@ The systems implication is clear: every tensor should reside on the device where
\index{Asynchronous Execution!hiding transfer latency}
When transfers are unavoidable, the next optimization is to hide their latency by executing them concurrently with computation. Modern GPUs contain independent hardware units for computation (SM clusters) and data transfer (copy engines), enabling true simultaneous execution. The framework abstraction that exposes this hardware parallelism is the *CUDA stream*\index{Execution Streams!definition}: an independent execution queue where operations execute sequentially within a stream but concurrently across streams.
Without explicit concurrency control, the GPU serializes all operations on a single default stream, leaving execution units idle while data transfers complete. By placing data transfers on one stream and computation on another, the effective latency approaches the theoretical minimum of $\max(\text{compute\_time}, \text{transfer\_time})$ rather than their sum. Stream-based overlap effectively hides the $D_{vol}/BW$ penalty when computation is the longer operation (see @lst-overlap-compute-transfer):
Without explicit concurrency control, the GPU serializes all operations on a single default stream, leaving execution units idle while data transfers complete. By placing data transfers on one stream and computation on another, the effective latency approaches the theoretical minimum of $\max(\text{compute\_time}, \text{transfer\_time})$ rather than their sum. Stream-based overlap effectively hides the $D_{\text{vol}}/BW$ penalty when computation is the longer operation (see @lst-overlap-compute-transfer):
::: {#lst-overlap-compute-transfer lst-cap="**Overlapping Computation and Transfer**: Use separate streams for data transfer and computation to hide transfer latency. Pinned memory enables truly asynchronous non-blocking transfers."}
@@ -3782,7 +3782,7 @@ JAX makes the most radical trade-off by treating the model as a pure function[^f
[^fn-pure-function-jax]: **Pure Function**: Has no side effects and always returns the same output for the same inputs. In JAX, purity is not a style preference but a compiler requirement: `jax.jit` traces the function once and caches the compiled result, so any side effect (printing, modifying global state, random number generation without explicit key threading) would execute only during the first trace and silently vanish from subsequent calls. This constraint is the cost JAX pays for composable, whole-program optimization. \index{Pure Function!JIT requirement}
[^fn-xla-compiler]: **XLA (Accelerated Linear Algebra)**: The "optimized machine code" in the triggering sentence means XLA fuses an entire subgraph into one kernel, eliminating both launch overhead ($L_{lat}$) and intermediate memory writes ($D_{vol}$). The 1.5--2$\times$ speedup for transformer blocks is modest because their large GEMM operations are already compute-bound, leaving little overhead for fusion to remove. Memory-bound models see 3--10$\times$ gains, where fusion hides the relative cost of many small, sequential operations. \index{XLA!compilation speedup}
[^fn-xla-compiler]: **XLA (Accelerated Linear Algebra)**: The "optimized machine code" in the triggering sentence means XLA fuses an entire subgraph into one kernel, eliminating both launch overhead ($L_{\text{lat}}$) and intermediate memory writes ($D_{\text{vol}}$). The 1.5--2$\times$ speedup for transformer blocks is modest because their large GEMM operations are already compute-bound, leaving little overhead for fusion to remove. Memory-bound models see 3--10$\times$ gains, where fusion hides the relative cost of many small, sequential operations. \index{XLA!compilation speedup}
[^fn-onnx-portability]: **ONNX (Open Neural Network Exchange)**: The "fragmentation" ONNX addresses is that the best training framework (often PyTorch for research velocity) rarely matches the best serving runtime (often TensorRT for latency, TF Lite for mobile). ONNX defines a hardware-agnostic graph representation that decouples the two, eliminating the engineer-months of manual model conversion that would otherwise be required each time a deployment target changes. The accepted trade-off is that ONNX export can lose framework-specific optimizations or custom operators, requiring fallback implementations. \index{ONNX!framework portability}
@@ -4368,12 +4368,12 @@ Machine learning frameworks exist to solve three fundamental problems that would
3. **The Abstraction Problem**: How do we target diverse hardware from a single interface? Frameworks provide tensor abstractions, intermediate representations, and runtime systems that hide hardware complexity while enabling efficient utilization across CPUs, GPUs, TPUs, and specialized accelerators.
These problems are interconnected and constrained by the **Iron Law** of performance (@sec-introduction-iron-law-ml-systems-c32a): execution strategy determines dispatch overhead ($L_{lat}$), differentiation determines memory traffic ($D_{vol}$), and abstraction determines hardware utilization ($\eta$). The memory wall makes data movement often more expensive than computation, explaining why frameworks invest in kernel fusion, activation checkpointing, mixed-precision training, and compilation pipelines.
These problems are interconnected and constrained by the **Iron Law** of performance (@sec-introduction-iron-law-ml-systems-c32a): execution strategy determines dispatch overhead ($L_{\text{lat}}$), differentiation determines memory traffic ($D_{\text{vol}}$), and abstraction determines hardware utilization ($\eta$). The memory wall makes data movement often more expensive than computation, explaining why frameworks invest in kernel fusion, activation checkpointing, mixed-precision training, and compilation pipelines.
::: {.callout-takeaways title="The Layer Between Math and Hardware"}
* **Three problems define every framework**: Execution (how to run), differentiation (how to train), and abstraction (how to express). TensorFlow prioritizes abstraction for deployment breadth, PyTorch prioritizes execution for research velocity, and JAX reframes differentiation through composable function transformations. These are infrastructure commitments, not tooling preferences.
* **The memory wall drives optimization**: Compute has grown approximately 1000$\times$ faster than memory bandwidth. Kernel fusion, activation checkpointing, mixed-precision training, and data layout optimizations all target the data movement term ($D_{vol}$) in the Iron Law, not the compute term.
* **The memory wall drives optimization**: Compute has grown approximately 1000$\times$ faster than memory bandwidth. Kernel fusion, activation checkpointing, mixed-precision training, and data layout optimizations all target the data movement term ($D_{\text{vol}}$) in the Iron Law, not the compute term.
* **Compilation pays off only at scale**: The Compilation Continuum principle (@eq-compilation-benefit) quantifies when compilation benefits exceed costs. Research prototyping favors eager mode; production training and inference favor progressive compilation from JIT to AOT. The Dispatch Overhead Law (@eq-dispatch-overhead) explains why small models benefit disproportionately.
* **The nn.Module pattern is widely adopted**: Automatic parameter discovery, mode-dependent behavior, and hierarchical composition with serialization appear across major frameworks, enabling million-parameter optimization in a single `optimizer.step()` call regardless of API syntax.
* **Framework choice constrains deployment by orders of magnitude**: A 17$\times$ latency gap (PyTorch vs. TensorRT) and 7,040$\times$ memory gap (PyTorch Mobile vs. TFLite Micro) on identical models demonstrate that frameworks are not interchangeable. Deployment target must be evaluated before framework selection.

20
book/quarto/contents/vol1/hw_acceleration/hw_acceleration.qmd
View File

@@ -54,7 +54,7 @@ Data was optimized in @sec-data-selection and the Algorithm (Model) was compress
::: {.callout-definition title="Hardware Acceleration"}
***Hardware Acceleration***\index{Hardware Acceleration!definition} is the practice of replacing general-purpose processor logic with domain-specific silicon optimized for a narrow class of operations, trading programmability for the compute density ($R_{peak}$) and energy efficiency ($\eta$) gains that regular, data-parallel workloads like matrix multiplication can exploit.
***Hardware Acceleration***\index{Hardware Acceleration!definition} is the practice of replacing general-purpose processor logic with domain-specific silicon optimized for a narrow class of operations, trading programmability for the compute density ($R_{\text{peak}}$) and energy efficiency ($\eta$) gains that regular, data-parallel workloads like matrix multiplication can exploit.
1. **Significance (Quantitative):** The throughput gain is orders of magnitude. An A100 GPU delivers 312 TFLOPS BF16 for matrix multiplication, while a server-class CPU delivers roughly 1–2 TFLOPS for the same operation — a 150–300$\times$ gap achieved by dedicating 80+ billion transistors to parallel arithmetic units rather than to branch predictors, out-of-order schedulers, and large caches.
2. **Distinction (Durable):** Unlike a general-purpose CPU, which is optimized to minimize latency for any single instruction in an arbitrary serial program, an accelerator is optimized to maximize throughput for a specific operation class — meaning it achieves its gains only when the workload presents enough parallel work to keep all arithmetic units busy simultaneously.
@@ -80,9 +80,9 @@ Hardware alone, however, cannot achieve these gains. The algorithms must be desi
Co-design explains *why* the compression techniques introduced in @sec-model-compression deliver real speedups. Quantization from FP32 to INT8 (as described in @sec-model-compression) yields 2--4$\times$ acceleration not because of fewer bits in the abstract, but because accelerators pack 4$\times$ more INT8 operations into the same silicon area. Structured pruning improves performance while unstructured pruning often does not, because structured patterns preserve the regular memory access patterns that hardware can optimize. Throughout this chapter, the physical constraints of silicon will reveal *why* some theoretically promising algorithmic optimizations succeed in practice and others fail.
\index{Iron Law!ML systems performance}\index{Amdahl's Law!acceleration ceiling}
Hardware acceleration targets specific terms in the **Iron Law of ML Systems** (@sec-introduction-iron-law-ml-systems-c32a), which decomposes end-to-end time into data volume ($D_{vol}/BW$), computation ($O / R_{peak} \cdot \eta$), and fixed latency ($L_{lat}$). While data selection reduced the total data and model compression reduced the ops per sample, hardware acceleration increases the rate at which those ops execute by maximizing the Throughput and Bandwidth denominators. Yet acceleration has a hard ceiling, established by *Amdahl's Law*[^fn-amdahls-law-acceleration].
Hardware acceleration targets specific terms in the **Iron Law of ML Systems** (@sec-introduction-iron-law-ml-systems-c32a), which decomposes end-to-end time into data volume ($D_{\text{vol}}/BW$), computation ($O / R_{\text{peak}} \cdot \eta$), and fixed latency ($L_{\text{lat}}$). While data selection reduced the total data and model compression reduced the ops per sample, hardware acceleration increases the rate at which those ops execute by maximizing the Throughput and Bandwidth denominators. Yet acceleration has a hard ceiling, established by *Amdahl's Law*[^fn-amdahls-law-acceleration].
[^fn-amdahls-law-acceleration]: **Amdahl's Law**: Dictates that accelerating one component of a system (computation) yields diminishing returns as the un-accelerated components (data movement, latency) come to dominate the total time. Even if hardware makes the computation term ($O/R_{peak}$) instantaneous, the system is still bottlenecked by the serial data loading ($D_{vol}/BW$) and fixed latency ($L_{lat}$) terms from the Iron Law. This is why a 100$\times$ improvement in raw accelerator throughput often produces only a 5--20$\times$ improvement in end-to-end task time. \index{Amdahl's Law!acceleration ceiling}
[^fn-amdahls-law-acceleration]: **Amdahl's Law**: Dictates that accelerating one component of a system (computation) yields diminishing returns as the un-accelerated components (data movement, latency) come to dominate the total time. Even if hardware makes the computation term ($O/R_{\text{peak}}$) instantaneous, the system is still bottlenecked by the serial data loading ($D_{\text{vol}}/BW$) and fixed latency ($L_{\text{lat}}$) terms from the Iron Law. This is why a 100$\times$ improvement in raw accelerator throughput often produces only a 5--20$\times$ improvement in end-to-end task time. \index{Amdahl's Law!acceleration ceiling}
To quantify this ceiling, consider the formalization of Amdahl's Law applied to accelerator speedup.
@@ -633,7 +633,7 @@ Machine learning constitutes a computational domain with unique characteristics
::: {.callout-definition title="ML Accelerator"}
***Machine Learning Accelerators***\index{ML Accelerator!definition} are domain-specific processors whose silicon is designed primarily for the dense matrix operations and regular data flow of neural networks, achieving high $R_{peak}$ and memory bandwidth utilization for these workloads by devoting die area to arithmetic units rather than to general-purpose control logic.
***Machine Learning Accelerators***\index{ML Accelerator!definition} are domain-specific processors whose silicon is designed primarily for the dense matrix operations and regular data flow of neural networks, achieving high $R_{\text{peak}}$ and memory bandwidth utilization for these workloads by devoting die area to arithmetic units rather than to general-purpose control logic.
1. **Significance (Quantitative):** The efficiency differential over CPUs is quantifiable. An A100 GPU delivers 312 TFLOPS BF16 with 2 TB/s memory bandwidth, while a high-end server CPU delivers roughly 1–2 TFLOPS FP32 with 200 GB/s bandwidth — a 150–300$\times$ compute throughput gap and a 10$\times$ bandwidth gap for the same matrix-multiply workloads that dominate neural network training and inference.
2. **Distinction (Durable):** Unlike a general-purpose CPU, which executes complex, branch-dependent serial programs efficiently by minimizing per-instruction latency, an ML accelerator processes thousands of independent arithmetic operations in parallel with predictable memory access patterns — making it 100–1,000$\times$ faster at matrix multiplication but potentially slower than a CPU for irregular control flow programs like tree traversal or dynamic programming.
@@ -2214,9 +2214,9 @@ Unlike conventional workloads, ML models require frequent access to large volume
::: {.callout-definition title="AI Memory Wall"}
***The AI Memory Wall***\index{AI Memory Wall!definition} is the performance constraint that arises when arithmetic throughput ($R_{peak}$) outpaces memory bandwidth ($BW$).
***The AI Memory Wall***\index{AI Memory Wall!definition} is the performance constraint that arises when arithmetic throughput ($R_{\text{peak}}$) outpaces memory bandwidth ($BW$).
1. **Significance (Quantitative):** It dictates that system performance is no longer bounded by FLOPs, but by the **Energy and Latency Cost** of moving data. Within the **Iron Law**, it is the point where the $\frac{D_{vol}}{BW}$ term dominates the total execution time ($T$).
1. **Significance (Quantitative):** It dictates that system performance is no longer bounded by FLOPs, but by the **Energy and Latency Cost** of moving data. Within the **Iron Law**, it is the point where the $\frac{D_{\text{vol}}}{BW}$ term dominates the total execution time ($T$).
2. **Distinction (Durable):** Unlike a **General-Purpose Memory Wall**, which affects all computing, the AI Memory Wall is driven by the **Massive Model State** and activation storage required by deep learning.
3. **Common Pitfall:** A frequent misconception is that the Memory Wall is "fixed" by more memory. In reality, it is a **Bandwidth-Latency Gap**: even with infinite capacity, the speed of moving data between memory and compute remains the fundamental physical bottleneck.
@@ -2297,7 +2297,7 @@ The memory wall manifests through three critical constraints. First, the energy
Different paradigms inhabit different regions of this "Memory Wall." We quantify this using the **Hardware Balance ($B$)\index{Hardware Balance ($B$)}**, defined as the number of operations required to hide the cost of fetching one byte of data:
$$ B = \frac{R_{peak}}{BW} $$
$$ B = \frac{R_{\text{peak}}}{BW} $$
This ratio partitions the deployment spectrum into two distinct regimes. High-end accelerators like the NVIDIA H100 have a balance of $\approx 150$--$300$, making them "Bandwidth-Hungry" giants where the challenge is moving data fast enough to saturate the ALUs. In contrast, TinyML microcontrollers often have a balance of $< 10$, making them "Compute-Starved" but relatively bandwidth-efficient. This explains why an architecture that is efficient in the cloud (where we optimize for $BW$ limits) can be a disaster at the edge: the hardware balance has shifted under the model, transforming a memory-bound success into a compute-bound failure.
@@ -2954,9 +2954,9 @@ The key metric that determines which ceiling a workload hits is *arithmetic inte
::: {.callout-definition title="Arithmetic Intensity"}
***Arithmetic Intensity***\index{Arithmetic Intensity!definition} is the ratio of floating-point operations to bytes of memory traffic for a given computation ($\text{FLOP}/\text{byte}$), determining whether the workload is limited by compute throughput ($R_{peak}$) or memory bandwidth ($BW$) on a given accelerator.
***Arithmetic Intensity***\index{Arithmetic Intensity!definition} is the ratio of floating-point operations to bytes of memory traffic for a given computation ($\text{FLOP}/\text{byte}$), determining whether the workload is limited by compute throughput ($R_{\text{peak}}$) or memory bandwidth ($BW$) on a given accelerator.
1. **Significance (Quantitative):** The intensity threshold separating memory-bound from compute-bound regimes is the roofline ridge point: $R_{peak} / BW$. For an A100 (312 TFLOPS BF16, 2 TB/s), the ridge point is $312 \times 10^{12} / (2 \times 10^{12}) = 156$ FLOP/byte. A large matrix-multiply achieves $\sim$100–200 FLOP/byte (compute-bound); a pointwise ReLU achieves $\sim$0.5 FLOP/byte (memory-bound) — placing these two operations in completely different optimization regimes on the same hardware.
1. **Significance (Quantitative):** The intensity threshold separating memory-bound from compute-bound regimes is the roofline ridge point: $R_{\text{peak}} / BW$. For an A100 (312 TFLOPS BF16, 2 TB/s), the ridge point is $312 \times 10^{12} / (2 \times 10^{12}) = 156$ FLOP/byte. A large matrix-multiply achieves $\sim$100–200 FLOP/byte (compute-bound); a pointwise ReLU achieves $\sim$0.5 FLOP/byte (memory-bound) — placing these two operations in completely different optimization regimes on the same hardware.
2. **Distinction (Durable):** Unlike total FLOPs (a count of operations), arithmetic intensity is a ratio that characterizes the shape of a workload's hardware demand. Two kernels with identical FLOPs but different memory access patterns have different arithmetic intensities and will be bottlenecked by different hardware resources.
3. **Common Pitfall:** A frequent misconception is that arithmetic intensity is a fixed property of an operation. In practice, it depends on implementation details: a naive matrix-multiply that reloads operands from DRAM for each output element has low arithmetic intensity; a blocked (tiled) implementation that reuses data from fast SRAM achieves high arithmetic intensity — the same mathematical operation, orders of magnitude apart in hardware efficiency.
@@ -3629,7 +3629,7 @@ Efficient execution of machine learning models on specialized AI acceleration ha
***Mapping in AI Acceleration***\index{Mapping!definition} is the process of binding the **Logical Computation Graph** to the **Physical Hardware Topology** by deciding which operations execute on which processing elements, which data resides in which memory tier, and in what temporal order.
1. **Significance (Quantitative):** Within the D·A·M taxonomy, mapping is the Machine-axis decision that determines whether the Algorithm's operations run at $R_{peak}$ or at $BW \times I$. Specifically, a GEMM with arithmetic intensity $I$ runs at $\min(R_{peak},\; BW \times I)$; a poor tiling choice that forces unnecessary DRAM accesses can reduce effective $I$ by 10–50$\times$, collapsing a compute-bound operation into a bandwidth-bound one and causing a commensurate drop in sustained throughput.
1. **Significance (Quantitative):** Within the D·A·M taxonomy, mapping is the Machine-axis decision that determines whether the Algorithm's operations run at $R_{\text{peak}}$ or at $BW \times I$. Specifically, a GEMM with arithmetic intensity $I$ runs at $\min(R_{\text{peak}},\; BW \times I)$; a poor tiling choice that forces unnecessary DRAM accesses can reduce effective $I$ by 10–50$\times$, collapsing a compute-bound operation into a bandwidth-bound one and causing a commensurate drop in sustained throughput.
2. **Distinction (Durable):** Unlike **Traditional Compilation** (which targets a linear instruction stream on a von Neumann processor), Mapping targets a **Dataflow Architecture** where the movement of data is as costly as the computation itself: off-chip DRAM access consumes ~200$\times$ more energy than a multiply-accumulate in local registers.
3. **Common Pitfall:** A frequent misconception is that mapping is automatically handled by frameworks. For general GPU workloads, compilers like XLA achieve near-optimal mappings; for specialized accelerators (systolic arrays, custom ASICs), compiler-generated mappings are often 20–50% suboptimal compared to hand-tuned schedules, because the compiler's search space is limited by the time budget at compilation.

52
book/quarto/contents/vol1/introduction/introduction.qmd
View File

@@ -151,7 +151,7 @@ The critical implication: *Data is Source Code* (Principle \ref{pri-data-as-code
[^fn-sgd-sampling]: **Stochastic Gradient Descent (SGD)**: The algorithm implements the "compilation" of logic from data by processing one small, random data sample (a "batch") at a time, instead of the entire dataset. This trade-off, statistical noise for computational speed, is the core engine of the training "compiler." The choice of batch size becomes a critical compilation flag; a batch too small (e.g., < 32) fails to saturate the parallel processors of an accelerator, wasting over 90% of its potential computation. \index{Stochastic Gradient Descent (SGD)!systems trade-off}
[^fn-model-weights-inference]: **Model Weights**: The learned numerical parameters of a neural network, one value per connection between units. A GPT-3-scale model stores 175 billion such values, consuming 350 GB in FP16 precision. Because every inference request must load these weights through the memory hierarchy, weight count is the single largest determinant of both memory footprint ($D_{vol}$) and serving cost (see @sec-neural-computation). \index{Model Weights!definition}
[^fn-model-weights-inference]: **Model Weights**: The learned numerical parameters of a neural network, one value per connection between units. A GPT-3-scale model stores 175 billion such values, consuming 350 GB in FP16 precision. Because every inference request must load these weights through the memory hierarchy, weight count is the single largest determinant of both memory footprint ($D_{\text{vol}}$) and serving cost (see @sec-neural-computation). \index{Model Weights!definition}
From a systems perspective, this represents a transition from *instruction-centric* to *data-centric* computing\index{Data-Centric Computing}:
@@ -1136,7 +1136,7 @@ The table reveals two additional insights. Training time initially increased (ho
\index{Deep Blue}
The principle finds further validation across AI breakthroughs. In chess, IBM's Deep Blue defeated world champion Garry Kasparov[^fn-deepblue-compute] in 1997 [@campbell2002deep] not by encoding chess strategies, but through brute-force search evaluating millions of positions per second.
[^fn-deepblue-compute]: **Deep Blue**: IBM's chess system defeated World Champion Garry Kasparov in 1997 not through strategic understanding but through raw computational power: 200 million positions per second on 480 custom chess processors. Deep Blue was the first public demonstration that purpose-built silicon ($R_{peak}$) could substitute for human expertise, foreshadowing the domain-specific accelerator strategy that defines modern ML hardware. \index{Deep Blue!computational scale}
[^fn-deepblue-compute]: **Deep Blue**: IBM's chess system defeated World Champion Garry Kasparov in 1997 not through strategic understanding but through raw computational power: 200 million positions per second on 480 custom chess processors. Deep Blue was the first public demonstration that purpose-built silicon ($R_{\text{peak}}$) could substitute for human expertise, foreshadowing the domain-specific accelerator strategy that defines modern ML hardware. \index{Deep Blue!computational scale}
\index{AlphaGo}
In Go, DeepMind's AlphaGo[^fn-alphago-scale] [@silver2016mastering] achieved superhuman performance by learning from self-play rather than studying centuries of human Go wisdom.
@@ -1147,7 +1147,7 @@ The lesson is "bitter" because our intuition misleads us\index{Bitter Lesson, Th
Modern language models like GPT-4 and image generation systems like DALL-E illustrate this principle directly. Their capabilities emerge not from linguistic or artistic theories encoded by humans but from training general-purpose neural networks on vast amounts of data using substantial computational resources. Estimates for models at GPT-3's scale suggest thousands of megawatt-hours of energy[^fn-gpt3-training-energy] [@patterson2021carbon], and serving these models to millions of users demands data centers consuming power comparable to small cities.
[^fn-gpt3-training-energy]: **GPT-3 Training Energy**: Patterson et al. estimated GPT-3's single training run consumed approximately 1,287 MWh and emitted 552 tonnes of CO2-equivalent, roughly the annual electricity of 120 average US households. The energy cost is dominated not by arithmetic but by data movement through the memory hierarchy, making the $D_{vol}/BW$ term of the Iron Law, not the compute term, the primary driver of the power bill at this scale. \index{GPT-3!training energy}
[^fn-gpt3-training-energy]: **GPT-3 Training Energy**: Patterson et al. estimated GPT-3's single training run consumed approximately 1,287 MWh and emitted 552 tonnes of CO2-equivalent, roughly the annual electricity of 120 average US households. The energy cost is dominated not by arithmetic but by data movement through the memory hierarchy, making the $D_{\text{vol}}/BW$ term of the Iron Law, not the compute term, the primary driver of the power bill at this scale. \index{GPT-3!training energy}
The implication is that realizing the Bitter Lesson's promise requires expertise in data engineering, hardware optimization, and systems coordination[^fn-memory-bandwidth-bottleneck] that goes far beyond algorithmic innovation. We explore these hardware constraints quantitatively in @sec-hardware-acceleration, where students will have the prerequisite background to analyze memory bandwidth limitations and their implications for system design.
@@ -1210,7 +1210,7 @@ These three interconnected concerns, obtaining and managing training data at sca
***Machine Learning Systems***\index{Machine Learning Systems!definition} are software systems whose core behavior is determined by parameters learned from data rather than explicitly programmed rules, making performance a function of data quality, algorithm choice, and hardware capacity simultaneously.
1. **Significance (Quantitative):** Every performance budget traces back to the Iron Law: $T = D_{vol}/BW + O/(R_{peak} \cdot \eta) + L_{lat}$. In a production recommendation system serving 10 million requests per day, a 10% reduction in $D_{vol}$ per request (via feature compression) reduces total data movement proportionally, while a 2$\times$ increase in $R_{peak}$ (via hardware upgrade) only improves the compute term, meaning the binding constraint must be identified before any optimization investment yields returns.
1. **Significance (Quantitative):** Every performance budget traces back to the Iron Law: $T = D_{\text{vol}}/BW + O/(R_{\text{peak}} \cdot \eta) + L_{\text{lat}}$. In a production recommendation system serving 10 million requests per day, a 10% reduction in $D_{\text{vol}}$ per request (via feature compression) reduces total data movement proportionally, while a 2$\times$ increase in $R_{\text{peak}}$ (via hardware upgrade) only improves the compute term, meaning the binding constraint must be identified before any optimization investment yields returns.
2. **Distinction (Durable):** Unlike traditional software, whose correctness degrades only when code changes, an ML system's accuracy degrades when the world changes. Model weights are fixed after deployment, but the distribution of inputs relative to what the model learned shifts continuously, eroding accuracy silently without any error or exception.
3. **Common Pitfall:** A frequent misconception is that an ML system is the model. Google's 2015 audit of production ML systems found that model code comprises approximately 5% of total lines; the other 95% is data pipelines, serving infrastructure, and monitoring [@sculley2015hidden].
@@ -1220,9 +1220,9 @@ Recall the **Data · Algorithm · Machine (D·A·M) taxonomy** introduced at the
::: {.callout-definition title="The D·A·M Taxonomy"}
The ***D·A·M Taxonomy***\index{D·A·M taxonomy!definition} is a diagnostic framework that classifies any machine learning system performance bottleneck along three axes: **Data** ($D_{vol}$, $BW$), **Algorithm** ($O$, model architecture), and **Machine** ($R_{peak}$, memory capacity). The goal is to identify which axis is the binding constraint.
The ***D·A·M Taxonomy***\index{D·A·M taxonomy!definition} is a diagnostic framework that classifies any machine learning system performance bottleneck along three axes: **Data** ($D_{\text{vol}}$, $BW$), **Algorithm** ($O$, model architecture), and **Machine** ($R_{\text{peak}}$, memory capacity). The goal is to identify which axis is the binding constraint.
1. **Significance (Quantitative):** The diagnostic power is concrete. A ResNet-50 inference run at batch size 1 is memory-bandwidth-bound (Machine axis): the A100's 2 TB/s bandwidth moves 25 MB of weights per forward pass in 12.5 μs, while the 4 GFLOP compute finishes in 2 μs. This 6$\times$ gap means hardware upgrades to $R_{peak}$ yield no improvement until the bandwidth bottleneck is resolved first.
1. **Significance (Quantitative):** The diagnostic power is concrete. A ResNet-50 inference run at batch size 1 is memory-bandwidth-bound (Machine axis): the A100's 2 TB/s bandwidth moves 25 MB of weights per forward pass in 12.5 μs, while the 4 GFLOP compute finishes in 2 μs. This 6$\times$ gap means hardware upgrades to $R_{\text{peak}}$ yield no improvement until the bandwidth bottleneck is resolved first.
2. **Distinction (Durable):** Unlike traditional software performance analysis, which treats code and data as separate concerns, the D·A·M taxonomy recognizes that algorithm choice directly determines both the data volume required (a Transformer needs orders of magnitude more data than a linear model to generalize) and the machine required to run it.
3. **Common Pitfall:** A frequent misconception is that the three axes are independent. Changing the algorithm (e.g., switching from a CNN to a Transformer) typically mandates a different machine (more memory for $O(N^2)$ attention) and a different data distribution (more diverse training data to avoid overfitting the flexible hypothesis space).
@@ -1384,9 +1384,9 @@ The D·A·M taxonomy provides the diagnostic lens, but to build systems, we must
\index{Engineering Crux!hierarchy}
Every machine learning system analyzed in this text is constructed from four hierarchical layers. This **Engineering Crux** transforms raw physical constraints into functional user applications, ensuring that a decision made at the silicon level is traceable to its impact on the final mission.
1. **Hardware (The Silicon)**: The physical foundation (The Engine). This layer defines the raw capabilities: $R_{peak}$, $BW$, and memory capacity. We use real-world hardware "Twins" like the **NVIDIA H100** and **ESP32-S3**.
1. **Hardware (The Silicon)**: The physical foundation (The Engine). This layer defines the raw capabilities: $R_{\text{peak}}$, $BW$, and memory capacity. We use real-world hardware "Twins" like the **NVIDIA H100** and **ESP32-S3**.
2. **Systems (The Platforms)**: The integrated deployment unit (The Car). This layer defines the "Envelope" in which hardware operates: power budget, thermal limits, and node-level interconnects. Examples include the **Training Cluster Node** or the **Sub-Watt Sensor Node**.
3. **Workloads (The Models)**: The algorithmic demand (The Route). This layer defines the mathematical workload: operation count ($O$), parameter volume ($D_{vol}$), and data layout. We use **Lighthouse Workloads** like **GPT-4** and **Wake Vision**.
3. **Workloads (The Models)**: The algorithmic demand (The Route). This layer defines the mathematical workload: operation count ($O$), parameter volume ($D_{\text{vol}}$), and data layout. We use **Lighthouse Workloads** like **GPT-4** and **Wake Vision**.
4. **Missions (The Scenarios)**: The application context (The Destination). This is the top of the stack, where a system is deployed to solve a specific problem. A **Mission**, such as the **Smart Doorbell**, introduces high-level requirements (e.g., "1-year battery life") that dictate the configuration of every layer below.
This hierarchy ensures that when we build a lab or a case study, we are not starting from scratch. We are "inheriting" the constraints of a System Archetype and applying a Lighthouse Model to a specific mission. For instance, the **Smart Doorbell** scenario (@sec-introduction-deployment-case-studies-636f) inherits the **TinyML Archetype**, uses the **Wake Vision** model, and operates on **ESP32** hardware. This structured approach allows us to reason about the "Physics of ML" across any application domain.
@@ -1398,7 +1398,7 @@ These three components interact through a single economic constraint that system
::: {.callout-perspective title="Samples per Dollar"}
**The Systems View**: While researchers optimize for *accuracy*, systems engineers optimize for **Samples per Dollar**. Let **Model Size** be the parameter count, **Dataset Size** the number of training samples, and **Hardware Efficiency** the compute throughput per dollar (FLOPs/$, where FLOPS are floating-point operations per second, formalized in the Iron Law below). This metric unifies the three axes of the D·A·M taxonomy into a single constraint equation, shown in @eq-cost-scaling:
**The Systems View**: While researchers optimize for *accuracy*, systems engineers optimize for **Samples per Dollar**. Let **Model Size** be the parameter count, **Dataset Size** the number of training samples, and **Hardware Efficiency** the compute throughput per dollar (FLOPs/\$, where FLOPS are floating-point operations per second, formalized in the Iron Law below). This metric unifies the three axes of the D·A·M taxonomy into a single constraint equation, shown in @eq-cost-scaling:
$$ \text{Cost} \propto \frac{\text{Model Size} \times \text{Dataset Size}}{\text{Hardware Efficiency}} $$ {#eq-cost-scaling}
@@ -1458,13 +1458,13 @@ The Degradation Equation reveals *what goes wrong* with ML systems: silent relia
\index{Iron Law of ML Systems!definition}
To reason about ML systems as engineers, we need more than qualitative descriptions. We need a quantitative framework that connects every layer of the stack. Just as classical mechanics is governed by Newton's laws and processor performance is governed by the Iron Law of Processor Performance, machine learning system performance is governed by the **Iron Law of ML Systems**\index{Iron Law of ML Systems!equation}, formalized in @eq-intro-iron-law:
$$\text{Time}_{\text{total}} = \underbrace{ \frac{\text{Data} (D_{vol})}{\text{Bandwidth} (BW)} }_{\text{The Data Term}} + \underbrace{ \frac{\text{Ops} (O)}{\text{Peak} (R_{peak}) \times \text{Efficiency} (\eta)} }_{\text{The Compute Term}} + \underbrace{ \text{Overhead} (L_{lat}) }_{\text{The Latency Term}}$$ {#eq-intro-iron-law}
$$\text{Time}_{\text{total}} = \underbrace{ \frac{\text{Data} (D_{\text{vol}})}{\text{Bandwidth} (BW)} }_{\text{The Data Term}} + \underbrace{ \frac{\text{Ops} (O)}{\text{Peak} (R_{\text{peak}}) \times \text{Efficiency} (\eta)} }_{\text{The Compute Term}} + \underbrace{ \text{Overhead} (L_{\text{lat}}) }_{\text{The Latency Term}}$$ {#eq-intro-iron-law}
*This equation is the mathematical spine of this book.* It decomposes the total time required for any ML task, whether training a model for weeks or serving an inference in milliseconds, into three terms that correspond directly to the physical constraints of the Dual Mandate introduced earlier:
1. **The Data Term ($D_{vol}/BW$)**: The physical cost of moving bits. $D_{vol}$ is the volume of data moved (bytes), and $BW$ is the memory or network bandwidth (bytes/sec). Whether loading terabytes from cloud storage or fetching weights from high-bandwidth memory, performance is often limited by I/O physics. We address this in **Part I: Foundations**.
2. **The Compute Term ($O / (R_{peak} \cdot \eta)$)**: The cost of arithmetic. $O$ is the number of floating-point operations. $R_{peak}$ is the hardware's theoretical peak throughput (FLOPS). $\eta$ (eta) is the utilization factor ($0 \le \eta \le 1$), representing software efficiency. We address this in **Part II: Build** and **Part III: Optimize**.
3. **The Overhead Term ($L_{lat}$)**: The irreducible "tax" of system orchestration, networking, and serialization. This fixed latency dominates in real-time deployment. We address this in **Part IV: Deploy**.
1. **The Data Term ($D_{\text{vol}}/BW$)**: The physical cost of moving bits. $D_{\text{vol}}$ is the volume of data moved (bytes), and $BW$ is the memory or network bandwidth (bytes/sec). Whether loading terabytes from cloud storage or fetching weights from high-bandwidth memory, performance is often limited by I/O physics. We address this in **Part I: Foundations**.
2. **The Compute Term ($O / (R_{\text{peak}} \cdot \eta)$)**: The cost of arithmetic. $O$ is the number of floating-point operations. $R_{\text{peak}}$ is the hardware's theoretical peak throughput (FLOPS). $\eta$ (eta) is the utilization factor ($0 \le \eta \le 1$), representing software efficiency. We address this in **Part II: Build** and **Part III: Optimize**.
3. **The Overhead Term ($L_{\text{lat}}$)**: The irreducible "tax" of system orchestration, networking, and serialization. This fixed latency dominates in real-time deployment. We address this in **Part IV: Deploy**.
::: {.callout-perspective title="The Iron Law Analogy"}
@@ -1472,7 +1472,7 @@ We call this the "Iron Law" by analogy to Patterson & Hennessy's Iron Law of Pro
The additive form assumes sequential execution; in practice, systems can **overlap** these terms, transforming the sum into a max as @eq-intro-iron-law-pipelined shows:
$$T_{pipelined} = \max\left(\frac{D_{vol}}{BW}, \frac{O}{R_{peak} \cdot \eta}\right) + L_{lat}$$ {#eq-intro-iron-law-pipelined}
$$T_{pipelined} = \max\left(\frac{D_{\text{vol}}}{BW}, \frac{O}{R_{\text{peak}} \cdot \eta}\right) + L_{\text{lat}}$$ {#eq-intro-iron-law-pipelined}
\index{Amdahl's Law!diagnostic analogy}
We retain "Iron Law" because, like **Amdahl's Law**, its value lies in **diagnostic power**: identifying which physical constraint dominates before optimizing. The Iron Law is useful precisely because it simplifies the complexity of the full stack into three manageable terms. @sec-dam-taxonomy presents the refined treatment, including pipelining and overlap techniques that transform the additive model into the max-based formulation used in practice.
@@ -1588,7 +1588,7 @@ class GPT3Training:
peak_tflops_str = fmt(peak_tflops, precision=0, commas=False)
# Equation assembly (inline refs use GPT3Training.* form)
time_formula_math = md_math(r"\text{Time} \approx \frac{O}{N \cdot R_{peak} \cdot \eta}")
time_formula_math = md_math(r"\text{Time} \approx \frac{O}{N \cdot R_{\text{peak}} \cdot \eta}")
time_value_math = md_math(
rf"\approx \frac{{{ops_coeff_str} \times 10^{{{ops_exp_value}}}}}{{{num_gpus} \times ({peak_tflops_str} \times 10^{{12}}) \times {eta_base}}}"
)
@@ -1624,23 +1624,23 @@ The equation is dimensionally consistent: each term resolves to seconds. One can
The **Iron Law** governs *time*, but time is not the only constraint. For mobile devices, edge systems, and large-scale training clusters, *energy* often matters more than raw speed.
Just as time is governed by physics, so is energy. We must add a fourth term to our mental model: **The Energy Tax.**\index{Energy Tax!data movement cost} In many modern systems (mobile, edge, and large-scale training), energy, not time, is the hard constraint. Let $D_{vol}$ be the total data volume moved (bytes), $E_{\text{move}}$ the energy per byte moved, $O$ the total operation count, and $E_{\text{compute}}$ the energy per operation. @eq-energy-cost formalizes this relationship:
Just as time is governed by physics, so is energy. We must add a fourth term to our mental model: **The Energy Tax.**\index{Energy Tax!data movement cost} In many modern systems (mobile, edge, and large-scale training), energy, not time, is the hard constraint. Let $D_{\text{vol}}$ be the total data volume moved (bytes), $E_{\text{move}}$ the energy per byte moved, $O$ the total operation count, and $E_{\text{compute}}$ the energy per operation. @eq-energy-cost formalizes this relationship:
$$ \text{Energy}_{\text{total}} \approx \underbrace{ D_{vol} \times E_{\text{move}} }_{\text{Dominant Term}} + \underbrace{ O \times E_{\text{compute}} }_{\text{Secondary Term}} $$ {#eq-energy-cost}
$$ \text{Energy}_{\text{total}} \approx \underbrace{ D_{\text{vol}} \times E_{\text{move}} }_{\text{Dominant Term}} + \underbrace{ O \times E_{\text{compute}} }_{\text{Secondary Term}} $$ {#eq-energy-cost}
The dominant term is data movement: $E_{\text{move}} \gg E_{\text{compute}}$. Moving a byte of data from memory often costs 100$\times$ more energy than performing a floating-point operation on it. The physical reason is that data movement requires charging and discharging wires over macroscopic distances, while arithmetic is performed locally within a processing unit's circuits. Therefore, **minimizing data movement ($D_{vol}$)** is the primary lever for both speed *and* energy efficiency.
The dominant term is data movement: $E_{\text{move}} \gg E_{\text{compute}}$. Moving a byte of data from memory often costs 100$\times$ more energy than performing a floating-point operation on it. The physical reason is that data movement requires charging and discharging wires over macroscopic distances, while arithmetic is performed locally within a processing unit's circuits. Therefore, **minimizing data movement ($D_{\text{vol}}$)** is the primary lever for both speed *and* energy efficiency.
The relationship between time, energy, and data movement forms the central analytical tool of this book.
::: {.callout-checkpoint title="The Iron Law"}
The **Iron Law** ($T \approx \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat}$) is the analytical backbone of this book. Before proceeding, verify you can manipulate its terms:
The **Iron Law** ($T \approx \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$) is the analytical backbone of this book. Before proceeding, verify you can manipulate its terms:
- [ ] **Data Term ($D_{vol}/BW$)**: Bound by memory bandwidth. Dominates in Transformers and Large Language Models where we move massive weights for every token.
- [ ] **Compute Term ($O/R_{peak}$)**: Bound by processor speed. Dominates in ConvNets (ResNet) where we reuse weights many times.
- [ ] **Latency Term ($L_{lat}$)**: Bound by physics and software overhead. Dominates in Inference and small-batch regimes.
- [ ] **Data Term ($D_{\text{vol}}/BW$)**: Bound by memory bandwidth. Dominates in Transformers and Large Language Models where we move massive weights for every token.
- [ ] **Compute Term ($O/R_{\text{peak}}$)**: Bound by processor speed. Dominates in ConvNets (ResNet) where we reuse weights many times.
- [ ] **Latency Term ($L_{\text{lat}}$)**: Bound by physics and software overhead. Dominates in Inference and small-batch regimes.
*Self-Test: If you double the processor speed ($R_{peak}$), which term does it improve?*
*Self-Test: If you double the processor speed ($R_{\text{peak}}$), which term does it improve?*
:::
@@ -1672,7 +1672,7 @@ Each archetype represents a distinct extreme of the Iron Law. For instance, **Re
: **Lighthouse Models as Systems Detectives**: Each workload isolates a distinct bottleneck, enabling systematic investigation of how system constraints affect different architectural patterns. Quantitative specifications and architectural details appear in @sec-network-architectures. {#tbl-lighthouse-examples}
The Iron Law makes these differences precise. ResNet-50 applies the same small weight filters across millions of spatial positions, reusing each weight thousands of times; its $O/(R_{peak} \cdot \eta)$ term dominates because the processor must sustain enormous arithmetic throughput while the data footprint remains modest. GPT-2, by contrast, loads billions of unique weight parameters for every token it generates, and each weight is used only once before the next must be fetched; its $D_{vol}/BW$ term dominates because memory bandwidth, not arithmetic, is the binding constraint. The same equation, applied to two different workloads, yields opposite diagnoses and therefore opposite optimization strategies: doubling $R_{peak}$ accelerates ResNet-50 but barely affects GPT-2, while doubling $BW$ has the reverse effect.
The Iron Law makes these differences precise. ResNet-50 applies the same small weight filters across millions of spatial positions, reusing each weight thousands of times; its $O/(R_{\text{peak}} \cdot \eta)$ term dominates because the processor must sustain enormous arithmetic throughput while the data footprint remains modest. GPT-2, by contrast, loads billions of unique weight parameters for every token it generates, and each weight is used only once before the next must be fetched; its $D_{\text{vol}}/BW$ term dominates because memory bandwidth, not arithmetic, is the binding constraint. The same equation, applied to two different workloads, yields opposite diagnoses and therefore opposite optimization strategies: doubling $R_{\text{peak}}$ accelerates ResNet-50 but barely affects GPT-2, while doubling $BW$ has the reverse effect.
Each archetype manifests different constraints along the D·A·M axes, ensuring that the principles developed throughout this text are tested against the diversity of real-world systems engineering challenges. Later in this chapter, we complement these technical workloads with three deployment case studies, Waymo, FarmBeats, and AlphaFold, that illustrate how the same core challenges manifest in production systems under radically different constraints.
@@ -2057,7 +2057,7 @@ The Iron Law decomposes performance into physical terms, the Degradation Equatio
***AI Engineering***\index{AI Engineering!definition} is the engineering discipline of designing, deploying, and maintaining systems whose outputs are inherently probabilistic (stochastic) to meet deterministic reliability targets by simultaneously satisfying constraints on all three D·A·M axes (Data quality, Algorithm correctness, Machine efficiency) in production.
1. **Significance (Quantitative):** ML Research typically optimizes only the Algorithm axis ($O$ and convergence). AI Engineering jointly optimizes all three: it bounds $D_{vol}$ by data governance requirements, bounds $O/(R_{peak} \cdot \eta)$ by latency SLOs, and bounds total power draw by energy and cost budgets. A production system that achieves 95% accuracy in research but violates a 100 ms p99 SLO is a failed system, regardless of its Algorithm score.
1. **Significance (Quantitative):** ML Research typically optimizes only the Algorithm axis ($O$ and convergence). AI Engineering jointly optimizes all three: it bounds $D_{\text{vol}}$ by data governance requirements, bounds $O/(R_{\text{peak}} \cdot \eta)$ by latency SLOs, and bounds total power draw by energy and cost budgets. A production system that achieves 95% accuracy in research but violates a 100 ms p99 SLO is a failed system, regardless of its Algorithm score.
2. **Distinction (Durable):** Unlike **Machine Learning Research**, which targets a single objective (validation loss) on a static dataset, AI Engineering targets a multi-objective constraint surface (latency, throughput, accuracy, cost, fairness, and robustness) on a distribution that shifts continuously after deployment.
3. **Common Pitfall:** A frequent misconception is that AI Engineering is just "Software Engineering for ML." The critical difference is that the system specification is probabilistic: an ML system's output is statistically valid or invalid relative to a shifting distribution, not correct or incorrect relative to a fixed deterministic contract. This makes continuous monitoring a structural requirement, not an operational choice.
@@ -2198,7 +2198,7 @@ At the other end, TinyML systems\index{TinyML} run on microcontrollers[^fn-micro
\index{Latency!etymology}
The space between these poles contains a rich variety of ML systems adapted for specific contexts. Edge ML systems\index{Edge ML!latency and bandwidth} bring computation closer to data sources, reducing latency[^fn-latency-responsiveness] and bandwidth requirements while managing local computing resources. Mobile ML systems\index{Mobile ML!resource constraints} must balance sophisticated capabilities with severe constraints. Modern smartphones typically have 4–12 GB RAM, ARM processors operating at 1.5–3 GHz, and power budgets of 2–5 W that must be shared across all system functions. For example, running a state-of-the-art image classification model on a smartphone might consume 100–500 mW and complete inference in 10–100 ms, compared to cloud servers that can use 200+ W but deliver results in under 1 ms. Enterprise ML systems often operate within specific business constraints, focusing on particular tasks while integrating with existing infrastructure. Some organizations employ hybrid approaches, distributing ML capabilities across multiple tiers to balance various requirements.
[^fn-latency-responsiveness]: **Latency**: From Latin *latere* ("to lie hidden"), a fitting etymology because delay is invisible until it causes failure. Autonomous braking requires <10 ms end-to-end; at highway speeds, every additional millisecond adds roughly 3 cm of stopping distance. This makes $L_{lat}$ rather than throughput the binding constraint for edge deployment, and explains why latency-critical systems cannot offload inference to distant cloud servers regardless of their superior compute. \index{Latency!edge deployment}
[^fn-latency-responsiveness]: **Latency**: From Latin *latere* ("to lie hidden"), a fitting etymology because delay is invisible until it causes failure. Autonomous braking requires <10 ms end-to-end; at highway speeds, every additional millisecond adds roughly 3 cm of stopping distance. This makes $L_{\text{lat}}$ rather than throughput the binding constraint for edge deployment, and explains why latency-critical systems cannot offload inference to distant cloud servers regardless of their superior compute. \index{Latency!edge deployment}
Each position on this deployment spectrum creates distinct bottlenecks that determine which efficiency dimensions matter most, as summarized in @tbl-efficiency-priorities:

6
book/quarto/contents/vol1/ml_ops/ml_ops.qmd
View File

@@ -349,7 +349,7 @@ In the ML context, this concept takes on a specific meaning:
1. **Significance (Quantitative):** It arises because ML systems have all the maintenance problems of traditional code plus new ML-specific drivers: **Entanglement** (changing one feature affects everything), **Correction Cascades**, and **Undeclared Consumers**.
2. **Distinction (Durable):** Unlike **Software Technical Debt** (which manifest as **Lower Productivity**), ML Technical Debt manifests as **Silent Performance Degradation** and unpredictable failures.
3. **Common Pitfall:** A frequent misconception is that "Better Code" solves Technical Debt in ML. In reality, it is a **Systems Architecture Problem**: it occurs when the assumptions of the training data ($D_{vol}$) and the model architecture ($O$) are no longer enforced at the system boundary.
3. **Common Pitfall:** A frequent misconception is that "Better Code" solves Technical Debt in ML. In reality, it is a **Systems Architecture Problem**: it occurs when the assumptions of the training data ($D_{\text{vol}}$) and the model architecture ($O$) are no longer enforced at the system boundary.
:::
@@ -2170,8 +2170,8 @@ Infrastructure-level monitoring tracks indicators such as CPU and GPU utilizatio
These utilization patterns map directly to the Iron Law of ML Systems (@sec-introduction-iron-law-ml-systems-c32a). Monitoring reveals which term dominates:
- **Compute-bound** (high GPU util, low memory BW util): Limited by $O/(R_{peak} \cdot \eta)$. Optimize kernels, use Tensor Cores, or upgrade hardware.
- **Memory-bound** (moderate GPU util, high memory BW util): Limited by $D_{vol}/BW$. Optimize with quantization, pruning, or batching.
- **Compute-bound** (high GPU util, low memory BW util): Limited by $O/(R_{\text{peak}} \cdot \eta)$. Optimize kernels, use Tensor Cores, or upgrade hardware.
- **Memory-bound** (moderate GPU util, high memory BW util): Limited by $D_{\text{vol}}/BW$. Optimize with quantization, pruning, or batching.
- **I/O-bound** (low GPU util, low memory BW util): Limited by data pipeline latency. Fix the DataLoader, not the model.
The Iron Law doubles as a *diagnostic framework* for production systems. When latency SLAs are violated, the monitoring dashboard indicates which term to investigate.

88
book/quarto/contents/vol1/ml_systems/ml_systems.qmd
View File

@@ -527,9 +527,9 @@ Knowing *that* these barriers exist is necessary but not sufficient. Given a spe
\index{Silicon Contract!Iron Law}The central analytical tool for this chapter is the **Iron Law of ML Systems**, established in @sec-introduction (@sec-introduction-iron-law-ml-systems-c32a) and restated here as @eq-iron-law:
$$T = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat}$$ {#eq-iron-law}
$$T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$$ {#eq-iron-law}
This equation decomposes total latency into three terms: data movement ($D_{vol}/BW$), compute ($O / (R_{peak} \cdot \eta)$), and fixed overhead ($L_{lat}$). For a single inference, these costs simply add up—you pay each one sequentially. In production systems, however, tasks are processed continuously as a stream, and the question shifts from "*how* long does one task take?" to "*which* of these three terms actually limits the system?" The answer depends entirely on the deployment environment: a model that is compute-bound during training may become memory-bound during inference; a system that runs efficiently in the cloud may hit power limits on mobile devices. To determine which term dominates, we need a companion principle.
This equation decomposes total latency into three terms: data movement ($D_{\text{vol}}/BW$), compute ($O / (R_{\text{peak}} \cdot \eta)$), and fixed overhead ($L_{\text{lat}}$). For a single inference, these costs simply add up—you pay each one sequentially. In production systems, however, tasks are processed continuously as a stream, and the question shifts from "*how* long does one task take?" to "*which* of these three terms actually limits the system?" The answer depends entirely on the deployment environment: a model that is compute-bound during training may become memory-bound during inference; a system that runs efficiently in the cloud may hit power limits on mobile devices. To determine which term dominates, we need a companion principle.
### The Bottleneck Principle {#sec-ml-systems-bottleneck-principle-3514}
@@ -537,14 +537,14 @@ This equation decomposes total latency into three terms: data movement ($D_{vol}
\index{pipelined execution!throughput analysis}
The Iron Law tells us the cost of each term. The **Bottleneck Principle** tells us which term *matters*. Unlike traditional software where optimizing the average case works, ML systems are dominated by their slowest component: optimizing fast operations yields zero benefit while the slowest stage remains unchanged. Modern accelerators use **pipelined execution** to overlap data movement with computation: while the accelerator computes on batch $n$, the memory system prefetches batch $n+1$. With this overlap, whichever operation is slower determines the system's throughput—the faster one "hides" behind it. The Iron Law's sum becomes a maximum, as @eq-bottleneck formalizes:
$$ T_{bottleneck} = \max\left(\frac{D_{vol}}{BW}, \frac{O}{R_{peak} \cdot \eta}, T_{network}\right) + L_{lat} $$ {#eq-bottleneck}
$$ T_{bottleneck} = \max\left(\frac{D_{\text{vol}}}{BW}, \frac{O}{R_{\text{peak}} \cdot \eta}, T_{network}\right) + L_{\text{lat}} $$ {#eq-bottleneck}
* **$\frac{D_{vol}}{BW}$ (Memory)**: Time to move data between memory and processor.
* **$\frac{O}{R_{peak} \cdot \eta}$ (Compute)**: Time to execute calculations.
* **$\frac{D_{\text{vol}}}{BW}$ (Memory)**: Time to move data between memory and processor.
* **$\frac{O}{R_{\text{peak}} \cdot \eta}$ (Compute)**: Time to execute calculations.
* **$T_{network}$**: Time for network communication (if offloading).
* **$L_{lat}$ (Overhead)**: Fixed latency (kernel launch, runtime overhead).
* **$L_{\text{lat}}$ (Overhead)**: Fixed latency (kernel launch, runtime overhead).
This principle dictates that if your system is **Memory Bound**\index{memory-bound workloads!optimization strategy}\index{compute-bound vs memory-bound!memory-bound} ($D_{vol}/BW > O/(R_{peak} \cdot \eta)$), buying faster processors ($R_{peak}$) yields exactly **0% speedup**—just as widening a six-lane highway yields no benefit when all traffic must funnel through a two-lane bridge. You must identify the dominant term before optimizing. This trade-off is governed by *the energy of transmission*.
This principle dictates that if your system is **Memory Bound**\index{memory-bound workloads!optimization strategy}\index{compute-bound vs memory-bound!memory-bound} ($D_{\text{vol}}/BW > O/(R_{\text{peak}} \cdot \eta)$), buying faster processors ($R_{\text{peak}}$) yields exactly **0% speedup**—just as widening a six-lane highway yields no benefit when all traffic must funnel through a two-lane bridge. You must identify the dominant term before optimizing. This trade-off is governed by *the energy of transmission*.
```{python}
#| label: energy-transmission-calc
@@ -602,13 +602,13 @@ class EnergyTransmission:
**The Variables**:
* **Data ($D_{vol}$)**: `{python} EnergyTransmission.data_mb_str` MB (e.g., 1 second of audio).
* **Data ($D_{\text{vol}}$)**: `{python} EnergyTransmission.data_mb_str` MB (e.g., 1 second of audio).
* **Transmission Energy ($E_{tx}$)**: `{python} EnergyTransmission.tx_energy_str` mJ/MB (Wi-Fi/LTE).
* **Compute Energy ($E_{op}$)**: `{python} EnergyTransmission.compute_energy_str` mJ/inference (MobileNet on NPU).
**The Calculation**:
1. **Cloud Approach**: $E_{cloud} \approx D_{vol} \times E_{tx}$ = `{python} EnergyTransmission.data_mb_str` MB$\times$ `{python} EnergyTransmission.tx_energy_str` mJ/MB = **`{python} EnergyTransmission.cloud_total_str` mJ**.
1. **Cloud Approach**: $E_{cloud} \approx D_{\text{vol}} \times E_{tx}$ = `{python} EnergyTransmission.data_mb_str` MB$\times$ `{python} EnergyTransmission.tx_energy_str` mJ/MB = **`{python} EnergyTransmission.cloud_total_str` mJ**.
2. **Local Approach**: $E_{local} \approx$ Inference = **`{python} EnergyTransmission.local_total_str` mJ**.
**The Systems Conclusion**: Transmitting raw data is **`{python} EnergyTransmission.ratio_str`$\times$ more expensive** than processing it locally. Even if the cloud had infinite speed ($Time \approx 0$), the **Energy Wall** makes cloud offloading physically impossible for always-on battery devices. The "Machine" constraint (Battery) dictates the "Algorithm" choice (TinyML).
@@ -620,11 +620,11 @@ The **Iron Law's** variables interact differently across deployment scenarios. B
::: {.callout-definition title="The Iron Law"}
***The Iron Law***\index{Iron Law!definition} is the fundamental physical constraint governing all machine learning performance, expressed as the total time $T$ required for a workload:
$$T = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat}$$
$$T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$$
1. **Significance (Quantitative):** It defines the **Physical Ceiling** for any system by quantifying the relationship between data volume ($D_{vol}$), compute capacity ($R_{peak}$), and communication overhead ($L_{lat}$).
1. **Significance (Quantitative):** It defines the **Physical Ceiling** for any system by quantifying the relationship between data volume ($D_{\text{vol}}$), compute capacity ($R_{\text{peak}}$), and communication overhead ($L_{\text{lat}}$).
2. **Distinction (Durable):** Unlike **Amdahl's Law**, which focuses on **Parallel Speedup**, the Iron Law addresses the **Total Energy and Time** required to move and transform data.
3. **Common Pitfall:** A frequent misconception is that these terms are independent. In reality, they are **Trade-off Axes**: for example, increasing batch size may improve the duty cycle ($\eta$) but also increase the data volume ($D_{vol}$) per request, potentially shifting a compute-bound problem to a memory-bound one.
3. **Common Pitfall:** A frequent misconception is that these terms are independent. In reality, they are **Trade-off Axes**: for example, increasing batch size may improve the duty cycle ($\eta$) but also increase the data volume ($D_{\text{vol}}$) per request, potentially shifting a compute-bound problem to a memory-bound one.
:::
@@ -649,7 +649,7 @@ These archetypes map naturally to deployment paradigms: Compute Beasts and Spars
\index{archetype!workload classification}
[^fn-archetype-bottleneck]: **Workload Archetype**: A classification of ML workloads by their dominant Iron Law bottleneck rather than their model family. The distinction matters because the optimization strategy differs fundamentally: a compute-bound workload benefits from faster arithmetic ($R_{peak}$), while a bandwidth-bound workload benefits only from wider memory buses ($BW$). Misidentifying the archetype wastes optimization effort on the wrong term of the Iron Law, as when teams add accelerator FLOPS to a memory-bound inference pipeline and observe zero speedup. \index{Archetype!workload classification}
[^fn-archetype-bottleneck]: **Workload Archetype**: A classification of ML workloads by their dominant Iron Law bottleneck rather than their model family. The distinction matters because the optimization strategy differs fundamentally: a compute-bound workload benefits from faster arithmetic ($R_{\text{peak}}$), while a bandwidth-bound workload benefits only from wider memory buses ($BW$). Misidentifying the archetype wastes optimization effort on the wrong term of the Iron Law, as when teams add accelerator FLOPS to a memory-bound inference pipeline and observe zero speedup. \index{Archetype!workload classification}
\index{paradigm!deployment regimes}
@@ -657,11 +657,11 @@ These archetypes map naturally to deployment paradigms: Compute Beasts and Spars
\index{latency!responsiveness constraint}
[^fn-latency-systems]: **Latency**: The time between issuing a request and receiving a result, corresponding to $L_{lat}$ in the Iron Law. The Light Barrier makes this floor irreducible: the speed of light in fiber imposes a ~36 ms minimum round trip across the continental US, consuming the entire latency budget of a 10 ms safety-critical system before any computation begins. Every millisecond consumed by distance is a millisecond unavailable for model inference, which is why the Light Barrier forces paradigm selection rather than mere optimization. \index{Latency!deployment constraint}
[^fn-latency-systems]: **Latency**: The time between issuing a request and receiving a result, corresponding to $L_{\text{lat}}$ in the Iron Law. The Light Barrier makes this floor irreducible: the speed of light in fiber imposes a ~36 ms minimum round trip across the continental US, consuming the entire latency budget of a 10 ms safety-critical system before any computation begins. Every millisecond consumed by distance is a millisecond unavailable for model inference, which is why the Light Barrier forces paradigm selection rather than mere optimization. \index{Latency!deployment constraint}
\index{bandwidth!memory wall}
[^fn-bandwidth-memory-wall]: **Memory Bandwidth (The Memory Wall)**: The term "Memory Wall" was coined by Wulf and McKee in 1995, who predicted that the processor-memory performance gap would eventually dominate system performance---a prediction that proved prescient for ML workloads where weight loading, not arithmetic, is the typical bottleneck. In the Iron Law, bandwidth ($BW$) appears in the denominator of the data term $D_{vol}/BW$, so every doubling of model size that is not matched by a doubling of memory bandwidth directly increases wall-clock time. This asymmetry, growing at roughly 1.33$\times$ per year, is why modern ML systems are more often memory-bound than compute-bound. \index{Bandwidth!memory wall}
[^fn-bandwidth-memory-wall]: **Memory Bandwidth (The Memory Wall)**: The term "Memory Wall" was coined by Wulf and McKee in 1995, who predicted that the processor-memory performance gap would eventually dominate system performance---a prediction that proved prescient for ML workloads where weight loading, not arithmetic, is the typical bottleneck. In the Iron Law, bandwidth ($BW$) appears in the denominator of the data term $D_{\text{vol}}/BW$, so every doubling of model size that is not matched by a doubling of memory bandwidth directly increases wall-clock time. This asymmetry, growing at roughly 1.33$\times$ per year, is why modern ML systems are more often memory-bound than compute-bound. \index{Bandwidth!memory wall}
\index{critical path!latency determination}
@@ -868,19 +868,19 @@ These hardware differences translate directly into performance bottlenecks. To u
\index{system bottlenecks!dominant constraints}The pipelined form of the **Iron Law of ML Systems** from @sec-introduction-iron-law-ml-systems-c32a states that execution time is bounded by the slowest resource, as @eq-iron-law-extended formalizes:
$$T = \max\left( \frac{O}{R_{peak} \cdot \eta}, \frac{D_{vol}}{BW}, \frac{D_{vol}}{BW_{IO}} \right) + L_{lat}$$ {#eq-iron-law-extended}
$$T = \max\left( \frac{O}{R_{\text{peak}} \cdot \eta}, \frac{D_{\text{vol}}}{BW}, \frac{D_{\text{vol}}}{BW_{IO}} \right) + L_{\text{lat}}$$ {#eq-iron-law-extended}
Here, $O$ represents total operations, $R_{peak}$ is peak compute rate, $\eta$ is hardware utilization efficiency, $D_{vol}$ is data volume, $BW$ is memory bandwidth, $BW_{IO}$ is I/O bandwidth (storage or network), and $L_{lat}$ is fixed overhead. The equation identifies which resource (compute, memory, or I/O) limits performance. For a systematic diagnostic guide to identifying these bottlenecks, consult the D·A·M taxonomy\index{D·A·M taxonomy!bottleneck diagnosis} (@sec-dam-taxonomy).
Here, $O$ represents total operations, $R_{\text{peak}}$ is peak compute rate, $\eta$ is hardware utilization efficiency, $D_{\text{vol}}$ is data volume, $BW$ is memory bandwidth, $BW_{IO}$ is I/O bandwidth (storage or network), and $L_{\text{lat}}$ is fixed overhead. The equation identifies which resource (compute, memory, or I/O) limits performance. For a systematic diagnostic guide to identifying these bottlenecks, consult the D·A·M taxonomy\index{D·A·M taxonomy!bottleneck diagnosis} (@sec-dam-taxonomy).
The **dominant term varies by paradigm and workload**, changing the optimization strategy entirely:
| **Paradigm** | **Dominant Constraint** | **Why** | **Optimization Focus** |
|:------------------------|:--------------------------|:-----------------------------------------------------------|:---------------------------------------------|
| **Cloud Training** | $O/R_{peak}$ (Compute) | Abundant memory/network; FLOPS limit throughput | Maximize accelerator utilization, batch size |
| **Cloud LLM Inference** | $D_{vol}/BW$ (Memory BW) | Autoregressive: ~1 FLOP/byte, memory-bound | KV-caching, quantization, batching |
| **Edge Inference** | $D_{vol}/BW$ (Memory BW) | Limited HBM; models often memory-bound | Model compression, operator fusion |
| **Mobile** | Energy (implicit) | Battery = $\int \text{Power} \cdot dt$; thermal throttling | Reduced precision, duty cycling |
| **TinyML** | $D_{vol}/\text{Capacity}$ | 256 KB total; model must fit on-chip | Extreme compression, binary networks |
| **Paradigm** | **Dominant Constraint** | **Why** | **Optimization Focus** |
|:------------------------|:---------------------------------|:-----------------------------------------------------------|:---------------------------------------------|
| **Cloud Training** | $O/R_{\text{peak}}$ (Compute) | Abundant memory/network; FLOPS limit throughput | Maximize accelerator utilization, batch size |
| **Cloud LLM Inference** | $D_{\text{vol}}/BW$ (Memory BW) | Autoregressive: ~1 FLOP/byte, memory-bound | KV-caching, quantization, batching |
| **Edge Inference** | $D_{\text{vol}}/BW$ (Memory BW) | Limited HBM; models often memory-bound | Model compression, operator fusion |
| **Mobile** | Energy (implicit) | Battery = $\int \text{Power} \cdot dt$; thermal throttling | Reduced precision, duty cycling |
| **TinyML** | $D_{\text{vol}}/\text{Capacity}$ | 256 KB total; model must fit on-chip | Extreme compression, binary networks |
The same ResNet-50 model is **compute-bound**\index{compute-bound vs memory-bound!training vs inference}\index{roofline model!bottleneck analysis} during cloud training (high batch size, high arithmetic intensity) but **memory-bound** during single-image inference (batch=1, low arithmetic intensity) [@williams2009roofline]. Deployment paradigm selection must account for this shift.
@@ -1097,7 +1097,7 @@ class ResnetMobile:
- Compute time: $T_{\text{comp}}$ = `{python} ResnetCloud.cloud_compute_frac`
- Memory time: $T_{\text{mem}}$ = `{python} ResnetCloud.cloud_memory_frac`
- **Bottleneck**: `{python} ResnetCloud.cloud_bottleneck_str` (`{python} ResnetCloud.cloud_ratio_x_str`$\times$ slower than compute)
- **Arithmetic Intensity**: `{python} ResnetCloud.cloud_ai_frac` — this ratio of compute operations to bytes loaded measures how efficiently a workload uses the hardware. When arithmetic intensity exceeds the hardware's *compute-to-bandwidth ratio* ($R_{peak}/BW$), the workload is compute-bound; below it, the workload is memory-bound. For single-image inference, the low batch size yields low arithmetic intensity, explaining why even powerful GPUs are memory-bound at batch=1.
- **Arithmetic Intensity**: `{python} ResnetCloud.cloud_ai_frac` — this ratio of compute operations to bytes loaded measures how efficiently a workload uses the hardware. When arithmetic intensity exceeds the hardware's *compute-to-bandwidth ratio* ($R_{\text{peak}}/BW$), the workload is compute-bound; below it, the workload is memory-bound. For single-image inference, the low batch size yields low arithmetic intensity, explaining why even powerful GPUs are memory-bound at batch=1.
**(b) Mobile: Flagship NPU (batch=1, INT8)**
@@ -1336,7 +1336,7 @@ Cloud deployments range from single-machine instances (workstations, multi-GPU s
[^fn-cloud-utility-model]: **Cloud as Utility Computing**: The utility model allows providers to offer a specialized hardware portfolio that is economically infeasible for a single organization to maintain. This provides direct, on-demand access to the specific architectures required by each workload archetype: dense accelerator pods for Compute Beasts, HBM-equipped nodes for Bandwidth Hogs, and high-memory systems with fast interconnects for Sparse Scatter. A team can therefore rent a purpose-built, $10M+ supercomputing pod for a few hours rather than owning it. \index{Cloud Infrastructure!utility model}
[^fn-nlp-training-scale]: **LLM Training Scale**: GPT-3 required approximately 3,640 petaflop-days, 10,000 V100 GPUs, and an estimated \$4.6M in compute at 2020 cloud rates. This scale illustrates the core Cloud ML trade-off: only centralized infrastructure can aggregate enough $R_{peak}$ for peta-scale training, but the resulting $L_{lat}$ penalty (100--500 ms network round trip) makes that same infrastructure unsuitable for real-time inference. \index{LLM!training scale}
[^fn-nlp-training-scale]: **LLM Training Scale**: GPT-3 required approximately 3,640 petaflop-days, 10,000 V100 GPUs, and an estimated \$4.6M in compute at 2020 cloud rates. This scale illustrates the core Cloud ML trade-off: only centralized infrastructure can aggregate enough $R_{\text{peak}}$ for peta-scale training, but the resulting $L_{\text{lat}}$ penalty (100--500 ms network round trip) makes that same infrastructure unsuitable for real-time inference. \index{LLM!training scale}
What unifies these diverse cloud workloads is a single defining trade-off:
@@ -1344,9 +1344,9 @@ What unifies these diverse cloud workloads is a single defining trade-off:
***Cloud Machine Learning***\index{Cloud ML!definition} is the deployment paradigm that optimizes for **Resource Elasticity** by decoupling computational capacity from physical location.
1. **Significance (Quantitative):** It enables systems to scale resources ($R_{peak}$) proportional to workload variance, allowing for bursts of peta-flops that would be economically unfeasible to maintain locally.
1. **Significance (Quantitative):** It enables systems to scale resources ($R_{\text{peak}}$) proportional to workload variance, allowing for bursts of peta-flops that would be economically unfeasible to maintain locally.
2. **Distinction (Durable):** Unlike **Edge ML**, which prioritizes **Data Locality**, Cloud ML prioritizes **Computational Density** and centralized management.
3. **Common Pitfall:** A frequent misconception is that Cloud ML is "unlimited compute." In reality, it is constrained by the **Distance Penalty** ($L_{lat}$) and the **Ingestion Bottleneck** ($BW$), making it unsuitable for sub-10 ms real-time control loops.
3. **Common Pitfall:** A frequent misconception is that Cloud ML is "unlimited compute." In reality, it is constrained by the **Distance Penalty** ($L_{\text{lat}}$) and the **Ingestion Bottleneck** ($BW$), making it unsuitable for sub-10 ms real-time control loops.
:::
@@ -1499,7 +1499,7 @@ Beyond raw computation, cloud infrastructure creates deployment flexibility thro
A common misconception holds that Cloud ML's vast computational resources make it universally superior. Exceptional computational power and storage do not automatically translate to optimal solutions for all applications. The **Data Gravity Invariant**\index{Data Gravity Invariant!cloud limitations} (Part I) explains why: as data scales, the cost of moving it to compute ($C_{move}(D) \gg C_{move}(Compute)$) eventually dominates. The trade-offs listed in the definition above become concrete when we consider where edge and embedded deployments excel: real-time response with sub-10 ms decision making in autonomous control loops, strict data privacy for medical devices processing patient data, predictable costs through one-time hardware investment versus recurring cloud fees, or operation in disconnected environments such as industrial equipment in remote locations. The optimal deployment paradigm depends on specific application requirements rather than raw computational capability.
[^fn-cloud-elastic-cost]: **Pay-as-You-Go Pricing**: A cloud economic model where users pay for accelerator-hours consumed rather than hardware owned. Elastic pricing converts the fixed cost of idle $R_{peak}$ into a variable cost proportional to actual utilization, but the inverse also holds: sustained 24/7 workloads (continuous inference serving) often cost 2--3$\times$ more on cloud than equivalent on-premises hardware amortized over three years, a crossover that drives the TCO analysis later in this section. \index{Cloud Economics!elastic pricing}
[^fn-cloud-elastic-cost]: **Pay-as-You-Go Pricing**: A cloud economic model where users pay for accelerator-hours consumed rather than hardware owned. Elastic pricing converts the fixed cost of idle $R_{\text{peak}}$ into a variable cost proportional to actual utilization, but the inverse also holds: sustained 24/7 workloads (continuous inference serving) often cost 2--3$\times$ more on cloud than equivalent on-premises hardware amortized over three years, a crossover that drives the TCO analysis later in this section. \index{Cloud Economics!elastic pricing}
### Cloud ML Trade-offs and Constraints {#sec-ml-systems-cloud-ml-tradeoffs-constraints-96ed}
@@ -1827,17 +1827,17 @@ Recommendation engines deployed by Netflix and Amazon demonstrate another compel
These applications share a common thread: they trade latency for scale, accepting hundreds of milliseconds of round-trip delay in exchange for access to computational resources that no other paradigm can provide. Fraud detection systems analyzing millions of transactions, recommendation engines processing terabytes of embedding tables, and language models generating text one token at a time all depend on this bargain. Yet as the Voice Assistant Wall demonstrated, there exist applications where no amount of cloud compute can compensate for the physics of distance. When latency budgets drop below what the speed of light permits, or when data volumes exceed what networks can carry, the computation must move closer to the data source.
[^fn-dlrm-memory-bound]: **Deep Learning Recommendation Model (DLRM)**: Meta's 2019 architecture that exemplifies the "Sparse Scatter" archetype. Embedding tables for production recommendation systems can exceed 100 TB, making DLRM constrained by memory capacity and communication $BW$ rather than raw $R_{peak}$. This inversion of the typical compute-bound assumption forces specialized cluster designs where memory, not arithmetic, is the scarce resource. \index{DLRM!memory-bound constraint}
[^fn-dlrm-memory-bound]: **Deep Learning Recommendation Model (DLRM)**: Meta's 2019 architecture that exemplifies the "Sparse Scatter" archetype. Embedding tables for production recommendation systems can exceed 100 TB, making DLRM constrained by memory capacity and communication $BW$ rather than raw $R_{\text{peak}}$. This inversion of the typical compute-bound assumption forces specialized cluster designs where memory, not arithmetic, is the scarce resource. \index{DLRM!memory-bound constraint}
## Edge ML: Latency and Privacy {#sec-ml-systems-edge-ml-reducing-latency-privacy-risk-2625}
\index{Edge ML!distance penalty} \index{Edge ML!data sovereignty}When latency budgets drop below 100 ms, cloud infrastructure hits a hard physical wall. The Distance Penalty means the speed of light alone imposes minimum latencies of 40--150 ms for cross-region requests—before any computation begins. When an autonomous vehicle needs to decide whether to brake, or an industrial robot needs to stop before hitting an obstacle, 100 ms is an eternity. The logical engineering response is to move the computation closer to the data source.
Edge ML emerged from this constraint, trading unlimited computational resources for sub-100 ms latency and local data retention. In Archetype terms, edge deployment transforms the optimization target: a Bandwidth Hog workload like LLM inference that is memory-bound in the cloud becomes *latency-bound* at the edge, where the 50–100 ms network penalty dominates the 10–20 ms compute time. Edge hardware with sufficient local memory can eliminate this penalty entirely, shifting the bottleneck back to the underlying memory bandwidth constraint. Recall the Iron Law from @eq-iron-law-extended: by processing locally, edge deployment eliminates the $D_{vol}/BW_{IO}$ (network I/O) term entirely, collapsing the latency to $\max(D_{vol}/BW, O/(R_{peak} \cdot \eta)) + L_{lat}$—the same memory-vs-compute trade-off, but without the network penalty that dominates cloud inference.
Edge ML emerged from this constraint, trading unlimited computational resources for sub-100 ms latency and local data retention. In Archetype terms, edge deployment transforms the optimization target: a Bandwidth Hog workload like LLM inference that is memory-bound in the cloud becomes *latency-bound* at the edge, where the 50–100 ms network penalty dominates the 10–20 ms compute time. Edge hardware with sufficient local memory can eliminate this penalty entirely, shifting the bottleneck back to the underlying memory bandwidth constraint. Recall the Iron Law from @eq-iron-law-extended: by processing locally, edge deployment eliminates the $D_{\text{vol}}/BW_{IO}$ (network I/O) term entirely, collapsing the latency to $\max(D_{\text{vol}}/BW, O/(R_{\text{peak}} \cdot \eta)) + L_{\text{lat}}$—the same memory-vs-compute trade-off, but without the network penalty that dominates cloud inference.
This paradigm shift is essential for applications where cloud's 100--500 ms round-trip delays are unacceptable. Autonomous systems requiring split-second decisions and industrial IoT[^fn-iiot-edge-latency] applications demanding real-time response cannot tolerate network delays. Similarly, applications subject to strict data privacy regulations must process information locally rather than transmitting it to remote data centers. Edge devices (gateways and IoT hubs) occupy a middle ground in the deployment spectrum, maintaining acceptable performance while operating under intermediate resource constraints.
[^fn-iiot-edge-latency]: **Industrial IoT (IIoT)**: A domain where latency constraints are set by physical safety, not user perception. The 100+ ms round-trip delay mentioned is intolerable for a robotic arm that must halt within 5 ms of detecting a human. This forces computation to the edge, trading near-zero network latency for significant on-device compute ($R_{peak}$) constraints. \index{IIoT!latency constraint}
[^fn-iiot-edge-latency]: **Industrial IoT (IIoT)**: A domain where latency constraints are set by physical safety, not user perception. The 100+ ms round-trip delay mentioned is intolerable for a robotic arm that must halt within 5 ms of detecting a human. This forces computation to the edge, trading near-zero network latency for significant on-device compute ($R_{\text{peak}}$) constraints. \index{IIoT!latency constraint}
We define this paradigm formally as *Edge ML*.
@@ -1845,7 +1845,7 @@ We define this paradigm formally as *Edge ML*.
***Edge Machine Learning***\index{Edge ML!definition} is the deployment paradigm optimized for **Latency Determinism** and **Data Locality** by locating computation physically adjacent to data sources.
1. **Significance (Quantitative):** It circumvents the **Distance Penalty** ($L_{lat}$) of the cloud, trading elastic scale for a fixed **Local Compute Capacity** ($R_{peak}$).
1. **Significance (Quantitative):** It circumvents the **Distance Penalty** ($L_{\text{lat}}$) of the cloud, trading elastic scale for a fixed **Local Compute Capacity** ($R_{\text{peak}}$).
2. **Distinction (Durable):** Unlike **Cloud ML**, which prioritizes **Throughput**, Edge ML prioritizes **Determinism** and privacy. Unlike **TinyML**, Edge ML may still use workstation-class accelerators (GPGPUs).
3. **Common Pitfall:** A frequent misconception is that Edge ML refers to a specific hardware class. In reality, it is a **Location Paradigm**: it spans from IoT gateways to on-premise servers, unified by physical proximity to the data source.
@@ -2059,7 +2059,7 @@ The bandwidth calculation above reveals why edge processing is mandatory for hig
\index{Edge ML!privacy benefits}
Edge ML spans wearables, industrial sensors, and smart home appliances that process data locally[^fn-iot-data-wall] without depending on central servers. @fig-energy-per-inference quantifies the physical imperative: full-system energy per inference spans eight orders of magnitude across deployment paradigms, from ~10 µJ for a TinyML keyword spotter to ~1 kJ for a cloud LLM query. This 100,000,000$\times$ gap is not an engineering shortcoming to be optimized away; it reflects the irreducible costs of data movement, cooling, and network overhead that separate deployment tiers. Because edge devices operate within tight power envelopes, their memory bandwidth of 25--100 GB/s constrains deployable models to 100 MB--1 GB of parameters. This constraint, in turn, motivates the optimization techniques covered in @sec-model-compression, which achieve 2--4$\times$ speedup by compressing models to fit within these hardware budgets. The payoff extends beyond compute: processing 1000 camera feeds locally avoids 1 Gbps uplink costs because raw data never leaves the device, reducing cloud expenses by \$10,000--100,000 annually.
[^fn-iot-data-wall]: **IoT Data Wall**: Connected devices are projected to exceed 25 billion by 2030, each generating continuous sensor streams. The aggregate $D_{vol}$ from these devices already exceeds global network $BW$ capacity for centralized ingestion, making local edge processing not an optimization but a physical necessity: the data simply cannot all reach the cloud. \index{IoT!Data Wall}
[^fn-iot-data-wall]: **IoT Data Wall**: Connected devices are projected to exceed 25 billion by 2030, each generating continuous sensor streams. The aggregate $D_{\text{vol}}$ from these devices already exceeds global network $BW$ capacity for centralized ingestion, making local edge processing not an optimization but a physical necessity: the data simply cannot all reach the cloud. \index{IoT!Data Wall}
::: {#fig-energy-per-inference fig-env="figure" fig-pos="htb" fig-cap="**Energy Per Inference Across Deployment Paradigms.** Full-system energy consumption per inference spans eight orders of magnitude, from ~10 µJ for TinyML keyword spotting to ~1 kJ for a cloud LLM query. This gap is not an engineering shortcoming—it reflects the physics of data movement, cooling, and network overhead that separates deployment tiers. The 100,000,000× difference explains why always-on sensing is only feasible at the TinyML tier." fig-alt="Horizontal log-scale bar chart showing energy per inference for five workloads across four deployment paradigms. TinyML keyword spotting at 10 microjoules, Mobile MobileNet at 50 millijoules, Edge ResNet-50 at 500 millijoules, Cloud ResNet-50 at 10 joules, and Cloud GPT-4 query at 1 kilojoule."}
@@ -2147,10 +2147,10 @@ plt.show()
::: {.callout-definition title="The Data Locality Invariant"}
***The Data Locality Invariant*** states that a workload necessitates local processing whenever the transmission delay ($D_{vol}/BW_{net}$) dominates the remote response time:
$\text{Data Locality} \iff \frac{D_{vol}}{BW_{net}} > L_{net} + \frac{O}{R_{peak, remote}}$
***The Data Locality Invariant*** states that a workload necessitates local processing whenever the transmission delay ($D_{\text{vol}}/BW_{net}$) dominates the remote response time:
$\text{Data Locality} \iff \frac{D_{\text{vol}}}{BW_{net}} > L_{net} + \frac{O}{R_{peak, remote}}$
1. **Significance (Quantitative):** It defines the **Locality Crossover**, the point where adding cloud compute (increasing $R_{peak}$) yields zero benefit because the "Pipe" ($BW_{net}$) is too narrow for the "Volume" ($D_{vol}$).
1. **Significance (Quantitative):** It defines the **Locality Crossover**, the point where adding cloud compute (increasing $R_{\text{peak}}$) yields zero benefit because the "Pipe" ($BW_{net}$) is too narrow for the "Volume" ($D_{\text{vol}}$).
2. **Distinction (Durable):** Unlike **The Iron Law**, which optimizes for time-to-result, the Locality Invariant optimizes for architectural feasibility by identifying when network physics forbids remote offloading.
3. **Common Pitfall:** A frequent misconception is that 5G/6G "solves" locality. While these improve $BW_{net}$, they do not reduce $L_{net}$ below the Light Barrier, meaning latency-critical tasks remain inherently local.
@@ -2162,7 +2162,7 @@ $\text{Data Locality} \iff \frac{D_{vol}}{BW_{net}} > L_{net} + \frac{O}{R_{peak
**The Variables**:
- **Data ($D_{vol}$)**: 4K frame ≈ `{python} DataLocalityInvariant.frame_mb_str` MB.
- **Data ($D_{\text{vol}}$)**: 4K frame ≈ `{python} DataLocalityInvariant.frame_mb_str` MB.
- **Bandwidth ($BW_{net}$)**: `{python} DataLocalityInvariant.net_bw_str` Mbps home broadband (up).
- **Remote Latency ($L_{net}$)**: `{python} DataLocalityInvariant.remote_ms_str` ms (round-trip + remote compute).
@@ -2171,7 +2171,7 @@ $\text{Data Locality} \iff \frac{D_{vol}}{BW_{net}} > L_{net} + \frac{O}{R_{peak
1. **Transmission Time**: `{python} DataLocalityInvariant.frame_mb_str` MB $\times$ 8 bits / `{python} DataLocalityInvariant.net_bw_str` Mbps = **`{python} DataLocalityInvariant.tx_time_ms_str` ms**.
2. **Remote Response**: **`{python} DataLocalityInvariant.remote_ms_str` ms**.
**The Systems Conclusion**: Since `{python} DataLocalityInvariant.tx_time_ms_str` ms $\gg$ `{python} DataLocalityInvariant.remote_ms_str` ms, the system is **Bandwidth Blocked**. The cloud could have an infinite processor ($R_{peak} = \infty$), but the drone would still crash because it cannot move the bits fast enough. This workload is **Locality Mandatory**.
**The Systems Conclusion**: Since `{python} DataLocalityInvariant.tx_time_ms_str` ms $\gg$ `{python} DataLocalityInvariant.remote_ms_str` ms, the system is **Bandwidth Blocked**. The cloud could have an infinite processor ($R_{\text{peak}} = \infty$), but the drone would still crash because it cannot move the bits fast enough. This workload is **Locality Mandatory**.
:::
@@ -2366,11 +2366,11 @@ The Industrial IoT[^fn-industry40-feedback] uses edge ML for applications where
Smart buildings use edge ML to optimize energy consumption while maintaining operational continuity during network outages. Commercial buildings equipped with edge-based building management systems process data from thousands of sensors monitoring temperature, occupancy, air quality, and energy usage. This reduces cloud transmission requirements by an order of magnitude or more while enabling sub-second response times. Healthcare applications similarly use edge ML for patient monitoring and surgical assistance, maintaining HIPAA compliance through local processing while supporting low-latency workflows for real-time guidance.
These applications share a common assumption: the edge device is stationary and plugged into wall power. Recall the Iron Law (@eq-iron-law-extended): edge deployment eliminated the $D_{vol}/BW_{IO}$ network term that dominated cloud inference, but it still assumes unlimited energy. A factory edge server consuming 500 W around the clock is unremarkable when connected to mains power. Billions of users, however, carry their computing devices with them, and those devices run on fixed battery budgets. When we shift from stationary edge infrastructure to the smartphone in a user's pocket, a new term enters the optimization: $\text{Energy} = \text{Power} \times T$. The dominant constraint changes from latency to *energy per inference*, and with it, the entire engineering calculus.
These applications share a common assumption: the edge device is stationary and plugged into wall power. Recall the Iron Law (@eq-iron-law-extended): edge deployment eliminated the $D_{\text{vol}}/BW_{IO}$ network term that dominated cloud inference, but it still assumes unlimited energy. A factory edge server consuming 500 W around the clock is unremarkable when connected to mains power. Billions of users, however, carry their computing devices with them, and those devices run on fixed battery budgets. When we shift from stationary edge infrastructure to the smartphone in a user's pocket, a new term enters the optimization: $\text{Energy} = \text{Power} \times T$. The dominant constraint changes from latency to *energy per inference*, and with it, the entire engineering calculus.
[^fn-amazon-go-edge]: **Amazon Go**: The system's use of local edge servers is a direct response to the immense data volume from hundreds of in-store cameras. This architecture avoids having to upload the raw video—which would saturate a multi-gigabit uplink—while also keeping sensitive customer footage on-premises. The edge-first design is necessitated by the sheer scale of data processed, which can exceed 1 TB per hour in a single store. \index{Amazon Go!bandwidth constraint}
[^fn-industry40-feedback]: **Industry 4.0**: The fourth industrial revolution integrates ML into the sensor-actuator feedback loop on factory floors. The systems consequence is that the control loop latency ($L_{lat}$) must be shorter than the physical process it governs: a welding robot that detects a defect at 60 Hz has 16.7 ms to halt, a budget only edge inference can meet. \index{Industry 4.0!control loop latency}
[^fn-industry40-feedback]: **Industry 4.0**: The fourth industrial revolution integrates ML into the sensor-actuator feedback loop on factory floors. The systems consequence is that the control loop latency ($L_{\text{lat}}$) must be shorter than the physical process it governs: a welding robot that detects a defect at 60 Hz has 16.7 ms to halt, a budget only edge inference can meet. \index{Industry 4.0!control loop latency}
[^fn-predictive-maint-edge]: **Predictive Maintenance**: Models that analyze high-frequency sensor data (e.g., vibration, thermal) to forecast equipment failure, enabling the simultaneous monitoring of thousands of assets. The "additional deployment complexity" mentioned stems directly from the edge requirement for continuous, 24/7 on-device inference. This imposes a strict power budget where the entire sensor and model must often operate on less than 1 watt, a major constraint driving model architecture and quantization choices. \index{Predictive Maintenance!edge duty cycle}
@@ -2393,7 +2393,7 @@ We define this paradigm formally as *Mobile ML*.
***Mobile Machine Learning***\index{Mobile ML!definition} is the deployment paradigm bounded by **Thermal Design Power (TDP)** and battery energy.
1. **Significance (Quantitative):** It is constrained by the **Heat Dissipation** capacity of passive cooling (typically 2–3 W), requiring architectures that prioritize **Sustained Energy Efficiency** over peak throughput ($R_{peak}$).
1. **Significance (Quantitative):** It is constrained by the **Heat Dissipation** capacity of passive cooling (typically 2–3 W), requiring architectures that prioritize **Sustained Energy Efficiency** over peak throughput ($R_{\text{peak}}$).
2. **Distinction (Durable):** Unlike **Edge ML**, which may have active cooling, Mobile ML must operate within a **Personal Energy Budget**. Unlike **TinyML**, it still provides a rich OS and multi-watt compute capacity.
3. **Common Pitfall:** A frequent misconception is that Mobile ML performance is a fixed value. In reality, it is a **Time-Varying Constraint**: performance often drops as the device hits its **Thermal Wall**, triggering throttling that reduces the duty cycle ($\eta$).
@@ -2692,7 +2692,7 @@ We define this paradigm formally as *TinyML*.
:::
TinyML's milliwatt-scale power consumption represents a six-order-of-magnitude reduction from cloud inference, a gap with profound implications for system design. In terms of the Iron Law (@eq-iron-law-extended), TinyML operates in a regime where the dominant constraint is neither $O/(R_{peak} \cdot \eta)$ nor $D_{vol}/BW$, but a term the equation does not explicitly capture: $D_{vol}/\text{Capacity}$. When total memory is measured in kilobytes, the model must fit entirely on-chip, and every byte of data movement costs energy measured in picojoules. The optimization objective shifts from minimizing latency to minimizing *energy per inference*—efficiency, not speed.
TinyML's milliwatt-scale power consumption represents a six-order-of-magnitude reduction from cloud inference, a gap with profound implications for system design. In terms of the Iron Law (@eq-iron-law-extended), TinyML operates in a regime where the dominant constraint is neither $O/(R_{\text{peak}} \cdot \eta)$ nor $D_{\text{vol}}/BW$, but a term the equation does not explicitly capture: $D_{\text{vol}}/\text{Capacity}$. When total memory is measured in kilobytes, the model must fit entirely on-chip, and every byte of data movement costs energy measured in picojoules. The optimization objective shifts from minimizing latency to minimizing *energy per inference*—efficiency, not speed.
```{python}
#| label: energy-inference-calc
@@ -3299,7 +3299,7 @@ Successful deployment balances technical optimization against organizational cap
***Hybrid Machine Learning***\index{Hybrid ML!definition} is the architectural strategy of **Hierarchical Distribution** across cloud and edge resources.
1. **Significance (Quantitative):** It partitions the ML workload across the **Latency-Compute Pareto Frontier**, minimizing the **Distance Penalty** ($L_{lat}$) for reactive tasks while utilizing cloud resources ($R_{peak}$) for heavy processing.
1. **Significance (Quantitative):** It partitions the ML workload across the **Latency-Compute Pareto Frontier**, minimizing the **Distance Penalty** ($L_{\text{lat}}$) for reactive tasks while utilizing cloud resources ($R_{\text{peak}}$) for heavy processing.
2. **Distinction (Durable):** Unlike **Cloud-Only** or **Edge-Only** deployments, Hybrid ML is defined by **Dynamic Task Offloading** based on resource availability and network status.
3. **Common Pitfall:** A frequent misconception is that Hybrid ML is just "running two models." In reality, it is a **Unified Data Fabric** where the state must be synchronized across disparate hardware to ensure consistency.

2
book/quarto/contents/vol1/ml_systems/ml_systems_quizzes.json
View File

@@ -52,7 +52,7 @@
{
"question_type": "CALC",
"question": "If a cloud-based ML system uses 1,000 NVIDIA V100 GPUs continuously for 355 days to train a model, calculate the total petaflop-days of compute used. Assume each V100 GPU provides 125 teraflops.",
"answer": "Each V100 GPU provides 125 TFLOPS ($R_{peak}$). For 1,000 GPUs, the total peak is 125,000 TFLOPS. Over 355 days, the total operations ($O$) is 125,000 TFLOPS \u00d7 24 hours/day \u00d7 355 days = 1,065,000,000 TFLOP-hours. Converting to petaflop-days: 1,065,000,000 / (1,000 TFLOPS/PFLOPS) / 24 hours = 44,375 petaflop-days. This demonstrates the massive computational power required for training large ML models.",
"answer": "Each V100 GPU provides 125 TFLOPS ($R_{\\text{peak}}$). For 1,000 GPUs, the total peak is 125,000 TFLOPS. Over 355 days, the total operations ($O$) is 125,000 TFLOPS \u00d7 24 hours/day \u00d7 355 days = 1,065,000,000 TFLOP-hours. Converting to petaflop-days: 1,065,000,000 / (1,000 TFLOPS/PFLOPS) / 24 hours = 44,375 petaflop-days. This demonstrates the massive computational power required for training large ML models.",
"learning_objective": "Apply knowledge of computational requirements to calculate the resources used in cloud-based ML training.",
"_hidden_at": "2025-09-11T18:18:10.011235",
"_manually_shown": true,

30
book/quarto/contents/vol1/ml_workflow/ml_workflow.qmd
View File

@@ -409,7 +409,7 @@ This shift from code-centric to data-centric development erodes more than just p
[^fn-locality-violation-cost]: **Locality of Reference**: Formalized by Peter Denning in 1968 as the principle governing virtual memory design. The cost of violating locality is quantitative: an L1 cache hit costs approximately 1 ns, while a DRAM access costs 50--100 ns (50--100$\times$ penalty), and an NVMe SSD read costs 10--100 $\mu$s (10,000--100,000$\times$ penalty). Random shuffling of multi-terabyte datasets during each training epoch triggers the worst case at every level of the memory hierarchy, explaining why ML data loaders must implement their own prefetching logic rather than relying on OS page cache heuristics. \index{Locality!ML workload violation}
ML workflows violate these abstractions at scale. A multi-terabyte dataset being randomly shuffled during every training epoch presents a "worst-case" workload for traditional file system buffers and virtual memory prefetchers. When every "instruction" (a sample) is fetched stochastically from a massive pool, the OS's predictive caching logic fails, and the system defaults to expensive disk I/O or network transfers. A systems engineer must acknowledge that the "Abstractions of the 1970s," once designed to hide hardware latency, are often the primary sources of the **Overhead Term ($L_{lat}$)** in the **Iron Law** for Software 2.0. Bridging this gap requires the specialized data engineering and hardware-aware optimizations we examine in the following Parts.
ML workflows violate these abstractions at scale. A multi-terabyte dataset being randomly shuffled during every training epoch presents a "worst-case" workload for traditional file system buffers and virtual memory prefetchers. When every "instruction" (a sample) is fetched stochastically from a massive pool, the OS's predictive caching logic fails, and the system defaults to expensive disk I/O or network transfers. A systems engineer must acknowledge that the "Abstractions of the 1970s," once designed to hide hardware latency, are often the primary sources of the **Overhead Term ($L_{\text{lat}}$)** in the **Iron Law** for Software 2.0. Bridging this gap requires the specialized data engineering and hardware-aware optimizations we examine in the following Parts.
These distinctions translate directly into the structured six-stage framework that organizes how ML projects unfold, each stage presenting unique challenges that traditional software methodologies cannot address. The differences just covered should be clear before examining that framework.
@@ -528,12 +528,12 @@ The ML lifecycle is not a straight line; it is a spiral of continuous refinement
::: {.callout-perspective title="The Iron Law of Workflow"}
The six lifecycle stages are not merely procedural steps; they are the engineering levers used to optimize the variables in the **Iron Law of ML Systems** ($T = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat}$):
The six lifecycle stages are not merely procedural steps; they are the engineering levers used to optimize the variables in the **Iron Law of ML Systems** ($T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$):
- **Data Collection & Preparation**: Primarily determines the **Data ($D_{vol}$)** term. High-quality curation reduces the volume of data needed to reach a target accuracy.
- **Data Collection & Preparation**: Primarily determines the **Data ($D_{\text{vol}}$)** term. High-quality curation reduces the volume of data needed to reach a target accuracy.
- **Model Development & Training**: Defines the **Operations ($O$)** term. Architectural choices (e.g., Transformers vs. CNNs) set the computational floor.
- **Evaluation & Validation**: Verifies whether the achieved **Efficiency ($\eta$)** and model accuracy jointly meet deployment requirements on the target hardware.
- **Deployment & Integration**: Focuses on minimizing the **Overhead ($L_{lat}$)** tax through efficient serving infrastructure.
- **Deployment & Integration**: Focuses on minimizing the **Overhead ($L_{\text{lat}}$)** tax through efficient serving infrastructure.
Viewed this way, managing the workflow is mathematically equivalent to minimizing the total system latency and cost.
@@ -547,7 +547,7 @@ The binding constraint differs dramatically across workload archetypes, causing
| **Training** | *Compute Bound*: Maximize Model FLOPs Utilization ($\eta$); mixed precision to saturate Tensor Cores | *I/O Bound*: Optimize sparse embedding lookups; memory bandwidth ($BW$) limits throughput | *Model Search*: Neural Architecture Search (NAS) for smallest architecture; quantization-aware training (QAT) required |
| **Deploy** | *Batching*: Batch size **> 128** to maximize throughput; latency secondary to cost | *SLA*: Strict **< 10 ms p99** latency; feature freshness requirements | *Energy*: **< 1 mW** budget; always-on inference without battery drain |
: **Workflow Variations by Lighthouse Model**: The same lifecycle stages target different Iron Law terms depending on the workload's binding constraint. ResNet-50 optimizes for Throughput ($O/s$); DLRM is bound by Memory Bandwidth ($D_{vol}/BW$); TinyML is strictly bound by Energy ($J$) and Memory Capacity. {#tbl-lighthouse-workflow-comparison}
: **Workflow Variations by Lighthouse Model**: The same lifecycle stages target different Iron Law terms depending on the workload's binding constraint. ResNet-50 optimizes for Throughput ($O/s$); DLRM is bound by Memory Bandwidth ($D_{\text{vol}}/BW$); TinyML is strictly bound by Energy ($J$) and Memory Capacity. {#tbl-lighthouse-workflow-comparison}
Production systems rarely fall neatly into a single archetype. A medical imaging classifier, for instance, is compute-bound during training (like ResNet-50, requiring sustained GPU utilization over large image datasets) yet faces strict energy and memory constraints when deployed to portable clinic devices (like the TinyML archetype). Understanding *how* the same workflow framework adapts to each archetype, and *how* a single project can span multiple archetypes simultaneously, is essential for making sound engineering decisions.
@@ -673,7 +673,7 @@ With objectives defined and constraints layered, the next question becomes immed
## Data Collection {#sec-ml-workflow-data-collection-preparation-stage-ae99}
\index{Data Collection!determines Iron Law Data term}
The constraints, metrics, and deployment targets from problem definition exist only on paper until a team acquires the data that will teach the model to satisfy them. This transition from defining goals to acquiring training data marks a critical juncture where many projects fail. As the quantitative data in @sec-ml-workflow-quantifying-ml-lifecycle-bd69 established, data-related activities consume the majority of project time, making decisions at this stage disproportionately consequential. In Iron Law terms, this stage primarily determines the Data ($D_{vol}$) term: the volume, quality, and format of training data that downstream stages must work with. The deployment constraints established during problem definition now become data requirements: if the model must run on edge devices, the data pipeline must produce inputs compatible with edge preprocessing. If the model must achieve 90% sensitivity across diverse populations, the data must include sufficient examples from each population.
The constraints, metrics, and deployment targets from problem definition exist only on paper until a team acquires the data that will teach the model to satisfy them. This transition from defining goals to acquiring training data marks a critical juncture where many projects fail. As the quantitative data in @sec-ml-workflow-quantifying-ml-lifecycle-bd69 established, data-related activities consume the majority of project time, making decisions at this stage disproportionately consequential. In Iron Law terms, this stage primarily determines the Data ($D_{\text{vol}}$) term: the volume, quality, and format of training data that downstream stages must work with. The deployment constraints established during problem definition now become data requirements: if the model must run on edge devices, the data pipeline must produce inputs compatible with edge preprocessing. If the model must achieve 90% sensitivity across diverse populations, the data must include sufficient examples from each population.
Data collection and preparation is not a preliminary step but the *primary engineering activity* of most ML projects. @sec-data-engineering addresses data engineering as its core focus. For DR screening, the challenge is substantial: the data must be statistically diverse enough to train a model that generalizes across populations, operationally feasible to collect in resource-limited clinics, and annotated with enough clinical rigor to satisfy regulatory scrutiny.
@@ -827,7 +827,7 @@ class StorageCosts:
\index{Tiered Storage!hot warm cold architecture}
Different data access patterns demand different storage solutions. Teams typically implement tiered storage architectures[^fn-tiered-storage-ml]:
[^fn-tiered-storage-ml]: **Tiered Storage**: Places data on different storage media based on access frequency and performance requirements. The cost-performance gap is an order of magnitude: NVMe SSDs deliver 500,000+ IOPS at ~$`{python} StorageCosts.cost_nvme_str`/GB/month, while object storage costs ~$`{python} StorageCosts.cost_s3_str`/GB/month but with 100--200 ms latency. For ML training loops requiring sustained sequential reads at 1--10 GB/s, choosing the wrong tier converts a compute-bound training pipeline into an I/O-bound one, directly inflating the Iron Law's Data term ($D_{vol}/BW$). \index{Tiered Storage!ML I/O bottleneck}
[^fn-tiered-storage-ml]: **Tiered Storage**: Places data on different storage media based on access frequency and performance requirements. The cost-performance gap is an order of magnitude: NVMe SSDs deliver 500,000+ IOPS at ~$`{python} StorageCosts.cost_nvme_str`/GB/month, while object storage costs ~$`{python} StorageCosts.cost_s3_str`/GB/month but with 100--200 ms latency. For ML training loops requiring sustained sequential reads at 1--10 GB/s, choosing the wrong tier converts a compute-bound training pipeline into an I/O-bound one, directly inflating the Iron Law's Data term ($D_{\text{vol}}/BW$). \index{Tiered Storage!ML I/O bottleneck}
- **Hot Storage**: High-throughput NVMe SSDs for data currently used in training loops.
- **Warm Storage**: S3-compatible object storage for recent datasets and active validation sets.
@@ -916,7 +916,7 @@ The DR team has 128,000 labeled retinal images, a validated preprocessing pipeli
\index{Transfer Learning!definition and history}
The DR system faces a sharp optimization challenge: achieve expert-level diagnostic accuracy while fitting within edge device memory and latency budgets. Data and compute budgets are finite, so techniques that reduce both requirements without sacrificing accuracy become essential design choices. Transfer learning[^fn-transfer-learning-efficiency] addresses exactly this constraint: rather than training a model from scratch, it adapts models pre-trained on large datasets (like ImageNet's 14 million images) to specific tasks [@alexnet2012; @deng2009imagenet]. Because transfer learning reuses representations already learned from millions of general images, practitioners can achieve expert-level performance with thousands rather than millions of domain-specific training examples, sharply reducing both training time and data collection effort. This approach became widespread in the 2013–2014 era through influential papers by Yosinski et al. and Oquab et al., establishing it as the foundation for practical computer vision applications.
[^fn-transfer-learning-efficiency]: **Transfer Learning**: Addresses the DR system's sharp optimization challenge by reusing representations already learned from ImageNet's 14 million general images. Fine-tuning with thousands of domain-specific retinal images rather than training from scratch on millions reduces both the Data term ($D_{vol}$) and Operations term ($O$) of the Iron Law, compressing the annotation budget and compute budget simultaneously. Without this technique, the DR project would need 10--100$\times$ more labeled retinal images to reach equivalent accuracy, making the annotation cost alone prohibitive. \index{Transfer Learning!compute efficiency}
[^fn-transfer-learning-efficiency]: **Transfer Learning**: Addresses the DR system's sharp optimization challenge by reusing representations already learned from ImageNet's 14 million general images. Fine-tuning with thousands of domain-specific retinal images rather than training from scratch on millions reduces both the Data term ($D_{\text{vol}}$) and Operations term ($O$) of the Iron Law, compressing the annotation budget and compute budget simultaneously. Without this technique, the DR project would need 10--100$\times$ more labeled retinal images to reach equivalent accuracy, making the annotation cost alone prohibitive. \index{Transfer Learning!compute efficiency}
Using transfer learning combined with a meticulously labeled dataset of 128,000 images, developers in DR projects achieve AUC[^fn-auc-threshold-independence] of 0.99 with sensitivity of 97.5% and specificity of 93.4% [@gulshan2016deep], comparable to or exceeding ophthalmologist performance in controlled settings. This result validates approaches that combine large-scale pre-training with domain-specific fine-tuning. The training strategy uses the gradient-based optimization principles @sec-neural-computation establishes to adapt the pre-trained convolutional architectures @sec-network-architectures presents for medical imaging.
@@ -954,9 +954,9 @@ Medical applications demand specific performance metrics[^fn-medical-metrics-ppv
[^fn-medical-metrics-ppv]: **Medical AI Performance Metrics**: Medical AI demands sensitivity (true positive rate) and specificity (true negative rate) rather than aggregate accuracy. For DR screening, >90% sensitivity is mandatory because missed cases cause blindness. The subtler systems trap is positive predictive value (PPV): a model with 95% accuracy in a lab can drop to 50% PPV in a low-prevalence population, making it clinically useless despite strong technical metrics. This prevalence dependence means a single model requires different operating thresholds per deployment site, a constraint invisible in standard ML evaluation. \index{Medical Metrics!PPV prevalence trap}
Optimizing for clinical performance alone is not enough. Edge deployment constraints from the data collection phase impose additional requirements: the model must run efficiently on resource-limited hardware while maintaining inference speeds compatible with clinical workflows. Improvements in one dimension often come at the cost of others: the Operations ($O$) term and the Overhead ($L_{lat}$) term from the Iron Law pull in opposite directions. @sec-network-architectures explores model capacity, while @sec-ml-systems discusses deployment feasibility, and the inherent tension between them drives architectural decisions. Systematic application of quantization, pruning, and knowledge distillation[^fn-compression-iterative-workflow] techniques can bridge the gap, meeting deployment requirements while aiming to preserve clinical utility.
Optimizing for clinical performance alone is not enough. Edge deployment constraints from the data collection phase impose additional requirements: the model must run efficiently on resource-limited hardware while maintaining inference speeds compatible with clinical workflows. Improvements in one dimension often come at the cost of others: the Operations ($O$) term and the Overhead ($L_{\text{lat}}$) term from the Iron Law pull in opposite directions. @sec-network-architectures explores model capacity, while @sec-ml-systems discusses deployment feasibility, and the inherent tension between them drives architectural decisions. Systematic application of quantization, pruning, and knowledge distillation[^fn-compression-iterative-workflow] techniques can bridge the gap, meeting deployment requirements while aiming to preserve clinical utility.
[^fn-compression-iterative-workflow]: **Model Compression Pipeline**: Quantization, pruning, and distillation each alter the Operations ($O$) and Overhead ($L_{lat}$) terms differently, so bridging the gap between research accuracy and edge deployment requires an iterative "compress-validate-adjust" loop. Finding a model that fits in device memory while preserving clinical sensitivity typically requires 3--5 iterations, because each compression step can silently degrade accuracy below the 90% sensitivity threshold in ways invisible until the full validation suite runs. \index{Model Compression!iterative workflow}
[^fn-compression-iterative-workflow]: **Model Compression Pipeline**: Quantization, pruning, and distillation each alter the Operations ($O$) and Overhead ($L_{\text{lat}}$) terms differently, so bridging the gap between research accuracy and edge deployment requires an iterative "compress-validate-adjust" loop. Finding a model that fits in device memory while preserving clinical sensitivity typically requires 3--5 iterations, because each compression step can silently degrade accuracy below the 90% sensitivity threshold in ways invisible until the full validation suite runs. \index{Model Compression!iterative workflow}
The ensemble trade-off illustrates a broader pattern: choosing an ensemble of lightweight models over a single large model reduces per-model complexity (enabling edge deployment) but increases pipeline complexity (requiring orchestration logic and multi-model monitoring). Every architectural decision creates this kind of downstream ripple.
@@ -1077,7 +1077,7 @@ Evaluation and validation address different questions. Evaluation measures model
***Model Validation***\index{Model Validation!definition} is the rigorous verification that a model meets **Business Constraints** (SLA, Fairness, Cost) on **Production-Representative Data**.
1. **Significance (Quantitative):** It moves beyond "Test Set Accuracy" to test for **Robustness** against distribution shift ($D_{vol}$) and **Efficiency** against hardware limits ($R_{peak}, BW$).
1. **Significance (Quantitative):** It moves beyond "Test Set Accuracy" to test for **Robustness** against distribution shift ($D_{\text{vol}}$) and **Efficiency** against hardware limits ($R_{\text{peak}}, BW$).
2. **Distinction (Durable):** Unlike **Model Evaluation**, which measures performance on a **Static Test Set**, Model Validation confirms that the model generalizes to the **Dynamic Conditions** of production.
3. **Common Pitfall:** A frequent misconception is that validation is "one more test." In reality, it is a **Risk Management Framework** for detecting silent failure before it compounds into business impact.
@@ -1133,7 +1133,7 @@ Validation failures drive model architecture revisions, training data augmentati
## Deployment and Integration {#sec-ml-workflow-deployment-integration-stage-d549}
A model that passes every validation test in the lab still faces its hardest exam when it meets the real world. Consider the DR system: a validated model must now run on tablets in rural clinics with intermittent connectivity, integrate with hospital information systems it was never tested against, and produce results that clinicians trust enough to act on — all within latency budgets that leave no room for cloud round-trips. Deployment is where the abstract constraints specified during problem definition become concrete engineering requirements. In Iron Law terms, this stage focuses on minimizing the Overhead ($L_{lat}$) term through efficient serving infrastructure, and the binding constraint varies by archetype: ResNet-50-class workloads optimize batch size for throughput, DLRM-class workloads enforce strict SLA latency, and TinyML-class workloads operate under sub-milliwatt energy budgets (@tbl-lighthouse-workflow-comparison). @sec-ml-operations covers the operational aspects of deployment and maintenance in depth.
A model that passes every validation test in the lab still faces its hardest exam when it meets the real world. Consider the DR system: a validated model must now run on tablets in rural clinics with intermittent connectivity, integrate with hospital information systems it was never tested against, and produce results that clinicians trust enough to act on — all within latency budgets that leave no room for cloud round-trips. Deployment is where the abstract constraints specified during problem definition become concrete engineering requirements. In Iron Law terms, this stage focuses on minimizing the Overhead ($L_{\text{lat}}$) term through efficient serving infrastructure, and the binding constraint varies by archetype: ResNet-50-class workloads optimize batch size for throughput, DLRM-class workloads enforce strict SLA latency, and TinyML-class workloads operate under sub-milliwatt energy budgets (@tbl-lighthouse-workflow-comparison). @sec-ml-operations covers the operational aspects of deployment and maintenance in depth.
### Deployment Requirements {#sec-ml-workflow-technical-operational-requirements-36ab}
@@ -1361,7 +1361,7 @@ The DR case study illustrated constraint propagation repeatedly: bandwidth limit
***The Constraint Propagation Principle***\index{Constraint Propagation Principle!definition} states that constraints discovered late in the lifecycle ($N$) incur an exponential cost relative to the stage where they should have been defined ($\text{Correction Cost} \approx 2^{(N-1)} \times \text{Base Effort}$).
1. **Significance (Quantitative):** It dictates that system design must proceed **End-to-End**. Within the **Iron Law**, a constraint on $R_{peak}$ at deployment (Stage 5) propagates backward to redefine the Data Volume ($D_{vol}$) and Algorithm Complexity ($O$) required at Stage 1.
1. **Significance (Quantitative):** It dictates that system design must proceed **End-to-End**. Within the **Iron Law**, a constraint on $R_{\text{peak}}$ at deployment (Stage 5) propagates backward to redefine the Data Volume ($D_{\text{vol}}$) and Algorithm Complexity ($O$) required at Stage 1.
2. **Distinction (Durable):** Unlike **Modular Decomposition**, which encourages isolation, this principle mandates **Global Optimization**: a "local maxima" in model accuracy may lead to a "global minima" in system feasibility.
3. **Common Pitfall:** A frequent misconception is that deployment is "the last step." In reality, the deployment environment is the **Day 1 Constraint** that defines the boundaries of every other decision.
@@ -1436,7 +1436,7 @@ Teams assume they can "figure out deployment later" and focus first on model acc
This chapter established the ML lifecycle as the systematic framework for engineering machine learning systems, the mental roadmap organizing how data, models, and deployment infrastructure interconnect throughout development. Return to @fig-ml-lifecycle one final time: the two parallel pipelines now carry richer meaning. The data pipeline transforms raw inputs through collection, ingestion, analysis, labeling, validation, and preparation into ML-ready datasets. The model development pipeline takes these datasets through training, evaluation, validation, and deployment to create production systems. With the full chapter as context, the feedback arrows tell a deeper story—each one represents a lesson learned in production flowing back to strengthen earlier stages, creating the continuous improvement cycles that distinguish ML from traditional linear development.
Understanding this framework explains why machine learning systems demand specialized approaches distinct from traditional software. ML workflows replace deterministic specifications with probabilistic optimization, static behavior with dynamic adaptation, and isolated development with continuous feedback loops. The Iron Law of Workflow provides the quantitative backbone: each lifecycle stage maps to a specific term in the performance equation ($T = D_{vol}/BW + O/(R_{peak} \cdot \eta) + L_{lat}$), making workflow management mathematically equivalent to system optimization. This systematic perspective recognizes that success emerges not from perfecting individual stages in isolation, but from understanding how data quality affects model performance, how deployment constraints shape training strategies, and how production insights inform each subsequent development iteration.
Understanding this framework explains why machine learning systems demand specialized approaches distinct from traditional software. ML workflows replace deterministic specifications with probabilistic optimization, static behavior with dynamic adaptation, and isolated development with continuous feedback loops. The Iron Law of Workflow provides the quantitative backbone: each lifecycle stage maps to a specific term in the performance equation ($T = D_{\text{vol}}/BW + O/(R_{\text{peak}} \cdot \eta) + L_{\text{lat}}$), making workflow management mathematically equivalent to system optimization. This systematic perspective recognizes that success emerges not from perfecting individual stages in isolation, but from understanding how data quality affects model performance, how deployment constraints shape training strategies, and how production insights inform each subsequent development iteration.
Three quantitative insights from this chapter guide engineering decisions:
@@ -1454,7 +1454,7 @@ Three quantitative insights from this chapter guide engineering decisions:
* **Stage interfaces are contracts**: Explicit inputs, outputs, and quality invariants at each stage help prevent the 60–70% of ML project failures caused by integration problems.
* **Feedback loops span multiple timescales**: Real-time inference monitoring (seconds), batch retraining triggers (days), and strategic model updates (months) all require distinct automation.
* **Constraint propagation is exponential across stages**: Deployment constraints (latency, memory) flow backward to model selection; data constraints (volume, quality) flow forward to architecture choices. A constraint discovered at stage $N$ costs $2^{N-1}$ times more to fix than if caught at stage 1.
* **Each lifecycle stage maps to an Iron Law term**: Data collection determines $D_{vol}$, model development defines $O$, and deployment minimizes $L_{lat}$—making workflow management mathematically equivalent to system optimization.
* **Each lifecycle stage maps to an Iron Law term**: Data collection determines $D_{\text{vol}}$, model development defines $O$, and deployment minimizes $L_{\text{lat}}$—making workflow management mathematically equivalent to system optimization.
:::

20
book/quarto/contents/vol1/model_serving/model_serving.qmd
View File

@@ -26,7 +26,7 @@ engine: jupyter
_Why does serving invert every optimization priority that made training successful?_
Training and serving demand opposite physics. Training maximizes throughput (\(T\), in samples per second): large batches and long epochs where latency spikes get absorbed invisibly. Serving minimizes latency (\(L_{lat}\), in milliseconds per request): individual requests answered fast enough that a single slow response is a *broken product*. Training amortizes hardware costs across billions of examples; serving pays a tax on every request, where small inefficiencies compound into operational debt. This inversion is why models that train beautifully often serve poorly: the batch-heavy architectures and memory-intensive optimizations designed to saturate accelerators during training are fundamentally ill-suited for the bursty, latency-critical, cost-sensitive reality of production traffic. But serving is more than a latency problem. A serving system must handle traffic that varies by orders of magnitude between peak and trough, route requests across model versions during progressive rollouts, degrade gracefully when upstream dependencies fail, and do all of this continuously—not for the duration of a training run but for the lifetime of the product. Every model that proved its value during training and survived compression and benchmarking eventually arrives at the serving layer—the deployment and integration stage of the ML lifecycle—where the question shifts from "does it work?" to "does it work *reliably, at scale, under production conditions, every second of every day*?" The serving infrastructure is where ML systems finally meet users, and the engineering that sustains that meeting is qualitatively different from the engineering that created the model.
Training and serving demand opposite physics. Training maximizes throughput (\(T\), in samples per second): large batches and long epochs where latency spikes get absorbed invisibly. Serving minimizes latency (\(L_{\text{lat}}\), in milliseconds per request): individual requests answered fast enough that a single slow response is a *broken product*. Training amortizes hardware costs across billions of examples; serving pays a tax on every request, where small inefficiencies compound into operational debt. This inversion is why models that train beautifully often serve poorly: the batch-heavy architectures and memory-intensive optimizations designed to saturate accelerators during training are fundamentally ill-suited for the bursty, latency-critical, cost-sensitive reality of production traffic. But serving is more than a latency problem. A serving system must handle traffic that varies by orders of magnitude between peak and trough, route requests across model versions during progressive rollouts, degrade gracefully when upstream dependencies fail, and do all of this continuously—not for the duration of a training run but for the lifetime of the product. Every model that proved its value during training and survived compression and benchmarking eventually arrives at the serving layer—the deployment and integration stage of the ML lifecycle—where the question shifts from "does it work?" to "does it work *reliably, at scale, under production conditions, every second of every day*?" The serving infrastructure is where ML systems finally meet users, and the engineering that sustains that meeting is qualitatively different from the engineering that created the model.
::: {.content-visible when-format="pdf"}
@@ -48,7 +48,7 @@ Training and serving demand opposite physics. Training maximizes throughput (\(T
## Serving Paradigm {#sec-model-serving-serving-paradigm-9634}
Serving\index{Serving!production deployment}\index{Model Serving!paradigm shift} marks the transition from model development to production deployment. The four deployment paradigms introduced in @sec-ml-systems (Cloud, Edge, Mobile, and TinyML) each impose distinct serving challenges, but all share a common inversion: the throughput-to-latency shift introduced in the Purpose. This inversion has concrete engineering implications that ripple through every technique established in prior chapters. The Iron Law of ML Systems (@sec-introduction-iron-law-ml-systems-c32a) undergoes a decisive shift: the latency term\index{Latency!serving constraint} ($L_{lat}$), representing the irreducible overhead of request scheduling, network round-trips, and system orchestration, becomes the dominant constraint rather than a rounding error. @sec-benchmarking measured performance under controlled conditions, but serving faces traffic patterns that no benchmark could anticipate; @sec-model-compression provided quantization methods that reduced model size, but serving must confirm those optimizations preserve accuracy under real traffic distributions. These revalidations define the *serving inversion*\index{Serving Inversion!throughput to latency}.
Serving\index{Serving!production deployment}\index{Model Serving!paradigm shift} marks the transition from model development to production deployment. The four deployment paradigms introduced in @sec-ml-systems (Cloud, Edge, Mobile, and TinyML) each impose distinct serving challenges, but all share a common inversion: the throughput-to-latency shift introduced in the Purpose. This inversion has concrete engineering implications that ripple through every technique established in prior chapters. The Iron Law of ML Systems (@sec-introduction-iron-law-ml-systems-c32a) undergoes a decisive shift: the latency term\index{Latency!serving constraint} ($L_{\text{lat}}$), representing the irreducible overhead of request scheduling, network round-trips, and system orchestration, becomes the dominant constraint rather than a rounding error. @sec-benchmarking measured performance under controlled conditions, but serving faces traffic patterns that no benchmark could anticipate; @sec-model-compression provided quantization methods that reduced model size, but serving must confirm those optimizations preserve accuracy under real traffic distributions. These revalidations define the *serving inversion*\index{Serving Inversion!throughput to latency}.
::: {.callout-perspective title="The Serving Inversion"}
@@ -255,7 +255,7 @@ These priorities motivate a formal definition of model serving.
***Model Serving***\index{Model Serving!definition} is the operational phase that provides model predictions to end-users or downstream systems under strict latency constraints.
1. **Significance (Quantitative):** It inverts the throughput priority ($\eta$) of training into a **Latency Constraint ($L_{lat}$)**, requiring an architectural stack designed to minimize the **Tail Latency** (p99) of individual inferences.
1. **Significance (Quantitative):** It inverts the throughput priority ($\eta$) of training into a **Latency Constraint ($L_{\text{lat}}$)**, requiring an architectural stack designed to minimize the **Tail Latency** (p99) of individual inferences.
2. **Distinction (Durable):** Unlike **Model Training**, which processes large, predictable batches of data, Model Serving must handle **Stochastic Request Patterns** and unpredictable load.
3. **Common Pitfall:** A frequent misconception is that serving is "just the forward pass." In reality, it is a **Distributed System Problem**: the model execution is only one component of a stack that includes request routing, load balancing, and data transformation.
@@ -1110,7 +1110,7 @@ Managing these percentile constraints requires decomposing the total allowed res
***Latency Budget***\index{Latency Budget!definition} is the **Time Capital** allocated to a request, strictly bounded by the end-to-end **Service Level Objective (SLO)**.
1. **Significance (Quantitative):** It acts as a **Zero-Sum Constraint System** where any milliseconds consumed by serialization or network overhead directly reduce the computational budget ($L_{lat}$) available for model inference.
1. **Significance (Quantitative):** It acts as a **Zero-Sum Constraint System** where any milliseconds consumed by serialization or network overhead directly reduce the computational budget ($L_{\text{lat}}$) available for model inference.
2. **Distinction (Durable):** Unlike **Average Latency**, which hides variance, a Latency Budget is a **Hard Bound** that must be maintained for the slowest requests (e.g., p99).
3. **Common Pitfall:** A frequent misconception is that the "model" has the entire budget. In reality, the model often has less than **50% of the total budget**; the remainder is consumed by the **Request Lifecycle** (DNS, TLS, Load Balancing, Serialization).
@@ -1823,7 +1823,7 @@ The resulting **Latency-Throughput Pareto Frontier**\index{Pareto Frontier!servi
$$ W_{total} \approx \underbrace{ \frac{B-1}{2\lambda} }_{\text{Formation Delay}} + \underbrace{ T_{inf}(B) }_{\text{Inference Time}} $$ {#eq-batching-tax}
This equation reveals the "Cost of Throughput." Increasing $B$ to saturate the GPU amortizes the hardware cost, but inflates the per-request latency. Concretely, at 500 QPS, moving from batch-1 to batch-32 increases wait-time from **`{python} BatchingTax.wait_time_b1_ms_str`ms** to **`{python} BatchingTax.wait_time_b32_ms_str`ms**, contributing to a **`{python} BatchingTax.penalty_ratio_str`$\times$** total latency penalty (`{python} BatchingTax.lat_b1_ms_str`ms → `{python} BatchingTax.lat_b32_ms_str`ms). For a systems engineer, this tax is the primary regulator of **Economic Efficiency**: the engineer chooses the batch size that maximizes throughput (minimizing cost per query) without violating the **Latency SLO** ($L_{lat}$).
This equation reveals the "Cost of Throughput." Increasing $B$ to saturate the GPU amortizes the hardware cost, but inflates the per-request latency. Concretely, at 500 QPS, moving from batch-1 to batch-32 increases wait-time from **`{python} BatchingTax.wait_time_b1_ms_str`ms** to **`{python} BatchingTax.wait_time_b32_ms_str`ms**, contributing to a **`{python} BatchingTax.penalty_ratio_str`$\times$** total latency penalty (`{python} BatchingTax.lat_b1_ms_str`ms → `{python} BatchingTax.lat_b32_ms_str`ms). For a systems engineer, this tax is the primary regulator of **Economic Efficiency**: the engineer chooses the batch size that maximizes throughput (minimizing cost per query) without violating the **Latency SLO** ($L_{\text{lat}}$).
Little's Law has immediate practical implications. If an inference service averages 10 ms per request ($W = 0.01$s) and the system shows 50 concurrent requests on average ($L = 50$), then the arrival rate must be $\lambda = L/W = 5000$ requests per second. Conversely, if the system must limit concurrent requests to 10 (perhaps due to GPU memory constraints) and the service time is 10 ms, it can sustain at most 1000 requests per second.
@@ -2089,7 +2089,7 @@ With preprocessing pipelines designed to avoid training-serving skew, the next c
***Cold Start***\index{Cold Start!definition} is the **Initialization Latency** incurred when instantiating a new model replica.
1. **Significance (Quantitative):** It represents the fixed cost of **State Hydration** (loading weights, compiling graphs), which can take seconds or minutes, effectively blocking the system's ability to scale elastically in response to traffic bursts.
2. **Distinction (Durable):** Unlike **Inference Latency** ($L_{lat}$), which is a **Per-Request Cost**, Cold Start is a **Per-Replica Cost** that occurs only during deployment or scaling events.
2. **Distinction (Durable):** Unlike **Inference Latency** ($L_{\text{lat}}$), which is a **Per-Request Cost**, Cold Start is a **Per-Replica Cost** that occurs only during deployment or scaling events.
3. **Common Pitfall:** A frequent misconception is that Cold Start is "just loading weights." In reality, it includes **Graph Compilation** and **Memory Allocation**, which can often take longer than the data transfer itself ($BW$).
:::
@@ -2254,7 +2254,7 @@ Consider a ResNet-50 classifier running on a V100 GPU at batch size 1: the GPU p
***Dynamic Batching***\index{Dynamic Batching!definition} is the runtime optimization of trading **Latency** for **Throughput** under stochastic arrival patterns.
1. **Significance (Quantitative):** By buffering requests into a **Batching Window**, the scheduler amortizes fixed overheads ($L_{lat}$) across multiple inputs, pushing the system away from the memory-bound regime ($BW$) toward the compute-bound regime ($R_{peak}$).
1. **Significance (Quantitative):** By buffering requests into a **Batching Window**, the scheduler amortizes fixed overheads ($L_{\text{lat}}$) across multiple inputs, pushing the system away from the memory-bound regime ($BW$) toward the compute-bound regime ($R_{\text{peak}}$).
2. **Distinction (Durable):** Unlike **Static Batching**, which is fixed during training, Dynamic Batching adaptively adjusts the batch size at **Inference Time** based on real-time traffic volume.
3. **Common Pitfall:** A frequent misconception is that batching "always helps." In reality, there is a **Latency-Throughput Pareto Frontier**: if the batching window is too large, the increased **Queuing Delay** may violate the system's SLO before the throughput gains are realized.
@@ -2634,9 +2634,9 @@ The throughput gains in @tbl-batching-throughput trace directly back to *the Iro
::: {.callout-notebook title="The Iron Law of Batching Efficiency"}
**The Iron Law Connection:**
In serving, we maximize throughput by amortizing the **Latency Term** ($L_{lat}$), as shown in @eq-compute-time:
In serving, we maximize throughput by amortizing the **Latency Term** ($L_{\text{lat}}$), as shown in @eq-compute-time:
$$ T = \frac{O}{R_{peak} \cdot \eta} + L_{lat} $$ {#eq-compute-time}
$$ T = \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}} $$ {#eq-compute-time}
**Deriving the Sweet Spot:**
@@ -3227,7 +3227,7 @@ The continuous batching and PagedAttention techniques covered in @sec-model-serv
#### Prefix Caching and Memory Offloading {#sec-model-serving-prefix-caching-offloading}
The memory pressure from KV caches can be further mitigated through architectural strategies that exploit request patterns. **Prefix Caching**\index{Prefix Caching!KV cache reuse} stores the KV cache of common instruction prefixes (such as a 2,000-token system prompt or a shared RAG context), allowing many independent requests to reuse the same pre-computed hidden states. This eliminates redundant prefill compute ($R_{peak}$) and reduces memory traffic ($BW$). For multi-turn conversations, this "caching of the past" allows the system to process only the *new* tokens in each turn.
The memory pressure from KV caches can be further mitigated through architectural strategies that exploit request patterns. **Prefix Caching**\index{Prefix Caching!KV cache reuse} stores the KV cache of common instruction prefixes (such as a 2,000-token system prompt or a shared RAG context), allowing many independent requests to reuse the same pre-computed hidden states. This eliminates redundant prefill compute ($R_{\text{peak}}$) and reduces memory traffic ($BW$). For multi-turn conversations, this "caching of the past" allows the system to process only the *new* tokens in each turn.
When the aggregate KV cache exceeds GPU VRAM, systems can employ **KV Cache Offloading**\index{KV Cache!offloading}. This strategy spills inactive or low-priority context windows to host CPU RAM or NVMe SSD, freeing VRAM for active generation. While retrieving offloaded context introduces a latency "tax" due to PCIe bandwidth limits (@sec-model-serving-model-swapping-host-memory-c54f), it prevents Out-of-Memory (OOM) failures and enables handling much larger context windows than the hardware could otherwise support.

28
book/quarto/contents/vol1/nn_architectures/nn_architectures.qmd
View File

@@ -55,7 +55,7 @@ When you select a neural network architecture, you are not making a modeling dec
\index{Architecture!systems engineering}
The mathematical operators established in @sec-neural-computation (matrix multiplication, activation functions, and gradient computation) form the "verbs" of neural networks. Those operators are the atoms; this chapter examines how they assemble into **architectures**: specialized structures optimized for specific data types and computational constraints. As defined in the Silicon Contract (Principle \ref{pri-silicon-contract}) (@sec-introduction-iron-law-ml-systems-c32a), every architecture makes an implicit agreement with hardware, trading computational patterns for efficiency on particular problem classes.
Every neural network architecture answers one central question: *how should we structure computation to match the structure in our data?* Images have spatial locality, language has sequential dependencies, and tabular records have no inherent structure at all. The architecture encodes assumptions about these patterns directly into the computational graph, and those assumptions determine everything from parameter count to hardware utilization to deployment feasibility. Architecture selection is therefore a systems engineering problem that directly determines the Iron Law terms (the number of operations $O$ and the volume of data movement $D_{vol}$) defined in @sec-introduction-iron-law-ml-systems-c32a.
Every neural network architecture answers one central question: *how should we structure computation to match the structure in our data?* Images have spatial locality, language has sequential dependencies, and tabular records have no inherent structure at all. The architecture encodes assumptions about these patterns directly into the computational graph, and those assumptions determine everything from parameter count to hardware utilization to deployment feasibility. Architecture selection is therefore a systems engineering problem that directly determines the Iron Law terms (the number of operations $O$ and the volume of data movement $D_{\text{vol}}$) defined in @sec-introduction-iron-law-ml-systems-c32a.
The structural assumptions that each architecture encodes are known as inductive biases[^fn-inductive-bias], and they serve as the *unifying concept* for this entire chapter.
@@ -63,13 +63,13 @@ The structural assumptions that each architecture encodes are known as inductive
***Inductive Bias***\index{Inductive Bias!definition} is a structural constraint built into a model architecture that restricts the hypothesis space, enabling generalization from finite data by encoding domain-specific assumptions (such as spatial locality or sequential ordering) directly into the computational graph.
1. **Significance (Quantitative):** Inductive bias directly reduces the data volume ($D_{vol}$) required for generalization. A CNN's spatial locality bias reduces the hypothesis space from $O(P^2)$ (fully connected) to $O(P \cdot K^2)$ (local filters), where $K \ll P$: for a 224×224 image, a 3×3 CNN kernel needs roughly 1,000× fewer parameters than an equivalent MLP, cutting both the memory footprint and the data required to avoid overfitting by the same factor.
1. **Significance (Quantitative):** Inductive bias directly reduces the data volume ($D_{\text{vol}}$) required for generalization. A CNN's spatial locality bias reduces the hypothesis space from $O(P^2)$ (fully connected) to $O(P \cdot K^2)$ (local filters), where $K \ll P$: for a 224×224 image, a 3×3 CNN kernel needs roughly 1,000× fewer parameters than an equivalent MLP, cutting both the memory footprint and the data required to avoid overfitting by the same factor.
2. **Distinction (Durable):** Unlike **Regularization** (which penalizes hypothesis complexity at training time via L1/L2 terms), Inductive Bias eliminates entire hypothesis classes at architecture design time — a CNN cannot represent arbitrary non-local functions regardless of training data, while regularization merely discourages them.
3. **Common Pitfall:** A frequent misconception is that stronger inductive bias is always better. A strong locality bias (CNN) excels on spatial data but fails to represent long-range dependencies in language, where a Transformer's lack of spatial bias — at the cost of $O(N^2)$ memory scaling — is necessary to achieve state-of-the-art performance.
:::
[^fn-inductive-bias]: **Inductive Bias**: From Latin *inducere*, "to lead into" -- encoding a structural assumption "leads" the model toward a smaller solution space, which is why this concept unifies the entire chapter: every architecture discussed here (MLP, CNN, RNN, Transformer) is defined by its choice of bias. A CNN's locality bias cuts parameters by orders of magnitude versus an equivalent MLP, directly shrinking the Iron Law's $O$ and $D_{vol}$ terms, while a Transformer's lack of spatial bias demands quadratic memory in exchange for flexible long-range connectivity. \index{Inductive Bias!etymology}\index{Inductive Bias!systems consequence}
[^fn-inductive-bias]: **Inductive Bias**: From Latin *inducere*, "to lead into" -- encoding a structural assumption "leads" the model toward a smaller solution space, which is why this concept unifies the entire chapter: every architecture discussed here (MLP, CNN, RNN, Transformer) is defined by its choice of bias. A CNN's locality bias cuts parameters by orders of magnitude versus an equivalent MLP, directly shrinking the Iron Law's $O$ and $D_{\text{vol}}$ terms, while a Transformer's lack of spatial bias demands quadratic memory in exchange for flexible long-range connectivity. \index{Inductive Bias!etymology}\index{Inductive Bias!systems consequence}
A CNN's inductive bias is spatial locality: nearby pixels matter more than distant ones. A Transformer's inductive bias is that any element may attend to any other, enabling flexible long-range relationships at the cost of quadratic memory scaling. These biases are not incidental design choices; they are the mechanism through which architectures achieve efficiency by restricting the space of functions they can represent. Without these biases, the hypothesis space is so large that learning even simple tasks would require effectively infinite data and compute. We formalize how inductive biases unify all architectural families in @sec-network-architectures-unified-framework-inductive-biases-257d, after examining how each architecture's bias manifests in practice.
@@ -415,12 +415,12 @@ class TransformerScaling:
: **Lighthouse Model Comparison**: Quantitative characteristics and pedagogical roles of the five canonical workloads. Parameters and memory represent model weights at FP32 precision. FLOPs measured per single inference. The bottleneck column indicates the primary system constraint each model reveals: compute-bound models like ResNet stress arithmetic throughput, while bandwidth-bound models like GPT-2 stress memory transfer rates. {#tbl-lighthouse-comparison}
The "Bottleneck" column in @tbl-lighthouse-comparison deserves particular attention: it identifies which system resource (compute throughput, memory bandwidth, memory capacity, latency, or power) limits performance for each workload class. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), the bottleneck tells you whether $O$ (operations) or $D_{vol}$ (data movement) dominates the runtime. These distinctions determine which optimization strategies prove effective, a theme we return to throughout subsequent chapters.
The "Bottleneck" column in @tbl-lighthouse-comparison deserves particular attention: it identifies which system resource (compute throughput, memory bandwidth, memory capacity, latency, or power) limits performance for each workload class. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), the bottleneck tells you whether $O$ (operations) or $D_{\text{vol}}$ (data movement) dominates the runtime. These distinctions determine which optimization strategies prove effective, a theme we return to throughout subsequent chapters.
\index{Architecture!compute vs. memory trade-off}
\index{ResNet-50!arithmetic intensity}
\index{GPT-2!arithmetic intensity}
Architecture selection is ultimately an engineering trade-off between **Math** ($O$) and **Memory Movement** ($D_{vol}$). By comparing our Lighthouses, we can see how architectural choices shift a model's position on the intensity spectrum:
Architecture selection is ultimately an engineering trade-off between **Math** ($O$) and **Memory Movement** ($D_{\text{vol}}$). By comparing our Lighthouses, we can see how architectural choices shift a model's position on the intensity spectrum:
- **ResNet-50 (Compute-Bound)**: High intensity ($\approx 50\text{--}200+$ FLOPs/byte, varying by layer). Convolutional layers reuse each weight many times across the spatial dimensions of an image. Deep bottleneck layers achieve intensity above 200, while early layers are lower. Its performance is limited by how fast the hardware can do math.
- **GPT-2 (Bandwidth-Bound)**: Low intensity ($\approx 1$ FLOPs/byte). Each token produces only a matrix-vector multiplication rather than the matrix-matrix operations of batch processing, so the system must load massive weights from memory for a single token’s math. Its performance is limited by how fast memory can move bits.
@@ -1072,7 +1072,7 @@ Spatial locality produces two key innovations that enhance efficiency for spatia
***Convolutional Neural Networks (CNNs)***\index{CNN!definition} are architectures defined by **Translation Equivariance** and **Spatial Locality**.
1. **Significance (Quantitative):** They exploit weight sharing to decouple parameter count from input size, enabling **$O(1)$** scaling for high-dimensional grid data (e.g., images) while maximizing **Compute Density** ($R_{peak}$).
1. **Significance (Quantitative):** They exploit weight sharing to decouple parameter count from input size, enabling **$O(1)$** scaling for high-dimensional grid data (e.g., images) while maximizing **Compute Density** ($R_{\text{peak}}$).
2. **Distinction (Durable):** Unlike **MLPs**, which have **Global Connectivity**, CNNs restrict connections to spatially adjacent regions, reflecting the insight that proximity correlates with feature relevance.
3. **Common Pitfall:** A frequent misconception is that CNNs are "vision-only" models. In reality, they are a **Symmetry-Aware Architecture**: they can be applied to any data with a grid-like topology, including audio (spectrograms) and text (1D-convolutions).
@@ -1856,7 +1856,7 @@ The assumption of sequential dependence guides the introduction of memory as a c
***Recurrent Neural Networks (RNNs)***\index{RNN!definition} are sequence-processing architectures that maintain a hidden state $h_t = f(h_{t-1}, x_t)$ updated at each time step, encoding the assumption that the current output depends on all prior inputs through this fixed-size state vector.
1. **Significance (Quantitative):** The fixed-size state provides $O(1)$ inference memory regardless of sequence length — processing a 10,000-token sequence requires the same memory as a 10-token sequence — but the sequential update rule creates a sequential bottleneck where all $T$ steps must execute in order, directly contributing to the $L_{lat}$ term of the Iron Law and making RNNs unable to exploit GPU parallelism during training.
1. **Significance (Quantitative):** The fixed-size state provides $O(1)$ inference memory regardless of sequence length — processing a 10,000-token sequence requires the same memory as a 10-token sequence — but the sequential update rule creates a sequential bottleneck where all $T$ steps must execute in order, directly contributing to the $L_{\text{lat}}$ term of the Iron Law and making RNNs unable to exploit GPU parallelism during training.
2. **Distinction (Durable):** Unlike Attention Mechanisms, which access the entire token history simultaneously with $O(N^2)$ memory cost, RNNs compress history into a bottleneck state, meaning gradient signal must propagate back through all $T$ steps — causing $\partial L / \partial h_0 \propto \prod_{t=1}^{T} \partial h_t / \partial h_{t-1}$, a product of $T$ Jacobians that vanishes or explodes exponentially with sequence length.
3. **Common Pitfall:** A frequent misconception is that RNNs are obsolete. For streaming inference on resource-constrained hardware where $O(N^2)$ attention memory is prohibitive — such as keyword spotting on a microcontroller — an RNN's $O(1)$ state size remains the systems-justified choice.
@@ -2075,7 +2075,7 @@ For a sequence processing task with input dimension 100 and hidden state dimensi
### System Implications {#sec-network-architectures-system-implications-18c3}
RNNs introduce an inescapable system constraint: **Sequential Dependency**. Unlike MLPs and CNNs where parallelism scales with the number of neurons or pixels, RNN parallelism is limited by the sequence length. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), neither increasing $O$ (compute throughput) nor $D_{vol}$ (memory bandwidth) can help—the bottleneck is *latency* along the sequential critical path.
RNNs introduce an inescapable system constraint: **Sequential Dependency**. Unlike MLPs and CNNs where parallelism scales with the number of neurons or pixels, RNN parallelism is limited by the sequence length. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), neither increasing $O$ (compute throughput) nor $D_{\text{vol}}$ (memory bandwidth) can help—the bottleneck is *latency* along the sequential critical path.
#### Computation Needs: The Wall of Time {#sec-network-architectures-computation-needs-wall-time-3457}
@@ -2107,7 +2107,7 @@ Attention mechanisms[^fn-bahdanau-attention] address precisely this challenge [@
***Attention Mechanisms***\index{Attention Mechanism!definition} are neural network operations that compute a weighted sum of value vectors, where the weights are derived from learned similarity scores between a query vector and a set of key vectors, enabling dynamic, content-dependent information routing between any two positions in a sequence.
1. **Significance (Quantitative):** Attention connects any two tokens in $O(1)$ depth, but the similarity matrix requires $O(N^2)$ memory: for a 4,096-token sequence with 16-bit scores, the attention matrix alone consumes $4096^2 \times 2 \approx 32$ MB per layer per head — a direct contribution to the $D_{vol}$ and $BW$ terms of the Iron Law that ultimately caps practical context window length.
1. **Significance (Quantitative):** Attention connects any two tokens in $O(1)$ depth, but the similarity matrix requires $O(N^2)$ memory: for a 4,096-token sequence with 16-bit scores, the attention matrix alone consumes $4096^2 \times 2 \approx 32$ MB per layer per head — a direct contribution to the $D_{\text{vol}}$ and $BW$ terms of the Iron Law that ultimately caps practical context window length.
2. **Distinction (Durable):** Unlike RNNs, which compress all prior context into a single fixed-size state vector, attention mechanisms retain all $N$ prior token representations and compute relevance scores at inference time, trading RNN's $O(1)$ memory for $O(N^2)$ memory in exchange for eliminating the sequential bottleneck on long-range dependencies.
3. **Common Pitfall:** A frequent misconception is that attention is a general-purpose weighting scheme that can be applied freely. The $O(N^2)$ memory growth is a hard physical constraint: doubling the context window quadruples the attention memory, which is why FlashAttention and sparse attention variants exist — they recompute rather than store the attention matrix to break this memory wall.
@@ -2572,7 +2572,7 @@ The translation from attention's mathematical elegance to hardware execution rev
### System Implications {#sec-network-architectures-system-implications-05a3}
Attention mechanisms exhibit distinctive system-level patterns that differ from previous architectures through their dynamic connectivity requirements. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), attention shifts the bottleneck from the latency-bound sequential path of RNNs to $D_{vol}$ (data volume) -- the $O(N^2)$ attention matrix must be materialized in memory, making attention **memory-bound** rather than compute-bound for large sequences.
Attention mechanisms exhibit distinctive system-level patterns that differ from previous architectures through their dynamic connectivity requirements. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), attention shifts the bottleneck from the latency-bound sequential path of RNNs to $D_{\text{vol}}$ (data volume) -- the $O(N^2)$ attention matrix must be materialized in memory, making attention **memory-bound** rather than compute-bound for large sequences.
#### Memory Requirements {#sec-network-architectures-memory-requirements-72c3}
@@ -2906,7 +2906,7 @@ The self-attention implementation above shows how Transformers process entire se
### System Implications {#sec-network-architectures-system-implications-77ac}
The quadratic bottleneck analyzed above manifests differently during training and inference, creating a bifurcation in system behavior defined by two distinct Iron Law regimes (@sec-introduction-iron-law-ml-systems-c32a): training is dominated by $O$ (compute), while inference is dominated by $D_{vol}$ (data movement).
The quadratic bottleneck analyzed above manifests differently during training and inference, creating a bifurcation in system behavior defined by two distinct Iron Law regimes (@sec-introduction-iron-law-ml-systems-c32a): training is dominated by $O$ (compute), while inference is dominated by $D_{\text{vol}}$ (data movement).
#### Training: The Quadratic Compute Wall {#sec-network-architectures-training-quadratic-compute-wall-de9d}
@@ -3088,7 +3088,7 @@ DLRM's computational mapping splits into two regimes that stress different hardw
### System Implications {#sec-network-architectures-system-implications-d8ee}
DLRM creates a unique systems challenge: **the model is too big to fit on a single GPU**. While a ResNet-50 (`{python} LighthouseSpecs.resnet_fp32_mb_str` MB) or even GPT-3 (`{python} LighthouseSpecs.gpt3_fp16_gb_str` GB) might fit on a single node, industrial recommendation models can reach terabytes or petabytes due to massive embedding tables. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), neither $O$ nor $D_{vol}$ is the binding constraint—it is raw **memory capacity** that limits the system, a regime the Iron Law was not designed to capture.
DLRM creates a unique systems challenge: **the model is too big to fit on a single GPU**. While a ResNet-50 (`{python} LighthouseSpecs.resnet_fp32_mb_str` MB) or even GPT-3 (`{python} LighthouseSpecs.gpt3_fp16_gb_str` GB) might fit on a single node, industrial recommendation models can reach terabytes or petabytes due to massive embedding tables. In Iron Law terms (@sec-introduction-iron-law-ml-systems-c32a), neither $O$ nor $D_{\text{vol}}$ is the binding constraint—it is raw **memory capacity** that limits the system, a regime the Iron Law was not designed to capture.
\index{Model Parallelism!embedding sharding}
This forces a specific parallelization strategy called *model parallelism* (specifically, **embedding sharding**\index{Embedding!sharding}):
@@ -3468,7 +3468,7 @@ Gating mechanisms were born in RNNs, where early sequence models hit a "temporal
[^fn-lstm-invention]: **LSTM (Long Short-Term Memory)**\index{LSTM!gating mechanism}: Invented by Hochreiter and Schmidhuber in 1997, LSTMs introduced a "Constant Error Carousel" -- a gated cell state that protects error signals from exponential decay during backpropagation through time. The systems cost of this solution: three gates per cell (forget, input, output) triple the parameter count and GEMM operations compared to a vanilla RNN, making each LSTM time step ~4$\times$ more expensive. This compute overhead is why Transformers, which solve long-range dependencies through parallelizable attention, displaced LSTMs in most production systems. \index{LSTM!compute overhead}
[^fn-gru]: **GRU (Gated Recurrent Unit)**\index{GRU!efficiency}: Cho et al. (2014) simplified the LSTM from 3 gates to 2, reducing parameters by ~25% and GEMM operations per step proportionally. GRUs match LSTM accuracy on most benchmarks while fitting more easily into memory-constrained deployments. The broader systems lesson: architectural simplification that reduces parameters without sacrificing task performance directly lowers $D_{vol}$ and training time, a principle that recurs in every efficiency-oriented design from MobileNet to distilled Transformers. \index{GRU!parameter efficiency}
[^fn-gru]: **GRU (Gated Recurrent Unit)**\index{GRU!efficiency}: Cho et al. (2014) simplified the LSTM from 3 gates to 2, reducing parameters by ~25% and GEMM operations per step proportionally. GRUs match LSTM accuracy on most benchmarks while fitting more easily into memory-constrained deployments. The broader systems lesson: architectural simplification that reduces parameters without sacrificing task performance directly lowers $D_{\text{vol}}$ and training time, a principle that recurs in every efficiency-oriented design from MobileNet to distilled Transformers. \index{GRU!parameter efficiency}
The key insight is that gating is *not* an RNN-specific technique—it is a general principle of using neural networks to modulate other neural networks. This concept has since migrated well beyond sequence processing:
@@ -3965,7 +3965,7 @@ RNNs require different optimization approaches due to their temporal dependencie
Transformer attention demands specialized optimizations that reduce memory usage and complexity. Techniques such as FlashAttention[^fn-flashattention] and sparse attention patterns, which can dramatically reduce resource requirements, are examined in @sec-model-compression.
[^fn-flashattention]: **FlashAttention**\index{FlashAttention!IO-aware}: An IO-aware algorithm (Dao et al., 2022) that avoids materializing the full $N \times N$ attention matrix in HBM by fusing computation into a single kernel tiled to fit in SRAM. The result: 2--4$\times$ wall-clock speedup and memory reduction from $O(N^2)$ to $O(N)$, enabling training on sequences 4--16$\times$ longer than standard attention. FlashAttention demonstrates that algorithmic optimization of data movement ($D_{vol}$) can yield larger speedups than increasing raw compute ($R_{peak}$) -- a concrete validation of the Iron Law's data term. \index{FlashAttention!memory reduction}
[^fn-flashattention]: **FlashAttention**\index{FlashAttention!IO-aware}: An IO-aware algorithm (Dao et al., 2022) that avoids materializing the full $N \times N$ attention matrix in HBM by fusing computation into a single kernel tiled to fit in SRAM. The result: 2--4$\times$ wall-clock speedup and memory reduction from $O(N^2)$ to $O(N)$, enabling training on sequences 4--16$\times$ longer than standard attention. FlashAttention demonstrates that algorithmic optimization of data movement ($D_{\text{vol}}$) can yield larger speedups than increasing raw compute ($R_{\text{peak}}$) -- a concrete validation of the Iron Law's data term. \index{FlashAttention!memory reduction}
The complexity patterns detailed in each architecture's System Implications section define optimal domains. MLPs excel when parameter efficiency is not critical, CNNs dominate for moderate-resolution spatial data, RNNs remain viable for very long sequences where memory is constrained, and Transformers excel for complex relational tasks where their computational cost is justified through superior performance. With these quantitative foundations established, we can construct a systematic decision framework for architecture selection.

6
book/quarto/contents/vol1/nn_computation/nn_computation.qmd
View File

@@ -120,7 +120,7 @@ The operators that follow are not abstract theory but a specification for comput
***Deep Learning***\index{Deep Learning!definition} is the computational paradigm of **Hierarchical Feature Learning** from raw data.
1. **Significance (Quantitative):** By stacking nonlinear transformations, it replaces manual **Feature Engineering** with **Architecture Engineering**, enabling models to scale with both **Data Volume ($D_{vol}$)** and **Compute ($R_{peak}$)**.
1. **Significance (Quantitative):** By stacking nonlinear transformations, it replaces manual **Feature Engineering** with **Architecture Engineering**, enabling models to scale with both **Data Volume ($D_{\text{vol}}$)** and **Compute ($R_{\text{peak}}$)**.
2. **Distinction (Durable):** Unlike **Shallow Learning**, which learns a single transformation, Deep Learning learns a **Hierarchy of Abstractions** that can be fine-tuned for different tasks.
3. **Common Pitfall:** A frequent misconception is that Deep Learning is "just a big neural network." In reality, it is a **Systems Strategy**: it uses the **Iron Law** to trade computation ($O$) for the ability to generalize from high-dimensional inputs.
@@ -2453,7 +2453,7 @@ class MemoryExplosionCalc:
::: {.callout-notebook title="The Memory Explosion"}
How does the scale of our Lighthouse Models affect the **Data ($D_{vol}$)** term of the Iron Law? Compare our MNIST classifier to **GPT-2**.
How does the scale of our Lighthouse Models affect the **Data ($D_{\text{vol}}$)** term of the Iron Law? Compare our MNIST classifier to **GPT-2**.
- **MNIST Archetype**: `{python} MemoryExplosionCalc.mnist_params_count_str` parameters$\times$ 4 bytes (FP32) ≈ **`{python} MemoryExplosionCalc.mnist_mem_str` KB**. This entire model fits inside the L2 cache of a modern processor.
- **GPT-2 Archetype**: `{python} MemoryExplosionCalc.gpt2_params_count_str` parameters$\times$ 4 bytes (FP32) ≈ **`{python} MemoryExplosionCalc.gpt2_mem_str` GB**. This requires dedicated GPU VRAM and high-speed memory bandwidth.
@@ -3299,7 +3299,7 @@ A network that achieves 99.5% accuracy on training data but only 85% on new data
***Overfitting***\index{Overfitting!definition} is the failure of **Generalization** caused by memorizing **Noise** instead of **Signal**.
1. **Significance (Quantitative):** It occurs when a model's **Capacity** exceeds the information content of the training data ($D_{vol}$), allowing it to satisfy the training objective without learning the underlying distribution.
1. **Significance (Quantitative):** It occurs when a model's **Capacity** exceeds the information content of the training data ($D_{\text{vol}}$), allowing it to satisfy the training objective without learning the underlying distribution.
2. **Distinction (Durable):** Unlike **Underfitting** (where the model is too simple), Overfitting is a **Symmetry Breaking** problem: the model becomes too specialized to the specific training sample.
3. **Common Pitfall:** A frequent misconception is that Overfitting is "solved" by more data. In reality, it is a **Capacity-Data Gap**: without proper **Regularization**, larger models will eventually overfit even large datasets.

10
book/quarto/contents/vol1/optimizations/model_compression.qmd
View File

@@ -140,7 +140,7 @@ Throughout this chapter, we ground each technique in concrete systems: ResNet-50
***Model Compression***\index{Model Compression!definition} is a family of techniques that reduce a trained model's computational cost and memory footprint by eliminating redundant parameters (pruning), reducing numerical precision (quantization), or transferring learned behavior into a smaller architecture (distillation), while preserving as much predictive accuracy as possible.
1. **Significance (Quantitative):** Compression directly reduces both Iron Law terms. INT8 quantization of a 175B-parameter LLM cuts weight memory from 350 GB (FP16) to 175 GB — a 2$\times$ reduction in $D_{vol}$ — while enabling Tensor Core INT8 paths that are 2$\times$ faster than FP16, yielding up to 4$\times$ combined throughput improvement. Unstructured pruning to 50% sparsity theoretically halves $O$, but hardware speedup only materializes when sparsity is structured (e.g., 2:4 sparse) to match accelerator capabilities.
1. **Significance (Quantitative):** Compression directly reduces both Iron Law terms. INT8 quantization of a 175B-parameter LLM cuts weight memory from 350 GB (FP16) to 175 GB — a 2$\times$ reduction in $D_{\text{vol}}$ — while enabling Tensor Core INT8 paths that are 2$\times$ faster than FP16, yielding up to 4$\times$ combined throughput improvement. Unstructured pruning to 50% sparsity theoretically halves $O$, but hardware speedup only materializes when sparsity is structured (e.g., 2:4 sparse) to match accelerator capabilities.
2. **Distinction (Durable):** Unlike Neural Architecture Search, which discovers efficient architectures from scratch by exploring a design space, Model Compression starts from an existing trained model and reduces its cost post hoc — meaning the accuracy ceiling is bounded by the base model, and the compression ratio is constrained by the architecture's inherent redundancy.
3. **Common Pitfall:** A frequent misconception is that compression techniques compose without interference. In practice, applying quantization after pruning can amplify quantization error in near-zero weight regions that pruning left behind, causing accuracy degradation that neither technique produces alone.
@@ -194,7 +194,7 @@ To understand *why* numerics matter so deeply, consider the *physics of quantiza
The **Energy-Movement Invariant** ($E_{move} \gg E_{compute}$) means that in the physics of silicon, **bits represent energy** (see @sec-machine-foundations-numerical-representations-c889 for a detailed comparison of FP32 vs. INT8 energy costs).
According to the **Iron Law** ($T = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat}$), which decomposes execution time into data volume moved, operations performed, and fixed latency, reducing the bit-width of a weight has a quadratic effect on efficiency:
According to the **Iron Law** ($T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}}$), which decomposes execution time into data volume moved, operations performed, and fixed latency, reducing the bit-width of a weight has a quadratic effect on efficiency:
1. **Memory Energy (Dvol)**: Fetching a 32-bit float from DRAM costs ≈ **`{python} CompressionSetup.energy_dram_str` pJ**. Fetching an 8-bit integer costs ≈ **`{python} CompressionSetup.energy_dram_per_byte_str` pJ**.
2. **Compute Energy (O)**: A 32-bit FLOP costs ≈ **`{python} CompressionSetup.energy_flop_fp32_str` pJ**. An 8-bit integer OP costs ≈ **`{python} CompressionSetup.energy_flop_int8_str` pJ**.
@@ -615,9 +615,9 @@ Pruning[^fn-pruning-lecun-1989] directly addresses memory efficiency constraints
***Pruning***\index{Pruning!definition} is the sparsification of the **Parameter Space** by removing weights that contribute minimal information to the loss landscape.
1. **Significance (Quantitative):** It converts dense matrices into sparse structures, reducing the **Memory Footprint** and the total **Data Volume ($D_{vol}$)** by as much as 10$\times$ without significant accuracy loss.
1. **Significance (Quantitative):** It converts dense matrices into sparse structures, reducing the **Memory Footprint** and the total **Data Volume ($D_{\text{vol}}$)** by as much as 10$\times$ without significant accuracy loss.
2. **Distinction (Durable):** Unlike **Quantization**, which reduces the **Precision** of every weight, Pruning reduces the **Count** of weights by identifying and eliminating redundancy.
3. **Common Pitfall:** A frequent misconception is that Pruning "automatically" speeds up execution. In reality, without **Specialized Hardware Support** ($R_{peak}$), the resulting sparse matrices may actually run *slower* than dense ones due to irregular memory access patterns.
3. **Common Pitfall:** A frequent misconception is that Pruning "automatically" speeds up execution. In reality, without **Specialized Hardware Support** ($R_{\text{peak}}$), the resulting sparse matrices may actually run *slower* than dense ones due to irregular memory access patterns.
:::
@@ -3099,7 +3099,7 @@ While DLRM operates at the terabyte scale, our Smart Doorbell Lighthouse faces t
**The Energy and Storage Constraint**: Our **Smart Doorbell Lighthouse** operates at the opposite extreme of the Iron Law from DLRM. While DLRM optimizes for terabyte-scale capacity, the Smart Doorbell's Keyword Spotting (KWS) model must operate within a 100 KB budget to run on a microcontroller with 256 KB RAM.
In FP32, even the compact DS-CNN architecture consumes 4$\times$ more memory bandwidth and energy per inference than in INT8. For an always-on device running on a coin cell battery, this 4$\times$ energy difference translates directly to battery life: a device that lasts 1 month on FP32 might last 4 months on INT8. Here, quantization is the primary lever for the **Energy Term** ($O / (R_{peak} \cdot \eta)$) of the Iron Law.
In FP32, even the compact DS-CNN architecture consumes 4$\times$ more memory bandwidth and energy per inference than in INT8. For an always-on device running on a coin cell battery, this 4$\times$ energy difference translates directly to battery life: a device that lasts 1 month on FP32 might last 4 months on INT8. Here, quantization is the primary lever for the **Energy Term** ($O / (R_{\text{peak}} \cdot \eta)$) of the Iron Law.
:::

10
book/quarto/contents/vol1/parts/build_principles.qmd
View File

@@ -4,12 +4,12 @@ Building a machine learning system is not merely about stacking layers; it is ab
::: {#pri-iron-law .callout-principle title="The Iron Law of ML Systems"}
**The Invariant**: The total time ($T$) of any machine learning operation is governed by three components — data movement, compute, and fixed system overhead:
$$ T = \frac{D_{vol}}{BW} + \frac{O}{R_{peak} \cdot \eta} + L_{lat} $$
where $D_{vol}$ is data volume (bytes moved), $BW$ is memory bandwidth, $O$ is total floating-point operations, $R_{peak}$ is peak compute rate, $\eta$ is hardware utilization efficiency, and $L_{lat}$ is fixed latency overhead such as kernel launch or network round-trip time. (For the full notation rationale, see the Notation and Conventions section.)
$$ T = \frac{D_{\text{vol}}}{BW} + \frac{O}{R_{\text{peak}} \cdot \eta} + L_{\text{lat}} $$
where $D_{\text{vol}}$ is data volume (bytes moved), $BW$ is memory bandwidth, $O$ is total floating-point operations, $R_{\text{peak}}$ is peak compute rate, $\eta$ is hardware utilization efficiency, and $L_{\text{lat}}$ is fixed latency overhead such as kernel launch or network round-trip time. (For the full notation rationale, see the Notation and Conventions section.)
When these stages overlap on modern hardware, wall-clock time is dominated by whichever term is largest. This is why the equation's practical lesson is about **dominance**, not summation: the term that takes longest sets the floor.
**The Implication**: Optimization is rarely free of trade-offs. Reducing one term often shifts the bottleneck to another. For example, unstructured pruning reduces compute ($O$) but introduces irregular memory access patterns that can increase data movement ($D_{vol}/BW$). A "faster" algorithm on paper is only faster in reality if it reduces the **dominant** term for your specific hardware.
**The Implication**: Optimization is rarely free of trade-offs. Reducing one term often shifts the bottleneck to another. For example, unstructured pruning reduces compute ($O$) but introduces irregular memory access patterns that can increase data movement ($D_{\text{vol}}/BW$). A "faster" algorithm on paper is only faster in reality if it reduces the **dominant** term for your specific hardware.
:::
The Iron Law tells us *what* to optimize, but not *how*. The answer depends on which hardware resource your architecture will saturate — a choice that defines an implicit contract:
@@ -17,8 +17,8 @@ The Iron Law tells us *what* to optimize, but not *how*. The answer depends on w
::: {#pri-silicon-contract .callout-principle title="The Silicon Contract"}
**The Invariant**: Every model architecture makes an implicit commitment to the hardware — a wager on which resource it will saturate first.
- **ResNet-50** assumes high-density floating-point compute. It is **Compute-Bound**: performance is limited by $O / (R_{peak} \cdot \eta)$.
- **Llama-3-8B** assumes high-bandwidth memory access. It is **Bandwidth-Bound**: performance is limited by $D_{vol} / BW$.
- **ResNet-50** assumes high-density floating-point compute. It is **Compute-Bound**: performance is limited by $O / (R_{\text{peak}} \cdot \eta)$.
- **Llama-3-8B** assumes high-bandwidth memory access. It is **Bandwidth-Bound**: performance is limited by $D_{\text{vol}} / BW$.
- **DLRM** assumes massive memory capacity for embedding lookup tables. It is **Capacity-Bound**: performance is limited by whether the working set fits in fast memory at all.
**The Implication**: Designing a model without knowing which hardware resource it will saturate is like designing a bridge without knowing the strength of the steel. You must design for the **Bottleneck**.

14
book/quarto/contents/vol1/parts/optimize_principles.qmd
View File

@@ -15,19 +15,19 @@ A working model is rarely an efficient one. Part II established how to construct
Navigating the Pareto Frontier requires knowing *which* resource to optimize. Before selecting a technique, you must diagnose the bottleneck:
::: {.callout-principle #pri-arithmetic-intensity title="Arithmetic Intensity Law" icon="false"}
**The Invariant**: Attainable throughput ($R$) is bounded by the minimum of peak compute ($R_{peak}$) and memory bandwidth ($BW$) relative to the workload's arithmetic intensity ($I$) [@williams2009roofline]:
$$ R = \min(R_{peak}, I \times BW) $$
**The Invariant**: Attainable throughput ($R$) is bounded by the minimum of peak compute ($R_{\text{peak}}$) and memory bandwidth ($BW$) relative to the workload's arithmetic intensity ($I$) [@williams2009roofline]:
$$ R = \min(R_{\text{peak}}, I \times BW) $$
**The Implication**: Adding compute power to a memory-bound model yields **zero** performance gain. You must identify whether your bottleneck is Math (Compute-Bound) or Memory (Bandwidth-Bound) before selecting an optimization technique.
:::
@tbl-bottleneck-diagnostic maps each bottleneck type to the optimization that addresses it — and, equally important, the optimization that would be wasted.
| **If You're...** | **Dominant Term** | **Optimization That Works** | **Optimization That is Wasted** |
|:------------------|:--------------------------|:-----------------------------------------|:--------------------------------|
| **Memory-Bound** | $D_{vol}/BW$ | Quantization, pruning, batching | Faster GPU (more FLOP/s) |
| **Compute-Bound** | $O/(R_{peak} \cdot \eta)$ | Better kernels, Tensor Cores, faster GPU | More memory bandwidth |
| **Latency-Bound** | $L_{lat}$ | Batching, kernel fusion, async dispatch | More FLOP/s or bandwidth alone |
| **If You're...** | **Dominant Term** | **Optimization That Works** | **Optimization That is Wasted** |
|:------------------|:---------------------------------|:-----------------------------------------|:--------------------------------|
| **Memory-Bound** | $D_{\text{vol}}/BW$ | Quantization, pruning, batching | Faster GPU (more FLOP/s) |
| **Compute-Bound** | $O/(R_{\text{peak}} \cdot \eta)$ | Better kernels, Tensor Cores, faster GPU | More memory bandwidth |
| **Latency-Bound** | $L_{\text{lat}}$ | Batching, kernel fusion, async dispatch | More FLOP/s or bandwidth alone |
: **The Bottleneck Diagnostic.** Before optimizing, identify which Iron Law term dominates. Optimizing the wrong term yields zero improvement. {#tbl-bottleneck-diagnostic}

4
book/quarto/contents/vol1/responsible_engr/responsible_engr.qmd
View File

@@ -393,9 +393,9 @@ When Amazon's ethics board finally reviewed the recruiting tool, the model had a
::: {.callout-definition title="Responsible AI Engineering"}
***Responsible AI Engineering***\index{Responsible AI Engineering!definition} is the engineering discipline of designing, deploying, and maintaining systems with probabilistic outputs by operationalizing societal and regulatory requirements as testable constraints on the D·A·M axes, bounding which values of $D_{vol}$, $O$, and $R_{peak} \cdot \eta$ are permissible.
***Responsible AI Engineering***\index{Responsible AI Engineering!definition} is the engineering discipline of designing, deploying, and maintaining systems with probabilistic outputs by operationalizing societal and regulatory requirements as testable constraints on the D·A·M axes, bounding which values of $D_{\text{vol}}$, $O$, and $R_{\text{peak}} \cdot \eta$ are permissible.
1. **Significance (Quantitative):** Each D·A·M axis acquires concrete governance constraints: the Data axis is bounded by privacy regulations (e.g., GDPR limits which $D_{vol}$ can be collected), the Algorithm axis is bounded by fairness metrics (e.g., demographic parity within $\varepsilon = 5\%$ across protected groups, meaning positive prediction rates must not differ by more than 5 percentage points), and the Machine axis is bounded by robustness budgets (e.g., accuracy degradation less than 2% under adversarial perturbation $\|\delta\|_\infty \leq 0.01$). Violating these bounds is a system failure, not a research shortcoming.
1. **Significance (Quantitative):** Each D·A·M axis acquires concrete governance constraints: the Data axis is bounded by privacy regulations (e.g., GDPR limits which $D_{\text{vol}}$ can be collected), the Algorithm axis is bounded by fairness metrics (e.g., demographic parity within $\varepsilon = 5\%$ across protected groups, meaning positive prediction rates must not differ by more than 5 percentage points), and the Machine axis is bounded by robustness budgets (e.g., accuracy degradation less than 2% under adversarial perturbation $\|\delta\|_\infty \leq 0.01$). Violating these bounds is a system failure, not a research shortcoming.
2. **Distinction (Durable):** Unlike **AI Ethics** (which articulates aspirational values), Responsible AI Engineering translates those values into **Measurable, Testable Invariants** that can be verified through automated testing and continuous monitoring, using the same lifecycle practices that enforce latency SLOs.
3. **Common Pitfall:** A frequent misconception is that responsibility is "added" at the end of development. The constraints imposed on the Data axis (what data can be collected) propagate forward to constrain the Algorithm axis (what biases will be encoded) and the Machine axis (what audit trails must be kept), making late-stage remediation structurally impossible.

24
book/quarto/contents/vol1/training/training.qmd
View File

@@ -192,12 +192,12 @@ Frameworks provide abstractions for expressing training algorithms, but training
***The Iron Law of Training Performance***\index{Iron Law of Training Performance!definition} is the simplified form of the general Iron Law (see @sec-introduction-iron-law-ml-systems-c32a) that isolates the computational bottleneck of iterative optimization:
$$T_{train} = \frac{O}{R_{peak} \times \eta}$$ {#eq-training-iron-law}
$$T_{train} = \frac{O}{R_{\text{peak}} \times \eta}$$ {#eq-training-iron-law}
The simplification is valid when the pipeline is correctly staged: at training scale with large batches, data movement ($D_{vol}/BW$) is overlapped with compute via prefetching pipelines, and communication overhead ($L_{lat}$) is absorbed by gradient overlap strategies, leaving hardware utilization as the dominant remaining lever. When pipelines are poorly staged, $D_{vol}/BW$ resurfaces as the bottleneck and the simplified form no longer applies.
The simplification is valid when the pipeline is correctly staged: at training scale with large batches, data movement ($D_{\text{vol}}/BW$) is overlapped with compute via prefetching pipelines, and communication overhead ($L_{\text{lat}}$) is absorbed by gradient overlap strategies, leaving hardware utilization as the dominant remaining lever. When pipelines are poorly staged, $D_{\text{vol}}/BW$ resurfaces as the bottleneck and the simplified form no longer applies.
1. **Significance (Quantitative):** The three factors identify three distinct optimization levers: $O$ (reducible by algorithmic changes such as mixed precision, pruning, and distillation), $R_{peak}$ (a hardware property set by procurement), and $\eta$ (the utilization fraction and primary engineering target; GPT-3 training achieved $\eta \approx 0.45$ [@narayanan2021efficient] while current systems target $\eta > 0.55$).
2. **Distinction (Durable):** Unlike the **General Iron Law**, which models all three cost terms ($D_{vol}/BW$, $O/(R_{peak} \cdot \eta)$, $L_{lat}$), this simplified form assumes data movement and communication are not the binding constraint, an assumption that breaks for small-batch workloads or bandwidth-limited deployments.
1. **Significance (Quantitative):** The three factors identify three distinct optimization levers: $O$ (reducible by algorithmic changes such as mixed precision, pruning, and distillation), $R_{\text{peak}}$ (a hardware property set by procurement), and $\eta$ (the utilization fraction and primary engineering target; GPT-3 training achieved $\eta \approx 0.45$ [@narayanan2021efficient] while current systems target $\eta > 0.55$).
2. **Distinction (Durable):** Unlike the **General Iron Law**, which models all three cost terms ($D_{\text{vol}}/BW$, $O/(R_{\text{peak}} \cdot \eta)$, $L_{\text{lat}}$), this simplified form assumes data movement and communication are not the binding constraint, an assumption that breaks for small-batch workloads or bandwidth-limited deployments.
3. **Common Pitfall:** A frequent error is treating $\eta$ as fixed by hardware. System efficiency is a pipeline property: memory bandwidth saturation, kernel launch overhead, and synchronization barriers each reduce $\eta$ independently, and diagnosing which factor dominates requires profiling rather than reading hardware specs.
:::
@@ -237,7 +237,7 @@ Training speed is governed by the utilization of hardware peaks.
:::
\index{Transformer!training evolution}
The Iron Law provides a static framework for reasoning about training performance, but the history of deep learning reveals how the *binding constraint* has shifted over time as hardware and algorithms co-evolved. In 1986, backpropagation was formalized [@rumelhart1986learning], and training a 3-layer network on toy datasets required days on CPU workstations---the bottleneck was raw compute throughput ($R_{peak}$). In 2012, AlexNet demonstrated GPU training [@alexnet2012], reducing ImageNet training from weeks to days and launching the deep learning era. By 2017, Transformers and NVIDIA Volta Tensor Cores enabled mixed-precision training with a further 5$\times$ speedup [@vaswani2017attention]. GPT-3 in 2020 used over `{python} TrainingScenarios.gpt3_gpu_count_str` V100 GPUs at an estimated \$`{python} TrainingScenarios.gpt3_compute_cost_str`M cost [@brown2020language], making utilization ($\eta$) critical. By 2023, training efficiency improved 10$\times$ through the techniques examined in this chapter: FlashAttention reduces $O$ while improving $\eta$; gradient checkpointing trades $O$ for memory capacity; mixed precision increases $R_{peak}$. Each innovation was motivated by a specific Iron Law bottleneck.
The Iron Law provides a static framework for reasoning about training performance, but the history of deep learning reveals how the *binding constraint* has shifted over time as hardware and algorithms co-evolved. In 1986, backpropagation was formalized [@rumelhart1986learning], and training a 3-layer network on toy datasets required days on CPU workstations---the bottleneck was raw compute throughput ($R_{\text{peak}}$). In 2012, AlexNet demonstrated GPU training [@alexnet2012], reducing ImageNet training from weeks to days and launching the deep learning era. By 2017, Transformers and NVIDIA Volta Tensor Cores enabled mixed-precision training with a further 5$\times$ speedup [@vaswani2017attention]. GPT-3 in 2020 used over `{python} TrainingScenarios.gpt3_gpu_count_str` V100 GPUs at an estimated \$`{python} TrainingScenarios.gpt3_compute_cost_str`M cost [@brown2020language], making utilization ($\eta$) critical. By 2023, training efficiency improved 10$\times$ through the techniques examined in this chapter: FlashAttention reduces $O$ while improving $\eta$; gradient checkpointing trades $O$ for memory capacity; mixed precision increases $R_{\text{peak}}$. Each innovation was motivated by a specific Iron Law bottleneck.
```{python}
#| label: gpt2-lighthouse-hardware-calc
@@ -320,7 +320,7 @@ Training systems occupy a critical position in the machine learning pipeline: th
::: {.callout-perspective title="The 10 GB to 10 TB Scale Factor"}
- **At 10 GB**: The entire dataset often fits in system RAM. Data loading is a one-time "startup cost," and the disk bandwidth ($BW$) does not matter after the first few seconds.
- **At 10 TB**: Data becomes a continuous, high-pressure stream. The system can no longer "load" the data; it must **orchestrate** its movement. The $D_{vol}$ term shifts from a storage bottleneck to a *networking and I/O bottleneck*, requiring zero-copy paths and multi-worker prefetching just to keep the accelerator from starving.
- **At 10 TB**: Data becomes a continuous, high-pressure stream. The system can no longer "load" the data; it must **orchestrate** its movement. The $D_{\text{vol}}$ term shifts from a storage bottleneck to a *networking and I/O bottleneck*, requiring zero-copy paths and multi-worker prefetching just to keep the accelerator from starving.
Scale is not just "more data"; it is a transformation of the system's physics.
@@ -827,8 +827,8 @@ However, processing single examples creates new system challenges. Modern accele
***Batch Processing***\index{Batch Processing!definition} is the aggregation of multiple training examples into a single tensor operation to amortize fixed per-step overhead (kernel launch, optimizer update) across $B$ examples, shifting the workload from memory-bandwidth-bound to compute-bound as $B$ increases.
1. **Significance (Quantitative):** Throughput increases with batch size up to the critical batch size, beyond which additional examples provide diminishing gradient quality without proportional convergence benefit. For ResNet-50 on ImageNet, empirical studies find the critical batch size near $B \approx 8{,}192$: at this batch size, throughput approaches $R_{peak}$ while validation accuracy is preserved; larger batches require learning rate scaling (linear rule: $\text{lr} \propto B$) to compensate for reduced update frequency.
2. **Distinction (Durable):** Unlike stochastic gradient descent ($B=1$), which updates parameters after every example with maximum noise, mini-batch processing averages gradients over $B$ examples, reducing gradient variance by $1/\sqrt{B}$ — lowering the $D_{vol}$ transfers per effective update while giving the hardware enough parallel work to reach compute-bound utilization.
1. **Significance (Quantitative):** Throughput increases with batch size up to the critical batch size, beyond which additional examples provide diminishing gradient quality without proportional convergence benefit. For ResNet-50 on ImageNet, empirical studies find the critical batch size near $B \approx 8{,}192$: at this batch size, throughput approaches $R_{\text{peak}}$ while validation accuracy is preserved; larger batches require learning rate scaling (linear rule: $\text{lr} \propto B$) to compensate for reduced update frequency.
2. **Distinction (Durable):** Unlike stochastic gradient descent ($B=1$), which updates parameters after every example with maximum noise, mini-batch processing averages gradients over $B$ examples, reducing gradient variance by $1/\sqrt{B}$ — lowering the $D_{\text{vol}}$ transfers per effective update while giving the hardware enough parallel work to reach compute-bound utilization.
3. **Common Pitfall:** A frequent misconception is that linear learning rate scaling (multiplying lr by $B/B_0$) works at any batch size. The linear rule holds only up to the critical batch size; beyond it, the noise reduction from larger batches no longer compensates for the reduced number of updates per epoch, and validation accuracy degrades even with perfectly scaled learning rates.
:::
@@ -2900,7 +2900,7 @@ The diagnostic methodology that transforms blueprint knowledge into actionable o
::: {.callout-definition title="Model FLOPs Utilization (MFU)"}
***Model FLOPs Utilization (MFU)***\index{Model FLOPs Utilization!definition} is the efficiency metric $\text{MFU} = C_{\text{model}} / (R_{peak} \cdot T_{\text{step}})$, where $C_{\text{model}}$ is the forward-pass FLOP count of the model and $T_{\text{step}}$ is the measured wall-clock time per training step, expressing what fraction of peak hardware throughput is doing useful model computation.
***Model FLOPs Utilization (MFU)***\index{Model FLOPs Utilization!definition} is the efficiency metric $\text{MFU} = C_{\text{model}} / (R_{\text{peak}} \cdot T_{\text{step}})$, where $C_{\text{model}}$ is the forward-pass FLOP count of the model and $T_{\text{step}}$ is the measured wall-clock time per training step, expressing what fraction of peak hardware throughput is doing useful model computation.
1. **Significance (Quantitative):** MFU is the $\eta$ term in the Iron Law made concrete. For a 7B-parameter Transformer on an A100 (312 TFLOPS BF16), a step time of 1.2 ms yields $\text{MFU} = 7\text{B} \times 6 \text{ FLOPs/param} / (312\text{e}12 \times 1.2\text{e}{-3}) \approx 0.11$ (11%) — meaning 89% of peak compute is lost to memory stalls, communication, and scheduling overhead. Production systems typically reach 30–50% MFU; values below 30% indicate a specific addressable bottleneck.
2. **Distinction (Durable):** Unlike hardware utilization reported by profilers (which counts all cycles where the compute units are active, including gradient checkpointing recomputation and padding FLOPs), MFU counts only the FLOPs that directly advance the model toward convergence — providing a hardware-agnostic efficiency score that is comparable across different accelerator generations.
@@ -5060,14 +5060,14 @@ Within a single node, GPUs communicate via high-bandwidth interconnects like NVL
**1. Data Parallelism (Split the Batch)**
* **Compute ($O$)**: Split by $N$ (Each GPU does $1/N$ of the batch).
* **Memory ($D_{vol}$)**: **Replicated**. Every GPU must hold the full model weights $P$.
* **Memory ($D_{\text{vol}}$)**: **Replicated**. Every GPU must hold the full model weights $P$.
* **Communication**: **Gradients**. Size $\propto P$. Occurs at end of backward pass.
* **Bottleneck**: When Model Size $P >$ GPU Memory.
**2. Model Parallelism (Split the Weights)**
* **Compute ($O$)**: Split by $N$ (Each accelerator computes part of the layer).
* **Memory ($D_{vol}$)**: **Split**. Each GPU holds $P/N$ weights.
* **Memory ($D_{\text{vol}}$)**: **Split**. Each GPU holds $P/N$ weights.
* **Communication**: **Activations**. Size $\propto B \times \text{Width}$. Occurs at every layer boundary.
* **Bottleneck**: When Activation Size is large (high communication frequency).
@@ -5506,7 +5506,7 @@ This co-design principle, where algorithms, software frameworks, and hardware ar
::: {.callout-takeaways title="Why Training Costs Millions"}
* **The Iron Law governs training**: $T_{train} = \frac{O}{R_{peak} \times \eta}$. Every optimization affects one of these terms. Identifying which term is affected is essential for effective optimization.
* **The Iron Law governs training**: $T_{train} = \frac{O}{R_{\text{peak}} \times \eta}$. Every optimization affects one of these terms. Identifying which term is affected is essential for effective optimization.
* **Memory is dominated by optimizer state and activations, not weights**: Adam's two state vectors per parameter create a 3$\times$ multiplier over model size, and activation memory scales linearly with batch size and depth. Together, these determine whether a model fits on a given GPU---not the parameter count alone.
* **Optimizer selection is a memory-convergence tradeoff**: Adam converges in roughly one-third the iterations of SGD but requires 3$\times$ the memory for per-parameter state, making the choice a binding constraint for large model training. Variants like AdamW and 8-bit Adam shift this tradeoff without eliminating it.
* **Profiling precedes optimization**: The iterative loop is: profile → identify bottleneck → apply targeted fix → re-profile. Optimization without profiling typically wastes effort on non-bottlenecks.

4
book/quarto/contents/vol2/data_storage/data_storage.qmd
View File

@@ -683,7 +683,7 @@ Local NVMe provides high bandwidth and low latency within a single node, but dis
***Parallel File System (PFS)***\index{Parallel File System!definition} is a distributed storage architecture that stripes data across many storage servers to provide aggregate throughput exceeding the capacity of any single device.
1. **Significance (Quantitative):** A PFS aggregates $BW_{io}$ linearly with the number of storage servers (Object Storage Servers). A Lustre cluster with 20 OSS nodes each delivering 10 GB/s provides 200 GB/s aggregate — versus a single NAS server capped at 10 GB/s — enabling an 8,000-GPU training job to load each 512-token batch in under 50 ms rather than 1 second. This aggregate bandwidth directly reduces the $D_{vol}/BW$ term in the Iron Law.
1. **Significance (Quantitative):** A PFS aggregates $BW_{io}$ linearly with the number of storage servers (Object Storage Servers). A Lustre cluster with 20 OSS nodes each delivering 10 GB/s provides 200 GB/s aggregate — versus a single NAS server capped at 10 GB/s — enabling an 8,000-GPU training job to load each 512-token batch in under 50 ms rather than 1 second. This aggregate bandwidth directly reduces the $D_{\text{vol}}/BW$ term in the Iron Law.
2. **Distinction (Durable):** Unlike Network Attached Storage (NAS), where every I/O request routes through a single server, a PFS client receives stripe location metadata from a dedicated Metadata Server (MDS) and then reads data directly from multiple OSS nodes in parallel — the MDS and OSS paths are architecturally separated, so data bandwidth scales with OSS count while metadata operations scale with MDS count.
3. **Common Pitfall:** A frequent misconception is that a PFS has unlimited throughput if enough OSS nodes are added. In reality, a Lustre MDS handles roughly 100,000–300,000 metadata operations per second; at 10,000 workers each opening one small file, the MDS saturates in under 1 second and becomes the serialization point that idles the entire cluster regardless of how many OSS nodes are present.
@@ -1434,7 +1434,7 @@ A `{python} StorageHierarchyAnalysis.gpt3_params_b`B parameter model with Adam o
***Checkpoint Storm***\index{Checkpoint Storm!definition} is a burst of synchronized network and storage traffic that occurs when all nodes in a training fleet save model state simultaneously.
1. **Significance (Quantitative):** The storm magnitude scales as $T_{save} = N \times \text{per-node-shard} / BW_{fabric}$. For a 70B-parameter model trained across 1,000 nodes with ZeRO-3 sharding ($\approx$140 GB per node at FP16), a checkpoint generates 140 TB of simultaneous writes; at 100 GB/s fabric bandwidth, $T_{save} \approx 1{,}400$ seconds — over 23 minutes of training stall per checkpoint event. This adds directly to the $L_{lat}$ term and can dwarf the compute time between checkpoints.
1. **Significance (Quantitative):** The storm magnitude scales as $T_{save} = N \times \text{per-node-shard} / BW_{fabric}$. For a 70B-parameter model trained across 1,000 nodes with ZeRO-3 sharding ($\approx$140 GB per node at FP16), a checkpoint generates 140 TB of simultaneous writes; at 100 GB/s fabric bandwidth, $T_{save} \approx 1{,}400$ seconds — over 23 minutes of training stall per checkpoint event. This adds directly to the $L_{\text{lat}}$ term and can dwarf the compute time between checkpoints.
2. **Distinction (Durable):** Unlike general I/O contention (which is stochastic and unpredictable), a Checkpoint Storm is synchronous and periodic — every node writes at the same moment because the training orchestrator triggers checkpoint after a fixed step count. Its predictability makes it both more damaging (all nodes compete simultaneously) and more tractable (it can be prevented by design through staggered scheduling or asynchronous serialization).
3. **Common Pitfall:** A frequent misconception is that checkpointing every 100 steps is "low overhead." At 70B scale, this assumption is catastrophically wrong: if each training step takes 10 seconds and a checkpoint storm takes 1,400 seconds, checkpointing every 100 steps means spending $1400/(100 \times 10) = 140\%$ of compute time on checkpointing — more time saving state than doing useful training.

412
book/quarto/contents/vol2/introduction/introduction.qmd
View File

@@ -381,79 +381,391 @@ These scale-induced challenges drive infrastructure investment by the largest AI
[^fn-infiniband-rdma-v2]: **InfiniBand (IB)**: Born in 1999 from the merger of Intel's NGIO and the Compaq/IBM Future I/O initiatives, InfiniBand was originally designed to replace the PCI bus. Its defining feature for ML is RDMA (Remote Direct Memory Access), which bypasses the OS kernel to transfer data directly between application memory on different machines at sub-microsecond latency. HDR IB delivers `{python} InfiniBandScenario.ib_hdr_gbs` GB/s usable bandwidth per link; NDR reaches `{python} InfiniBandScenario.ib_ndr_gbs` GB/s. This bandwidth gap versus standard Ethernet (12--50 GB/s) determines whether large-model training is compute-bound or communication-bound. \index{InfiniBand!RDMA}
## A Breed Apart: The ML Workload Character {#sec-vol2-introduction-breed-apart}
Existing distributed systems like Apache Spark and standard web microservices cannot run the Machine Learning Fleet. The **workload characteristics** of ML systems differ fundamentally from traditional distributed systems, even though the underlying hardware (network, compute, storage) is identical.
## The Engineering Crux: A Hierarchy of Architecture {#sec-vol2-introduction-engineering-crux}
::: {#sec-vol2-introduction-breed-apart}
:::
\index{Engineering Crux!hierarchy of scale}
Building machine learning systems at fleet scale requires a reproducible hierarchy of components. We formalize this throughout Volume II as the **Engineering Crux**: a four-layer stack that transforms raw cluster resources into global-scale AI applications.
Existing distributed systems like Apache Spark and standard web microservices cannot run the Machine Learning Fleet. The workload characteristics of ML systems differ fundamentally from traditional distributed systems, even though the underlying hardware (network, compute, storage) is identical. To understand why, we need a reproducible hierarchy of components. We formalize this throughout Volume II as the **Engineering Crux**: a four-layer stack that transforms raw cluster resources into global-scale AI applications.
@fig-vol2-system-scaling-regimes visualizes the transition from single-node to fleet. Volume I covered the left side of this diagram: 1–8 accelerators connected by shared memory, where the binding constraint is the **Memory Wall**\index{Memory Wall}. This volume crosses the scaling arrow into the **Distributed Fleet** regime, where thousands of nodes coordinate across a high-speed switch fabric and the bottleneck shifts to the **Bisection Bandwidth Wall**\index{Bisection Bandwidth Wall}: network congestion and message-passing latency dominate.
::: {#fig-vol2-system-scaling-regimes fig-env="figure" fig-pos="htb" fig-cap="**The Scaling Regimes of ML Systems**: Machine learning engineering is partitioned into two distinct physical regimes. Single-node systems are limited by local memory bandwidth (**Memory Wall**), while distributed fleets are limited by network communication (**Bisection Bandwidth Wall**). Mastery of intra-node data movement is the prerequisite for distributed scaling." fig-alt="Diagram comparing Single-Node Stack (Application, ML Framework, System Software, Hardware) to Distributed Fleet Stack (Governance, Serving/Ops, Distribution, Infrastructure), separated by a scaling arrow."}
```{.tikz}
\begin{tikzpicture}[font=\usefont{T1}{phv}{m}{n}\footnotesize, x=1cm, y=1.1cm]
\tikzset{
stack/.style={rectangle, rounded corners=2pt, draw, align=center, minimum width=3.5cm, minimum height=0.8cm, line width=0.8pt},
label/.style={font=\usefont{T1}{phv}{b}{n}\small, align=center},
sublabel/.style={font=\usefont{T1}{phv}{m}{n}\scriptsize, color=black!70, align=center},
bottleneck/.style={rectangle, rounded corners=2pt, draw, dashed, align=center, minimum width=3.5cm, minimum height=0.6cm, line width=0.6pt, fill=RedLine!5},
arrow/.style={-latex, line width=1.5pt, draw=GreenLine}
\begin{tikzpicture}[font=\usefont{T1}{phv}{m}{n}\small]
\tikzset{
Box/.style={align=center, inner xsep=2pt,draw=GreenLine, line width=0.5pt,
node distance=8.5mm,fill=none, minimum width=27mm, minimum height=20mm},
Box1/.style={Box,draw=OrangeLine,fill=none},
Box2/.style={align=center, inner sep=0pt,draw=GreenLine, line width=0.5pt,
anchor=south east,fill=none, minimum width=29mm, minimum height=6mm},
BoxD/.style={Box,font=\small\usefont{T1}{phv}{b}{n},anchor=north,
draw=red,dashed,text=black,fill=none, line width=1pt,
minimum width=59mm,minimum height=9mm},
BoxD1/.style={Box,font=\small\usefont{T1}{phv}{b}{n},align=center,
draw=none,text=black,fill=black!007, line width=1pt,
minimum width=59mm,minimum height=12mm},
Circle1/.style={circle, minimum size=33mm, draw=none, fill=BrownLine!20},
LineD/.style={BrownLine!60!black!20,line width=4.0pt,dashed,dash pattern=on 5pt off 2pt,
{-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=5pt,shorten >=1pt},
LineA/.style={GreenLine!90,line width=7.0pt,font=\usefont{T1}{phv}{b}{n}\small,
{-{Triangle[width=1.5*8pt,length=2.0*5pt]}},shorten <=7pt,shorten >=8pt},
ALineA/.style={black!60,{Circle[line width=1.0pt,fill=white,round,length=5pt,width=5pt]}-,
line width=1.0pt,shorten <=-3pt,shorten >=-1pt},
Text1/.style={font=\usefont{T1}{phv}{m}{n}\footnotesize,text=black!70}
}
%CPU
\tikzset{%
pics/cpu/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
\node[fill=\filllcolor,minimum width=66, minimum height=66,
rounded corners=2,outer sep=2pt] (C1) {};
\node[fill=white,minimum width=54, minimum height=54] (C2) {};
\node[fill=\filllcolor!40,minimum width=44, minimum height=44] (C3) {\large GPU};
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=3, minimum height=15,
inner sep=0pt,anchor=south](GO\y)at($(C1.north west)!\x!(C1.north east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=3, minimum height=15,
inner sep=0pt,anchor=north](DO\y)at($(C1.south west)!\x!(C1.south east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=3,
inner sep=0pt,anchor=east](LE\y)at($(C1.north west)!\x!(C1.south west)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=3,
inner sep=0pt,anchor=west](DE\y)at($(C1.north east)!\x!(C1.south east)$){};
}
\end{scope}
}
}
}
% Single Node Stack
\node[label] at (0, 4.5) {Single-Node Stack};
\node[sublabel] at (0, 4.1) {1--8 GPUs, Shared Memory};
%Token style
\tikzset{
pics/token/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[draw=\drawcolor,fill=\filllcirclecolor,circle,minimum size=40mm,
line width=\Linewidth](T-\picname){};
\node[draw=white,fill=none,circle,minimum size=0.925*40mm,line width=0.6*\Linewidth]{};
\clip[] circle (0.925*20mm);
\draw[step=5mm,draw=white] (-2,-2) grid (2,2);
\foreach \x/\y[count=\a] in {0/0,1.3/0,1/1,-0.5/1.5,-1.5/0.5,-1.0/-0.9,0.5/-1.3,2.0/0}{
\fill[fill=white,draw=none](\x,0.8*\y)circle(5pt)coordinate(C\a);
}
\draw[white,line width=\Linewidth,fill opacity=0.5,fill=\filllcolor!40](C2)--(C3)--(C4)--(C5)--(C6)--(C7)--cycle;
\foreach \x in {2,...,7}{
\draw[white,line width=\Linewidth](C1)--(C\x);
}
\foreach \x/\y\col[count=\a] in {0/0/red,1.3/0/green,1/1/blue,-0.5/1.5/violet,
-1.5/0.5/magenta,-1.0/-0.9/brown,0.5/-1.3/yellow}{
\fill[fill=\col,draw=none](\x,0.8*\y)circle(5pt)coordinate(C\a);
}
\end{scope}
}
}
}
%display
\tikzset{%
comp/.style = {draw,
minimum width =18mm,
minimum height = 15mm,
inner sep = 0pt,
rounded corners,
draw = \drawcolor,
fill=\filllcolor!10,
line width=2.0pt
},
pics/displayG/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[comp](\picname-COM){};
% \draw[draw = \drawcolor,line width=1.0pt]
% ($(\picname-COM.north west)!0.85!(\picname-COM.south west)$)-- ($(\picname-COM.north east)!0.85!(\picname-COM.south east)$);
\draw[draw = \drawcolor,line width=\Linewidth]($(\picname-COM.south west)!0.4!(\picname-COM.south east)$)--++(270:0.2)coordinate(DL);
\draw[draw = \drawcolor,line width=\Linewidth]($(\picname-COM.south west)!0.6!(\picname-COM.south east)$)--++(270:0.2)coordinate(DD);
\draw[draw = \drawcolor,line width=3*\Linewidth,shorten <=-3mm,shorten >=-3mm](DL)--(DD);
\end{scope}
}
}
}
\node[stack, fill=RedLine!15, draw=RedLine] (app1) at (0, 3.2) {Application};
\node[sublabel] at (0, 3.2) {\\[0.4em]Training Loop / Inference};
%gear
% #1 number of teeths
% #2 radius intern
% #3 radius extern
% #4 angle from start to end of the first arc
% #5 angle to decale the second arc from the first
% #6 inner radius to cut off
\tikzset{
pics/gear/.style args={#1/#2/#3/#4/#5/#6/#7}{
code={
\pgfkeys{/channel/.cd, #7}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\pgfmathtruncatemacro{\N}{#1}%
\def\rin{#2}\def\rout{#3}\def\aA{#4}\def\aOff{#5}\def\rcut{#6}%
\path[rounded corners=1.5pt,draw=\drawcolor,fill=\filllcolor]
(0:\rin)
\foreach \i [evaluate=\i as \n using (\i-1)*360/\N] in {1,...,\N}{%
arc (\n:\n+\aA:\rin)
-- (\n+\aA+\aOff:\rout)
arc (\n+\aA+\aOff:\n+360/\N-\aOff:\rout)
-- (\n+360/\N:\rin)
} -- cycle;
\draw[draw=none,fill=white](0,0) circle[radius=\rcut];
\end{scope}
}}
}
%Infinity_loops
\tikzset{%
radius=2, start angle=-90, line cap=round,
arr node/.style={sloped, allow upside down, single arrow,
single arrow head extend=+2mm, thick, minimum height=+9mm, fill=white},
arr/.style ={ edge node={node[arr node, pos={#1}]{}}},
arr'/.style={insert path={node[arr node, pos={#1}]{}}},
pics/infinityL/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=LOOPS,scale=\scalefac, every node/.append style={transform shape}]
\draw[line width=+2.5mm, sloped, text=white,draw=\drawcolor]
(0, 2) edge[preaction={line cap=butt, line width=+5mm, draw=white, overlay},draw=\filllcirclecolor,
out=0, in=180, arr=.2, arr=.8] (6, -2)
(6, -2) arc[delta angle= 180]
[arr'=.5,draw=\filllcolor] node[very near start]{} node[very near end]{}
to[out=180, in=0, arr=.2, arr=.8] (0, -2) arc[delta angle=-180,]
[arr'=.5,draw=\filllcolor] node[very near start]{} node[very near end]{};
\node[stack, fill=OrangeLine!15, draw=OrangeLine] (fw1) at (0, 2.2) {ML Framework};
\node[sublabel] at (0, 2.2) {\\[0.4em]PyTorch / JAX / Kernels};
\end{scope}
}
}
}
\node[stack, fill=GreenLine!15, draw=GreenLine] (os1) at (0, 1.2) {System Software};
\node[sublabel] at (0, 1.2) {\\[0.4em]CUDA Runtime / PCIe DMA};
%server
\tikzset {
pics/server/.style = {
code = {
% \colorlet{red}{black}
\pgfkeys{/channel/.cd, #1}
\begin{scope}[anchor=center, transform shape,scale=\scalefac, every node/.append style={transform shape}]
\draw[draw=\drawcolor,line width=\Linewidth,fill=\filllcolor](-0.55,-0.5) rectangle (0.55,0.5);
\foreach \i in {-0.25,0,0.25} {
\draw[line width=\Linewidth]( -0.55,\i) -- (0.55, \i);
}
\foreach \i in {-0.375, -0.125, 0.125, 0.375} {
\draw[line width=\Linewidth](-0.45,\i)--(0,\i);
\fill[](0.35,\i) circle (1.5pt);
}
\node[stack, fill=BlueLine!15, draw=BlueLine] (hw1) at (0, 0.2) {Hardware};
\node[sublabel] at (0, 0.2) {\\[0.4em]HBM / NVLink (900 GB/s)};
\draw[draw=\drawcolor,line width=1.5*\Linewidth](0,-0.53) |- (-0.55,-0.7);
\draw[draw=\drawcolor,line width=1.5*\Linewidth](0,-0.53) |- (0.55,-0.7);
\end{scope}
}
}
}
\tikzset{
pics/llm/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[circle,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=0.5*\Linewidth](C\picname) at (0,0){};
\def\startangle{90}
\def\radius{1.15}
\def\radiusI{1.1}
\foreach \i [evaluate=\i as \j using \i+1] [count =\k] in {0,2,4,6,8} {
\pgfmathsetmacro{\angle}{\startangle - \i * (360/8)}
\draw[draw=black,-{Circle[black ,fill=\filllcirclecolor,length=5.5pt,line width=0.5*\Linewidth]},line width=1.5*\Linewidth](C\picname)--++(\startangle - \i*45:\radius) ;
\node[circle,draw=black,fill=\filllcirclecolor!80!red!50,inner sep=3pt,line width=0.5*\Linewidth](2C\k)at(\startangle - \j*45:\radiusI) {};
}
\draw[line width=1.5*\Linewidth](2C1)--++(-0.5,0)|-(2C2);
\draw[line width=1.5*\Linewidth](2C3)--++(0.5,0)|-(2C4);
\node[circle,,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=0.5*\Linewidth]at (0,0){};
\node[draw,rectangle,rounded corners=1pt,minimum width=7mm,minimum height=4mm,fill=orange!10](R1)at(0.1,0.1){};
\draw[BrownLine,shorten <=2pt,shorten >=2pt ]($(R1.north west)!0.35!(R1.south west)$)--($(R1.north east)!0.35!(R1.south east)$);
\draw[BrownLine,shorten <=2pt,shorten >=2pt ]($(R1.north west)!0.7!(R1.south west)$)--($(R1.north east)!0.7!(R1.south east)$);
\node[draw,rectangle,rounded corners=1pt,minimum width=6mm,minimum height=4mm,fill=orange!10](R2)at(-0.05,-0.15){};
\draw[BrownLine,shorten <=2pt,shorten >=2pt ]($(R2.north west)!0.35!(R2.south west)$)--($(R2.north east)!0.35!(R2.south east)$);
\draw[BrownLine,shorten <=2pt,shorten >=2pt ]($(R2.north west)!0.7!(R2.south west)$)--($(R2.north east)!0.7!(R2.south east)$);
\end{scope}
}
}
}
\def\inset{3.2pt} %
\def\myshape{%
(0,1.34) to[out=220,in=0] (-1.20,1.03) --
(-1.20,-0.23) to[out=280,in=160] (0,-1.53) to[out=20,in=260] (1.20,-0.23) --
(1.20,1.03) to[out=180,in=320] cycle
}
\tikzset{
pics/stitC/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
%\draw[draw=none,fill=\filllcolor!60](0,1.34)to[out=220,in=0](-1.20,1.03)to(-1.20,-0.23)
%to[out=280,in=160](0,-1.53)to[out=20,in=260](C1)to(1.20,1.03)to[out=180,in=320]cycle;
\fill[fill=\filllcolor!60] \myshape;
%
\begin{scope}
\clip \myshape;
\draw[draw=\filllcolor, line width=2*\Linewidth,fill=white] \myshape; % boja i debljina po želji
\end{scope}
%\fill[fill=\filllcolor!60](0,0)circle(0.4);
\draw[draw=\filllcirclecolor,line join=round,line cap=round,line width=1.85*\Linewidth](-0.65,-0.35)--++(320:0.5)--++(50:1.4);
\end{scope}
}
}
}
%code
\tikzset{
pics/interpreter/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[black,font=\Large\bfseries]at(-0.75,0.65){\textless\,/\,\textgreater};
\draw[line cap=round,line join=round,green!99!black!90,line width=\Linewidth](-1.32,0.17)--(-1.05,0.17);
\draw[line cap=round,line join=round,red,line width=\Linewidth](-0.8,0.17)--(-0.1,0.17);
\draw[line cap=round,line join=round,green!99!black!90,line width=\Linewidth](-1.15,-0.15)--(-0.45,-0.15);
\draw[line cap=round,line join=round,green!99!black!90,line width=\Linewidth](-1.15,-0.47)--(-0.75,-0.47);
\draw[line cap=round,line join=round,red,line width=\Linewidth](-0.45,-0.47)--(0.45,-0.47);
\draw[line cap=round,line join=round,cyan,line width=\Linewidth](0.75,-0.47)--(1.1,-0.47);
\draw[line cap=round,line join=round,green!99!black!90,line width=\Linewidth](-1.15,-0.79)--(-1,-0.79);
\draw[line cap=round,line join=round,red,line width=\Linewidth](-0.65,-0.79)--(-0.10,-0.79);
\draw[line cap=round,line join=round,cyan,line width=\Linewidth](0.2,-0.79)--(1.1,-0.79);
\draw[line cap=round,line join=round,green!99!black!90,line width=\Linewidth](-1.15,-1.11)--(-0.4,-1.11);
\draw[line cap=round,line join=round,blue!99!black!90,line width=\Linewidth](-0.15,-1.11)--(1.1,-1.11);
\end{scope}
}
}
}
\pgfkeys{
/channel/.cd,
Depth/.store in=\Depth,
Height/.store in=\Height,
Width/.store in=\Width,
filllcirclecolor/.store in=\filllcirclecolor,
filllcolor/.store in=\filllcolor,
drawcolor/.store in=\drawcolor,
drawcircle/.store in=\drawcircle,
scalefac/.store in=\scalefac,
Linewidth/.store in=\Linewidth,
picname/.store in=\picname,
tiecolor/.store in=\tiecolor,
bodycolor/.store in=\bodycolor,
stetcolor/.store in=\stetcolor,
tiecolor=red, % derfault tie color
bodycolor=blue!30, % derfault body color
stetcolor=green, % derfault stet color
filllcolor=BrownLine,
filllcirclecolor=violet!20,
drawcolor=black,
drawcircle=violet,
scalefac=1,
Linewidth=0.5pt,
Depth=0.2,
Height=0.5,
Width=0.25,
picname=C
}
\node[bottleneck, draw=RedLine] (bn1) at (0, -0.8) {\textbf{Bottleneck: Memory Wall}};
% Scaling Arrow
\draw[arrow] (2, 1.7) -- (3, 1.7) node[midway, above, color=GreenLine, font=\usefont{T1}{phv}{b}{n}] {Scaling};
% Distributed Fleet Stack
\node[label] at (5, 4.5) {Distributed Fleet Stack};
\node[sublabel] at (5, 4.1) {1,000--100,000+ GPUs};
\node[stack, fill=VioletLine!15, draw=VioletLine] (app2) at (5, 3.2) {Governance};
\node[sublabel] at (5, 3.2) {\\[0.4em]Responsible AI / Security};
\node[stack, fill=OrangeLine!15, draw=OrangeLine] (fw2) at (5, 2.2) {Serving / Ops};
\node[sublabel] at (5, 2.2) {\\[0.4em]Orchestration / CI/CD};
\node[stack, fill=GreenLine!15, draw=GreenLine] (os2) at (5, 1.2) {Distribution};
\node[sublabel] at (5, 1.2) {\\[0.4em]NCCL / RDMA / Comms};
\node[stack, fill=BlueLine!15, draw=BlueLine] (hw2) at (5, 0.2) {Infrastructure};
\node[sublabel] at (5, 0.2) {\\[0.4em]InfiniBand / Ethernet Fabric};
\node[bottleneck, draw=RedLine] (bn2) at (5, -0.8) {\textbf{Bottleneck: Network Wall}};
% Application
\node[Box1,draw=none](B1){};
\node[above=-1pt of B1.north,Text1]{Training Loop / inference};
\node[Box2,draw=OrangeLine](BB1)at($(B1.south west)+(-3mm,0)$){Application};
\draw[ALineA](B1.west)--(BB1.north);
\pic[shift={(0,0.08)}] at (B1){displayG={scalefac=0.95,picname=DD,
filllcolor=brown!60!, drawcolor=BrownLine,Linewidth=0.7pt}};
%\pic[shift={(-0.20,0.23)}] at (D-COM) {gear={11/1.25/1.7/11/2.0/0.6/scalefac=0.26,drawcolor=black,filllcolor=OrangeLine!90}};
\pic[shift={(0.47,0.32)}] at (DD-COM) {gear={10/1.3/1.8/17/1/0.6/scalefac=0.18,drawcolor=RedLine,filllcolor=RedLine}};
\pic[shift={(0.05,0.05)}] at (DD-COM){interpreter={scalefac=0.5,picname=1,filllcolor=cyan!30!, Linewidth=2.0pt,filllcirclecolor=orange}};
%ML Framework
\node[Box1,draw=none,below=of B1](B2){};
\node[above=-1pt of B2.north,Text1]{PyTorch / JAX / Kernels};
\node[Box2,draw=GreenLine](BB2)at($(B2.south west)+(-3mm,0)$){ML Framework};
\draw[ALineA](B2.west)--(BB2.north);
\pic[shift={(0,0)}] at (B2){token={scalefac=0.43,picname=2,drawcolor=green!55!black,
filllcirclecolor=green!55!black!90,filllcolor=green!55!black,Linewidth=1.25pt}};
%System Software
\node[Box1,draw=none,below=of B2](B3){};
\node[above=-1pt of B3.north,Text1]{CUDA / PCIe DMA};
\node[Box2,draw=VioletLine](BB3)at($(B3.south west)+(-3mm,0)$){System Software};
\draw[ALineA](B3.west)--(BB3.north);
\pic[shift={(0,0.1)}] at (B3){displayG={scalefac=0.98,picname=D,
filllcolor=BlueLine, drawcolor=BlueLine,Linewidth=0.7pt}};
\pic[shift={(-0.20,0.23)}] at (D-COM) {gear={11/1.25/1.7/11/2.0/0.6/
scalefac=0.26,drawcolor=black,filllcolor=OrangeLine!90}};
\pic[shift={(0.2,-0.3)}] at (D-COM) {gear={10/1.3/1.8/17/1/0.6/
scalefac=0.2,drawcolor=black,filllcolor=OrangeLine!60}};
%Hardware
\node[Box1,draw=none,below=of B3](B4){};
\node[above=-1pt of B4.north,Text1]{HBM / NVLink (900 GB/s)};
\node[Box2,draw=RedLine](BB4)at($(B4.south west)+(-3mm,0)$){Hardware};
\draw[ALineA](B4.west)--(BB4.north);
\pic[shift={(0,0)}] at (B4){cpu={scalefac=0.49,picname=1,filllcolor=GreenLine, Linewidth=0.7pt}};
%%%%%%%%%%%%%%%
% Governance
%%%%%%%%%%%%%%%
\node[Box1,right=3.2 of B1,draw=none](B1B){};
\node[above=-1pt of B1B.north,Text1]{Responsible AI / Security};
\node[Box2,draw=OrangeLine,anchor=south west](BB1B)at($(B1B.south east)+(3mm,0)$){Governance};
\draw[ALineA](B1B.east)--(BB1B.north);
\pic[shift={(0,0.05)}] at (B1B){stitC={scalefac=0.63,picname=1,drawcolor=orange,
filllcolor=purple,filllcirclecolor=GreenLine,Linewidth=3.1pt}};
%Serving / Ops
\node[Box1,draw=none,below=of B1B](B2B){};
\node[above=-1pt of B2B.north,Text1]{Orchestration / CI/CD};
\node[Box2,draw=GreenLine,anchor=south west](BB2B)at($(B2B.south east)+(3mm,0)$){Serving / Ops};
\draw[ALineA](B2B.east)--(BB2B.north);
\pic[shift={(-0.6,0)}] at (B2B){infinityL={scalefac=0.2,picname=1,Linewidth=1.0pt,
filllcolor=BlueLine,filllcirclecolor=red,drawcolor=GreenLine}};
%Distribution
\node[Box1,draw=none,below=of B2B](B3B){};
\node[above=-1pt of B3B.north,Text1]{NCCL / RDMA / Comms};
\node[Box2,draw=VioletLine,anchor=south west](BB3B)at($(B3B.south east)+(3mm,0)$){Distribution};
\draw[ALineA](B3B.east)--(BB3B.north);
\pic[shift={(0,0)}] at (B3B){llm={scalefac=0.78,drawcolor=BlueLine,filllcolor=BlueLine!50!, Linewidth=1pt,filllcirclecolor=red}};
%Infrastructure
\node[Box1,draw=none,below=of B3B](B4B){};
\node[above=-1pt of B4B.north,Text1]{Fabric / RDMA (InfiniBand)};
\node[Box2,draw=RedLine,anchor=south west](BB4B)at($(B4B.south east)+(3mm,0)$){Infrastructure};
\draw[ALineA](B4B.east)--(BB4B.north);
\pic[shift={(0,0.1)}] at (B4B){server={scalefac=1.2,picname=1,
drawcolor=black,filllcolor=cyan!15!,Linewidth=0.57pt,Linewidth=1pt}};
%fitting
\node[draw=none,fit=(B1)(B4)(BB4)](F1){};
\node[draw=none,fit=(B1B)(B4B)(BB4B)](F2){};
\draw[LineA](F1)--node[above]{Scaling}(F2);
%
\node[BoxD,below=10pt of F1]{Bottleneck: Memory Wall};
\node[BoxD,below=10pt of F2]{Bottleneck: Network Wall};
%
\node[BoxD1,above=20pt of F1]{Single-Node Stack \\
{\small\usefont{T1}{phv}{m}{n} 1--8 GPUs, Shared Memory}};
\node[BoxD1,above=20pt of F2]{Distributed Fleet Stack \\
{\small\usefont{T1}{phv}{m}{n} 1,000--100,000+ GPUs}};
\end{tikzpicture}
```
:::
The architecture does not change when we scale: every ML system still has Hardware, a System envelope, a Workload, and a Mission. What changes is the physics at each layer. At the Hardware layer, local HBM and NVLink (900 GB/s within a node) give way to InfiniBand fabrics (400 Gb/s per link across racks). At the System layer, a single CUDA runtime is replaced by distributed communication libraries (NCCL, RDMA) coordinating thousands of processes. At the Workload layer, a training loop that fits in one process becomes a distributed job requiring gradient synchronization after every batch. At the Mission layer, new concerns emerge that do not exist on a single node: responsible AI governance, fleet-wide orchestration, and multi-tenant scheduling. The four layers of the Engineering Crux at fleet scale are:
The stack architecture in @fig-vol2-system-scaling-regimes does not change when we scale: every ML system still has Hardware, a System envelope, a Workload, and a Mission. What changes is the physics at each layer. Read the figure from bottom to top. At the bottom row, **Hardware** (HBM and NVLink at 900 GB/s within one node) becomes **Infrastructure** (InfiniBand RDMA fabric spanning racks at 400 Gb/s per link), and the bottleneck shifts from the Memory Wall to the Bisection Bandwidth Wall. One row up, **System Software** (a single CUDA runtime managing PCIe DMA) becomes **Distribution** (NCCL and RDMA libraries coordinating thousands of processes across the fabric).
1. **Hardware (The Silicon)**: The physical foundation (The Engine). This layer defines the raw capabilities of individual nodes ($R_{peak}$, $\text{BW}$). Our primary "Hardware Twins" are the **NVIDIA H100** and **B200**.
2. **Systems (The Platforms)**: The integrated deployment unit (The Car). This layer defines the cluster "Envelope": bisection bandwidth, power usage effectiveness (PUE), and failure rates (MTBF). Examples include the **H100 Training Cluster**.
3. **Workloads (The Models)**: The algorithmic demand (The Route). This layer defines the mathematical workload sharded across the cluster ($O$, $D_{vol}$, $CI$). We use **Lighthouse Workloads** like **GPT-4** and **DLRM**.
4. **Missions (The Scenarios)**: The global application context (The Destination). This is the top of the stack, where a fleet is deployed to solve a mission-critical problem. A **Mission** (such as **Frontier Model Training**) introduces high-level requirements (e.g., "99.99% service availability") that dictate the configuration of every layer below.
The upper two layers undergo an equally profound transformation. **ML Framework** (PyTorch or JAX executing a training loop on one node) becomes **Serving / Ops** (orchestration and CI/CD pipelines that schedule distributed jobs and manage rolling deployments). At the top, **Application** (a single training script or inference service) becomes **Governance** (responsible AI policy, security auditing, and multi-tenant access control), because fleet-scale deployment introduces organizational concerns absent from a single machine. The four layers of the Engineering Crux at fleet scale are:
1. **Infrastructure** (Hardware — The Engine): The physical foundation. This layer defines the fleet's raw capabilities: per-node $R_{\text{peak}}$ and $\text{BW}$, interconnected by InfiniBand RDMA fabric. Our primary "Hardware Twins" are the **NVIDIA H100** and **B200**.
2. **Distribution** (Systems — The Car): The communication substrate. This layer defines the cluster envelope: NCCL and RDMA collectives that coordinate thousands of accelerators, along with bisection bandwidth, power usage effectiveness (PUE), and failure rates (MTBF).
3. **Serving / Ops** (Workloads — The Route): The orchestration layer. This layer manages the mathematical workload sharded across the cluster ($O$, $D_{\text{vol}}$, $CI$) through CI/CD pipelines and scheduling. We use **Lighthouse Workloads** like **GPT-4** and **DLRM**.
4. **Governance** (Missions — The Destination): The mission context. This is the top of the stack, where responsible AI policy, security, and multi-tenant access control shape fleet-wide behavior. A **Mission** (such as **Frontier Model Training**) introduces high-level requirements (e.g., "99.99% service availability") that dictate the configuration of every layer below.
This hierarchy ensures that every distributed engineering decision is grounded in its "Mission Context." For example, the **Frontier Training** mission inherits the **Cloud Archetype**, uses the **GPT-4** model, and operates on a cluster of **H100** hardware. By standardizing these protagonists, we ensure that the "Physics of Scale" remains traceable across every chapter.

2
book/quarto/contents/vol2/robust_ai/robust_ai.qmd
View File

@@ -843,7 +843,7 @@ Covariate shift occurs when the input distribution changes while the relationshi
***Concept Drift***\index{Concept Drift!definition} is the subtype of **Distribution Shift** (see @sec-robust-ai-distribution-shift-concept-drift-55e2) in which the statistical relationship $P(Y|X)$ changes over time, meaning the decision boundary itself becomes incorrect rather than merely the input distribution. Its sibling is **Data Drift** (see @sec-ml-operations-scale-monitoring-scale-73c5), in which $P(X)$ changes while $P(Y|X)$ remains stable.
1. **Significance (Quantitative):** It causes **Silent Model Degradation** because the historical mapping learned by the model is no longer representative of current reality. Within the **Iron Law**, it compresses the effective deployment window before retraining is required: for credit card fraud systems, $P(Y|X)$ shifts have been measured as 6-month correlation decay rates of 0.2–0.4, requiring retraining every 90–120 days to hold precision above 85%. Each forced retraining cycle incurs the full $O/(R_{peak} \cdot \eta)$ cost of the original training run, making the amortized per-prediction cost a direct function of drift velocity.
1. **Significance (Quantitative):** It causes **Silent Model Degradation** because the historical mapping learned by the model is no longer representative of current reality. Within the **Iron Law**, it compresses the effective deployment window before retraining is required: for credit card fraud systems, $P(Y|X)$ shifts have been measured as 6-month correlation decay rates of 0.2–0.4, requiring retraining every 90–120 days to hold precision above 85%. Each forced retraining cycle incurs the full $O/(R_{\text{peak}} \cdot \eta)$ cost of the original training run, making the amortized per-prediction cost a direct function of drift velocity.
2. **Distinction (Durable):** Unlike **Data Drift** (where fresh $P(X)$ data with unchanged labels fully restores performance), Concept Drift requires relabeling under the new $P(Y|X)$, because the same ground-truth labeling procedure that cures Data Drift is insufficient when the correct answer for the same input has changed. This makes Concept Drift structurally more expensive to remediate: it demands human annotation of recent examples, not merely resampling of the existing labeled distribution.
3. **Common Pitfall:** A frequent misconception is that Concept Drift is detectable by monitoring input feature statistics. Because $P(X)$ may be entirely unchanged, input-level monitoring (PSI, KL divergence on features) will show no signal. Concept Drift can only be confirmed by comparing **Predictions to Ground Truth Outcomes**, making it significantly harder to detect in real-time and requiring a ground-truth feedback loop before remediation can begin.