Refines content for Volume 1 conclusion

Enhances the conclusion of Volume 1, improving clarity and flow by:

- Refining wording and structure for better readability
- Clarifying the connection between theoretical invariants and practical applications
- Adding information for clarity and context

book/quarto/contents/vol1/conclusion/conclusion.qmd

@@ -124,8 +124,8 @@ t_comp_eq = ConclusionRoofline.t_comp_eq
 
 _What does mastering the full stack enable that expertise in any single layer cannot?_
 
-Data pipelines, architectures, training systems, compression techniques, accelerators, serving infrastructure, operational practices, responsible engineering—mastered individually, each is a valuable skill. Mastered together, they become something qualitatively different: the ability to *reason across boundaries*. An engineer who understands only compression can shrink a model, but cannot predict whether the accuracy loss matters for the deployment context. An engineer who understands only serving can optimize latency, but cannot trace a performance regression to a data pipeline change three stages upstream. The discipline of ML systems engineering is the discipline of seeing these connections: understanding that a model architecture choice determines memory requirements that constrain hardware selection, which influences quantization strategy, which affects accuracy, which feeds back to architecture design. Concretely, mastering the full stack enables you to see the invisible connections where one team's optimization becomes another team's constraint. Each layer of the stack interacts with every other, and the interactions are where the hardest problems live: not in any single component but in the spaces between them, where one team's optimization becomes another team's constraint.
- The principles governing these interactions—constraint propagation, the memory wall, the training-serving inversion, the iteration tax—are not tied to any specific framework, hardware generation, or model family. Technologies will change; the physics and the trade-offs will not. What endures is the ability to look at a system that does not yet exist and reason about how its pieces will interact, where its bottlenecks will emerge, and which design decisions will prove irreversible. That ability—to think in systems rather than components—is what separates an engineer who can build a part from one who can build the whole.
+Data pipelines, architectures, training systems, compression techniques, accelerators, serving infrastructure, operational practices, responsible engineering: mastered individually, each is a valuable skill. Mastered together, they become something qualitatively different: the ability to *reason across boundaries*. An engineer who understands only compression can shrink a model, but cannot predict whether the accuracy loss matters for the deployment context. An engineer who understands only serving can optimize latency, but cannot trace a performance regression to a data pipeline change three stages upstream. The discipline of ML systems engineering is the discipline of seeing these connections: understanding that a model architecture choice determines memory requirements that constrain hardware selection, which influences quantization strategy, which affects accuracy, which feeds back to architecture design. Each layer of the stack interacts with every other, and the interactions are where the hardest problems live; not in any single component but in the spaces between them, where one team's optimization becomes another team's constraint.
+ The principles governing these interactions, including constraint propagation, the memory wall, the training-serving inversion, and the iteration tax, are not tied to any specific framework, hardware generation, or model family. Technologies will change; the physics and the trade-offs will not. What endures is the ability to look at a system that does not yet exist and reason about how its pieces will interact, where its bottlenecks will emerge, and which design decisions will prove irreversible. That ability—to think in systems rather than components—is what separates an engineer who can build a part from one who can build the whole.
 
 ::: {.content-visible when-format="pdf"}
 \newpage
@@ -151,7 +151,7 @@ This is the central lesson of every chapter in this book. The introduction (@sec
 
 \index{Iron Law of ML Systems!synthesis}
 \index{Silicon Contract!constraint propagation}
-This book began with a mathematical formula: the **Iron Law of ML Systems** (@sec-introduction-iron-law-ml-systems-c32a). Its terms, **Data Movement**, **Compute**, and **Overhead**, once seemed abstract. Now they are primary engineering levers for quantitative analysis of systems that once seemed opaque. Building intelligence requires more than writing algorithms; it requires honoring the **Silicon Contract**, the *physical and economic agreement* between the model and the machine. @sec-hardware-acceleration equipped us to calculate arithmetic intensity and identify whether workloads are memory-bound or compute-bound, transforming vague performance intuitions into quantitative engineering decisions.
+This book began with a mathematical formula: the Iron Law of ML Systems (@sec-introduction-iron-law-ml-systems-c32a). Its terms, Data Movement, Compute, and Overhead, once seemed abstract. Now they are primary engineering levers for quantitative analysis of systems that once seemed opaque. Building intelligence requires more than writing algorithms; it requires honoring the Silicon Contract, the *physical and economic agreement* between the model and the machine. @sec-hardware-acceleration equipped us to calculate arithmetic intensity and identify whether workloads are memory-bound or compute-bound, transforming vague performance intuitions into quantitative engineering decisions.
 
 \index{Transformer!systems integration}
 This quantitative foundation leads to a broader point: contemporary artificial intelligence[^fn-ai-systems-view] achievements are not the product of any single algorithmic insight but of careful integration across interacting components. This systems perspective places machine learning within the same engineering tradition that built reliable computers, where powerful capabilities arise from coordinating many parts together. The Transformer architectures [@vaswani2017attention] enabling large language models exemplify this principle—their mathematical elegance alone does not explain their dominance. Their practical utility depends on integrating attention mechanisms with distributed training infrastructure, memory-efficient optimization techniques, and reliable operational frameworks that keep them reliable in production.
@@ -173,7 +173,13 @@ An ML system is greater than the sum of its parts.
 - [ ] **End-to-End**: Can you trace a user request from the UI, through the network, preprocessing, model, postprocessing, and back to the UI?
 :::
 
-This insight, that system boundaries define model capabilities, has guided our exploration throughout this book. The journey from Part I's foundations through Part IV's production reality traced a deliberate arc. We began with the mathematical primitives of neural computation (@sec-neural-computation) and the architectural families built from them (@sec-network-architectures), establishing *what* ML systems compute. We then turned to *how* they compute it: training systems (@sec-model-training) that orchestrate billions of optimization steps, frameworks (@sec-ml-frameworks) that compile high-level model definitions into hardware-specific execution plans, and the data engineering (@sec-data-engineering) and data selection (@sec-data-selection) pipelines that determine what the model learns from. Part III shifted from building to optimizing: model compression (@sec-model-compression) renegotiated the Silicon Contract for deployment, hardware acceleration (@sec-hardware-acceleration) maximized the throughput those compressed models could achieve, and benchmarking (@sec-benchmarking) provided the measurement discipline to verify that optimizations delivered real improvements. Finally, Part IV confronted production reality: serving systems (@sec-model-serving) that meet latency budgets under load, operational practices (@sec-ml-operations) that maintain model health over time, and responsible engineering (@sec-responsible-engineering) that ensures systems serve all users fairly.
+This insight, that system boundaries define model capabilities, has guided our exploration throughout this book. The journey from Part I's foundations through Part IV's production reality traced a deliberate arc, and each stage built on what came before.
+
+Part II established *what* ML systems compute and *how*. The mathematical primitives of neural computation (@sec-neural-computation) and the architectural families built from them (@sec-network-architectures) defined the computational substrate. Training systems (@sec-model-training) orchestrated billions of optimization steps on that substrate, while frameworks (@sec-ml-frameworks) compiled high-level model definitions into hardware-specific execution plans. Upstream of the model, the data engineering (@sec-data-engineering) and data selection (@sec-data-selection) pipelines determined what the model learns from.
+
+Part III shifted from building to optimizing. Model compression (@sec-model-compression) renegotiated the Silicon Contract for deployment. Hardware acceleration (@sec-hardware-acceleration) maximized the throughput those compressed models could achieve. Benchmarking (@sec-benchmarking) provided the measurement discipline to verify that optimizations delivered real improvements rather than illusory gains.
+
+Finally, Part IV confronted production reality. Serving systems (@sec-model-serving) had to meet latency budgets under load. Operational practices (@sec-ml-operations) maintained model health over time as data distributions shifted. Responsible engineering (@sec-responsible-engineering) ensured that systems serve all users fairly, not just the populations best represented in training data.
 
 Each chapter contributed a piece. But the real lesson is not in any individual piece—it is in *how the pieces constrain each other*. An architecture choice enabled a compression choice, which enabled an acceleration choice, which shaped a serving constraint, which defined an operational requirement. Depthwise separable convolutions in MobileNetV2 allowed INT8 quantization with minimal accuracy loss. That in turn enabled mobile NPU deployment, which shaped a P99 < 50 ms latency constraint and required drift monitoring across heterogeneous device populations. Every decision propagated forward, and the engineer who understands only one layer cannot predict how changes ripple through the rest.
 
@@ -203,7 +209,7 @@ Together, these five workloads span the full deployment spectrum from datacenter
 
 The table reveals a pattern: every row's decisions constrain the next row's options. Architecture choices (depthwise separable convolutions) enabled compression choices (INT8 quantization), which in turn enabled acceleration choices (mobile NPU deployment). This propagation of constraints governs every ML system—but the MobileNetV2 journey is one instance of a deeper structure. What quantitative invariants transcend specific models and technologies? The answer lies in twelve principles, each grounded in physics, information theory, or statistics, that recur across every Lighthouse model and every deployment context.
 
-## Twelve Quantitative Invariants {#sec--twelve-quantitative-invariants-0dd2}
+## Twelve Quantitative Invariants {#sec-twelve-quantitative-invariants-0dd2}
 
 \index{Twelve Invariants!quantitative framework}
 Throughout this book, each Part introduced quantitative principles that govern ML system behavior. These are not rules of thumb or best practices that evolve with fashion. They are invariants—constraints rooted in physics, information theory, and statistics. @tbl-twelve-principles collects all twelve in one place, organized by the four Parts that revealed them. Read the table as a reference framework: the first two columns identify each principle, the third locates where it was introduced, and the final two columns capture its mathematical essence and predictive power.
@@ -233,7 +239,7 @@ These twelve invariants are not independent axioms. They form an integrated fram
 
 The **Data as Code Invariant** (1) and the **Data Gravity Invariant** (2), established in Part I and developed quantitatively in @sec-data-engineering where data pipelines determine model quality, establish that data is simultaneously the logical program and the physical anchor of every ML system.
 
-The Lighthouse models illustrate both invariants directly. ResNet-50 and GPT-2 are **Data as Code** embodied: their capabilities derive from what they were trained on, not from their architectures alone. DLRM is **Data Gravity** embodied: its terabyte-scale embedding tables force the system architecture to be designed around where the data physically resides. These two invariants explain why the "compute-to-data" pattern recurs in every deployment context from cloud to edge.
+The Lighthouse models illustrate both invariants directly. ResNet-50 and GPT-2 are Data as Code embodied: their capabilities derive from what they were trained on, not from their architectures alone. DLRM is Data Gravity embodied: its terabyte-scale embedding tables force the system architecture to be designed around where the data physically resides. These two invariants explain why the "compute-to-data" pattern recurs in every deployment context from cloud to edge.
 
 ### Build: How Complexity Becomes Computation (Invariants 3–4) { .unnumbered}
 
@@ -245,7 +251,7 @@ The **Iron Law** (3) and the **Silicon Contract** (4) govern every decision in c
 \index{Energy-Movement Invariant!data locality}
 The four optimization invariants form a tightly coupled diagnostic chain. The **Pareto Frontier** (5) establishes that no free improvements exist: quantization trades precision for bandwidth, pruning trades capacity for speed, and distillation trades training compute for inference efficiency. The **Arithmetic Intensity Law** (6) diagnoses which resource is the bottleneck, revealing whether optimization should target compute or memory. The **Energy-Movement Invariant** (7) explains why data locality dominates efficiency: moving a bit from DRAM costs 100 to 1,000 times more energy than computing on it. **Amdahl's Law** (8) sets the ceiling on any parallelism gain, explaining why data loading and preprocessing become the ultimate bottlenecks in highly optimized systems.
 
-MobileNetV2 (our Lighthouse from @sec-network-architectures) navigates all four simultaneously: depthwise separable convolutions reshape the **Pareto Frontier**, quantization to INT8 exploits the **Arithmetic Intensity Law** by fitting more operations per byte of bandwidth, and the resulting energy savings respect the **Energy-Movement Invariant** while **Amdahl's Law** explains why the non-accelerable preprocessing stage limits end-to-end speedup. The KWS Lighthouse pushes these trade-offs to their extreme, where sub-megabyte models on microcontrollers leave zero margin for waste on any axis.
+MobileNetV2 (our Lighthouse from @sec-network-architectures) navigates all four simultaneously: depthwise separable convolutions reshape the Pareto Frontier, quantization to INT8 exploits the Arithmetic Intensity Law by fitting more operations per byte of bandwidth, and the resulting energy savings respect the Energy-Movement Invariant while Amdahl's Law explains why the non-accelerable preprocessing stage limits end-to-end speedup. The KWS Lighthouse pushes these trade-offs to their extreme, where sub-megabyte models on microcontrollers leave zero margin for waste on any axis.
 
 ### Deploy: How Reality Defeats Assumptions (Invariants 9–12) { .unnumbered}
 
@@ -253,15 +259,17 @@ MobileNetV2 (our Lighthouse from @sec-network-architectures) navigates all four
 \index{Statistical Drift Invariant!accuracy erosion}
 The deployment invariants address a category of failure that the first eight invariants cannot prevent: the system works correctly on the bench but degrades silently in production. The **Verification Gap** (9) establishes that ML testing is fundamentally statistical; you bound error rather than prove correctness. The **Statistical Drift Invariant** (10) quantifies how accuracy erodes as the world drifts from the training distribution, even when no code changes. The **Training-Serving Skew Law** (11) warns that even subtle differences between training and serving code paths (a different image resize library, a float32 versus float64 normalization) silently degrade accuracy. The **Latency Budget Invariant** (12) constrains the entire serving architecture: P99 latency is the hard constraint, and throughput is optimized within that envelope, never at its expense.
 
-These four invariants explain why @sec-ml-operations devoted extensive attention to monitoring, drift detection, and feature stores (the operational infrastructure that catches silent failures before they reach users). A DLRM recommendation system that achieves excellent offline accuracy will lose revenue if **Training-Serving Skew** corrupts feature values in production (Invariant 11) or if user behavior drifts seasonally without triggering retraining (Invariant 10). GPT-2/Llama serving must respect the **Latency Budget** (Invariant 12) through techniques like continuous batching and speculative decoding, as detailed in @sec-model-serving where we examined inference optimization at scale, because a chatbot that responds in ten seconds is a chatbot nobody uses.
+These four invariants explain why @sec-ml-operations devoted extensive attention to monitoring, drift detection, and feature stores (the operational infrastructure that catches silent failures before they reach users). A DLRM recommendation system that achieves excellent offline accuracy will lose revenue if Training-Serving Skew corrupts feature values in production (Invariant 11) or if user behavior drifts seasonally without triggering retraining (Invariant 10). GPT-2/Llama serving must respect the Latency Budget (Invariant 12) through techniques like continuous batching and speculative decoding, as detailed in @sec-model-serving where we examined inference optimization at scale, because a chatbot that responds in ten seconds is a chatbot nobody uses.
 
 ### The Integrated Framework { .unnumbered}
 
-These principles are not a checklist to apply sequentially. They form a web of mutual constraints. As the **Conservation of Complexity** dictates, a single engineering decision ripples through multiple invariants simultaneously.
+These principles are not a checklist to apply sequentially. They form a web of mutual constraints. As the Conservation of Complexity dictates, a single engineering decision ripples through multiple invariants simultaneously.
 
-To see this concretely, trace what happens when you quantize a model from FP16 to INT8. This single decision navigates the **Pareto Frontier** (Invariant 5), trading precision for bandwidth. But the consequences do not stop there: quantization changes the model's **Silicon Contract** (Invariant 4), shifting where it sits on the **Arithmetic Intensity** curve (Invariant 6) and altering its energy profile (Invariant 7). When you deploy that quantized model, the **Latency Budget** (Invariant 12) governs whether the speedup meets the SLO, while the **Training-Serving Skew Law** (Invariant 11) demands verification that reduced precision did not introduce a divergence between training and serving behavior. Concretely, a single quantization decision ripples through the Pareto Frontier, Silicon Contract, and Latency Budget simultaneously, where a win in one (bandwidth) must be validated against a risk in another (numerical skew). The **Verification Gap** (Invariant 9) reminds us that statistical tests can only *bound* the resulting accuracy loss, and the **Statistical Drift Invariant** (Invariant 10) warns that even a validated deployment will degrade over time. Meanwhile, the **Data Gravity Invariant** (2) determines where the model runs, the **Data as Code Invariant** (1) determines what it learned, the **Iron Law** (3) determines how fast it runs, and **Amdahl's Law** (8) determines how much faster it can ever run. Complexity is conserved; the engineer's task is to allocate it wisely.
+To see this concretely, trace what happens when you quantize a model from FP16 to INT8. This single decision navigates the Pareto Frontier (Invariant 5), trading precision for bandwidth. But the consequences do not stop there: quantization changes the model's Silicon Contract (Invariant 4), shifting where it sits on the Arithmetic Intensity curve (Invariant 6) and altering its energy profile (Invariant 7). When you deploy that quantized model, the Latency Budget (Invariant 12) governs whether the speedup meets the SLO, while the Training-Serving Skew Law (Invariant 11) demands verification that reduced precision did not introduce a divergence between training and serving behavior. A single quantization decision ripples through the Pareto Frontier, Silicon Contract, and Latency Budget simultaneously, where a win in one (bandwidth) must be validated against a risk in another (numerical skew).
 
-To see this cycle of mutual constraint in action, trace the flow in @fig-invariants-cycle. The four phases (Foundations, Build, Optimize, Deploy) surround a central hub representing the Conservation of Complexity, and the arrows map the perpetual flow of engineering decisions: each phase's choices constrain what becomes possible in the next, and the cycle eventually feeds back to the beginning. Decisions in the Build phase (governed by the **Iron Law**) constrain the Optimize phase (bounded by **Arithmetic Intensity**). Operational realities like Drift and Skew force feedback into the **Foundations**, requiring new data to stabilize the system. The engineer's role is to manage this flow, ensuring that complexity lands where it can be handled most efficiently.
+Meanwhile, the Data Gravity Invariant (2) determines where the model runs, the Data as Code Invariant (1) determines what it learned, the Iron Law (3) determines how fast it runs, and Amdahl's Law (8) determines how much faster it can ever run. The Verification Gap (Invariant 9) reminds us that statistical tests can only *bound* the resulting accuracy loss, and the Statistical Drift Invariant (Invariant 10) warns that even a validated deployment will degrade over time. Complexity is conserved; the engineer's task is to allocate it wisely.
+
+To see this cycle of mutual constraint in action, trace the flow in @fig-invariants-cycle. The four phases (Foundations, Build, Optimize, Deploy) surround a central hub representing the Conservation of Complexity, and the arrows map the perpetual flow of engineering decisions: each phase's choices constrain what becomes possible in the next, and the cycle eventually feeds back to the beginning. Decisions in the Build phase (governed by the Iron Law) constrain the Optimize phase (bounded by Arithmetic Intensity). Operational realities like Drift and Skew force feedback into the Foundations, requiring new data to stabilize the system. The engineer's role is to manage this flow, ensuring that complexity lands where it can be handled most efficiently.
 
 ::: {#fig-invariants-cycle fig-env="figure" fig-pos="htb" fig-cap="**The Cycle of ML Systems (The 12 Invariants)**: The complete systems engineering lifecycle. The meta-principle of *Conservation of Complexity* (center) unifies the process: complexity is neither created nor destroyed, only shifted between Data, Model, Hardware, and Operations. Each transition is governed by specific quantitative invariants that constrain valid engineering decisions." fig-alt="Circular diagram with four phases: Foundations (Data) in green, Build (Model) in blue, Optimize (Hardware) in orange, and Deploy (Operations) in violet. Arrows connect each phase in a cycle, with the 12 invariants labeled on each transition. Conservation of Complexity is shown in the center as a dashed circle."}
 ```{.tikz}
@@ -311,7 +319,7 @@ A colleague proposes quantizing your model from FP32 to INT8 to reduce serving c
 :::
 
 ::: {.callout-perspective title="The Cost of a Token"}
-We can apply the **Iron Law** (Invariant 3) and **Arithmetic Intensity** (Invariant 6) to a real-world problem: serving one token from a `{python} llama_params_str` parameter model (like Llama-2-70B) on an NVIDIA H100.
+We can apply the Iron Law (Invariant 3) and Arithmetic Intensity (Invariant 6) to a real-world problem: serving one token from a `{python} llama_params_str` parameter model (like Llama-2-70B) on an NVIDIA H100.
 
 **The Physics:**
 
@@ -326,7 +334,7 @@ We can apply the **Iron Law** (Invariant 3) and **Arithmetic Intensity** (Invari
 
 **The Systems Insight:**
 
-The memory time $T_{mem}$ is `{python} ratio_str` $\times$ larger than compute time $T_{comp}$. The system is heavily memory-bound (arithmetic intensity $\approx$ 1). To honor the **Silicon Contract**, we must either increase **Arithmetic Intensity** (via batching users to reuse $D_{vol}$) or reduce Data Volume (via quantization to INT4). A systems engineer who optimizes compute kernels ($T_{comp}$) without addressing memory ($T_{mem}$) wastes 100% of their effort.
+The memory time $T_{mem}$ is `{python} ratio_str` $\times$ larger than compute time $T_{comp}$. The system is heavily memory-bound (arithmetic intensity $\approx$ 1). To honor the Silicon Contract, we must either increase Arithmetic Intensity (via batching users to reuse $D_{vol}$) or reduce Data Volume (via quantization to INT4). A systems engineer who optimizes compute kernels ($T_{comp}$) without addressing memory ($T_{mem}$) wastes 100% of their effort.
 :::
 
 The Cost of a Token calculation illustrates a broader truth: the invariant framework is not an abstract taxonomy but a diagnostic instrument. Every chapter in this book applied these invariants to specific engineering decisions, often without naming them explicitly. The following section traces those applications across the three domains where they mattered most—building foundations, engineering for scale, and navigating production reality—to show how the framework we have just formalized has already been guiding our thinking throughout this book.
@@ -337,13 +345,13 @@ The twelve invariants gain their power not from theoretical elegance but from pr
 
 ### Building Technical Foundations { .unnumbered}
 
-The **Data as Code Invariant** (Invariant 1) shaped @sec-data-engineering in its entirety, explaining why "data is the new code" [@karpathy2017software] became a rallying cry for production ML teams—and why ResNet-50's and GPT-2's capabilities trace to their training data, not their architectures. Mathematical foundations (@sec-neural-computation) established the computational patterns that drive the **Silicon Contract**: the matrix multiplications at the heart of neural computation determine arithmetic intensity, which in turn determines whether a workload is memory-bound or compute-bound on any given hardware. Framework selection (@sec-ml-frameworks) illustrated the Silicon Contract's practical consequence: the chosen framework constrains which deployment paths remain open, because each framework makes different bets on graph optimization, memory management, and hardware backend support. An engineer who selects a framework without considering its Silicon Contract implications may discover, too late, that the chosen path forecloses the most efficient deployment option.
+The Data as Code Invariant (Invariant 1) shaped @sec-data-engineering in its entirety, explaining why "data is the new code" [@karpathy2017software] became a rallying cry for production ML teams, and why ResNet-50's and GPT-2's capabilities trace to their training data, not their architectures. Mathematical foundations (@sec-neural-computation) established the computational patterns that drive the Silicon Contract: the matrix multiplications at the heart of neural computation determine arithmetic intensity, which in turn determines whether a workload is memory-bound or compute-bound on any given hardware. Framework selection (@sec-ml-frameworks) illustrated the Silicon Contract's practical consequence: the chosen framework constrains which deployment paths remain open, because each framework makes different bets on graph optimization, memory management, and hardware backend support. An engineer who selects a framework without considering its Silicon Contract implications may discover, too late, that the chosen path forecloses the most efficient deployment option.
 
 These foundational choices—what data to curate, which computational primitives to rely on, which framework to adopt—propagate forward into every subsequent engineering decision. Nowhere is that propagation more visible than when a system must scale beyond a single machine, where the Iron Law's three terms expand from chip-level quantities to cluster-level constraints.
 
 ### Engineering for Scale { .unnumbered}
 
-Training systems (@sec-model-training) demonstrated the **Iron Law** in action: data parallelism reduces the Compute term by distributing work across GPUs, mixed precision halves the Data Movement term by using FP16 instead of FP32, and gradient checkpointing trades recomputation for memory capacity—each technique pulling a different lever of the same three-term equation. Model compression (@sec-model-compression) navigated the **Pareto Frontier** directly: MobileNetV2's INT8 quantization and DLRM's embedding pruning each traded one metric for another, while the **Arithmetic Intensity Law** diagnosed which trade-off would yield the greatest return for a given hardware target.
+Training systems (@sec-model-training) demonstrated the Iron Law in action: data parallelism reduces the Compute term by distributing work across GPUs, mixed precision halves the Data Movement term by using FP16 instead of FP32, and gradient checkpointing trades recomputation for memory capacity, each technique pulling a different lever of the same three-term equation. Model compression (@sec-model-compression) navigated the Pareto Frontier directly: MobileNetV2's INT8 quantization and DLRM's embedding pruning each traded one metric for another, while the Arithmetic Intensity Law diagnosed which trade-off would yield the greatest return for a given hardware target.
 
 Building and optimizing a model, however, is only half the engineering challenge. The other half begins the moment the model leaves the training cluster and enters production—where a new set of invariants governs behavior and where the optimizations that worked on the bench must survive the unpredictability of real-world traffic.
 
@@ -390,9 +398,9 @@ conclusion_tail_ratio_str = TailLatencyRatio.conclusion_tail_ratio_str
 
 ### Navigating Production Reality { .unnumbered}
 
-The transition from training to inference inverts optimization objectives: where training maximizes throughput over days, inference optimizes latency per request in milliseconds. The **Latency Budget Invariant** makes P99 the governing constraint, and tracking tail latencies reveals that mean latency tells little about user experience when one in a hundred users waits `{python} conclusion_tail_ratio_str` times longer than average. MLOps (@sec-ml-operations) orchestrates the full system lifecycle, transforming the **Statistical Drift Invariant** and the **Training-Serving Skew Law** from abstract equations into monitoring alerts and automated retraining triggers.
+The transition from training to inference inverts optimization objectives: where training maximizes throughput over days, inference optimizes latency per request in milliseconds. The Latency Budget Invariant makes P99 the governing constraint, and tracking tail latencies reveals that mean latency tells little about user experience when one in a hundred users waits `{python} conclusion_tail_ratio_str` times longer than average. MLOps (@sec-ml-operations) orchestrates the full system lifecycle, transforming the Statistical Drift Invariant and the Training-Serving Skew Law from abstract equations into monitoring alerts and automated retraining triggers.
 
-Beyond technical performance, @sec-responsible-engineering broadened the framework to include societal impact. The **Verification Gap** demands monitoring not just for performance but for fairness violations: tracking prediction distributions across demographic groups, detecting bias amplification over time, and alerting on unexplained accuracy disparities. The **Statistical Drift Invariant** applies equally to demographic subgroup performance, where accuracy may degrade for underrepresented populations even as aggregate metrics remain stable. These connections reveal that responsible AI is an integral dimension of systems engineering—not an afterthought but a first-class design constraint governed by the same invariants that govern performance.
+Beyond technical performance, @sec-responsible-engineering broadened the framework to include societal impact. The Verification Gap demands monitoring not just for performance but for fairness violations: tracking prediction distributions across demographic groups, detecting bias amplification over time, and alerting on unexplained accuracy disparities. The Statistical Drift Invariant applies equally to demographic subgroup performance, where accuracy may degrade for underrepresented populations even as aggregate metrics remain stable. These connections reveal that responsible AI is an integral dimension of systems engineering—not an afterthought but a first-class design constraint governed by the same invariants that govern performance.
 
 These three domains demonstrate that the twelve invariants are not theoretical constructs but working tools. The question now is where these tools will be tested next, as ML systems expand into new deployment contexts, confront new failure modes, and pursue increasingly ambitious goals.
 
@@ -411,7 +419,7 @@ In contrast, mobile and edge systems face stringent power, memory, and latency c
 
 [^fn-ai-democratization]: **AI Democratization**: Making AI accessible beyond a small number of well-resourced organizations through efficient systems engineering. Mobile-optimized models and cloud APIs can widen access, but doing so sustainably requires systematic optimization across hardware, algorithms, and infrastructure to maintain quality at scale.
 
-Generative AI systems—the frontier that GPT-2/Llama exposed—apply the principles at unprecedented scale. Autoregressive generation is inherently memory-bound (each token requires loading the full model weights), making the **Arithmetic Intensity Law** the governing constraint. Novel techniques like dynamic model partitioning and speculative decoding[^fn-speculative-decoding] reshape the **Silicon Contract** by trading compute for latency, demonstrating how our principles adapt even as technologies push infrastructure boundaries.
+Generative AI systems—the frontier that GPT-2/Llama exposed—apply the principles at unprecedented scale. Autoregressive generation is inherently memory-bound (each token requires loading the full model weights), making the Arithmetic Intensity Law the governing constraint. Novel techniques like dynamic model partitioning and speculative decoding[^fn-speculative-decoding] reshape the Silicon Contract by trading compute for latency, demonstrating how our principles adapt even as technologies push infrastructure boundaries.
 
 [^fn-speculative-decoding]: **Speculative Decoding**: Inference optimization where a smaller draft model generates candidate tokens that a larger target model verifies in parallel. Since autoregressive generation is memory-bound (each token requires loading the full model), speculative decoding trades compute for latency: the draft model proposes 4–8 tokens; the target verifies them in a single forward pass. Achieves 2–3 $\times$ speedup when draft acceptance rates exceed 70%, making it essential for interactive LLM applications.
 
@@ -423,7 +431,7 @@ What unites these four paradigms is not their hardware but their physics: the sa
 
 ML systems face unique failure modes that traditional software never encounters. A traditional web server either responds or crashes; a machine learning system can respond *confidently and incorrectly*, and no one may notice for weeks. Distribution shifts degrade accuracy without any code changes. Adversarial inputs exploit vulnerabilities invisible to standard testing. Edge cases reveal training data limitations that no amount of debugging can fix. These are not hypothetical risks—they are statistical certainties predicted by the deployment invariants.
 
-The **Verification Gap** (Invariant 9) guarantees that ML testing can only bound error, never prove correctness. The **Statistical Drift Invariant** (Invariant 10) guarantees that systems will degrade over time as the world drifts from the training distribution. Together, these two invariants establish that some failures *will* reach production and that system quality *will* erode. Continuous monitoring is therefore a design requirement, not an operational afterthought. The question is not whether the system will fail, but whether the failure will be detected before users do.
+The Verification Gap (Invariant 9) guarantees that ML testing can only bound error, never prove correctness. The Statistical Drift Invariant (Invariant 10) guarantees that systems will degrade over time as the world drifts from the training distribution. Together, these two invariants establish that some failures *will* reach production and that system quality *will* erode. Continuous monitoring is therefore a design requirement, not an operational afterthought. The question is not whether the system will fail, but whether the failure will be detected before users do.
 
 Robustness demands designing for failure from the ground up. Redundant hardware provides fault tolerance when individual components fail. Ensemble methods reduce single-point failures by distributing prediction responsibility across multiple models. Uncertainty quantification enables graceful degradation—a system that knows when it does not know can defer to a human or a fallback policy rather than producing a confident wrong answer. As AI systems assume increasingly autonomous roles in healthcare, transportation, and finance, the gap between "works in the lab" and "works in the world" becomes the critical engineering challenge. These robustness techniques become even more essential at distributed scale, where failure planning must account for coordination across hundreds or thousands of machines and where mean time between failures drops from years to hours.
 
@@ -431,7 +439,7 @@ Robustness demands designing for failure from the ground up. Redundant hardware
 
 Robust systems are the prerequisite for deploying AI where it can benefit society most. A medical AI that fails unpredictably cannot be trusted with patient care. An educational system that degrades under load cannot serve the students who need it most. A climate model that produces confident but uncalibrated predictions may misdirect policy decisions affecting millions of lives. In each domain, the twelve invariants converge, and robustness becomes not just an engineering virtue but an ethical imperative.
 
-The invariants manifest differently across these domains. Scientific discovery—protein folding, drug interaction modeling, materials science—requires massive throughput governed by the **Iron Law** and **Silicon Contract**, where distributed training across thousands of GPUs must be coordinated to explore vast parameter spaces. Healthcare AI demands explainable decisions and continuous monitoring, where the **Statistical Drift Invariant** takes on life-or-death significance: a diagnostic model trained on one hospital's population may silently degrade when deployed to another with different demographics, disease prevalence, or imaging equipment. Personalized education needs privacy-preserving inference at global scale, stressing the **Latency Budget** (responsiveness matters for learning engagement) and the **Data as Code Invariant** (the model must learn from student interactions without compromising student privacy).
+The invariants manifest differently across these domains. Scientific discovery—protein folding, drug interaction modeling, materials science—requires massive throughput governed by the Iron Law and Silicon Contract, where distributed training across thousands of GPUs must be coordinated to explore vast parameter spaces. Healthcare AI demands explainable decisions and continuous monitoring, where the Statistical Drift Invariant takes on life-or-death significance: a diagnostic model trained on one hospital's population may silently degrade when deployed to another with different demographics, disease prevalence, or imaging equipment. Personalized education needs privacy-preserving inference at global scale, stressing the Latency Budget (responsiveness matters for learning engagement) and the Data as Code Invariant (the model must learn from student interactions without compromising student privacy).
 
 These applications demonstrate that technical excellence alone is insufficient. Success requires interdisciplinary collaboration among technologists, domain experts, policymakers, and affected communities. The principles developed throughout this book—the D·A·M taxonomy, the twelve invariants, and the quantitative reasoning framework—provide the systems engineering foundation, but the *application* of that foundation requires domain knowledge that no single discipline can supply.
 
@@ -443,19 +451,19 @@ The most ambitious application of these invariants lies ahead: engineering the p
 
 [^fn-agi-def]: **Artificial General Intelligence (AGI)**: A system capable of universal cognitive generalization at or above human levels. Unlike narrow AI, which is optimized for specific domains, AGI requires transfer learning to novel situations and reasoning about unfamiliar problems without specific retraining. The term gained currency in the early 2000s to distinguish human-level AI from the task-specific systems that had dominated the field since the 1980s.
 
-Universal generalization imposes extraordinary systems demands. Every invariant becomes simultaneously active: the **Iron Law** governs computation at a scale where models may contain trillions of parameters. The **Silicon Contract** must be honored across heterogeneous hardware spanning GPUs, TPUs, and custom accelerators. The **Pareto Frontier** expands from two or three metrics (accuracy, latency, memory) to dozens (safety, fairness, reasoning quality, factuality, multilinguality). The **Statistical Drift Invariant** applies not to a single domain but to the entire distribution of human knowledge and interaction. No monolithic model can navigate this complexity alone.
+Universal generalization imposes extraordinary systems demands. Every invariant becomes simultaneously active: the Iron Law governs computation at a scale where models may contain trillions of parameters. The Silicon Contract must be honored across heterogeneous hardware spanning GPUs, TPUs, and custom accelerators. The Pareto Frontier expands from two or three metrics (accuracy, latency, memory) to dozens (safety, fairness, reasoning quality, factuality, multilinguality). The Statistical Drift Invariant applies not to a single domain but to the entire distribution of human knowledge and interaction. No monolithic model can navigate this complexity alone.
 
-This realization has driven the emergence of **compound AI systems**[^fn-compound-ai]\index{Compound AI Systems!reliability through composition}—architectures that chain multiple models and deterministic tools to achieve reliability exceeding their individual components. Rather than building a single model that does everything, compound systems decompose tasks into specialized steps: a retrieval component finds relevant information, a reasoning component processes it, and a verification component checks the output. Each step can be independently updated, monitored, and debugged. This decomposition trades latency and architectural complexity for control and correctness—a trade-off that the **Pareto Frontier** predicts and the **Conservation of Complexity** demands.
+This realization has driven the emergence of **compound AI systems**[^fn-compound-ai]\index{Compound AI Systems!reliability through composition}—architectures that chain multiple models and deterministic tools to achieve reliability exceeding their individual components. Rather than building a single model that does everything, compound systems decompose tasks into specialized steps: a retrieval component finds relevant information, a reasoning component processes it, and a verification component checks the output. Each step can be independently updated, monitored, and debugged. This decomposition trades latency and architectural complexity for control and correctness—a trade-off that the Pareto Frontier predicts and the Conservation of Complexity demands.
 
 [^fn-compound-ai]: **Compound AI Systems**\index{Compound AI Systems!etymology}: Coined by researchers at Berkeley AI Research (BAIR) in 2024 to describe systems that compose multiple AI components---models, retrievers, tools, and verifiers---into pipelines, rather than relying on a single monolithic model. Examples include retrieval-augmented generation (RAG) and tool-augmented agents. From a systems perspective, compound AI systems trade single-model simplicity for orchestration complexity, but gain independently updatable components, debuggable intermediate outputs, and the ability to enforce deterministic constraints alongside probabilistic generation.
 
-The compound AI systems framework aligns naturally with the systems engineering principles we have studied. Modular components can be independently compressed and accelerated using the techniques from @sec-model-compression and @sec-hardware-acceleration. Each component has its own **Silicon Contract** and **Arithmetic Intensity** profile, allowing hardware-specific optimization. The interfaces between components create natural monitoring points for detecting drift, skew, and degradation. The engineering challenges ahead—reliable orchestration of multiple models, efficient routing of requests across specialized components, maintaining consistency across distributed state—require mastery across the full stack we have explored, from data engineering and distributed training to model optimization and operational infrastructure. These quantitative invariants, not algorithmic breakthroughs alone, define the path toward artificial general intelligence—an endeavor that unfolds within what Hennessy and Patterson have called *a new golden age for computer architecture*.
+The compound AI systems framework aligns naturally with the systems engineering principles we have studied. Modular components can be independently compressed and accelerated using the techniques from @sec-model-compression and @sec-hardware-acceleration. Each component has its own Silicon Contract and Arithmetic Intensity profile, allowing hardware-specific optimization. The interfaces between components create natural monitoring points for detecting drift, skew, and degradation. The engineering challenges ahead—reliable orchestration of multiple models, efficient routing of requests across specialized components, maintaining consistency across distributed state—require mastery across the full stack we have explored, from data engineering and distributed training to model optimization and operational infrastructure. These quantitative invariants, not algorithmic breakthroughs alone, define the path toward artificial general intelligence—an endeavor that unfolds within what Hennessy and Patterson have called *a new golden age for computer architecture*.
 
 \index{Hennessy and Patterson}
 
 ::: {.callout-perspective title="A New Golden Age"}
 
-**Engineering the Future**: Hennessy and Patterson [@hennessy_patterson_2019] declared a **"New Golden Age for Computer Architecture,"** driven by the realization that general-purpose processors can no longer sustain the exponential growth required by AI. Reaching AGI will not be a matter of writing a better loss function; it will be an epic systems engineering challenge. It will require a thousand-fold improvement in energy efficiency, exascale interconnects that operate with the reliability of a single chip, and software stacks that can manage trillions of parameters as fluidly as we manage kilobytes today. The twelve invariants you have learned in this book, from the **Iron Law** and **Silicon Contract** to the **Statistical Drift Invariant** and **Latency Budget**, are the blueprints for this new era.
+**Engineering the Future**: Hennessy and Patterson [@hennessy_patterson_2019] declared a **"New Golden Age for Computer Architecture,"** driven by the realization that general-purpose processors can no longer sustain the exponential growth required by AI. Reaching AGI will not be a matter of writing a better loss function; it will be an epic systems engineering challenge. It will require a thousand-fold improvement in energy efficiency, exascale interconnects that operate with the reliability of a single chip, and software stacks that can manage trillions of parameters as fluidly as we manage kilobytes today. The twelve invariants you have learned in this book, from the Iron Law and Silicon Contract to the Statistical Drift Invariant and Latency Budget, are the blueprints for this new era.
 :::
 
 What does this golden age demand in concrete terms? Achieving exascale sustained throughput ($\geq 10^{18}$ FLOP/s) and beyond requires not just faster chips but entirely new approaches to power delivery, cooling, interconnects, and software coordination. These challenges await engineers who can apply systems thinking to unprecedented problems—and you are now among those engineers.
@@ -464,7 +472,7 @@ Whether or not AGI emerges in its fullest form, the systems principles establish
 
 ## Journey Forward {#sec--journey-forward-6453}
 
-Every frontier explored in the previous section—diverse deployment contexts, robust systems, societal applications, compound AI, and the path to AGI—rests on a common foundation: the engineering skills this book has developed. We have learned to manage the stochastic nature of data through the **Data as Code** and **Statistical Drift** invariants, while enforcing deterministic reliability through the **Iron Law**, **Silicon Contract**, and **Latency Budget**. We have bridged the gap between Software 1.0's explicit logic and Software 2.0's learned behaviors, mastering the engineering rigor required to make probabilistic systems dependable.
+Every frontier explored in the previous section—diverse deployment contexts, robust systems, societal applications, compound AI, and the path to AGI—rests on a common foundation: the engineering skills this book has developed. We have learned to manage the stochastic nature of data through the Data as Code and Statistical Drift invariants, while enforcing deterministic reliability through the Iron Law, Silicon Contract, and Latency Budget. We have bridged the gap between Software 1.0's explicit logic and Software 2.0's learned behaviors, mastering the engineering rigor required to make probabilistic systems dependable.
 
 Intelligence is a systems property. It emerges from integrating components rather than from any single breakthrough. GPT-4 [@openai2023gpt4] illustrates this directly: its success required reliable data pipelines processing petabytes of text, distributed training infrastructure[^fn-distributed-ml] coordinating thousands of GPUs, efficient architectures built on attention mechanisms and mixture-of-experts, secure deployment preventing prompt injection attacks[^fn-prompt-injection], and responsible governance implementing safety filters and usage policies. No single component made GPT-4 possible; the integration made it possible.
 
@@ -474,7 +482,7 @@ Intelligence is a systems property. It emerges from integrating components rathe
 
 ### The Engineering Responsibility { .unnumbered}
 
-Before looking to the horizon of scale, we must ground ourselves in responsibility. The systems integration perspective explains why ethical considerations cannot be separated from technical ones. The same **Iron Law** that enables efficient systems determines who can access them: a model requiring four H100 GPUs for inference excludes organizations that cannot afford that infrastructure. The same **Data as Code Invariant** that gives models their capabilities also encodes the biases present in training data. The same **Energy-Movement Invariant** that governs chip-level efficiency scales to datacenter-level carbon footprints that affect the planet. Technical decisions are ethical decisions, viewed through a wider lens.
+Before looking to the horizon of scale, we must ground ourselves in responsibility. The systems integration perspective explains why ethical considerations cannot be separated from technical ones. The same Iron Law that enables efficient systems determines who can access them: a model requiring four H100 GPUs for inference excludes organizations that cannot afford that infrastructure. The same Data as Code Invariant that gives models their capabilities also encodes the biases present in training data. The same Energy-Movement Invariant that governs chip-level efficiency scales to datacenter-level carbon footprints that affect the planet. Technical decisions are ethical decisions, viewed through a wider lens.
 
 The question confronting our generation is not whether increasingly capable AI will arrive, but whether it will be built well. Will it be efficient enough to democratize access beyond wealthy institutions? Secure enough to resist exploitation? Sustainable enough to preserve our planet? Responsible enough to serve all humanity equitably? The intelligent systems that will define the coming decades—from planetary-scale climate monitors to personalized medical assistants—require your engineering expertise, guided by the responsibility that @sec-responsible-engineering established as a first-class design constraint.
 
@@ -492,7 +500,7 @@ This is the frontier of the **Warehouse-Scale Computer**[^fn-warehouse-scale]\in
 
 [^fn-warehouse-scale]: **Warehouse-Scale Computer (WSC)**\index{Warehouse-Scale Computer!etymology}: Coined by Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle at Google [@barroso2019datacenter]. The term reframes a datacenter from a *building that houses computers* to a *single computer* distributed across thousands of racks. The "system bus" becomes the network fabric, "memory" spans petabytes of distributed storage, and component failures occur continuously rather than exceptionally. For ML training, the WSC perspective explains why frontier model training is fundamentally an infrastructure challenge requiring the entire fleet to function as a single programmable system.
 
-The transition from Node to Fleet is a fundamental shift in which physical constraints dominate, yet the foundation remains the same. The **Iron Law** still governs performance, but the variables now span racks and zones. The AI Triad still applies, but the "Machine" is now a global infrastructure. You have mastered the unit; you are now ready to build the collective.
+The transition from Node to Fleet is a fundamental shift in which physical constraints dominate, yet the foundation remains the same. The Iron Law still governs performance, but the variables now span racks and zones. The AI Triad still applies, but the "Machine" is now a global infrastructure. You have mastered the unit; you are now ready to build the collective.
 
 Mastery, however, carries a recurring temptation: the belief that understanding a system means understanding it completely. Before we close, we confront the misconceptions that even experienced engineers carry, the fallacies and pitfalls that arise when confidence outpaces humility.
 
@@ -510,7 +518,23 @@ When an optimization reduces latency by 50%, ask where the cost went. Did quanti
 
 Component expertise is necessary but insufficient. An engineer who understands data pipelines, training, serving, and operations as isolated domains will still struggle with systems where a data schema change cascades through training, breaks quantization assumptions, and triggers silent accuracy degradation in production. The integration complexity exceeds the sum of component complexities because interfaces multiply failure modes. Systems thinking means understanding how components interact, not just how they work individually.
 
-These fallacies share a common root: the temptation to reduce a system to its parts. The key takeaways that follow capture the integrated perspective that resists that reduction.
+**Fallacy:** *More data always improves model quality.*
+
+The intuition that more data yields better models is seductive because it holds true early in model development. @sec-data-selection demonstrated the diminishing returns that set in once a dataset achieves sufficient coverage: beyond that threshold, doubling dataset size yields marginal accuracy gains while doubling storage, preprocessing, and labeling costs. The Data Gravity Invariant (2) ensures that data scale decisions cascade through every downstream system, because larger datasets demand proportionally more I/O bandwidth, longer preprocessing pipelines, and costlier feature stores. The engineer who scales data without measuring the incremental return per additional sample optimizes the wrong variable.
+
+**Fallacy:** *A single accuracy metric captures model quality.*
+
+A model evaluated solely on accuracy inhabits a one-dimensional world. The Pareto Frontier Invariant (5) establishes that accuracy is one dimension of a multi-dimensional trade-off space encompassing latency, throughput, memory, energy, fairness, and cost. A model achieving 95% accuracy with 500 ms latency may be strictly worse for production serving than one achieving 93% accuracy at 50 ms latency, because the Latency Budget Invariant (12) enforces P99 as the hard constraint. @sec-responsible-engineering showed that even the accuracy dimension itself is misleading when aggregate metrics conceal 40 $\times$ error rate disparities across demographic groups. Evaluation must span the full Pareto surface, not a single axis.
+
+**Pitfall:** *Deploying models without automated rollback mechanisms.*
+
+The Statistical Drift Invariant (10) guarantees that accuracy degrades over time as the serving distribution drifts from the training distribution, even when no code changes. Without automated rollback, a silently degrading model continues serving bad predictions until a human notices, a delay that can extend for weeks when the degradation is gradual. @sec-ml-operations analyzed how drift detection pipelines must be coupled with automated response: monitoring without action is surveillance, not engineering. Rollback mechanisms close the loop between detection and correction, converting the Statistical Drift Invariant from a threat into a manageable constraint.
+
+**Pitfall:** *Optimizing a single pipeline stage without profiling the full system.*
+
+Amdahl's Law (Invariant 8) applies directly to end-to-end ML pipelines. Optimizing inference latency by 10 $\times$ yields only 1.1 $\times$ system speedup if data loading accounts for 90% of end-to-end latency. The Iron Law of ML Systems (Invariant 3) decomposes execution time into data movement, computation, and latency terms precisely so that engineers can identify the dominant term before investing optimization effort. @sec-benchmarking formalized this diagnostic process through profiling methodologies that measure where time actually goes. Engineers who optimize without profiling are guessing, and Amdahl's Law is unforgiving of guesses that target the wrong term.
+
+These fallacies and pitfalls share a common root: the temptation to reduce a system to its parts, whether by optimizing a single metric, a single stage, or a single moment in time. The key takeaways that follow capture the integrated perspective that resists that reduction.
 
 ## Summary {#sec--summary-2ef7}
 
@@ -518,13 +542,13 @@ This chapter distilled the integrated perspective that distinguishes ML systems
 
 ::: {.callout-takeaways title="Reasoning Across Boundaries"}
 
-- **Twelve quantitative invariants define ML systems engineering**: From the **Data as Code Invariant** through the **Latency Budget Invariant**, these principles quantify the constraints that govern every design decision, organized across Foundations (data physics), Build (computation physics), Optimize (efficiency physics), and Deploy (reliability physics).
+- **Twelve quantitative invariants define ML systems engineering**: From the Data as Code Invariant through the Latency Budget Invariant, these principles quantify the constraints that govern every design decision, organized across Foundations (data physics), Build (computation physics), Optimize (efficiency physics), and Deploy (reliability physics).
 - **The Conservation of Complexity unifies all twelve**: You cannot destroy complexity in an ML system; you can only move it between Data, Algorithm, and Machine. Every invariant quantifies a specific consequence of where complexity currently resides.
 - **The system is the model**: The true model is data pipeline + training infrastructure + serving system + monitoring loop. Optimize the system to improve the model.
-- **Production ML demands continuous operation and designed-in robustness**: The **Verification Gap**, **Statistical Drift Invariant**, and **Training-Serving Skew Law** guarantee that models degrade without code changes and that some failures reach production. Redundancy, uncertainty quantification, and continuous monitoring are first-class design requirements, not optional add-ons.
-- **Every deployment context stresses different invariants, but no context escapes them**: Cloud, edge, generative AI, and TinyML each foreground different terms of the **Iron Law**, but the **Pareto Frontier** and **Energy-Movement Invariant** govern all of them — success requires applying multiple principles simultaneously rather than optimizing any single metric.
-- **Technical excellence must combine with ethical commitment**: The **Verification Gap** and drift invariants apply equally to fairness metrics. Build systems that are efficient, accessible, sustainable, and beneficial.
-- **Mastering the node prepares you for the fleet**: The principles developed for single systems—bottleneck diagnosis, hardware co-design, drift monitoring—scale to the Warehouse-Scale Computer, where the datacenter becomes the computer and the **Iron Law** spans racks and zones.
+- **Production ML demands continuous operation and designed-in robustness**: The Verification Gap, Statistical Drift Invariant, and Training-Serving Skew Law guarantee that models degrade without code changes and that some failures reach production. Redundancy, uncertainty quantification, and continuous monitoring are first-class design requirements, not optional add-ons.
+- **Every deployment context stresses different invariants, but no context escapes them**: Cloud, edge, generative AI, and TinyML each foreground different terms of the Iron Law, but the Pareto Frontier and Energy-Movement Invariant govern all of them; success requires applying multiple principles simultaneously rather than optimizing any single metric.
+- **Technical excellence must combine with ethical commitment**: The Verification Gap and drift invariants apply equally to fairness metrics. Build systems that are efficient, accessible, sustainable, and beneficial.
+- **Mastering the node prepares you for the fleet**: The principles developed for single systems, from bottleneck diagnosis to hardware co-design and drift monitoring, scale to the Warehouse-Scale Computer, where the datacenter becomes the computer and the Iron Law spans racks and zones.
 
 :::
 
@@ -537,7 +561,7 @@ The future of intelligence is not a destiny we will simply witness. It is a syst
 *Prof. Vijay Janapa Reddi, Harvard University*
 
 ::: {.callout-chapter-connection title="From Node to Fleet"}
-This volume established the principles for mastering the ML node—the single system where data, algorithms, and hardware converge. Every invariant, every Lighthouse analysis, and every optimization strategy was developed within the scope of one machine. Volume II extends these principles to the *fleet*: the Warehouse-Scale Computer where thousands of nodes must function as a single programmable system. The **Iron Law** will expand from chip-level analysis to rack-level and zone-level decomposition. The **Silicon Contract** will generalize from CPU-GPU co-design to heterogeneous fleet orchestration. The **Statistical Drift Invariant** will scale from monitoring one model to governing thousands of models serving billions of users. The physics does not change; the scale does. You have mastered the unit. Now learn to orchestrate the collective.
+This volume established the principles for mastering the ML node—the single system where data, algorithms, and hardware converge. Every invariant, every Lighthouse analysis, and every optimization strategy was developed within the scope of one machine. Volume II extends these principles to the *fleet*: the Warehouse-Scale Computer where thousands of nodes must function as a single programmable system. The Iron Law will expand from chip-level analysis to rack-level and zone-level decomposition. The Silicon Contract will generalize from CPU-GPU co-design to heterogeneous fleet orchestration. The Statistical Drift Invariant will scale from monitoring one model to governing thousands of models serving billions of users. The physics does not change; the scale does. You have mastered the unit. Now learn to orchestrate the collective.
 :::
 
 ::: { .quiz-end }

book/quarto/contents/vol1/data_engineering/data_engineering.qmd

@@ -3349,7 +3349,7 @@ The patterns in this section—data debt accumulation, diagnostic debugging, and
 
 **Fallacy:** *More data always improves model performance.*
 
-Beyond a threshold, additional data yields diminishing returns while costs scale linearly. A dataset of 10 million examples may provide only marginal accuracy gains over 1 million examples, yet incur 10 $\times$ the storage, labeling, and processing costs. The Information Entropy concept from @sec-data-engineering-physics-data-cdcb explains why: if new examples are redundant (low entropy), they add mass without information. Smart data selection—active learning, deduplication, curriculum design—often outperforms naive data accumulation.
+Beyond a threshold, additional data yields diminishing returns while costs scale linearly. Empirical studies across image classification, translation, and language modeling confirm that test loss follows a power law in dataset size, with exponents so small that a 10 $\times$ increase in data often reduces error by less than one percentage point, while labeling and storage costs scale proportionally [@hestness2017deep]. The Information Entropy concept from @sec-data-engineering-physics-data-cdcb explains why: if new examples are redundant (low entropy), they add mass without information. Smart data selection, including active learning, deduplication, and curriculum design, often outperforms naive data accumulation.
 
 **Pitfall:** *Treating data preprocessing as a one-time task.*
 
@@ -3367,6 +3367,10 @@ Feature computation differences between training and serving environments are th
 
 Synthetic data excels at augmenting real data—generating rare edge cases, increasing diversity, reducing costs—but cannot replace it entirely. Synthetic generation inherits the biases and limitations of its generative models. A KWS system trained purely on synthesized speech will fail on accent patterns, background noises, and pronunciation variations that the generator never modeled. The optimal strategy combines real data for coverage with synthetic data for scale.
 
+**Pitfall:** *Neglecting data versioning until model debugging requires it.*
+
+Teams treat data versioning as optional infrastructure to add "when needed," then discover the need only after a deployed model produces unexpected results. Without versioning, reproducing a training run requires re-executing the entire data pipeline from scratch, a process that can consume days of compute and weeks of engineering time. Consider a model that performed well three months ago but degrades after retraining on updated data. Without versioned snapshots, the team cannot determine whether the regression stems from a labeling policy change, a schema migration error, or genuine distribution shift. The data lineage principles established in @sec-data-engineering-data-lineage-tracking-3e95 formalize this requirement: every training artifact must trace back to a specific, immutable dataset version. Organizations that defer versioning report 2--4 $\times$ longer debugging cycles when production issues arise, because each investigation begins with the question "what data did this model actually train on?" and no system can answer it.
+
 ## Summary {#sec-data-engineering-summary-4ac6}
 
 Data engineering provides the foundational infrastructure that transforms raw information into the basis of machine learning systems, determining model performance, system reliability, ethical compliance, and long-term maintainability. The Four Pillars framework—Quality, Reliability, Scalability, and Governance (@fig-four-pillars)—organizes design choices across acquisition, ingestion, validation, and storage, while the cascading nature of data quality failures (@fig-cascades) reveals why every pipeline stage requires careful engineering decisions. The task of "getting data ready" encompasses complex trade-offs quantified throughout this chapter: data engineering cost constants for budgeting, storage performance hierarchies (@tbl-storage-performance, @tbl-ml-latencies), and drift detection thresholds (PSI and KL divergence) that operationalize the Degradation Equation (@eq-degradation) into production monitoring infrastructure.

book/quarto/contents/vol1/data_selection/data_selection.qmd

@@ -967,7 +967,7 @@ Static pruning answers a question about *what* to keep, but it treats the answer
 ## Dynamic Selection {#sec-data-selection-dynamic-selection-edaa}
 
 \index{Dynamic Selection!training-time optimization}
-The answer is that they do. Early in training, the model benefits from diverse coverage to build broad feature representations; later, it benefits from focusing on hard examples near the decision boundary to refine its predictions. Dynamic selection exploits this insight by optimizing which samples to use *during* training, adapting the data diet based on the model's evolving state.
+The optimal training samples do change as the model learns. Early in training, the model benefits from diverse coverage to build broad feature representations; later, it benefits from focusing on hard examples near the decision boundary to refine its predictions. Dynamic selection exploits this insight by optimizing which samples to use *during* training, adapting the data diet based on the model's evolving state.
 
 ### Curriculum Learning: Easy to Hard {#sec-data-selection-curriculum-learning-easy-hard-2c4e}
 
@@ -2400,7 +2400,7 @@ This ratio, where data costs dominate, is typical for supervised learning. The r
 ### ROI Framework for Data Selection Techniques {#sec-data-selection-roi-framework-data-selection-techniques-dade}
 
 \index{Return on Investment!data selection framework}
-Understanding total costs enables rational decisions about which efficiency techniques merit investment. Every technique carries both a cost (implementation effort, compute overhead) and a benefit (reduced data requirements, faster training). Comparing these trade-offs requires a common framework: **Return on Investment (ROI)**.
+Understanding total costs enables rational decisions about which efficiency techniques merit investment. Every technique carries both a cost (implementation effort, compute overhead) and a benefit (reduced data requirements, faster training). Comparing these trade-offs requires a common framework: Return on Investment (ROI).
 
 $$
 \text{ROI} = \frac{\text{Savings} - \text{Investment}}{\text{Investment}} \times 100\%

book/quarto/contents/vol1/frameworks/frameworks.qmd

@@ -7,6 +7,14 @@ engine: jupyter
 
 # ML Frameworks {#sec-ml-frameworks}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:frameworks")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -2876,7 +2884,7 @@ We introduce these concepts here because they shape framework API design even fo
 
 \index{Data Parallelism!definition}
 \index{Pipeline Parallelism!definition}
-When models exceed single-device memory, frameworks combine multiple parallelism strategies simultaneously. A GPT-3 scale model, for instance, cannot fit on a single GPU---its `{python} GPT3Context.gpt3_params_b_str` B parameters alone require `{python} gpt3_fp16_gb_str` GB in FP16, far exceeding any GPU's memory. How do practitioners train such models? By distributing computation across multiple devices using three complementary strategies. @fig-3d-parallelism lays out how large-scale training distributes computation across three orthogonal dimensions to overcome this constraint. In the figure, look for how each dimension addresses a different scaling need: **Data Parallelism**\index{Data Parallelism!model replication} (replicating the model across columns) scales throughput by processing different batches in parallel; **Pipeline Parallelism**\index{Pipeline Parallelism!layer splitting} (splitting layers across rows) distributes a single model's depth across devices; and **Model Parallelism**\index{Model Parallelism!tensor sharding} (sharding tensors within each cluster) partitions individual layers that are too large for one device. This "3D" approach allows frameworks to scale beyond the memory limits of any single device [@mcmahan2017federated]. @sec-model-training examines these parallelism strategies in depth, including their implementation trade-offs and communication patterns.
+When models exceed single-device memory, frameworks combine multiple parallelism strategies simultaneously. A GPT-3 scale model, for instance, cannot fit on a single GPU---its `{python} GPT3Context.gpt3_params_b_str` B parameters alone require `{python} gpt3_fp16_gb_str` GB in FP16, far exceeding any GPU's memory. How do practitioners train such models? By distributing computation across multiple devices using three complementary strategies. @fig-3d-parallelism lays out how large-scale training distributes computation across three orthogonal dimensions to overcome this constraint. In the figure, look for how each dimension addresses a different scaling need: **Data Parallelism**\index{Data Parallelism!model replication} (replicating the model across columns) scales throughput by processing different batches in parallel; **Pipeline Parallelism**\index{Pipeline Parallelism!layer splitting} (splitting layers across rows) distributes a single model's depth across devices; and **Model Parallelism**\index{Model Parallelism!tensor sharding} (sharding tensors within each cluster) partitions individual layers that are too large for one device. This "3D" approach allows frameworks to scale beyond the memory limits of any single device. @sec-model-training examines these parallelism strategies in depth, including their implementation trade-offs and communication patterns.
 
 ::: {#fig-3d-parallelism fig-env="figure" fig-pos="htb" fig-cap="**3D Parallelism.** A grid of eight accelerator clusters arranged in two rows and four columns, each containing stacked computational units. Distinct colors encode the three parallelism dimensions: data parallelism across columns, pipeline parallelism across rows, and model parallelism within each cluster." fig-alt="Grid of 8 GPU clusters in 2 rows and 4 columns. Each cluster contains 4 stacked cubes. Colors vary: blue, red, green, orange in bottom row; olive, yellow, brown, pink in top row."}
 ```{.tikz}
@@ -3340,7 +3348,7 @@ Each major framework represents a distinct point in the design space defined by
 
 \index{TensorFlow!graph-first architecture}
 \index{TensorFlow!production deployment}
-TensorFlow's architecture reflects a comprehensive solution to the **Abstraction Problem**: how do you target diverse hardware---from cloud TPUs to microcontrollers---using a single interface? Its design philosophy is built on the **Static Graph** (or "Define-and-Run") principle. By requiring the model to be represented as a complete computational graph before execution, TensorFlow enables ahead-of-time (AOT) compilation and optimization.
+TensorFlow's architecture reflects a comprehensive solution to the **Abstraction Problem**: how do you target diverse hardware, from cloud TPUs to microcontrollers, using a single interface? Google's production environment demanded this breadth because the same model often needed to serve predictions on TPU pods in the datacenter, on Android phones via TensorFlow Lite, and in web browsers through TensorFlow.js. This deployment diversity drove the choice of a **Static Graph** (or "Define-and-Run") design. By requiring the model to be represented as a complete computational graph before execution, TensorFlow enables ahead-of-time (AOT) compilation and optimization for each target platform.
 
 This approach prioritizes the **Deployment Spectrum**. Because the framework sees the entire graph, it can perform aggressive optimizations like constant folding, operator fusion, and memory layout optimization before the first byte of data is processed. This is why TensorFlow remains the standard for complex production ecosystems. @fig-tensorflow-architecture maps the full training-to-deployment pipeline---trace how a model flows from data preprocessing through distributed training on the left, then fans out to serving, mobile (TF Lite), browser (TF.js), and language bindings on the right.
 
@@ -3415,15 +3423,15 @@ While TensorFlow 2.0 introduced eager execution to bridge the gap between resear
 
 \index{PyTorch!eager execution default}
 \index{PyTorch!research-first design}
-Where TensorFlow's graph-first approach prioritizes production optimization, PyTorch makes the opposite trade-off: it prioritizes *developer experience*. PyTorch's architecture represents a sharply different answer to the **Execution Problem**, built on **Dynamic Graphs** (or "Define-by-Run"). Instead of building a blueprint before execution, PyTorch builds the computational graph on-the-fly as the code runs.
+Where TensorFlow's graph-first approach prioritizes production optimization, PyTorch makes the opposite trade-off: it prioritizes *developer experience*. PyTorch's architecture represents a sharply different answer to the **Execution Problem**, built on **Dynamic Graphs** (or "Define-by-Run"). Instead of building a blueprint before execution, PyTorch builds the computational graph on-the-fly as the code runs. Facebook AI Research (FAIR) adopted this design because researchers need immediate feedback when experimenting with novel architectures; the define-then-run cycle of static graphs introduced a compilation delay that slowed the rapid prototyping essential to research workflows.
 
-This approach won the research community for a simple reason: it treats deep learning as standard Python programming. You can use Python loops, conditionals, and debuggers (like `pdb`) directly within your model's forward pass, with no special syntax, no separate compilation step, and no waiting to see if your code works. This "Eager Execution" model enables rapid iteration and intuitive model design, which is essential for the trial-and-error nature of frontier AI research.
+This approach won the broader research community for the same reason: it treats deep learning as standard Python programming. You can use Python loops, conditionals, and debuggers (like `pdb`) directly within your model's forward pass, with no special syntax, no separate compilation step, and no waiting to see if your code works. This "Eager Execution" model enables rapid iteration and intuitive model design, which is essential for the trial-and-error nature of frontier AI research.
 
 PyTorch's answer to the **Differentiation Problem** is the tape-based autograd system examined in @sec-ml-frameworks-pytorch-autograd-internals-4fa0: flexible and debuggable, but harder to optimize globally because the tape is rebuilt each iteration. Its answer to the **Abstraction Problem** is more pragmatic than comprehensive: strong GPU support through cuBLAS and cuDNN, but deployment to mobile, edge, and browser environments requires exporting through ONNX or specialized runtimes rather than a native path.
 
 The trade-off is therefore a more fragmented deployment path. Because the graph is dynamic, the framework cannot easily perform global optimizations before execution. A model that works perfectly in development may hit performance walls in production when dispatch overhead dominates small operations. To bridge this research-to-production gap, PyTorch introduced **TorchScript** and **PyTorch 2.0 (with `torch.compile`)**, which allow developers to capture a dynamic model and turn it into an optimized, static representation for deployment. This evolution shows PyTorch moving toward the production end of the compilation continuum while preserving the eager experience that made it dominant in research.
 
-### JAX: The Functional Transformation Engine {#sec-ml-frameworks-jax-functional-transformation-engine-242e-functional-transformation-engine-242e}
+### JAX: The Functional Transformation Engine {#sec-ml-frameworks-jax-functional-transformation-engine-242e}
 
 \index{JAX!functional transformations}
 \index{JAX!composable program transformations}
@@ -3431,7 +3439,7 @@ The trade-off is therefore a more fragmented deployment path. Because the graph
 \index{JAX!transformation composition}
 PyTorch's eager execution and TensorFlow's graph compilation represent two points on a spectrum, yet both share an imperative programming heritage where computation proceeds as a sequence of stateful operations. JAX represents a radically different approach, one built on functional programming principles and composable program transformations rather than computational graphs [@jax2018github]. Developed by Google Research, JAX has gained significant traction in research settings, particularly for work requiring custom differentiation, advanced optimization research, and large-scale distributed training.
 
-JAX's architecture reframes the **Differentiation Problem** entirely. Rather than implementing automatic differentiation as a tape-based system (PyTorch) or a graph transformation pass (TensorFlow), JAX treats differentiation as one of several *composable function transformations*. The `jax.grad` function does not compute gradients directly; it returns a *new function* that computes gradients. This subtle distinction enables powerful compositions: you can differentiate a differentiated function (higher-order derivatives), vectorize a gradient computation (`vmap(grad(f))`), or compile a vectorized gradient to XLA (`jit(vmap(grad(f)))`).
+JAX's architecture reframes the **Differentiation Problem** entirely. Google Research built JAX on a key observation: if functions are pure (no side effects, no mutable state), the compiler can safely reorder, fuse, and parallelize any operation, because outputs depend only on inputs. This constraint, borrowed from functional programming, is what makes JAX's composable transformations possible. Rather than implementing automatic differentiation as a tape-based system (PyTorch) or a graph transformation pass (TensorFlow), JAX treats differentiation as one of several *composable function transformations*. The `jax.grad` function does not compute gradients directly; it returns a *new function* that computes gradients. This subtle distinction enables powerful compositions: you can differentiate a differentiated function (higher-order derivatives), vectorize a gradient computation (`vmap(grad(f))`), or compile a vectorized gradient to XLA (`jit(vmap(grad(f)))`).
 
 JAX's functional paradigm requires a genuine mental shift from "tracking state through objects" to "transforming pure functions." The conceptual introduction here covers JAX's core design; transformation composition, pytree handling, and XLA tracing mechanics each warrant dedicated study for production use.
 
@@ -3988,7 +3996,7 @@ Framework selection involves subtle trade-offs where intuitions from conventiona
 # │ Context: Fallacies section demonstrating framework non-equivalence
 # │
 # │ Goal: Shows 17x performance gap (52ms vs 3ms) between PyTorch and TensorRT
-# │      on same model, and 7040x memory gap between PyTorch Mobile (220MB) and
+# │      on same model, and 6875x memory gap between PyTorch Mobile (220MB) and
 # │      TFLite Micro (32KB). Frameworks are NOT interchangeable.
 # │
 # │ Imports: mlsys.formatting (fmt)
@@ -4007,7 +4015,7 @@ tflite_micro_kb_value = 32
 
 # --- Process ---
 perf_gap_value = pytorch_ms_value / tensorrt_ms_value                          # ~17x gap
-memory_ratio_value = pytorch_mobile_mb_value * KIB_TO_BYTES / tflite_micro_kb_value    # ~7040x gap
+memory_ratio_value = pytorch_mobile_mb_value * 1000 / tflite_micro_kb_value             # ~6875x gap (decimal SI: 1 MB = 1000 KB)
 
 # --- Outputs (formatted strings for prose) ---
 pytorch_ms_str = fmt(pytorch_ms_value, precision=0, commas=False)              # e.g. "52"
@@ -4015,7 +4023,7 @@ tensorrt_ms_str = fmt(tensorrt_ms_value, precision=0, commas=False)            #
 pytorch_mobile_mb_str = fmt(pytorch_mobile_mb_value, precision=0, commas=False)  # e.g. "220"
 tflite_micro_kb_str = fmt(tflite_micro_kb_value, precision=0, commas=False)    # e.g. "32"
 perf_gap_str = fmt(perf_gap_value, precision=0, commas=False)                  # e.g. "17"
-memory_ratio_str = fmt(memory_ratio_value, precision=0, commas=False)          # e.g. "7040"
+memory_ratio_str = fmt(memory_ratio_value, precision=0, commas=False)          # e.g. "6875"
 ```
 
 **Fallacy:** *"All frameworks provide equivalent performance for the same model architecture."*

book/quarto/contents/vol1/hw_acceleration/hw_acceleration.qmd

@@ -54,7 +54,7 @@ start_chapter("vol1:hw_acceleration")
 ## Acceleration Fundamentals {#sec-hardware-acceleration-ai-hardware-acceleration-fundamentals-9b28}
 
 \index{D·A·M Taxonomy!machine axis}
-We have optimized the **Data** in @sec-data-selection and compressed the **Algorithm** (Model) in @sec-model-compression. Now we turn to the final axis of the D·A·M taxonomy (@sec-introduction): the **Machine**. Hardware acceleration exists because of a striking asymmetry in modern computing: arithmetic is *cheap*, but moving data is *expensive*. In the time a modern GPU computes a thousand floating-point operations, a single value travels from main memory. This inversion, where computation is the abundant resource and bandwidth is the scarce one, is the reason specialized hardware matters for machine learning.
+We have optimized the Data in @sec-data-selection and compressed the Algorithm (Model) in @sec-model-compression. Now we turn to the final axis of the D·A·M taxonomy (@sec-introduction): the Machine. Hardware acceleration exists because of a striking asymmetry in modern computing: arithmetic is *cheap*, but moving data is *expensive*. In the time a modern GPU computes a thousand floating-point operations, a single value travels from main memory. This inversion, where computation is the abundant resource and bandwidth is the scarce one, is the reason specialized hardware matters for machine learning.
 
 ::: {.callout-definition title="Hardware Acceleration"}
 
@@ -4535,12 +4535,14 @@ Understanding runtime adaptation mechanisms helps engineers design systems that
 
 :::
 
+To see how these runtime mechanisms work together in practice, consider a concrete scenario: a transformer inference request arrives at a production server. The runtime must adapt execution parameters such as tiling and memory allocation to current conditions (dynamic execution), determine which kernel implementation to use for each operation based on real-time hardware state (kernel selection), and schedule the selected kernels across available compute units to maximize utilization (kernel scheduling). These are not independent systems but three interrelated phases of a single runtime pipeline, and the following subsections examine each phase using this transformer inference request as a running example.
+
 ### Dynamic Kernel Execution {#sec-hardware-acceleration-dynamic-kernel-execution-33fc}
 
 \index{Dynamic Kernel Execution!runtime adaptation}
-While static compilation provides a solid foundation, efficient execution of machine learning workloads requires real-time adaptation to fluctuating conditions: available memory, data sizes, and computational loads all change during execution. The runtime continuously adjusts execution strategies to match both hardware constraints and workload characteristics.
+While static compilation provides a solid foundation, efficient execution of machine learning workloads requires real-time adaptation to fluctuating conditions. When our transformer inference request arrives, the runtime cannot simply execute a fixed plan: available memory, input sequence length, and computational load may differ from what the compiler assumed. The runtime continuously adjusts execution strategies to match both hardware constraints and workload characteristics.
 
-Individual computational operations (matrix multiplications, convolutions, activation functions) must be assigned to appropriate processing units, and this mapping is not fixed. As input data, memory availability, and system load change during execution, the runtime makes real-time decisions about kernel selection, execution order, and memory management to keep workloads efficient despite shifting conditions.
+Individual computational operations (matrix multiplications, convolutions, activation functions) must be assigned to appropriate processing units, and this mapping is not fixed. As input data, memory availability, and system load change during execution, the runtime makes real-time decisions about execution order and memory management to keep workloads efficient despite shifting conditions.
 
 For example, consider an AI accelerator executing a deep neural network (DNN) for image classification. If an incoming batch of high-resolution images requires significantly more memory than expected, a statically planned execution may cause cache thrashing or excessive off-chip memory accesses. Instead, a dynamic runtime can adjust tiling strategies on the fly, breaking down tensor operations into smaller tiles that fit within the high-speed on-chip memory. This prevents memory stalls and ensures optimal utilization of caches.
 
@@ -4551,28 +4553,28 @@ Overlapping computation with memory movement mitigates a common performance bott
 
 Consider the execution of convolutional layers in a CNN on a GPU. If multiple convolution kernels need scheduling, static approaches may lead to inefficient resource utilization due to variation in layer sizes and compute requirements. Dynamic scheduling allows AI runtimes to prioritize smaller kernels when compute units are partially occupied, improving hardware utilization. For instance, in NVIDIA's TensorRT runtime, fusion of small kernels into larger execution units is done dynamically to avoid launch overhead, optimizing latency-sensitive inference tasks.
 
-Dynamic adjustment of execution strategies in response to real-time system conditions optimizes both training and inference performance across hardware platforms. Beyond execution strategy adaptation, runtimes must also select which specific kernel implementations to invoke for each operation.
+Dynamic adjustment of execution strategies in response to real-time system conditions optimizes both training and inference performance across hardware platforms. These adaptations, however, depend on having the right kernel in the first place. Returning to our transformer inference example: before the runtime can adjust tiling or memory allocation for a matrix multiplication, it must first decide *which* kernel implementation to invoke.
 
 ### Runtime Kernel Selection {#sec-hardware-acceleration-runtime-kernel-selection-1ffe}
 
-While compilers perform an initial selection of kernels based on static analysis, AI runtimes often need to override these decisions during execution. Real-time factors, such as available memory, hardware utilization, and workload priorities, may differ significantly from the assumptions made during compilation. By dynamically selecting and switching kernels at runtime, AI runtimes can adapt to these changing conditions, ensuring that models continue to perform efficiently.
+While compilers perform an initial selection of kernels based on static analysis, AI runtimes often need to override these decisions during execution. Real-time factors, such as available memory, hardware utilization, and workload priorities, may differ significantly from the assumptions made during compilation. In our transformer example, the compiler may have selected an FP32 matrix multiplication kernel, but the runtime observes that Tensor Cores are available and the operation is numerically stable, prompting a switch to an FP16 kernel for higher throughput. By dynamically selecting and switching kernels at runtime, AI runtimes can adapt to these changing conditions, ensuring that models continue to perform efficiently.
 
 For instance, consider transformer-based language models, where a significant portion of execution time is spent on matrix multiplications. The AI runtime must determine the most efficient way to execute these operations based on the current system state. If the model is running on a GPU with specialized Tensor Cores, the runtime may switch from a standard FP32 kernel to an FP16 kernel to take advantage of hardware acceleration [@Shoeybi2019]. Conversely, if the lower precision of FP16 causes unacceptable numerical instability, the runtime can opt for mixed-precision execution, selectively using FP32 where higher precision is necessary.
 
 Memory constraints also influence kernel selection. When memory bandwidth is limited, the runtime may adjust its execution strategy, reordering operations or changing the tiling strategy to fit computations into the available cache rather than relying on slower main memory. For example, a large matrix multiplication may be broken into smaller chunks, ensuring that the computation fits into the on-chip memory of the GPU, reducing overall latency.
 
-Batch size also influences kernel selection. For workloads that handle a mix of small and large batches, the AI runtime may choose a latency-optimized kernel for small batches and a throughput-optimized kernel for large-scale batch processing. This adjustment ensures that the model continues to operate efficiently across different execution scenarios, without the need for manual tuning. Once the appropriate kernel is selected, the runtime must schedule its execution to maximize hardware utilization.
+Batch size also influences kernel selection. For workloads that handle a mix of small and large batches, the AI runtime may choose a latency-optimized kernel for small batches and a throughput-optimized kernel for large-scale batch processing. This adjustment ensures that the model continues to operate efficiently across different execution scenarios, without the need for manual tuning. With the appropriate kernels selected and their execution parameters adapted, the final pipeline stage determines *when and where* each kernel runs.
 
 ### Kernel Scheduling and Utilization {#sec-hardware-acceleration-kernel-scheduling-utilization-99d6}
 
 \index{Kernel Scheduling!hardware utilization}
-Kernel scheduling ensures that selected kernels execute in a way that maximizes parallelism and resource utilization. Unlike traditional task schedulers that manage CPU threads, AI runtimes coordinate a much larger number of tasks across parallel execution units: GPU cores, tensor processing units, or custom AI accelerators [@jouppi2017datacenter]. Keeping these resources fully engaged prevents bottlenecks and maximizes throughput.
+Kernel scheduling completes the runtime pipeline by determining how selected kernels execute across available hardware to maximize parallelism and resource utilization. Returning to the transformer inference request: the runtime has selected FP16 kernels for the attention matrix multiplications and adapted tiling to fit the current sequence length. Now the scheduler must distribute these operations across GPU streaming multiprocessors, interleave them with layer normalization and activation kernels, and ensure that intermediate data is prefetched before each operation needs it. Unlike traditional task schedulers that manage CPU threads, AI runtimes coordinate a much larger number of tasks across parallel execution units: GPU cores, tensor processing units, or custom AI accelerators [@jouppi2017datacenter]. Keeping these resources fully engaged prevents bottlenecks and maximizes throughput.
 
 For example, in image recognition models that use convolutional layers, operations can be distributed across multiple processing units, enabling different filters to run concurrently. This parallelization ensures that the available hardware is fully utilized, speeding up execution. Similarly, batch normalization and activation functions must be scheduled efficiently to avoid unnecessary delays. If these operations are not interleaved with other computations, they may block the pipeline and reduce overall throughput.
 
 Efficient kernel scheduling can also be influenced by real-time memory management. AI runtimes ensure that intermediate data, such as feature maps in deep neural networks, are preloaded into cache before they are needed. This proactive management helps prevent delays caused by waiting for data to be loaded from slower memory tiers, ensuring continuous execution.
 
-Together, these techniques ensure optimal resource utilization and efficient parallel computation across multiple hardware accelerators.
+Together, kernel selection, dynamic execution adaptation, and scheduling form a tightly coupled runtime pipeline. For our transformer inference request, the pipeline determined the best kernel for each operation, adapted tiling and precision to current memory and hardware conditions, and distributed work across compute units to sustain high utilization. These three phases operate continuously and interdependently: a scheduling decision may trigger re-selection of a different kernel, which in turn requires new execution parameter adaptation.
 
 The compiler and runtime systems examined thus far optimize execution within single accelerators, but the largest AI workloads exceed what any single chip can deliver. Single-chip optimizations achieve impressive results: ResNet-50 inference accelerates from 47 ms to 8 ms through compiler optimization alone, and the dataflow strategies we examined can push GPU utilization from 20% to 80% of peak throughput. Yet for the largest AI workloads, even perfectly optimized single-chip execution proves insufficient.
 
@@ -4647,9 +4649,11 @@ Beyond vertically integrated solutions, IP licensing models allow SoC designers
 \index{Workload Distribution!heterogeneous processor}
 With multiple specialized processors available on heterogeneous SoCs, the critical challenge becomes intelligently distributing neural network operations across these resources to maximize performance while respecting power and latency constraints.
 
-Modern neural networks require intelligent partitioning across heterogeneous processors based on operation characteristics and current system state. Convolutional layers with regular data access patterns typically execute efficiently on GPU shader cores, while fully connected layers with irregular sparsity patterns may perform better on general-purpose CPU cores with large caches. Attention mechanisms in transformers benefit from NPU matrix engines when sequences are long, but may execute more efficiently on CPU when sequence lengths are small due to the NPU setup overhead.
+Consider a concrete example: an engineer deploying a real-time object detection pipeline on a mobile SoC with a CPU, GPU, and NPU. The pipeline has three stages: a MobileNet backbone for feature extraction, non-maximum suppression (NMS) for post-processing, and a display overlay for rendering bounding boxes. The backbone consists of depthwise separable convolutions with regular, predictable data access patterns and high arithmetic intensity, making it an ideal fit for the NPU's matrix engines, which deliver 10 to 50 $\times$ better energy efficiency than the CPU on these operations. NMS, by contrast, involves conditional branching over variable-length candidate lists, with irregular memory access that maps poorly to the NPU's fixed dataflow. The CPU handles NMS more efficiently because its branch predictor and large caches accommodate the unpredictable control flow. Finally, the display overlay involves pixel-level compositing across the entire frame, a massively parallel but arithmetically simple workload that maps naturally to the GPU's shader cores. This three-way split, NPU for the backbone, CPU for NMS, GPU for the overlay, achieves lower latency and lower power than running the entire pipeline on any single processor.
 
-Beyond static operation-to-processor mapping, heterogeneous SoCs implement dynamic processor selection governed by real-time constraints. Power budget dictates routing: during battery operation, the system may route computations to lower-power DSP cores rather than high-performance GPU cores. Thermal state introduces another dimension: when approaching thermal limits, workloads shift from the power-hungry NPU to more efficient CPU execution. Safety-critical automotive applications add latency requirements that prioritize deterministic CPU execution over potentially faster but variable NPU processing. Finally, concurrent workload interference from multiple AI applications may require load balancing across available processors to maintain Quality of Service.
+This example illustrates the general principle: modern neural networks require intelligent partitioning across heterogeneous processors based on operation characteristics and current system state. Convolutional layers with regular data access patterns typically execute efficiently on GPU shader cores or NPU matrix engines, while operations with irregular sparsity patterns or conditional control flow may perform better on general-purpose CPU cores with large caches. Attention mechanisms in transformers benefit from NPU matrix engines when sequences are long, but may execute more efficiently on CPU when sequence lengths are small due to the NPU setup overhead.
+
+Beyond static operation-to-processor mapping, the optimal assignment can change moment to moment. Returning to the object detection example: during battery operation, the system might shift the MobileNet backbone from the NPU to lower-power DSP cores, accepting higher latency to extend battery life. Power budget dictates routing: during battery operation, the system may route computations to lower-power DSP cores rather than high-performance GPU cores. Thermal state introduces another dimension: when approaching thermal limits, workloads shift from the power-hungry NPU to more efficient CPU execution. Safety-critical automotive applications add latency requirements that prioritize deterministic CPU execution over potentially faster but variable NPU processing. Finally, concurrent workload interference from multiple AI applications may require load balancing across available processors to maintain Quality of Service.
 
 Compounding the processor selection challenge, shared memory architectures require priority-based arbitration when multiple processors access LPDDR simultaneously. The Snapdragon 8 Gen 3's memory controller implements priority-based scheduling where camera processing receives higher priority than background AI tasks, ensuring real-time video processing while background neural networks adapt their execution patterns to available memory bandwidth. This arbitration becomes critical during memory-intensive operations like large language model inference, where parameter streaming from DRAM must be carefully coordinated across processors.
 
@@ -4835,73 +4839,69 @@ Organizations optimize exclusively for specific vendors to maximize performance
 ```{python}
 #| label: feasibility-calc
 #| echo: false
+# ┌─────────────────────────────────────────────────────────────────────────────
+# │ HARDWARE FEASIBILITY ASSESSMENT
+# ├─────────────────────────────────────────────────────────────────────────────
+# │ Context: Callout "Feasibility Assessment: Can You Run It?"
+# │
+# │ Goal: Quantify three deployment feasibility checks.
+# │ Show: Memory headroom, bandwidth-limited latency, and real-time frame budget.
+# │ How: Three independent checks using model_memory() formula.
+# │
+# │ Imports: mlsys.formatting (fmt, check), mlsys.formulas (model_memory),
+# │          mlsys.constants (GB, BYTES_FP16)
+# │ Exports: headroom_str, token_latency_ms_str, frame_budget_str
+# └─────────────────────────────────────────────────────────────────────────────
+from mlsys.formatting import fmt, check
+from mlsys.formulas import model_memory
+from mlsys.constants import GB, BYTES_FP16
+
 # ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
 class FeasibilityAssessment:
     """
     Namespace for Hardware Feasibility Assessment.
-    Scenario: Deploying a 7B model on 16GB GPU.
+    Check 1: Memory — 7B model (FP16) on 16GB GPU.
+    Check 2: Bandwidth — 70B model on 1TB/s GPU.
+    Check 3: Compute — 30 FPS video processing.
     """
 
     # ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
-    params_b = 7
+    # Memory check: Llama-7B on 16GB GPU
+    params_7b = 7e9
     gpu_mem_gb = 16
-    mem_bw_gb_s = 500
+
+    # Bandwidth check: 70B model on 1TB/s GPU
+    params_70b = 70e9
+    mem_bw_gb_s = 1000  # 1 TB/s
+
+    # Compute check: video at 30 FPS
     fps_target = 30
 
     # ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
-    # Memory fit
-    model_size_gb = params_b * 2 # FP16
-    headroom_gb = gpu_mem_gb - model_size_gb
+    # Memory: does the model fit?
+    model_7b_gb = model_memory(params_7b, BYTES_FP16, GB)
+    headroom_gb = gpu_mem_gb - model_7b_gb
 
-    # Bandwidth Latency (Batch 1)
-    token_latency_ms = (model_size_gb / mem_bw_gb_s) * 1000
+    # Bandwidth: how fast can we generate tokens?
+    model_70b_gb = model_memory(params_70b, BYTES_FP16, GB)
+    token_latency_ms = (model_70b_gb / mem_bw_gb_s) * 1000
 
-    # Real-time frame budget
+    # Compute: real-time frame budget
     frame_budget_ms = 1000 / fps_target
 
     # ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
-    check(headroom_gb > 0, f"Model ({model_size_gb}GB) doesn't fit on GPU ({gpu_mem_gb}GB)!")
+    check(headroom_gb > 0, f"Model ({model_7b_gb}GB) doesn't fit on GPU ({gpu_mem_gb}GB)!")
+    check(token_latency_ms > 0, "Token latency must be positive.")
 
     # ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
-    headroom_str = fmt(headroom_gb, precision=1, commas=False)
-    token_latency_ms_str = fmt(token_latency_ms, precision=1, commas=False)
-    frame_budget_str = fmt(frame_budget_ms, precision=1, commas=False)
+    headroom_str = fmt(headroom_gb, precision=0, commas=False)
+    token_latency_ms_str = fmt(token_latency_ms, precision=0, commas=False)
+    frame_budget_str = fmt(frame_budget_ms, precision=0, commas=False)
 
 # ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
 headroom_str = FeasibilityAssessment.headroom_str
 token_latency_ms_str = FeasibilityAssessment.token_latency_ms_str
-frame_budget_str = FeasibilityAssessment.frame_budget_ms_str
-# └─────────────────────────────────────────────────────────────────────────────
-from mlsys.formatting import fmt
-from mlsys.formulas import model_memory
-from mlsys.constants import GB, BYTES_FP16
-
-# --- Inputs (model specs and hardware constraints) ---
-# Memory check: Llama-7B on 16GB GPU
-llama7b_params = 7e9                     # 7B parameter model
-gpu_mem_gb = 16                          # 16 GB GPU memory
-
-# Bandwidth check: 70B model on 1TB/s GPU
-model_70b_params = 70e9                  # 70B parameter model
-bw_check_tbps = 1.0                      # 1 TB/s memory bandwidth
-
-# Compute check: video at 30 FPS
-video_fps = 30                           # target frames per second
-
-# --- Outputs (formatted strings for prose) ---
-# Memory check calculation
-llama7b_weights_gb = model_memory(llama7b_params, BYTES_FP16, GB)
-headroom_gb = gpu_mem_gb - llama7b_weights_gb
-headroom_str = f"{headroom_gb}"          # e.g. "2" GB remaining for context
-
-# Bandwidth check calculation
-model_70b_size_gb = model_memory(model_70b_params, BYTES_FP16, GB)
-token_latency_ms = model_70b_size_gb / (bw_check_tbps * 1e3) * 1e3
-token_latency_ms_str = fmt(token_latency_ms, precision=0, commas=False)  # e.g. "140" ms
-
-# Compute check calculation
-frame_budget_ms = 1000 / video_fps
-frame_budget_str = fmt(frame_budget_ms, precision=0, commas=False)  # e.g. "33" ms
+frame_budget_str = FeasibilityAssessment.frame_budget_str
 ```
 
 ::: {.callout-checkpoint title="Feasibility Assessment: Can You Run It?" collapse="false"}
@@ -4935,22 +4935,88 @@ These fallacies and pitfalls reveal a recurring theme: optimizing for the wrong
 \index{Hardware Sustainability!carbon footprint}
 Beyond raw performance, we must evaluate hardware through the lens of *silicon sustainability*\index{Sustainability!silicon design}. In the era of planetary-scale AI, performance per watt\index{Performance per Watt!efficiency metric}\index{Energy Efficiency!TFLOPS per Watt} is not merely a mobile constraint but a global environmental mandate. Quantifying *the carbon ROI of specialized silicon* makes the case concrete.
 
+```{python}
+#| label: carbon-roi-calc
+#| echo: false
+# ┌─────────────────────────────────────────────────────────────────────────────
+# │ CARBON ROI OF SPECIALIZED SILICON
+# ├─────────────────────────────────────────────────────────────────────────────
+# │ Context: "The Carbon ROI of Specialized Silicon" callout
+# │
+# │ Goal: Quantify carbon savings of NPUs vs CPUs for inference fleets.
+# │ Show: The 200× efficiency gap and ~350 metric tons CO2 saved per year.
+# │ How: Compare power consumption and compute efficiency, then project
+# │      daily energy use and annual carbon savings.
+# │
+# │ Imports: mlsys.constants (DAYS_PER_YEAR), mlsys.formatting (fmt)
+# │ Exports: cpu_power_str, npu_power_str, cpu_tflops_str, npu_tflops_str,
+# │          cpu_eff_str, npu_eff_str, eff_gap_str, workload_str,
+# │          cpu_energy_day_str, npu_energy_day_str, carbon_intensity_str,
+# │          co2_saved_str
+# └─────────────────────────────────────────────────────────────────────────────
+from mlsys.formatting import fmt, check
+from mlsys.constants import DAYS_PER_YEAR
+
+# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────────
+# Power and compute specs (scenario values for illustrative comparison)
+cpu_power_w = 100          # Watts per CPU server doing inference
+cpu_tflops = 1             # Peak TFLOPS for CPU inference
+npu_power_w = 5            # Watts per NPU chip
+npu_tflops = 10            # Peak TFLOPS for NPU inference
+
+# Fleet workload and carbon intensity
+inferences_per_day = 1e9   # 1 billion inferences/day
+carbon_kg_per_kwh = 0.4    # Approximate global grid average (kg CO2/kWh)
+
+# Energy per inference derived from power and throughput
+# CPU energy/day and NPU energy/day are scenario estimates for the fleet
+cpu_energy_kwh_day = 2400  # kWh/day for CPU fleet serving 1B inferences
+npu_energy_kwh_day = 12    # kWh/day for NPU fleet serving 1B inferences
+
+# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────────
+cpu_eff = cpu_tflops / cpu_power_w        # TFLOPS/W
+npu_eff = npu_tflops / npu_power_w        # TFLOPS/W
+eff_gap = npu_eff / cpu_eff               # Efficiency ratio
+
+energy_savings_kwh_day = cpu_energy_kwh_day - npu_energy_kwh_day
+co2_saved_kg_year = energy_savings_kwh_day * DAYS_PER_YEAR * carbon_kg_per_kwh
+co2_saved_metric_tons = co2_saved_kg_year / 1000
+
+# ┌── 3. INVARIANTS (Guardrails) ──────────────────────────────────────────────
+check(eff_gap == 200, f"Efficiency gap should be 200×, got {eff_gap}×")
+check(co2_saved_metric_tons > 300, f"CO2 savings should exceed 300 tons, got {co2_saved_metric_tons}")
+
+# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────────
+cpu_power_str = fmt(cpu_power_w, precision=0)
+npu_power_str = fmt(npu_power_w, precision=0)
+cpu_tflops_str = fmt(cpu_tflops, precision=0)
+npu_tflops_str = fmt(npu_tflops, precision=0)
+cpu_eff_str = fmt(cpu_eff, precision=2)
+npu_eff_str = fmt(npu_eff, precision=1)
+eff_gap_str = fmt(eff_gap, precision=0)
+workload_str = fmt(inferences_per_day / 1e9, precision=0)
+cpu_energy_day_str = fmt(cpu_energy_kwh_day, precision=0)
+npu_energy_day_str = fmt(npu_energy_kwh_day, precision=0)
+carbon_intensity_str = fmt(carbon_kg_per_kwh, precision=1)
+co2_saved_str = fmt(co2_saved_metric_tons, precision=0)
+```
+
 ::: {.callout-notebook title="The Carbon ROI of Specialized Silicon"}
 
 **The Problem**: Should you run your inference fleet on generic CPUs or invest in specialized NPUs (Neural Processing Units)?
 
 **The Physics**: Specialized hardware achieves higher arithmetic intensity while using fewer transistors for control logic.
 
-*   **CPU Inference**: 100 Watts for 1 TFLOP (Efficiency = 0.01 TFLOPS/W).
-*   **NPU Inference**: 5 Watts for 10 TFLOPS (Efficiency = 2.0 TFLOPS/W).
-*   **The Gap**: The NPU is **200 $\times$ more energy-efficient** per operation.
+*   **CPU Inference**: `{python} cpu_power_str` Watts for `{python} cpu_tflops_str` TFLOP (Efficiency = `{python} cpu_eff_str` TFLOPS/W).
+*   **NPU Inference**: `{python} npu_power_str` Watts for `{python} npu_tflops_str` TFLOPS (Efficiency = `{python} npu_eff_str` TFLOPS/W).
+*   **The Gap**: The NPU is **`{python} eff_gap_str` $\times$ more energy-efficient** per operation.
 
 **The Calculation**:
 
-1.  **Workload**: 1 Billion inferences per day.
-2.  **CPU Energy**: ~2,400 kWh/day.
-3.  **NPU Energy**: ~12 kWh/day.
-4.  **Carbon Savings**: At 0.4 kg CO2/kWh, switching to NPUs saves **~350 metric tons of CO2 per year**.
+1.  **Workload**: `{python} workload_str` Billion inferences per day.
+2.  **CPU Energy**: ~`{python} cpu_energy_day_str` kWh/day.
+3.  **NPU Energy**: ~`{python} npu_energy_day_str` kWh/day.
+4.  **Carbon Savings**: At `{python} carbon_intensity_str` kg CO2/kWh, switching to NPUs saves **~`{python} co2_saved_str` metric tons of CO2 per year**.
 
 **The Systems Conclusion**: Custom silicon is the ultimate "Green" technology for ML. Investing in specialized accelerators is not just about speed; it is the single most effective way to reduce the carbon footprint of intelligence.
 :::

book/quarto/contents/vol1/introduction/introduction.qmd

@@ -1633,7 +1633,7 @@ If scale is the ultimate lever for performance, it is also the ultimate consumer
 
 The **Iron Law** provides more than a diagnostic framework. It organizes the entire discipline. Each term in the equation corresponds to a core engineering imperative. The Data Term demands that we *build* robust data pipelines and infrastructure (@sec-data-engineering). The Compute Term requires that we *optimize* algorithms and hardware utilization for efficiency (Part III). The Overhead Term necessitates that we *deploy* and *operate* systems reliably in production (@sec-model-serving, @sec-ml-operations). These three imperatives structure this textbook: Parts I and II address building, Part III addresses optimization, and Part IV addresses deployment and operations.
 
-Abstract equations become concrete through concrete workloads. This textbook employs five recurring **Lighthouse Models** as diagnostic tools for the Iron Law. We do not use these models merely as examples. They are **The Cast of Characters**, our **Systems Detectives**: canonical workloads that we will use in every chapter to "interrogate" the Iron Law (the **Silicon Contract** between algorithm and machine).
+Abstract equations become concrete through concrete workloads. This textbook employs five recurring **Lighthouse Models** as diagnostic tools for the Iron Law. These canonical workloads serve as the Cast of Characters, our Systems Detectives, reappearing in every chapter to interrogate the Iron Law (the Silicon Contract between algorithm and machine).
 
 Each archetype represents a distinct extreme of the Iron Law. For instance, **ResNet-50** allows us to investigate the **Compute Term** in its purest form, while **GPT-2/Llama** acts as our primary probe for **Memory Bandwidth** bottlenecks. By following these same workloads from data engineering through to edge deployment, you will see how a single architectural choice propagates physical and economic constraints across the entire system.
 
@@ -1706,7 +1706,7 @@ The Bitter Lesson establishes that scale drives AI progress, but it also creates
 
 Training GPT-4-class models reportedly consumed over two million A100 GPU-days, representing millions of dollars in compute costs and substantial environmental impact. Many research institutions and companies cannot afford to compete through brute-force scaling. The reality motivates a complementary approach: rather than asking "how much more compute can we apply?" we must also ask "how efficiently can we use the compute we have?"
 
-The answer defines the efficiency framework. Three complementary dimensions map directly to our **D·A·M taxonomy** (@tbl-dam-taxonomy), and they matured in a revealing historical sequence. **Algorithmic Efficiency**, the earliest frontier, reduces computational requirements through better model design and training procedures. Techniques such as **Model Compression**\index{Model Compression} (pruning, quantization, knowledge distillation), efficient architectures like MobileNet, and neural architecture search all deliver more capability per FLOP. As algorithms demanded ever more computation, **Compute Efficiency** became the second critical dimension. It maximizes hardware utilization by aligning algorithmic logic with machine physics, encompassing the evolution from general-purpose CPUs to specialized accelerators (GPUs, TPUs) and the hardware-software co-design principles that translate theoretical TFLOPS into real-world speedups. Most recently, **Data Selection** emerged as the third dimension, extracting more learning signal from limited examples and thereby reducing the data numerator of the Iron Law. Techniques including transfer learning\index{Transfer Learning}, active learning\index{Active Learning}, and curriculum design ensure every sample provides maximum learning value. Together, these three dimensions provide the engineering tools to overcome the Data, Algorithm, and Machine walls that pure scaling alone cannot address.
+The answer defines the efficiency framework. Three complementary dimensions map directly to our **D·A·M taxonomy** (@tbl-dam-taxonomy), and they matured in a revealing historical sequence. **Algorithmic Efficiency**, the earliest frontier, reduces computational requirements through better model design and training procedures. Techniques such as model compression\index{Model Compression} (pruning, quantization, knowledge distillation), efficient architectures like MobileNet, and neural architecture search all deliver more capability per FLOP. As algorithms demanded ever more computation, **Compute Efficiency** became the second critical dimension. It maximizes hardware utilization by aligning algorithmic logic with machine physics, encompassing the evolution from general-purpose CPUs to specialized accelerators (GPUs, TPUs) and the hardware-software co-design principles that translate theoretical TFLOPS into real-world speedups. Most recently, **Data Selection** emerged as the third dimension, extracting more learning signal from limited examples and thereby reducing the data numerator of the Iron Law. Techniques including transfer learning\index{Transfer Learning}, active learning\index{Active Learning}, and curriculum design ensure every sample provides maximum learning value. Together, these three dimensions provide the engineering tools to overcome the Data, Algorithm, and Machine walls that pure scaling alone cannot address.
 
 These three dimensions did not emerge simultaneously; as @fig-evolution-efficiency reveals, each progressed through distinct eras at different rates. Algorithmic efficiency led the way, compute efficiency followed as demand grew, and data-centric methods matured most recently. While history progressed from algorithmic breakthroughs to hardware acceleration to data-centric methods, Part III of this book reverses that sequence: we begin with data selection, then model compression, then hardware acceleration. This pedagogical order reflects how practitioners actually build systems: quality data is prerequisite to effective model optimization, and understanding the model is prerequisite to mapping it efficiently onto hardware.
 
@@ -2224,7 +2224,9 @@ waymo_data_low_str = WaymoStats.low_str
 waymo_data_high_str = WaymoStats.high_str
 ```
 
-Real-world data is often noisy and inconsistent, presenting the first category of challenges. Waymo's autonomous vehicles serve as roving data centers, processing between `{python} waymo_data_low_str` and `{python} waymo_data_high_str` terabytes of data per hour across their sensor suite, including LiDAR[^fn-lidar], radar[^fn-radar], and cameras. Engineers must solve for sensor interference, such as rain obscuring cameras, and temporal misalignment across asynchronous data streams. Scale compounds these quality issues: while FarmBeats operates under severe constraints (running inference on models under 500 KB transmitted over TV white-space bandwidth measured in kilobits per second), AlphaFold occupies the opposite extreme, requiring access to the entire Protein Data Bank containing over 180,000 experimentally determined structures to predict configurations for more than 200 million proteins. And data drift\index{Data Drift!operational burden} creates an ongoing operational burden atop both quality and scale. Waymo models trained on Phoenix's sun-drenched roads may fail in New York's snowstorms due to distribution shift[^fn-drift]; detecting these shifts requires continuous monitoring of input statistics before they manifest as system failures.
+Real-world data is often noisy and inconsistent, presenting the first category of challenges. Waymo's autonomous vehicles serve as roving data centers, processing between `{python} waymo_data_low_str` and `{python} waymo_data_high_str` terabytes of data per hour across their sensor suite, including LiDAR[^fn-lidar], radar[^fn-radar], and cameras. Engineers must solve for sensor interference, such as rain obscuring cameras, and temporal misalignment across asynchronous data streams. Scale compounds these quality issues: while FarmBeats operates under severe constraints (running inference on models under 500 KB transmitted over TV white-space bandwidth measured in kilobits per second), AlphaFold occupies the opposite extreme, requiring access to the entire Protein Data Bank containing over 180,000 experimentally determined structures to predict configurations for more than 200 million proteins.
+
+Data drift\index{Data Drift!operational burden} creates an ongoing operational burden atop both quality and scale. The statistical properties of input data change over time, and models are only as reliable as their alignment with the current distribution. Waymo models trained on Phoenix's sun-drenched roads may fail in New York's snowstorms due to distribution shift[^fn-drift]; detecting these shifts requires continuous monitoring of input statistics before they manifest as system failures.
 
 [^fn-lidar]: **LiDAR (Light Detection and Ranging)**: A remote sensing method that uses light in the form of a pulsed laser to measure ranges (variable distances) to the Earth.
 

book/quarto/contents/vol1/ml_ops/ml_ops.qmd

@@ -7,6 +7,14 @@ engine: jupyter
 
 # ML Operations {#sec-ml-operations}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:ml_ops")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -67,7 +75,7 @@ Machine Learning Operations (MLOps)[^fn-mlops-emergence] is the engineering disc
 
 A concrete example illustrates *why* this discipline is necessary. Consider deploying a demand prediction system for a ridesharing service. Your benchmark results (@sec-benchmarking) show 94% accuracy, 15ms P99 latency, and strong performance across test segments. You deploy. Week one: excellent. Week four: accuracy has dropped to 88%, but your infrastructure metrics show nothing wrong. Week eight: a product manager notices driver dispatch is inefficient; investigation reveals the model has not adapted to a competitor's new promotion that shifted user behavior. The model needed retraining six weeks ago, but no system was watching for this degradation. MLOps provides the framework to detect such drift, trigger retraining, and validate new models before users experience the impact.
 
-This framing connects directly to the book's analytical foundations. If benchmarking provides the *sensors* for our system, MLOps is the complete *control system*. It closes the *Verification Gap*\index{Verification Gap!MLOps closure} (the recognition that ML testing is statistical, not deterministic, formalized in @sec--twelve-quantitative-invariants-0dd2) by continuously recalibrating against a changing world. MLOps operationalizes the *Degradation Equation*\index{Degradation Equation!operational implementation} (formalized in @sec--twelve-quantitative-invariants-0dd2): accuracy decay is not a failure of the code, but an inevitable consequence of the distributional divergence between the world we trained on and the world we serve. It also formalizes interfaces and responsibilities across traditionally isolated domains — data science, machine learning engineering, and systems operations [@amershi2019software] — through continuous retraining, A/B evaluation, graduated rollout, and standardized artifact tracking that makes every deployed model reproducible and auditable.
+This framing connects directly to the book's analytical foundations. If benchmarking provides the *sensors* for our system, MLOps is the complete *control system*. It closes the *Verification Gap*\index{Verification Gap!MLOps closure} (the recognition that ML testing is statistical, not deterministic, formalized in @sec-twelve-quantitative-invariants-0dd2) by continuously recalibrating against a changing world. MLOps operationalizes the *Degradation Equation*\index{Degradation Equation!operational implementation} (formalized in @sec-twelve-quantitative-invariants-0dd2): accuracy decay is not a failure of the code, but an inevitable consequence of the distributional divergence between the world we trained on and the world we serve. It also formalizes interfaces and responsibilities across traditionally isolated domains — data science, machine learning engineering, and systems operations [@amershi2019software] — through continuous retraining, A/B evaluation, graduated rollout, and standardized artifact tracking that makes every deployed model reproducible and auditable.
 
 This chapter focuses on what we term *single-model operations*: the practices required to deploy, monitor, and maintain one ML system in production. This operational unit requires a dedicated term. We define the *ML Node*, a complete system comprising data pipelines, feature computation, model training, serving infrastructure, and monitoring for a single machine learning application. Platform operations at larger scale (managing hundreds of models, cross-model dependencies, multi-region coordination, and organization-wide ML platform engineering) constitute advanced topics that build on these single-model foundations.
 
@@ -762,23 +770,17 @@ Reliable machine learning systems depend on structured, scalable, and repeatable
 
 \index{Data Management!technical debt prevention}The technical debt patterns we examined stem largely from poor data management\index{Data Management!MLOps lifecycle}: unversioned datasets create boundary erosion, inconsistent feature computation causes correction cascades, and undocumented data dependencies breed hidden consumers. Data management infrastructure directly addresses these root causes. Building on the data engineering foundations from @sec-data-engineering, data collection, preprocessing, and feature transformation become formalized operational processes. Where data engineering focuses on single-pipeline correctness, MLOps data management emphasizes cross-pipeline consistency, ensuring that training and serving compute identical features. Data management thus extends beyond initial preparation to encompass the continuous handling of data artifacts throughout the ML system lifecycle.
 
-Central to this foundation is dataset versioning that enables reproducible model development by tracking data evolution. @sec-ml-operations-versioning-lineage-b1cf examines implementation details including Git integration, metadata tracking, and lineage preservation. Tools such as DVC (Data Version Control)\index{DVC (Data Version Control)!dataset versioning}\index{Data Versioning!DVC tool} [@dvc] enable teams to version large datasets alongside code repositories managed by Git [@git], ensuring data lineage is preserved and experiments are reproducible.
-
-This versioning foundation enables more sophisticated capabilities. Supervised learning pipelines, for instance, require consistent annotation workflows. Labeling tools such as Label Studio [@label_studio] support scalable, team-based annotation with integrated audit trails and version histories. These capabilities are essential in production settings, where labeling conventions evolve over time or require refinement across multiple iterations of a project.
+Three principles organize the infrastructure that addresses these root causes: *consistency*, *freshness*, and *quality*. Each principle motivates specific tooling rather than the reverse.
 
 {{< margin-video "https://www.youtube.com/watch?v=gz-44N3MMOA&list=PLkDaE6sCZn6GMoA0wbpJLi3t34Gd8l0aK&index=33" "Data Pipelines" "MIT 6.S191" >}}
 
-Beyond annotation workflows, operational environments require data storage supporting secure, scalable, and collaborative access. Cloud-based object storage systems such as [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage) offer durability and fine-grained access control for managing both raw and processed data artifacts.
+**Data consistency** requires that every artifact influencing model behavior, from raw datasets to engineered features, is versioned and reproducible. Without versioning, teams cannot trace which data produced which model, making debugging and rollback impossible. Dataset versioning tools such as DVC (Data Version Control)\index{DVC (Data Version Control)!dataset versioning}\index{Data Versioning!DVC tool} [@dvc] enable teams to version large datasets alongside code repositories managed by Git [@git], while cloud-based object storage systems such as [Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage) provide the durable, access-controlled backend for both raw and processed artifacts. @sec-ml-operations-versioning-lineage-b1cf examines implementation details including Git integration, metadata tracking, and lineage preservation. At the feature level, the **feature store**\index{Feature Store!Uber Michelangelo origin}, a concept pioneered by Uber's Michelangelo platform team in 2017, enforces consistency by computing features once and serving them identically to both training and serving pipelines. The team coined the term after realizing that feature engineering was duplicated across hundreds of ML models, and their solution became the template that inspired Feast, Tecton, and dozens of other platforms. @sec-ml-operations-feature-stores-c01c details implementation patterns for training-serving consistency.
 
-Building on this storage foundation, MLOps teams construct automated data pipelines\index{Data Pipelines!automated workflows} to transition from raw data to analysis- or inference-ready formats. These pipelines perform structured tasks such as data ingestion, schema validation, deduplication, transformation, and loading. Orchestration tools including Apache Airflow\index{Workflow Orchestration!pipeline automation} [@apache_airflow], Prefect [@prefect], and dbt [@dbt] are commonly used to define and manage these workflows. When managed as code, pipelines support versioning, modularity, and integration with CI/CD systems.
+**Data freshness** ensures that models train and serve on current data rather than stale snapshots. Automated data pipelines\index{Data Pipelines!automated workflows} maintain freshness by continuously transforming raw data into analysis-ready formats through structured stages: ingestion, schema validation, deduplication, transformation, and loading. Orchestration tools including Apache Airflow\index{Workflow Orchestration!pipeline automation} [@apache_airflow], Prefect [@prefect], and dbt [@dbt] define and manage these workflows. When managed as code, pipelines support versioning, modularity, and integration with CI/CD systems, so that data flows remain synchronized with evolving model requirements.
 
-As these automated pipelines scale across organizations, they encounter the challenge of feature management at scale. An increasingly important element of modern data infrastructure is the **feature store**\index{Feature Store!Uber Michelangelo origin}, a concept pioneered by Uber's Michelangelo platform team in 2017. The team coined the term after realizing that feature engineering was duplicated across hundreds of ML models. Their solution, a centralized "feature store," became the template that inspired Feast, Tecton, and dozens of other platforms.
+**Data quality** governs whether the data reaching models is accurate, complete, and consistently labeled. In supervised learning pipelines, labeling quality directly determines model ceilings. Labeling tools such as Label Studio [@label_studio] support scalable, team-based annotation with integrated audit trails and version histories, capabilities that become essential when labeling conventions evolve over time or require refinement across multiple project iterations.
 
-Feature stores centralize engineered features for reuse across models and teams. @sec-ml-operations-feature-stores-c01c details implementation patterns for training-serving consistency.
-
-To illustrate these concepts in practice, consider a predictive maintenance application in an industrial setting. A continuous stream of sensor data is ingested and joined with historical maintenance logs through a scheduled pipeline managed in Airflow. The resulting features, including rolling averages and statistical aggregates, are stored in a feature store for both retraining and low-latency inference. This pipeline is versioned, monitored, and integrated with the model registry, enabling full traceability from data to deployed model predictions.
-
-Data management establishes the operational backbone enabling model reproducibility, auditability, and sustained deployment at scale, making feature stores a critical infrastructure component.
+To illustrate how these three principles reinforce each other in practice, consider a predictive maintenance application in an industrial setting. A continuous stream of sensor data is ingested and joined with historical maintenance logs through a scheduled pipeline managed in Airflow (*freshness*). The resulting features, including rolling averages and statistical aggregates, are stored in a feature store for both retraining and low-latency inference (*consistency*). The entire pipeline is versioned, monitored, and integrated with the model registry (*quality*), enabling full traceability from data to deployed model predictions. Data management, organized around these three principles, establishes the operational backbone for model reproducibility, auditability, and sustained deployment at scale.
 
 #### Feature Stores {#sec-ml-operations-feature-stores-c01c}
 
@@ -1196,61 +1198,30 @@ class SilentFailureCost:
     """
 
     # ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
-    ttd_manual_days = 30 # monthly review
-    ttd_auto_days = 1 # daily monitoring
-    incidents_per_year = 4
+    annual_revenue = 50_000_000   # $50M/year recommendation engine
+    quality_drop = 0.05           # 5% conversion rate degradation
+    days_manual = 28              # monthly review cycle (~4 weeks)
+    days_auto = 1                 # daily automated monitoring
+    incidents_per_year = 4        # typical for high-drift domains
 
     # ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
-    daily_revenue = annual_revenue / DAYS_PER_YEAR
-    loss_per_day = daily_revenue * quality_drop
-
-    total_loss_manual = loss_per_day * ttd_manual_days * incidents_per_year
-    total_loss_auto = loss_per_day * ttd_auto_days * incidents_per_year
-    annual_savings = total_loss_manual - total_loss_auto
-
-    # ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
-    check(annual_savings >= 500_000, f"Annual savings (${annual_savings:,.0f}) are too low to justify MLOps investment.")
-
-    # ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
-    annual_revenue_str = fmt(annual_revenue / MILLION, precision=0, commas=False) + "M"
-    loss_manual_str = fmt(total_loss_manual, precision=0, commas=True)
-    loss_auto_str = fmt(total_loss_auto, precision=0, commas=True)
-    annual_savings_str = fmt(annual_savings, precision=0, commas=True)
-    ttd_manual_str = f"{ttd_manual_days}"
-    ttd_auto_str = f"{ttd_auto_days}"
-    incidents_str = f"{incidents_per_year}"
-
-# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
-annual_revenue_str = SilentFailureCost.annual_revenue_str
-loss_manual_str = SilentFailureCost.loss_manual_str
-loss_auto_str = SilentFailureCost.loss_auto_str
-annual_savings_str = SilentFailureCost.annual_savings_str
-ttd_manual_str = SilentFailureCost.ttd_manual_str
-ttd_auto_str = SilentFailureCost.ttd_auto_str
-incidents_str = SilentFailureCost.incidents_str
-
-    days_manual = 28
-    days_auto = 1
-    incidents_year = 5
-
-    # ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
-    # Loss = Rev * Drop * (Days / DAYS_PER_YEAR)
+    # Per-incident loss = Revenue × Quality Drop × (Detection Days / 365)
     loss_manual = annual_revenue * quality_drop * (days_manual / DAYS_PER_YEAR)
     loss_auto = annual_revenue * quality_drop * (days_auto / DAYS_PER_YEAR)
 
     savings_per_incident = loss_manual - loss_auto
-    annual_savings = savings_per_incident * incidents_year
+    annual_savings = savings_per_incident * incidents_per_year
 
     # ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
-    check(annual_savings >= 500_000, f"Annual savings (${annual_savings:,.0f}) is too low to justify MLOps investment.")
+    check(annual_savings >= 500_000, f"Annual savings (${annual_savings:,.0f}) too low to justify MLOps investment.")
 
     # ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
-    annual_revenue_str = f"{annual_revenue // 1_000_000:.0f}M"
+    annual_revenue_str = f"{int(annual_revenue // 1_000_000)}M"
     quality_drop_pct_str = f"{int(quality_drop * 100)}"
     quality_drop_str = f"{quality_drop}"
     manual_detect_days_str = f"{days_manual}"
     auto_detect_days_str = f"{days_auto}"
-    incidents_per_year_str = f"{incidents_year}"
+    incidents_per_year_str = f"{incidents_per_year}"
 
     loss_manual_str = fmt(loss_manual, precision=0, commas=True)
     loss_auto_str = fmt(loss_auto, precision=0, commas=True)
@@ -1436,7 +1407,7 @@ The following framework formalizes the derivation used in the calculation above,
 
 ###### The Staleness Cost Function {.unnumbered}
 
-\index{Staleness Cost!accuracy decay model}Model accuracy typically degrades over time due to distribution drift. While the mechanism of this degradation is the distributional divergence $D(P_t \| P_0)$ described by the **Degradation Equation** (formalized in @sec--twelve-quantitative-invariants-0dd2), for economic planning we can model the *observable impact* over time as an exponential decay process. Note that $\lambda$ represents sensitivity to distributional divergence; here we repurpose it as a temporal decay rate, assuming drift accumulates steadily over time. The exponential model is a simplification that enables closed-form economic analysis. Let $A(t)$ represent accuracy at time $t$ since last training, and $A_0$ represent initial accuracy. @eq-accuracy-decay captures this degradation, where the rate $\lambda$ depends on domain volatility:
+\index{Staleness Cost!accuracy decay model}Model accuracy typically degrades over time due to distribution drift. While the mechanism of this degradation is the distributional divergence $D(P_t \| P_0)$ described by the **Degradation Equation** (formalized in @sec-twelve-quantitative-invariants-0dd2), for economic planning we can model the *observable impact* over time as an exponential decay process. Note that $\lambda$ represents sensitivity to distributional divergence; here we repurpose it as a temporal decay rate, assuming drift accumulates steadily over time. The exponential model is a simplification that enables closed-form economic analysis. Let $A(t)$ represent accuracy at time $t$ since last training, and $A_0$ represent initial accuracy. @eq-accuracy-decay captures this degradation, where the rate $\lambda$ depends on domain volatility:
 
 $$A(t) = A_0 \cdot e^{-\lambda t}$$ {#eq-accuracy-decay}
 
@@ -1946,9 +1917,9 @@ Infrastructure as Code addresses how to provision resources; the challenge remai
 
 The optimization challenge intensifies when considering the interdependencies between training frequency, model complexity, and serving infrastructure costs. Effective resource management requires holistic approaches that model the entire system rather than optimizing individual components in isolation. Such approaches must account for data pipeline throughput, model retraining schedules, and serving capacity planning.
 
-Hardware-aware resource optimization emerges as a critical operational discipline that bridges infrastructure efficiency with model performance. Production MLOps teams must establish utilization targets that balance cost efficiency against operational reliability: GPU utilization should consistently exceed 80% for batch training workloads to justify hardware costs, while serving workloads require sustained utilization above 60% to maintain economically viable inference operations. Memory bandwidth utilization patterns become equally important, as underutilized memory interfaces indicate suboptimal data pipeline configurations that can degrade training throughput by 30–50%.
+Hardware-aware resource optimization emerges as a critical operational discipline that bridges infrastructure efficiency with model performance. Production MLOps teams must establish utilization targets that balance cost efficiency against operational reliability: batch training workloads should maintain high GPU utilization to justify hardware costs, while serving workloads require sufficient sustained utilization to maintain economically viable inference operations. Memory bandwidth utilization patterns become equally important, as underutilized memory interfaces indicate suboptimal data pipeline configurations that can substantially degrade training throughput.
 
-Operational resource allocation extends beyond simple utilization metrics to encompass power budget management across mixed workloads. Production deployments typically allocate 60–70% of power budgets to training operations during development cycles, reserving 30–40% for sustained inference workloads. This allocation shifts dynamically based on business priorities: recommendation systems might reallocate power toward inference during peak traffic periods, while research environments prioritize training resource availability. Thermal management considerations become operational constraints rather than hardware design concerns, as sustained high-utilization workloads must be scheduled with cooling capacity limitations and thermal throttling thresholds that can impact SLA compliance.
+Operational resource allocation extends beyond simple utilization metrics to encompass power budget management across mixed workloads. Production deployments typically allocate the majority of power budgets to training operations during development cycles, reserving the remainder for sustained inference workloads. This allocation shifts dynamically based on business priorities: recommendation systems might reallocate power toward inference during peak traffic periods, while research environments prioritize training resource availability. Thermal management considerations become operational constraints rather than hardware design concerns, as sustained high-utilization workloads must be scheduled with cooling capacity limitations and thermal throttling thresholds that can impact SLA compliance.
 
 ##### Hardware Utilization Patterns {#sec-ml-operations-hardware-utilization-patterns-5c10}
 
@@ -2678,41 +2649,13 @@ Technical monitoring capabilities alone do not ensure operational success. The m
 
 ### Governance and Team Coordination {#sec-ml-operations-model-governance-team-coordination-d86f}
 
-Technical monitoring capabilities are necessary but insufficient. Production ML also requires governance (policies ensuring models operate transparently and fairly), cross-functional collaboration (bridging data science, engineering, and business teams), and stakeholder communication (translating probabilistic system behavior into business decisions).
+\index{Model Governance!proactive policies}On-call practices address operational emergencies, but production ML also requires proactive governance\index{Model Governance!transparency and compliance} and cross-functional collaboration\index{Cross-Functional Collaboration!ML teams}. Governance encompasses the policies and practices ensuring that ML models operate transparently, fairly, and in compliance with ethical and regulatory standards. Without it, deployed models may produce biased or opaque decisions, creating legal, reputational, and societal risks. Governance focuses on three core objectives: transparency\index{Transparency!governance objective} (interpretable, auditable models), fairness\index{Fairness!governance objective} (equitable treatment across user groups), and compliance\index{Compliance!governance objective} (alignment with legal and organizational policies). The specific interpretability methods, fairness metrics, and bias detection techniques that operationalize these objectives are examined in @sec-responsible-engineering; MLOps provides the infrastructure to enforce these checks continuously throughout the deployment lifecycle.
 
-#### Model Governance {#sec-ml-operations-model-governance-363c}
+What makes ML governance uniquely challenging is its lifecycle scope. Unlike traditional software compliance, which can be verified at release time, ML governance must span development, deployment, and operation. During development, teams must document model assumptions and training data provenance. At deployment, pre-release audits evaluate fairness and robustness. Post-deployment, the monitoring systems discussed in the previous section must track not only performance degradation but also fairness drift, where concept drift disproportionately affects specific user subgroups. Governance policies encoded into automated pipelines ensure that these checks are applied consistently rather than relying on ad hoc human review.
 
-\index{Model Governance!proactive policies}On-call practices address operational emergencies, but production ML also requires proactive governance. Governance\index{Model Governance!transparency and compliance} encompasses the policies, practices, and tools ensuring that ML models operate transparently, fairly, and in compliance with ethical and regulatory standards. Without proper governance, deployed models may produce biased or opaque decisions, creating significant legal, reputational, and societal risks.
+Governance establishes policies, but cross-functional collaboration implements them. Machine learning systems are developed and maintained by multidisciplinary teams, and the boundaries between roles create the most failure-prone points in the entire lifecycle. Shared experiment tracking, model registries, and standardized documentation provide the connective tissue that enables reproducibility and eases handoff between specialists. Equally important is shared understanding of data semantics: glossaries, schema references, and lineage documentation ensure that all stakeholders interpret features, labels, and statistics consistently.
 
-Governance begins during model development, where teams implement techniques to increase transparency and explainability\index{Explainability!governance requirement}. Methods such as SHAP\index{SHAP!model interpretability} [@lundberg2017unified][^fn-shap-adoption] and LIME\index{LIME!local explanations} [@lime_github][^fn-lime-explanations] offer post hoc explanations of model predictions by identifying which input features were most influential in a particular decision. These interpretability techniques complement security measures that address how to protect both model integrity and data privacy in production environments. They allow auditors, developers, and non-technical stakeholders to better understand how and why a model behaves the way it does.
-
-[^fn-shap-adoption]: **SHAP (SHapley Additive exPlanations)**: Based on Shapley values from game theory (1953, Nobel Prize 2012), adapted for ML by Lundberg and Lee in 2017. Treats features as "players" and prediction impact as the "payoff." Adds 10--500 ms latency per prediction. See @sec-responsible-engineering for detailed treatment of SHAP and LIME.
-
-[^fn-lime-explanations]: **LIME (Local Interpretable Model-agnostic Explanations)**: Perturbation-based explanation method introduced by Ribeiro, Singh, and Guestrin in 2016. LIME generates explanations by fitting a local linear model around a prediction, making it faster than SHAP (50--200 ms vs. 100--500 ms) but less stable. See @sec-responsible-engineering for a detailed comparison.
-
-In addition to interpretability, fairness is a central concern in governance. Governance encompasses fairness and bias monitoring\index{Fairness Monitoring!governance requirement} to ensure equitable treatment across user groups. The specific fairness metrics and bias detection techniques are examined in @sec-responsible-engineering; MLOps provides the infrastructure to implement these checks throughout the deployment lifecycle, including pre-deployment audits that evaluate fairness, robustness, and overall model behavior before a system is put into production.
-
-Governance also extends into the post-deployment phase. As introduced in the previous section on monitoring, teams must track for concept drift, where the statistical relationships between features and labels evolve over time. Such drift can undermine the fairness or accuracy of a model, particularly if the shift disproportionately affects a specific subgroup. By analyzing logs and user feedback, teams can identify recurring failure modes, unexplained model outputs, or emerging disparities in treatment across user segments.
-
-Supporting this lifecycle approach to governance are platforms and toolkits that integrate governance functions into the broader MLOps stack. For example, Watson OpenScale [@watson_openscale] provides built-in modules for explainability, bias detection, and monitoring. These tools allow governance policies to be encoded as part of automated pipelines, ensuring that checks are consistently applied throughout development, evaluation, and production.
-
-Governance focuses on three core objectives: transparency\index{Transparency!governance objective} (interpretable, auditable models), fairness\index{Fairness!governance objective} (equitable treatment across user groups), and compliance\index{Compliance!governance objective} (alignment with legal and organizational policies). Embedding governance throughout the MLOps lifecycle transforms machine learning into a trustworthy system serving societal and organizational goals.
-
-#### Cross-Functional Collaboration {#sec-ml-operations-crossfunctional-collaboration-9f0d}
-
-Governance frameworks establish policies; cross-functional collaboration\index{Cross-Functional Collaboration!ML teams} implements them. Machine learning systems are developed and maintained by multidisciplinary teams: data scientists, ML engineers, software developers, infrastructure specialists, product managers, and compliance officers. MLOps fosters cross-functional integration by introducing shared tools, processes, and artifacts that promote transparency and coordination across the ML lifecycle.
-
-Collaboration begins with consistent tracking of experiments\index{Experiment Tracking!collaborative ML development}, model versions, and metadata. Tools such as [MLflow](https://mlflow.org/) [@zaharia2018accelerating] provide a structured environment for logging experiments, capturing parameters, recording evaluation metrics, and managing trained models through a centralized registry. This registry serves as a shared reference point for all team members, enabling reproducibility and easing handoff between roles. Integration with version control systems such as GitHub [@github] and GitLab [@gitlab] further streamlines collaboration by linking code changes with model updates and pipeline triggers.
-
-In addition to tracking infrastructure, teams benefit from platforms that support exploratory collaboration. Weights & Biases [@wandb] is one such platform that allows data scientists to visualize experiment metrics, compare training runs, and share insights with peers. Features such as live dashboards and experiment timelines facilitate discussion and decision-making around model improvements, hyperparameter tuning, or dataset refinements. These collaborative environments reduce friction in model development by making results interpretable and reproducible across the team.
-
-Beyond model tracking, collaboration also depends on shared understanding of data semantics and usage. Establishing common data contexts, by means of glossaries, data dictionaries, schema references, and lineage documentation, ensures that all stakeholders interpret features, labels, and statistics consistently. This is particularly important in large organizations, where data pipelines may evolve independently across teams or departments.
-
-For example, a data scientist working on an anomaly detection model may use Weights & Biases to log experiment results and visualize performance trends. These insights are shared with the broader team to inform feature engineering decisions. Once the model reaches an acceptable performance threshold, it is registered in MLflow along with its metadata and training lineage. This allows an ML engineer to pick up the model for deployment without ambiguity about its provenance or configuration.
-
-By integrating collaborative tools, standardized documentation, and transparent experiment tracking, MLOps removes communication barriers that have traditionally slowed down ML workflows. It enables distributed teams to operate cohesively, accelerating iteration cycles and improving the reliability of deployed systems.
-
-\index{ML Team Roles!organizational structure}Effective MLOps requires clear role definitions that align expertise with responsibilities. While titles vary across organizations, five core roles emerge consistently in ML teams. @tbl-ml-roles-matrix maps these roles to their primary responsibilities:
+\index{ML Team Roles!organizational structure}While titles vary across organizations, five core roles emerge consistently in ML teams. @tbl-ml-roles-matrix maps these roles to their primary responsibilities:
 
 | **Role**              | **Primary Focus**                                                 | **Key Deliverables**                                        | **Collaboration Points**                                                     |
 |:----------------------|:------------------------------------------------------------------|:------------------------------------------------------------|:-----------------------------------------------------------------------------|
@@ -2724,15 +2667,7 @@ By integrating collaborative tools, standardized documentation, and transparent
 
 : **ML Team Roles Matrix.** Clear role boundaries prevent gaps and overlaps. Data Scientists focus on model quality while ML Engineers handle productionization. Data Engineers own data pipelines while Platform Engineers own MLOps tooling. SREs ensure overall system reliability. {#tbl-ml-roles-matrix}
 
-Clear role definitions matter most at handoff points, where work transitions between specialists. The most failure-prone handoff occurs between Data Scientists and ML Engineers—the infamous "notebook to production" gap\index{Notebook to Production!handoff challenges}. A model that performs well in a Jupyter notebook may fail in production due to undocumented preprocessing steps, hardcoded file paths, or dependencies on the Data Scientist's local environment. Organizations mitigate this gap through standardized model interfaces that define expected inputs and outputs, required documentation that captures feature engineering logic and training procedures, and reproducibility requirements that must be verified before handoff. Without these guardrails, ML Engineers spend weeks reverse-engineering notebooks rather than productionizing models.
-
-The handoff from ML Engineers to SREs presents different challenges. Before SREs accept operational responsibility for a model, production readiness reviews\index{Production Readiness Review!handoff criteria} should verify that monitoring dashboards exist, alerting rules are configured with appropriate thresholds, runbooks document common failure scenarios and their resolutions, and rollback procedures have been tested. SREs cannot effectively support systems they do not understand, and ML Engineers cannot provide 24/7 coverage indefinitely. Formal handoff criteria protect both teams.
-
-Data Engineers hand off to the broader ML team through feature contracts\index{Feature Contracts!schema specification}, which are formal specifications of schema, freshness SLOs, and quality guarantees. When a Data Engineer needs to modify a feature pipeline, the contract identifies which downstream models depend on the affected features and what coordination is required. Breaking changes demand explicit communication rather than silent updates that surface as mysterious model degradation weeks later.
-
-Some organizations, particularly startups, attempt to avoid these handoff complexities by expecting individuals to span all roles: the "full-stack ML engineer" who handles everything from data pipelines to model serving. While appealing for small teams seeking to minimize coordination overhead, this approach creates expertise gaps in critical areas like security and reliability, burnout from constant context-switching between qualitatively different types of work, and insufficient depth in any single area to handle complex problems. A single engineer can prototype across roles during early experimentation, but production systems benefit from specialization. Small teams that cannot afford full specialization should explicitly acknowledge their coverage gaps and prioritize based on risk, perhaps accepting weaker data infrastructure while investing heavily in serving reliability for customer-facing applications.
-
-Team structure naturally evolves with organizational maturity and model portfolio size. Organizations operating one to three models typically rely on generalist ML Engineers who cover most responsibilities; Data Scientists may even deploy their own models with minimal handoff. As the portfolio grows to three to ten models, specialization becomes necessary: dedicated Data Engineers own feature pipelines while ML Engineers focus on serving infrastructure. Beyond ten models, a platform team typically emerges to provide shared infrastructure (experiment tracking, model registries, deployment pipelines) that individual product teams consume. Large organizations often embed ML Engineers within product teams while maintaining a central platform that provides common tooling and best practices.
+Clear role definitions matter most at handoff points\index{Notebook to Production!handoff challenges}, where work transitions between specialists. The most failure-prone handoff occurs between Data Scientists and ML Engineers: a model that performs well in a Jupyter notebook may fail in production due to undocumented preprocessing steps, hardcoded file paths, or environment dependencies. Similarly, the handoff from ML Engineers to SREs\index{Production Readiness Review!handoff criteria} requires verified monitoring dashboards, configured alerting rules, documented runbooks, and tested rollback procedures. Data Engineers hand off to the broader ML team through feature contracts\index{Feature Contracts!schema specification}, formal specifications of schema, freshness SLOs, and quality guarantees that prevent silent pipeline changes from surfacing as mysterious model degradation weeks later. Organizations mitigate these handoff risks through standardized model interfaces, required documentation, and reproducibility requirements that must be verified before each transition.
 
 Effective MLOps extends beyond internal team coordination, however, to encompass the broader communication challenges that arise when technical teams interface with business stakeholders.
 
@@ -3660,11 +3595,11 @@ The following fallacies and pitfalls capture common errors that waste engineerin
 
 **Fallacy:** *Automated retraining ensures optimal model performance without human oversight.*
 
-\index{MLOps Fallacies!automated retraining sufficiency}Engineers assume automated pipelines handle all maintenance scenarios, yet automation cannot detect all failure modes. Automated retraining perpetuates biases in corrupted training data, triggers updates during peak traffic, or deploys models that pass validation but degrade edge cases. Industry postmortems indicate that 30–40% of P1 incidents (accuracy drops exceeding 10%) originate from automated retraining propagating upstream data quality issues. A news recommendation system retrained on weekend data exhibited 22% lower engagement because user behavior differs sharply across weekday versus weekend contexts. Organizations implementing human checkpoints at validation boundaries report 50–70% fewer production incidents than fully automated pipelines. Effective MLOps requires escalation protocols for anomalous validation results, manual approval for unusual metric patterns, and override capabilities when automation produces questionable outcomes.
+\index{MLOps Fallacies!automated retraining sufficiency}Engineers assume automated pipelines handle all maintenance scenarios, yet automation cannot detect all failure modes. Automated retraining perpetuates biases in corrupted training data, triggers updates during peak traffic, or deploys models that pass validation but degrade edge cases. Industry postmortems suggest that a substantial fraction of P1 incidents (accuracy drops exceeding 10%) originate from automated retraining propagating upstream data quality issues. A news recommendation system retrained on weekend data exhibited lower engagement because user behavior differs sharply across weekday versus weekend contexts. Organizations implementing human checkpoints at validation boundaries consistently report fewer production incidents than fully automated pipelines. Effective MLOps requires escalation protocols for anomalous validation results, manual approval for unusual metric patterns, and override capabilities when automation produces questionable outcomes.
 
 **Pitfall:** *Focusing on technical infrastructure while neglecting organizational and process alignment.*
 
-\index{MLOps Pitfalls!organizational neglect}Organizations invest in MLOps platforms expecting tooling to solve deployment problems, but sophisticated infrastructure fails without cultural transformation. MLOps demands coordination between data scientists optimizing for accuracy, engineers prioritizing latency, and business stakeholders focused on impact. A retail company deployed feature stores and model registries but maintained quarterly deployment frequency because data scientists and engineers operated in isolation. Industry surveys indicate that unified ML teams deploy 4–8 $\times$ more frequently (weekly or daily versus quarterly) compared to siloed organizations. Time-to-production for new models averages 3–6 months in fragmented organizations but drops to 2–4 weeks with integrated teams. Successful MLOps requires cross-functional teams with unified objectives, shared on-call rotations building empathy across roles, and incentive structures rewarding production reliability alongside model performance.
+\index{MLOps Pitfalls!organizational neglect}Organizations invest in MLOps platforms expecting tooling to solve deployment problems, but sophisticated infrastructure fails without cultural transformation. MLOps demands coordination between data scientists optimizing for accuracy, engineers prioritizing latency, and business stakeholders focused on impact. A retail company deployed feature stores and model registries but maintained quarterly deployment frequency because data scientists and engineers operated in isolation. Practitioners commonly observe that unified ML teams deploy significantly more frequently (weekly or daily versus quarterly) compared to siloed organizations. Time-to-production for new models can stretch to months in fragmented organizations but drops considerably with integrated teams. Successful MLOps requires cross-functional teams with unified objectives, shared on-call rotations building empathy across roles, and incentive structures rewarding production reliability alongside model performance.
 
 **Fallacy:** *Training and serving environments automatically remain consistent once pipelines are established.*
 

book/quarto/contents/vol1/ml_systems/ml_systems.qmd

@@ -47,7 +47,7 @@ The defining insight of ML systems engineering is that constraints drive archite
 - Distinguish the four **deployment paradigms** (Cloud, Edge, Mobile, TinyML) by their operational characteristics and quantitative trade-offs.
 - Apply the **decision framework** to select deployment paradigms based on privacy, latency, computational, and cost requirements.
 - Analyze hybrid integration patterns to determine which combinations address specific system constraints.
-- Evaluate deployment decisions by identifying common fallacies (including **Amdahl's Law** limits on system speedup) and assessing alignment between architecture and requirements.
+- Evaluate deployment decisions by identifying common fallacies (including Amdahl's Law limits on system speedup) and assessing alignment between architecture and requirements.
 - Identify the universal principles (data pipelines, resource management, system architecture) that apply across deployment paradigms and explain why optimization techniques transfer between scales.
 
 :::
@@ -1870,7 +1870,7 @@ The bandwidth calculation above reveals why edge processing is mandatory for hig
 
 \index{Edge ML!distributed processing} \index{Edge ML!deployment challenges}
 \index{Edge ML!privacy benefits}
-Edge ML spans wearables, industrial sensors, and smart home appliances that process data locally[^fn-iot-growth] without depending on central servers. @fig-energy-per-inference quantifies the physical imperative: full-system energy per inference spans eight orders of magnitude across deployment paradigms, from ~10 µJ for a TinyML keyword spotter to ~1 kJ for a cloud LLM query. This 100,000,000 $\times$ gap is not an engineering shortcoming to be optimized away—it reflects the irreducible costs of data movement, cooling, and network overhead that separate deployment tiers. Memory bandwidth at 25-100 GB/s enables edge models requiring 100 MB-1 GB parameters. The optimization techniques covered in @sec-model-compression achieve 2–4 $\times$ speedup compared to cloud models. Local processing also generates substantial bandwidth savings: processing 1000 camera feeds locally avoids 1 Gbps uplink costs and reduces cloud expenses by $10,000-100,000 annually.
+Edge ML spans wearables, industrial sensors, and smart home appliances that process data locally[^fn-iot-growth] without depending on central servers. @fig-energy-per-inference quantifies the physical imperative: full-system energy per inference spans eight orders of magnitude across deployment paradigms, from ~10 µJ for a TinyML keyword spotter to ~1 kJ for a cloud LLM query. This 100,000,000 $\times$ gap is not an engineering shortcoming to be optimized away; it reflects the irreducible costs of data movement, cooling, and network overhead that separate deployment tiers. Because edge devices operate within tight power envelopes, their memory bandwidth of 25--100 GB/s constrains deployable models to 100 MB--1 GB of parameters. This constraint, in turn, motivates the optimization techniques covered in @sec-model-compression, which achieve 2--4 $\times$ speedup by compressing models to fit within these hardware budgets. The payoff extends beyond compute: processing 1000 camera feeds locally avoids 1 Gbps uplink costs because raw data never leaves the device, reducing cloud expenses by \$10,000--100,000 annually.
 
 [^fn-iot-growth]: **IoT Device Growth**: Explosive growth from 8.4B connected devices (2017) to projected 25.4B by 2030 [@mckinsey2021iot]. Daily data generation approaches 2.5 quintillion bytes, with 90% requiring real-time processing. Network bandwidth and cloud costs make edge processing economically essential; uploading raw sensor data would cost $10–100 per device monthly.
 
@@ -2429,7 +2429,7 @@ TinyML [@reddi2022widening] completes the deployment spectrum by pushing intelli
 \index{microcontroller development platforms!TinyML}
 Where mobile ML requires sophisticated hardware with gigabytes of memory and multi-core processors, TinyML operates on microcontrollers[^fn-microcontrollers-specs] with kilobytes of RAM and single-digit dollar price points [@banbury2021mlperftiny; @lin2020mcunet]. This radical constraint forces an entirely different approach to machine learning deployment, prioritizing ultra-low power consumption and minimal cost over computational sophistication. TinyML systems power applications such as predictive maintenance, environmental monitoring, and simple gesture recognition. The energy gap between TinyML and cloud inference spans six orders of magnitude[^fn-energy-efficiency]—a 1,000,000 $\times$ difference that drives entirely different system architectures and deployment models. This extraordinary efficiency enables operation for months or years on limited power sources such as coin-cell batteries[^fn-coin-cell], as exemplified by the device kits in @fig-TinyML-example. These systems deliver actionable insights in remote or disconnected environments where power, connectivity, and maintenance access are impractical.
 
-[^fn-microcontrollers-specs]: **Microcontrollers**: Single-chip computers with integrated CPU, memory, and peripherals, typically operating at 1–100 MHz with 32 KiB–2 MiB RAM. Arduino Uno uses an ATmega328P with 32 KiB flash and 2 KiB RAM, while ESP32 provides WiFi capability with 520 KiB RAM, still thousands of times less than a smartphone.
+[^fn-microcontrollers-specs]: **Microcontrollers**: Single-chip computers with integrated CPU, memory, and peripherals, typically operating at 1–100 MHz with 32 KB–2 MB RAM. Arduino Uno uses an ATmega328P with 32 KB flash and 2 KB RAM, while ESP32 provides WiFi capability with 520 KB RAM, still thousands of times less than a smartphone.
 
 [^fn-energy-efficiency]: **Energy Efficiency in TinyML**: Ultra-low power enables decade-long deployment in remote locations. ARM Cortex-M0+ consumes <1 µW in sleep, 100–300 µW/MHz active. Specialized accelerators (Syntiant NDP, MAX78000) achieve <1 µJ per inference.
 
@@ -2614,11 +2614,11 @@ above=1of $(B2.north east)!0.5!(B3.north west)$](B0){TinyML};
 
 ### TinyML Advantages and Operational Trade-offs {#sec-ml-systems-tinyml-advantages-operational-tradeoffs-2d40}
 
-\index{TinyML!resource constraints} \index{TinyML!model compression} \index{microcontrollers!ML deployment}TinyML operates at hardware extremes. Compared to cloud systems, TinyML deployments provide $10^4$ to $10^5$ times less memory, with power budgets in the milliwatt range. These strict limitations enable months or years of autonomous operation[^fn-on-device-training] but demand specialized algorithms and careful systems co-design. Devices range from palm-sized developer kits to millimeter-scale chips[^fn-device-size], enabling ubiquitous sensing in contexts where networking, power, or maintenance are costly. Representative developer kits include the Arduino Nano 33 BLE Sense (256 KiB RAM, 1 MiB flash, 20–40 mW) and ESP32-CAM (520 KiB RAM, 4 MiB flash, 50–250 mW).
+\index{TinyML!resource constraints} \index{TinyML!model compression} \index{microcontrollers!ML deployment}TinyML operates at hardware extremes. Compared to cloud systems, TinyML deployments provide $10^4$ to $10^5$ times less memory, with power budgets in the milliwatt range. These strict limitations enable months or years of autonomous operation[^fn-on-device-training] but demand specialized algorithms and careful systems co-design. Devices range from palm-sized developer kits to millimeter-scale chips[^fn-device-size], enabling ubiquitous sensing in contexts where networking, power, or maintenance are costly. Representative developer kits include the Arduino Nano 33 BLE Sense (256 KB RAM, 1 MB flash, 20–40 mW) and ESP32-CAM (520 KB RAM, 4 MB flash, 50–250 mW).
 
-[^fn-on-device-training]: **On-Device Training Constraints**: Microcontrollers (256KiB-2MiB RAM) cannot support full backpropagation through large networks. Alternatives include on-device fine-tuning of final layers, federated learning with local gradient computation, and TinyTL (memory-efficient training using <50KiB). Apple's on-device personalization adapts keyboard predictions without uploading typing data.
+[^fn-on-device-training]: **On-Device Training Constraints**: Microcontrollers (256 KB–2 MB RAM) cannot support full backpropagation through large networks. Alternatives include on-device fine-tuning of final layers, federated learning with local gradient computation, and TinyTL (memory-efficient training using <50 KB). Apple's on-device personalization adapts keyboard predictions without uploading typing data.
 
-[^fn-device-size]: **TinyML Device Scale**: ML-capable chips range from 5 $\times$ 5 mm (Syntiant NDP: 140 µW, 1 MiB SRAM) to full single-board computers (Coral Dev Board Mini: 40 $\times$ 48 mm, 4 TOPS). This 100 $\times$ size range reflects diverse deployment needs from implantable medical devices to industrial edge gateways processing multiple sensor streams simultaneously.
+[^fn-device-size]: **TinyML Device Scale**: ML-capable chips range from 5 $\times$ 5 mm (Syntiant NDP: 140 µW, 1 MB SRAM) to full single-board computers (Coral Dev Board Mini: 40 $\times$ 48 mm, 4 TOPS). This 100 $\times$ size range reflects diverse deployment needs from implantable medical devices to industrial edge gateways processing multiple sensor streams simultaneously.
 
 TinyML's extreme resource constraints paradoxically enable unique advantages. By avoiding network transmission entirely, TinyML devices achieve the lowest end-to-end latency in the deployment spectrum, enabling rapid local responses for sensing and control loops without communication overhead. This self-sufficiency also transforms the economics of large-scale deployments: when per-node costs drop to single-digit dollars, instrumenting an entire factory floor, farm, or building becomes financially viable in ways that edge or cloud alternatives cannot match. Energy efficiency compounds the economic case, enabling multi-year operation on small batteries or even indefinite operation through energy harvesting. Privacy benefits follow naturally from locality—raw data never leaves the device, reducing transmission risks and simplifying compliance—though on-device processing alone does not automatically provide formal privacy guarantees without additional security mechanisms.
 
@@ -3388,7 +3388,7 @@ cam_total_opt_str = fmt(cam_total_optimized_ms_value, precision=0, commas=False)
 
 **Fallacy:** *Model optimization translates linearly to system speedup.*
 
-**Amdahl's Law**\index{Amdahl's Law!speedup limits}\index{optimization!Amdahl's Law}[^fn-amdahls-law-systems] establishes hard limits that the Bottleneck Principle (@sec-ml-systems-bottleneck-principle-3514) formalizes: $Speedup_{overall} = \frac{1}{(1-p) + \frac{p}{s}}$ where $p$ is the fraction of work that can be improved and $s$ is the speedup of that fraction. Imagine you tap the shutter on a smartphone camera. The image passes through `{python} cam_isp_str` ms of signal processing (auto-exposure, white balance), `{python} cam_ml_str` ms of ML scene classification, and `{python} cam_post_str` ms of post-processing (tone mapping, HDR merge)---`{python} cam_total_str` ms total. You optimize the ML classifier to run 10 $\times$ faster (`{python} cam_ml_opt_str` ms instead of `{python} cam_ml_str` ms), but total time drops from `{python} cam_total_str` ms to `{python} cam_total_opt_str` ms---only `{python} cam_speedup_10x_str` $\times$ overall, not 10 $\times$. Even eliminating ML entirely ($s = \infty$) achieves only `{python} cam_speedup_inf_str` $\times$ speedup, because the remaining `{python} cam_non_ml_pct_str` percent of the pipeline is untouched. Effective optimization requires profiling the entire pipeline and addressing bottlenecks systematically, because system performance depends on the slowest unoptimized stage.
+Amdahl's Law\index{Amdahl's Law!speedup limits}\index{optimization!Amdahl's Law}[^fn-amdahls-law-systems] establishes hard limits that the Bottleneck Principle (@sec-ml-systems-bottleneck-principle-3514) formalizes: $Speedup_{overall} = \frac{1}{(1-p) + \frac{p}{s}}$ where $p$ is the fraction of work that can be improved and $s$ is the speedup of that fraction. Imagine you tap the shutter on a smartphone camera. The image passes through `{python} cam_isp_str` ms of signal processing (auto-exposure, white balance), `{python} cam_ml_str` ms of ML scene classification, and `{python} cam_post_str` ms of post-processing (tone mapping, HDR merge)---`{python} cam_total_str` ms total. You optimize the ML classifier to run 10 $\times$ faster (`{python} cam_ml_opt_str` ms instead of `{python} cam_ml_str` ms), but total time drops from `{python} cam_total_str` ms to `{python} cam_total_opt_str` ms---only `{python} cam_speedup_10x_str` $\times$ overall, not 10 $\times$. Even eliminating ML entirely ($s = \infty$) achieves only `{python} cam_speedup_inf_str` $\times$ speedup, because the remaining `{python} cam_non_ml_pct_str` percent of the pipeline is untouched. Effective optimization requires profiling the entire pipeline and addressing bottlenecks systematically, because system performance depends on the slowest unoptimized stage.
 
 [^fn-amdahls-law-systems]: **Amdahl's Law**: Formulated by Gene Amdahl in 1967 [@amdahl1967validity], this law quantifies theoretical speedup when only part of a system can be improved. The formula $S = 1/((1-p) + p/s)$ shows that even infinite speedup ($s \to \infty$) of the parallelizable fraction $p$ cannot exceed $1/(1-p)$. For ML systems, this explains why end-to-end optimization matters: a 10 $\times$ faster GPU yields minimal gains if data loading or preprocessing dominates total latency. See @sec-hardware-acceleration for a detailed treatment.
 
@@ -3396,6 +3396,10 @@ cam_total_opt_str = fmt(cam_total_optimized_ms_value, precision=0, commas=False)
 
 \index{scaling laws!data limitations}Three constraints limit data scaling benefits, as the workload archetypes in @sec-ml-systems-analyzing-workloads-cbb8 illustrate. First, model size limits what can be learned: a keyword spotting model with 250K parameters achieves 95% accuracy on 50K samples but only 96.5% on 1M samples, a 0.3% gain for 5 $\times$ more data, storage, and labeling cost. The model simply cannot represent more complex patterns. Second, data quality dominates quantity: 1M curated samples often outperform 100M noisy web-scraped samples, because mislabeled examples and misleading patterns degrade performance even as dataset size grows. Third, deployment distribution matters more than training scale: a model trained on 1B web images may perform worse on medical imaging than one trained on 100K domain-specific samples. Teams that maximize dataset scale without analyzing model capacity waste months of labeling effort for negligible accuracy gains.
 
+**Pitfall:** *Deploying the same model binary across all edge devices without hardware-specific optimization.*
+
+Teams build a single model artifact and deploy it identically to every target device, treating deployment as a packaging step rather than an optimization opportunity. In practice, hardware-specific optimizations yield 3--5 $\times$ efficiency gains that generic binaries cannot capture. An INT8 model running on a device with a dedicated Neural Processing Unit (NPU) achieves 3--4 $\times$ higher throughput per watt than the same model running in FP32 on a general-purpose CPU, because the NPU's fixed-function INT8 datapaths avoid the energy overhead of floating-point arithmetic. Similarly, operator fusion and memory layout tuning for a specific accelerator's cache hierarchy can halve inference latency without changing the model's weights. As the deployment paradigm analysis in @sec-ml-systems-deployment-paradigms-detailed-look-37c6 establishes, each paradigm imposes distinct hardware constraints; a model binary optimized for an Arm Cortex-A78 will underutilize the matrix acceleration units on a device equipped with an Arm Ethos-U NPU. Teams that skip per-target optimization either waste battery life on mobile devices or fail to meet latency SLAs on edge hardware, forcing costly post-deployment remediation.
+
 ## Summary {#sec-ml-systems-summary-d75c}
 
 This chapter answered a deceptively simple question: *why does the same model demand fundamentally different engineering on a phone versus a datacenter?* The answer is physics. Three immutable constraints—the speed of light, the power wall, and the memory wall—carve the deployment landscape into four distinct paradigms spanning nine orders of magnitude in power and memory. No single paradigm suffices for production systems; hybrid architectures that partition work across Cloud, Edge, Mobile, and TinyML tiers define the state of the art.

book/quarto/contents/vol1/ml_workflow/ml_workflow.qmd

@@ -81,9 +81,7 @@ The AI Triad—Data, Algorithm, Machine—names the components of every ML syste
 
 ## ML Lifecycle {#sec-ml-workflow-understanding-ml-lifecycle-ca87}
 
-The lifecycle stages that follow are not borrowed from software project management and adapted for ML. They are derived from the physical structure of the problem itself. Data has gravity (it resists movement across network boundaries) and drift, which means its statistical properties change over time. Models have computational cost governed by the Iron Law and statistical uncertainty that compounds through the pipeline. Deployment imposes latency, memory, and energy constraints that propagate *backward*, reshaping which architectures are worth training and which data is worth collecting. Each stage exists because the physics demands it.
-
-The previous chapters established *what* ML systems are made of and *where* they run. @sec-introduction introduced the AI Triad—Data, Algorithm, and Machine—as the core components of any ML system. @sec-ml-systems revealed the physical constraints that partition deployment into four paradigms: Cloud, Edge, Mobile, and TinyML. You now know the parts and the operating environments. The question this chapter answers is: *how do you orchestrate them?*
+The previous chapters established *what* ML systems are made of and *where* they run. @sec-introduction introduced the AI Triad as the core components of any ML system. @sec-ml-systems revealed the physical constraints that partition deployment into four paradigms: Cloud, Edge, Mobile, and TinyML. You now know the parts and the operating environments. The question this chapter answers is: *how do you orchestrate them?*
 
 Consider what happens without orchestration. Day 1: "Build a diagnostic model for rural clinics." Day 90: 95% accuracy on the test set. Day 120: 96% accuracy after a month of architecture tuning. Day 150: model handed to deployment engineers. Day 151: deployment engineers report the model requires 4 GB of memory. Day 152: someone checks the deployment target—tablets in mobile clinics with 512 MB available. Day 153: five months of work is discarded.
 
@@ -1405,19 +1403,23 @@ ML workflows introduce counterintuitive complexities that lead teams to apply fa
 
 **Fallacy:** *ML development can follow traditional software workflows without modification.*
 
-Engineers assume waterfall or standard agile processes will work for ML projects. In production, ML replaces deterministic specifications with probabilistic optimization, static behavior with dynamic adaptation, and isolated development with continuous feedback loops (@tbl-sw-ml-cycles). Traditional approaches treat requirements as fixed and testing as binary pass/fail, but ML systems require iterative experimentation where problem definitions evolve through exploration. Industry estimates suggest ML projects fail at 2–3 $\times$ the rate of traditional software, with 60–80% never reaching deployment. Projects forced into rigid phase gates miss the 4–8 iteration cycles that production-ready systems require. Organizations that adapt workflows to accommodate ML's experimental nature have reported 40–60% shorter time-to-deployment.
+Engineers assume waterfall or standard agile processes will work for ML projects. In production, ML replaces deterministic specifications with probabilistic optimization, static behavior with dynamic adaptation, and isolated development with continuous feedback loops (@tbl-sw-ml-cycles). Traditional approaches treat requirements as fixed and testing as binary pass/fail, but ML systems require iterative experimentation where problem definitions evolve through exploration. Industry estimates suggest ML projects fail at several $\times$ the rate of traditional software, with a majority never reaching deployment. Projects forced into rigid phase gates miss the 4–8 iteration cycles that production-ready systems require. Organizations that adapt workflows to accommodate ML's experimental nature have reported significantly shorter time-to-deployment.
 
 **Pitfall:** *Treating data preparation as a one-time preprocessing step.*
 
-Teams assume they can "finish" data preparation and move on to modeling. In production, data distributions shift continuously. The two-pipeline architecture in @fig-ml-lifecycle shows data and model pipelines running in parallel with continuous feedback, not sequentially. As @sec-ml-workflow-monitoring-maintenance-stage-e79a establishes, data quality decisions cascade through model training, validation, and deployment. Data quality issues account for 60–80% of production ML failures. Recommendation systems see 10–15% of features requiring updates monthly. Models degrade 5–10% within months as distributions shift, requiring emergency retraining that costs 3–5 $\times$ more than proactive monitoring. Organizations that build continuous data validation pipelines from the start detect drift within days rather than months, maintaining accuracy within 2–3% of development baselines.
+Teams assume they can "finish" data preparation and move on to modeling. In production, data distributions shift continuously. The two-pipeline architecture in @fig-ml-lifecycle shows data and model pipelines running in parallel with continuous feedback, not sequentially. As @sec-ml-workflow-monitoring-maintenance-stage-e79a establishes, data quality decisions cascade through model training, validation, and deployment. Data quality issues account for the majority of production ML failures. Recommendation systems see a significant fraction of features requiring updates monthly. Models degrade 5–10% within months as distributions shift, requiring emergency retraining that costs 3–5 $\times$ more than proactive monitoring. Organizations that build continuous data validation pipelines from the start detect drift within days rather than months, maintaining accuracy within 2–3% of development baselines.
 
 **Fallacy:** *Passing model evaluation means the system is ready for deployment.*
 
 Engineers treat the model development pipeline as the entire workflow, assuming strong evaluation metrics mean the system is complete. The two-pipeline architecture in @fig-ml-lifecycle exposes the blind spot: this mindset ignores half the lifecycle—data pipeline feedback loops, deployment integration, and production monitoring remain unaddressed. The diabetic retinopathy screening case study (@sec-ml-workflow-case-study-diabetic-retinopathy-screening-7d71) demonstrates the gap: the model passed evaluation but required additional validation to handle equipment variations across clinics, operator skill differences, and demographic diversity absent from curated development data. Evaluation metrics measure algorithm quality in isolation; production readiness requires verifying the complete system, including data freshness, preprocessing consistency, latency under load, and failure recovery. By the Constraint Propagation Principle, a deployment-stage discovery costs $2^{5-1} = 16\times$ the effort of catching it during evaluation design. Teams that equate strong evaluation metrics with deployment readiness consistently underestimate the integration effort by 3–5 $\times$.
 
+**Fallacy:** *More data always improves model performance.*
+
+Teams assume that scaling dataset size is the most reliable path to accuracy gains, treating data collection as a monotonically beneficial investment. In practice, returns diminish sharply after sufficient coverage of the target distribution. Doubling a dataset from 500K to 1M labeled examples often improves accuracy by less than 1 percentage point while doubling labeling costs, storage requirements, and preprocessing time. The feedback loops in @fig-ml-lifecycle illustrate why: model performance depends on the interaction between data quality, model capacity, and deployment conditions, not data volume alone. A 100K-example dataset with careful label quality control and balanced class representation routinely outperforms a 1M-example dataset with noisy labels and skewed distributions. The Data Collection and Preparation stage (@sec-ml-workflow-data-collection-preparation-stage-cf85) establishes that data quality decisions cascade through every subsequent stage. Teams that invest a second month of labeling budget in cleaning existing data rather than collecting new data report 2--3 $\times$ greater accuracy improvement per dollar spent.
+
 **Pitfall:** *Skipping validation stages to accelerate timelines.*
 
-Teams assume cutting validation time ships faster. In production, the multi-stage validation process exists because each stage catches different failure modes (@sec-ml-workflow-evaluation-validation-stage-b47d). Skipping shadow mode testing causes integration issues with 10–50 $\times$ latency spikes (@sec-ml-workflow-validation-production-conditions-a351). Bypassing canary deployment leads to incidents affecting millions of users. Post-deployment fixes cost 10–100 $\times$ more than catching issues during validation. Inadequate validation extends time-to-production by 2–5 months through unplanned remediation. A team that "saves" 2 weeks by skipping validation spends 6–8 weeks on emergency remediation. Organizations investing in systematic validation infrastructure reduce production incidents by 60–80% and achieve substantially higher first-deployment success rates.
+Teams assume cutting validation time ships faster. In production, the multi-stage validation process exists because each stage catches different failure modes (@sec-ml-workflow-evaluation-validation-stage-b47d). Skipping shadow mode testing causes integration issues with 10–50 $\times$ latency spikes (@sec-ml-workflow-validation-production-conditions-a351). Bypassing canary deployment leads to incidents affecting millions of users. Post-deployment fixes cost 10–100 $\times$ more than catching issues during validation. Inadequate validation extends time-to-production by 2–5 months through unplanned remediation. A team that "saves" 2 weeks by skipping validation spends 6–8 weeks on emergency remediation. Organizations investing in systematic validation infrastructure achieve substantially fewer production incidents and higher first-deployment success rates.
 
 **Pitfall:** *Deferring deployment paradigm selection until after model development.*
 

book/quarto/contents/vol1/model_serving/model_serving.qmd

@@ -3205,21 +3205,13 @@ On multi-socket servers[^fn-numa-serving]\index{NUMA!memory locality}\index{Thre
 
 CPUs often outperform GPUs at batch size 1 for small models\index{Small Batch!CPU advantage}. The overhead of launching a GPU kernel (~10 $\mu$s) and transferring data (~50 $\mu$s) can exceed the compute time for a tiny dense layer. For models under 50 MB serving single requests, a well-optimized CPU runtime often delivers lower latency than a GPU.
 
-### Fast Model Loading {#sec-model-serving-fast-model-loading-1109}
+### Model Serialization and Fast Loading {#sec-model-serving-fast-model-loading-1109}
 
-In autoscaling systems, the time to spin up a new node is critical. A major component of "Cold Start" (@sec-model-serving-model-loading-initialization-cc5a) is simply reading the model weights from disk into memory.
+In autoscaling systems, the time to spin up a new node is critical. A major component of "Cold Start" (@sec-model-serving-model-loading-initialization-cc5a) is simply reading the model weights from disk into memory. The choice of serialization format determines how quickly this loading can occur.
 
-#### The Pickle Problem {#sec-model-serving-pickle-problem-4550}
+The standard PyTorch `torch.load()` uses Python's `pickle` format\index{Pickle!loading overhead}. This approach is inefficient because it requires the CPU to unpickle objects one by one, copy them into memory, and then often copy them *again* to the GPU. A faster alternative is memory mapping\index{mmap!zero-copy loading}, which allows the OS to map a file directly into the process's virtual address space. The data is effectively "loaded" only when accessed, and the OS handles the transfer from disk to RAM efficiently.
 
-The standard PyTorch `torch.load()` uses Python's `pickle` format\index{Pickle!loading overhead}. This approach is inefficient because it requires the CPU to unpickle objects one by one, copy them into memory, and then often copy them *again* to the GPU.
-
-#### Zero-Copy with mmap {#sec-model-serving-zerocopy-mmap-95e4}
-
-Memory mapping\index{mmap!zero-copy loading} allows the OS to map a file directly into the process's virtual address space. The data is effectively "loaded" only when accessed, and the OS handles the transfer from disk to RAM efficiently.
-
-#### Safetensors {#sec-model-serving-safetensors-6461}
-
-Safetensors[^fn-safetensors]\index{Safetensors!zero-copy loading} is a modern format designed specifically for fast loading. It stores tensors as raw bytes with a minimal JSON header. This enables zero-copy\index{Zero-Copy!model loading} loading: the raw bytes on disk are mapped directly into the tensor's memory buffer.
+Building on this zero-copy principle, Safetensors[^fn-safetensors]\index{Safetensors!zero-copy loading} is a modern format designed specifically for fast loading. It stores tensors as raw bytes with a minimal JSON header. This enables zero-copy\index{Zero-Copy!model loading} loading: the raw bytes on disk are mapped directly into the tensor's memory buffer.
 
 ::: {.callout-example title="Loading Speed: Safetensors vs. Pickle"}
 Loading a 5GB Stable Diffusion model:

book/quarto/contents/vol1/nn_architectures/nn_architectures.qmd

@@ -8,6 +8,14 @@ engine: jupyter
 
 # Network Architectures {#sec-network-architectures}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:nn_architectures")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -53,7 +61,7 @@ The previous chapter established the mathematical operators—matrix multiplicat
 
 Every neural network architecture answers one central question: *how should we structure computation to match the structure in our data?* Images have spatial locality, language has sequential dependencies, and tabular records have no inherent structure at all. The architecture encodes assumptions about these patterns directly into the computational graph, and those assumptions determine everything from parameter count to hardware utilization to deployment feasibility. Architecture selection is therefore a systems engineering problem that directly determines the Iron Law terms (the number of operations $O$ and the volume of data movement $D_{vol}$) defined in @sec-introduction-iron-law-ml-systems-c32a.
 
-The structural assumptions that each architecture encodes are known as **inductive biases**[^fn-inductive-bias], and they serve as the *unifying concept* for this entire chapter.
+The structural assumptions that each architecture encodes are known as inductive biases[^fn-inductive-bias], and they serve as the *unifying concept* for this entire chapter.
 
 ::: {.callout-definition title="Inductive Bias"}
 
@@ -817,7 +825,7 @@ This translation from mathematical abstraction to concrete computation exposes h
 
 [^fn-mac]: **Multiply-Accumulate (MAC)**: The atomic operation in neural networks: multiply two values and add to a running sum. Modern accelerators measure performance in MACs per second: datacenter class accelerators can sustain on the order of \(10^{14}\) to \(10^{15}\) MAC/s on dense kernels (e.g., NVIDIA H100 at ~1 PFLOPS), while mobile class chips are often on the order of \(10^{12}\) to \(10^{13}\) MAC/s. At the hardware level, the energy cost of the arithmetic itself is typically in the picojoule range per MAC, while moving data to and from off chip memory can cost orders of magnitude more, which is why data movement is often the dominant systems concern.
 
-In our reference MNIST layer, each output neuron requires `{python} MLPLayerStats.mnist_mlp_macs_str` ÷ 100 = 784 multiply-accumulate operations and at least `{python} MLPLayerStats.mnist_neuron_mem_acc_str` memory accesses (784 for inputs, 784 for weights). While actual implementations use optimizations through libraries like BLAS[^fn-dnn-blas] or cuBLAS, these patterns drive key system design decisions. The hardware architectures that accelerate these matrix operations, including GPU tensor cores[^fn-tensor-cores] and specialized AI accelerators, are covered in @sec-hardware-acceleration.
+In our reference MNIST layer, each output neuron requires `{python} MLPLayerStats.mnist_mlp_macs_str` divided by 100, or 784, multiply-accumulate operations and at least `{python} MLPLayerStats.mnist_neuron_mem_acc_str` memory accesses (784 for inputs, 784 for weights). While actual implementations use optimizations through libraries like BLAS[^fn-dnn-blas] or cuBLAS, these patterns drive key system design decisions. The hardware architectures that accelerate these matrix operations, including GPU tensor cores[^fn-tensor-cores] and specialized AI accelerators, are covered in @sec-hardware-acceleration.
 
 ```{python}
 #| label: a100-tensor-core-specs
@@ -1608,15 +1616,79 @@ While using fewer operations per output, the spatial structure creates different
 
 The sliding window and im2col transformations above reveal *how* CNNs compute; the system implications below reveal *what that computation costs* in memory, compute, and data movement.
 
+```{python}
+#| label: cnn-system-implications
+#| echo: false
+
+# ┌─────────────────────────────────────────────────────────────────────────────
+# │ CNN SYSTEM IMPLICATIONS
+# ├─────────────────────────────────────────────────────────────────────────────
+# │ Context: Memory Requirements, Computation Needs, and Data Movement
+# │          subsections of §System Implications for CNNs
+# │
+# │ Goal: Derive weight counts, activation counts, and spatial positions
+# │       from architecture parameters so prose values stay consistent.
+# │ Show: 576 weights (3×3×1×64), 3.2M activations (224×224×64), 50,176 positions.
+# │ How:  Direct multiplication of kernel, channel, and spatial dimensions.
+# │
+# │ Imports: mlsys.constants (IMAGE_DIM_RESNET), mlsys.formatting (fmt, check)
+# │ Exports: cnn_weights_single_ch_str, cnn_spatial_positions_str, cnn_activations_m_str
+# └─────────────────────────────────────────────────────────────────────────────
+
+from mlsys.constants import IMAGE_DIM_RESNET
+from mlsys.formatting import fmt, check
+
+# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
+class CNNSysImpl:
+    """
+    Namespace for CNN System Implications calculations.
+    Scenario: ImageNet (224×224) with 3×3 filters and 64 output channels.
+    """
+
+    # ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
+    img_dim = IMAGE_DIM_RESNET          # 224
+    kernel_size = 3
+    c_out = 64                          # output channels
+    c_in_single = 1                     # single input channel (for weight illustration)
+
+    # ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
+    # Weight parameters for single input channel: K × K × C_in × C_out
+    weights_single_ch = kernel_size * kernel_size * c_in_single * c_out  # 576
+
+    # Spatial positions in the output feature map
+    spatial_positions = img_dim * img_dim                                # 50,176
+
+    # Activation values: H × W × C_out
+    activations = img_dim * img_dim * c_out                             # 3,211,264
+
+    # ┌── 3. INVARIANTS (Guardrails) ──────────────────────────────────────────
+    check(weights_single_ch == 576,
+          f"Expected 576 weight params, got {weights_single_ch}")
+    check(spatial_positions == 50_176,
+          f"Expected 50,176 spatial positions, got {spatial_positions}")
+    check(activations == 3_211_264,
+          f"Expected 3,211,264 activations, got {activations}")
+
+    # ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
+    weights_single_ch_str = fmt(weights_single_ch, precision=0, commas=True)
+    spatial_positions_str = fmt(spatial_positions, precision=0, commas=True)
+    activations_str = fmt(activations / 1e6, precision=1)       # "3.2" (millions)
+
+# ┌── EXPORTS (Bridge to Text) ────────────────────────────────────────────────
+cnn_weights_single_ch_str = CNNSysImpl.weights_single_ch_str
+cnn_spatial_positions_str = CNNSysImpl.spatial_positions_str
+cnn_activations_m_str = CNNSysImpl.activations_str
+```
+
 #### Memory Requirements {#sec-network-architectures-memory-requirements-a1a1}
 
-For convolutional layers, memory requirements center around two key components: filter weights and feature maps. Unlike MLPs that require storing full connection matrices, CNNs use small, reusable filters. For a typical CNN processing $224 \times 224$ ImageNet images, a convolutional layer with 64 filters of size $3 \times 3$ applied to a single input channel requires storing only 576 weight parameters ($3 \times 3 \times 1 \times 64$); for $C_{in}$ input channels, this becomes $3 \times 3 \times C_{in} \times 64$ parameters, still dramatically less than the millions of weights needed for equivalent fully-connected processing. The system must store feature maps for all spatial positions, creating a different memory demand. A $224 \times 224$ input with 64 output channels requires storing 3.2 million activation values ($224 \times 224 \times 64$).
+For convolutional layers, memory requirements center around two key components: filter weights and feature maps. Unlike MLPs that require storing full connection matrices, CNNs use small, reusable filters. For a typical CNN processing $224 \times 224$ ImageNet images, a convolutional layer with 64 filters of size $3 \times 3$ applied to a single input channel requires storing only `{python} cnn_weights_single_ch_str` weight parameters ($3 \times 3 \times 1 \times 64$); for $C_{in}$ input channels, this becomes $3 \times 3 \times C_{in} \times 64$ parameters, still dramatically less than the millions of weights needed for equivalent fully-connected processing. The system must store feature maps for all spatial positions, creating a different memory demand. A $224 \times 224$ input with 64 output channels requires storing `{python} cnn_activations_m_str` million activation values ($224 \times 224 \times 64$).
 
 These memory access patterns suggest opportunities for optimization through weight reuse and careful feature map management. Processors optimize these spatial patterns by caching filter weights for reuse across positions while streaming feature map data. CPUs use their cache hierarchy to keep frequently used filters resident, while GPUs employ specialized memory architectures designed for the spatial access patterns of image processing. The detailed architecture design principles for these specialized processors are covered in @sec-hardware-acceleration.
 
 #### Computation Needs {#sec-network-architectures-computation-needs-85cd}
 
-The core computation in CNNs involves repeatedly applying small filters across spatial positions. Each output value requires a local multiply-accumulate operation over the filter region. For ImageNet processing with $3 \times 3$ filters and 64 output channels, computing one spatial position involves $3 \times 3 \times C_{in} \times 64$ multiply-accumulates (576 per input channel), and this must be repeated for all 50,176 spatial positions ($224 \times 224$). While each individual computation involves fewer operations than an MLP layer, the total computational load remains large due to spatial repetition.
+The core computation in CNNs involves repeatedly applying small filters across spatial positions. Each output value requires a local multiply-accumulate operation over the filter region. For ImageNet processing with $3 \times 3$ filters and 64 output channels, computing one spatial position involves $3 \times 3 \times C_{in} \times 64$ multiply-accumulates (`{python} cnn_weights_single_ch_str` per input channel), and this must be repeated for all `{python} cnn_spatial_positions_str` spatial positions ($224 \times 224$). While each individual computation involves fewer operations than an MLP layer, the total computational load remains large due to spatial repetition.
 
 This computational pattern presents different optimization opportunities than MLPs. The regular, repeated nature of convolution operations enables efficient hardware utilization through structured parallelism. Modern processors exploit this pattern in various ways. CPUs use SIMD instructions[^fn-simd] to process multiple filter positions simultaneously, while GPUs parallelize computation across spatial positions and channels. The model optimization techniques that further reduce these computational demands, including specialized convolution optimizations and sparsity patterns, are detailed in @sec-model-compression.
 
@@ -1624,9 +1696,9 @@ This computational pattern presents different optimization opportunities than ML
 
 #### Data Movement {#sec-network-architectures-data-movement-36f6}
 
-The sliding window pattern of convolutions creates a distinctive data movement profile. Unlike MLPs where each weight is used once per forward pass, CNN filter weights are reused many times as the filter slides across spatial positions. For ImageNet processing, each $3 \times 3$ filter weight is reused 50,176 times (once for each position in the $224 \times 224$ feature map). This creates a different challenge: the system must stream input features through the computation unit while keeping filter weights stable.
+The sliding window pattern of convolutions creates a distinctive data movement profile. Unlike MLPs where each weight is used once per forward pass, CNN filter weights are reused many times as the filter slides across spatial positions. For ImageNet processing, each $3 \times 3$ filter weight is reused `{python} cnn_spatial_positions_str` times (once for each position in the $224 \times 224$ feature map). This creates a different challenge: the system must stream input features through the computation unit while keeping filter weights stable.
 
-The predictable spatial access pattern enables strategic data movement optimizations. The CPU/GPU caching strategies described above apply directly to data movement: frameworks orchestrate computation to maximize the 50,176 $\times$ filter weight reuse and minimize redundant feature map accesses, exploiting the same spatial locality that makes CNNs memory-efficient.
+The predictable spatial access pattern enables strategic data movement optimizations. The CPU/GPU caching strategies described above apply directly to data movement: frameworks orchestrate computation to maximize the `{python} cnn_spatial_positions_str` $\times$ filter weight reuse and minimize redundant feature map accesses, exploiting the same spatial locality that makes CNNs memory-efficient.
 
 ### Efficient Architectures: Keyword Spotting {#sec-network-architectures-efficient-architectures-keyword-spotting-0625}
 
@@ -1893,7 +1965,7 @@ RNNs are uniquely memory-efficient for long sequences. They maintain a fixed-siz
 RNNs exhibit high temporal locality for weights (reused every step) but low locality for activations. The weight matrices $W_{hh}$ and $W_{xh}$ stay in the cache (or on-chip memory) for the entire duration of the sequence processing, achieving high arithmetic intensity *if* the batch size is large enough. However, the requirement to read and write the hidden state at every step creates a constant stream of low-intensity updates that can strain memory bandwidth if not carefully managed.
 
 \index{Recurrent Neural Network!sequential dependency bottleneck}
-This tension between memory efficiency and sequential execution defined the pre-Transformer era. While RNNs enabled sequence modeling, their inability to exploit massive parallel hardware ultimately led to their replacement by attention-based architectures. The hardware strategies for managing sequential bottlenecks -- including pipeline parallelism and operator fusion for latency-sensitive workloads -- are analyzed in @sec-hardware-acceleration-dataflow-optimization-strategies-ce52.
+This tension between memory efficiency and sequential execution defined the pre-Transformer era. RNNs compress arbitrarily long histories into a fixed-size hidden state, which is memory efficient but creates two compounding problems: the sequential dependency prevents hardware from parallelizing across time steps, and the fixed-capacity state becomes an information bottleneck where early inputs fade as sequences grow (the vanishing gradient problem). Together, these limitations motivated a fundamental question: could an architecture access any position in a sequence directly, without processing all intervening elements? The answer, as the next section shows, is the attention mechanism. Hardware strategies for managing sequential bottlenecks in RNN workloads that remain in production, including pipeline parallelism and operator fusion, are analyzed in @sec-hardware-acceleration-dataflow-optimization-strategies-ce52.
 
 ## Attention: Dynamic Processing {#sec-network-architectures-attention-mechanisms-dynamic-pattern-processing-22df}
 
@@ -2492,7 +2564,7 @@ The attention mechanism analyzed above provides the computational primitive—dy
 While the attention mechanisms examined above introduced dynamic connectivity, they were initially applied as additions to existing architectures, particularly RNNs for sequence-to-sequence tasks. This hybrid approach still suffered from the inherent limitations of recurrent architectures: sequential processing constraints that prevented efficient parallelization and difficulties with very long sequences. The breakthrough insight was recognizing that attention mechanisms alone could replace both convolutional and recurrent processing entirely -- eliminating the sequential bottleneck while preserving dynamic pattern processing.
 
 \index{Vaswani, Ashish}
-Transformers, introduced in the landmark \"Attention is All You Need\" paper[^fn-attention-is-all-you-need] by @vaswani2017attention, embody a revolutionary inductive bias: **they assume no prior structure but allow the model to learn all pairwise relationships dynamically based on content**. Rather than adding attention to RNNs, Transformers built the entire architecture around attention mechanisms, introducing self-attention as the primary computational pattern. This architectural decision traded the parameter efficiency of CNNs and the sequential coherence of RNNs for maximum flexibility and parallelizability.
+Transformers, introduced in the landmark "Attention is All You Need" paper[^fn-attention-is-all-you-need] by @vaswani2017attention, embody a revolutionary inductive bias: **they assume no prior structure but allow the model to learn all pairwise relationships dynamically based on content**. Rather than adding attention to RNNs, Transformers built the entire architecture around attention mechanisms, introducing self-attention as the primary computational pattern. This architectural decision traded the parameter efficiency of CNNs and the sequential coherence of RNNs for maximum flexibility and parallelizability.
 
 [^fn-attention-is-all-you-need]: **"Attention is All You Need"**\index{Attention Is All You Need (paper)}: This 2017 paper by Google researchers eliminated recurrence entirely, showing that attention mechanisms alone could achieve state-of-the-art results. The paper's eight authors---Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin---were researchers at Google Brain and Google Research in Mountain View. Remarkably, nearly all later left to found AI companies: Gomez co-founded Cohere, Parmar and Vaswani co-founded Adept (later Essential AI), Shazeer co-founded Character.AI, Jones co-founded Sakana AI, and Polosukhin co-founded NEAR Protocol. This diaspora illustrates how foundational research at one institution can seed an entire industry.
 
@@ -3594,15 +3666,15 @@ energy_mac_pj_str = f"{energy_mac_pj_value}"                     # e.g. "4.6"
 energy_dram_str = fmt(energy_dram_value, precision=0, commas=False)  # e.g. "26"
 ```
 
-Dense matrix operations in MLPs achieve excellent arithmetic intensity[^fn-arithmetic-intensity-dnn] (computation per data movement) but consume significant absolute energy. Each multiply-accumulate operation consumes approximately `{python} energy_mac_pj_str` pJ, while data movement from DRAM costs `{python} energy_dram_str` pJ per 32-bit value [@horowitz2014computing]. For typical MLP inference, 70-80% of energy goes to data movement rather than computation, making memory bandwidth optimization critical for energy efficiency.
+Dense matrix operations in MLPs achieve excellent arithmetic intensity[^fn-arithmetic-intensity-dnn] (computation per data movement) but consume significant absolute energy. Each multiply-accumulate operation consumes approximately `{python} energy_mac_pj_str` pJ, while data movement from DRAM costs `{python} energy_dram_str` pJ per 32-bit value [@horowitz2014computing]. Given this energy ratio, typical MLP inference spends the majority of its energy budget on data movement rather than computation, making memory bandwidth optimization critical for energy efficiency.
 
 [^fn-arithmetic-intensity-dnn]: **Arithmetic Intensity**\index{Arithmetic Intensity!definition}: The ratio of floating-point operations to memory accesses, measured in FLOPs/byte. High arithmetic intensity (>10 FLOPs/byte) enables efficient hardware utilization, while low intensity (<1 FLOPs/byte) makes workloads memory-bound. Attention mechanisms typically have low arithmetic intensity, explaining their energy inefficiency. See @sec-hardware-acceleration for the roofline model that formalizes this metric.
 
-Convolutional operations reduce energy consumption through data reuse but exhibit variable efficiency depending on implementation. Im2col-based convolution implementations trade memory for simplicity, often doubling memory requirements and energy consumption. Direct convolution implementations achieve 3–5 $\times$ better energy efficiency by eliminating redundant data movement, particularly for larger kernel sizes.
+Convolutional operations reduce energy consumption through data reuse but exhibit variable efficiency depending on implementation. Im2col-based convolution implementations trade memory for simplicity, often doubling memory requirements and energy consumption. Direct convolution implementations can achieve substantially better energy efficiency by eliminating redundant data movement, particularly for larger kernel sizes where im2col duplication is most severe.
 
-Sequential processing in RNNs creates energy efficiency opportunities through temporal data reuse. The constant memory footprint of RNN hidden states allows aggressive caching strategies, reducing DRAM access energy by 80-90% for long sequences. The sequential dependencies limit parallelization opportunities, often resulting in suboptimal hardware utilization and higher energy per operation.
+Sequential processing in RNNs creates energy efficiency opportunities through temporal data reuse. The constant memory footprint of RNN hidden states allows aggressive caching strategies that can dramatically reduce DRAM access energy for long sequences by keeping the recurrent state in on-chip SRAM. The sequential dependencies limit parallelization opportunities, often resulting in suboptimal hardware utilization and higher energy per operation.
 
-Attention mechanisms in Transformers exhibit the highest energy consumption per operation due to irregular memory access patterns and the need to store attention matrices (the quadratic bottleneck from @sec-network-architectures-system-implications-05a3). Self-attention operations consume 2–3 $\times$ more energy per FLOP than standard matrix multiplication, making long-sequence processing energy-prohibitive without architectural modifications such as FlashAttention.
+Attention mechanisms in Transformers exhibit the highest energy consumption per operation due to irregular memory access patterns and the need to store attention matrices (the quadratic bottleneck from @sec-network-architectures-system-implications-05a3). The irregular access patterns of self-attention result in significantly higher energy per useful FLOP compared to standard matrix multiplication, making long-sequence processing energy-prohibitive without architectural modifications such as FlashAttention.
 
 System designers must balance competing trade-offs when supporting different primitives, each with unique characteristics that influence system design and performance. Optimizing for the dense matrix operations common in MLPs and CNNs might come at the cost of flexibility needed for the more dynamic computations in attention mechanisms. Supporting large working sets for Transformers might require sacrificing energy efficiency.
 
@@ -4038,6 +4110,10 @@ a100_8x_mem_str = f"{a100_8x_mem_value}"                         # e.g. "640"
 
 Teams design for high-end GPU clusters, then discover deployment failures on target hardware. An architecture exploiting 8 $\times$ A100 GPUs (`{python} a100_8x_mem_str` GB total memory) cannot deploy to edge devices with 4 GB—the 160 $\times$ reduction requires architectural changes, not just quantization. As @sec-network-architectures-decision-framework-a889 emphasizes, architecture selection must analyze the full system stack. Edge deployment compounds constraints: models must fit 10–100 MB storage, execute in 50–200 ms, and operate within 2–5 W power. Organizations deferring deployment considerations to "optimize later" encounter mismatches requiring costly redesigns that delay products by months.
 
+**Pitfall:** *Ignoring KV cache growth when estimating Transformer serving costs.*
+
+Teams budget Transformer deployment based on model weight memory alone, overlooking the key-value (KV) cache that self-attention requires during autoregressive generation. The KV cache scales as $O(\text{batch} \times \text{layers} \times \text{heads} \times \text{seq\_len} \times \text{head\_dim})$, and for large models this overhead dominates serving memory. Consider a Transformer with 32 layers and 32 attention heads, each with a 128-dimensional head, serving sequences of length 2048 in FP16. Each concurrent request stores $32 \times 32 \times 2048 \times 128 \times 2$ bytes $\approx$ 537 MB of KV cache. At even modest concurrency of 2--4 users, the KV cache alone consumes 1--2 GB, rivaling or exceeding the memory occupied by model weights. As the quadratic memory analysis in @sec-network-architectures-system-implications-77ac establishes, attention memory grows with sequence length, making the KV cache the binding constraint on serving throughput. Teams that size infrastructure based solely on weight memory discover at deployment that halving the batch size or truncating context length is the only way to fit within device memory, degrading either throughput or output quality.
+
 With these cautionary notes in mind, we now synthesize the key concepts that practitioners should carry forward from this chapter's systematic tour of architectural families, shared building blocks, computational primitives, and selection methodology.
 
 ## Summary {#sec-network-architectures-summary-e642}

book/quarto/contents/vol1/nn_computation/nn_computation.qmd

@@ -7,6 +7,14 @@ engine: jupyter
 
 # Neural Computation {#sec-neural-computation}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:nn_computation")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -2145,15 +2153,6 @@ bp_h1_kb_str = BackpropMemory.act_l1_kb_str
 bp_h2_kb_str = BackpropMemory.act_l2_kb_str
 bp_out_kb_str = fmt((BackpropMemory.act_counts[3] * 4) / 1024, precision=1, commas=False)
 ```
-total_train_mib = total_train_kib / KIB_TO_BYTES
-training_ratio_calc = total_train_kib / total_infer_kib
-
-grad_kib_str = fmt(grad_kib_value, precision=1, commas=False)
-opt_kib_str = fmt(opt_kib_value, precision=1, commas=False)
-total_train_mib_str = fmt(total_train_mib, precision=1, commas=False)
-total_infer_kib_str = fmt(total_infer_kib, precision=1, commas=False)
-training_ratio_str = fmt(training_ratio_calc, precision=1, commas=False)
-```
 
 ::: {.callout-example title="Memory: Training vs. Inference"}
 
@@ -3668,6 +3667,10 @@ Engineers assume neural networks lack the transparency of traditional code. In p
 
 Teams assume automatic feature learning removes the need for domain knowledge. Successful systems require domain expertise at every stage: architecture selection, training objective design, dataset curation, and output interpretation. The USPS system in @sec-neural-computation-case-study-usps-digit-recognition-97be succeeded because postal engineers specified confidence thresholds based on mail sorting economics, routing 10-15% of uncertain cases to human operators. Without domain knowledge, teams deploy networks that achieve 98% test accuracy but fail in production by routing 40% of cases to manual processing or misrouting 5% of mail.
 
+**Fallacy:** *Deeper networks are always more accurate than wider ones.*
+
+Engineers assume that stacking more layers is the primary path to higher accuracy, since depth enables hierarchical feature extraction. In practice, depth alone encounters diminishing returns. ResNet demonstrated that networks beyond 152 layers showed negligible accuracy improvement on ImageNet despite substantially increased training cost and inference latency. The vanishing gradient problem analyzed in @sec-neural-computation-learning-process-0b83 explains part of this ceiling: even with skip connections, very deep networks suffer from optimization difficulties as gradients traverse hundreds of layers. EfficientNet later demonstrated that balanced scaling of width, depth, and input resolution outperforms depth-only scaling by 2--3 percentage points at equivalent computational cost. Doubling depth from 50 to 100 layers roughly doubles training time and memory consumption while yielding less than 1 percentage point of accuracy gain, whereas distributing the same parameter budget across width and depth achieves greater accuracy without the optimization penalty. Teams that reflexively add layers before profiling their network's capacity utilization waste compute on diminishing returns when wider layers or higher-resolution inputs would deliver greater improvement per FLOP.
+
 **Pitfall:** *Using neural networks for problems solvable with simpler methods.*
 
 Teams assume deep learning always performs better. Logistic regression training in 10 ms often outperforms a neural network requiring 2 hours when data contains fewer than 1,000 examples or relationships are approximately linear. If logistic regression achieves 94% accuracy, a neural network achieving 95% rarely justifies the cost: 100–1,000 $\times$ longer training, 10–50 $\times$ more memory, and ongoing maintenance burden. As shown in @sec-neural-computation-learning-process-0b83, neural networks excel at hierarchical pattern discovery but impose substantial overhead. Reserve them for problems with spatial locality, temporal dependencies, or high-dimensional nonlinear interactions that simpler models cannot capture.

book/quarto/contents/vol1/optimizations/model_compression.qmd

@@ -8,6 +8,14 @@ engine: jupyter
 
 # Model Compression {#sec-model-compression}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:model_compression")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -18,6 +26,7 @@ engine: jupyter
 
 :::
 
+
 ## Purpose {.unnumbered}
 
 \begin{marginfigure}
@@ -164,9 +173,10 @@ mcu_ram_str = f"{mcu_ram_kb_value}"
 gpt3_training_flops_str = f"$3.14 \\times 10^{{{gpt3_training_flops_exp_value}}}$"
 ```
 
+
 ## Optimization Framework {#sec-model-compression-optimization-framework-9e21}
 
-A `{python} llm_7b_str`-billion parameter language model requires `{python} llm_7b_mem_str` GB just to store its weights in FP16. Your deployment target is a smartphone with `{python} smartphone_ram_str` GB of RAM shared across the operating system, applications, and your model. *The math does not work.* No amount of clever engineering changes this arithmetic: `{python} llm_7b_mem_str` GB cannot fit in `{python} smartphone_ram_str` GB. Yet users expect the model to run: responsively, offline, without draining their battery in an hour. The gap between what training produces and what deployment permits (the **Latency Budget**, the maximum allowable end-to-end inference time, defined formally in @sec-model-serving) is not a minor inconvenience but a defining challenge of model compression.
+A `{python} llm_7b_str`-billion parameter language model requires `{python} llm_7b_mem_str` GB just to store its weights in FP16. Your deployment target is a smartphone with `{python} smartphone_ram_str` GB of RAM shared across the operating system, applications, and your model. *The math does not work.* No amount of clever engineering changes this arithmetic: `{python} llm_7b_mem_str` GB cannot fit in `{python} smartphone_ram_str` GB. Yet users expect the model to run: responsively, offline, without draining their battery in an hour. The gap between what training produces and what deployment permits (the Latency Budget, the maximum allowable end-to-end inference time, defined formally in @sec-model-serving) is not a minor inconvenience but a defining challenge of model compression.
 
 Recall the **Silicon Contract** (@sec-introduction-iron-law-ml-systems-c32a), the implicit agreement every model makes with its hardware about which resource it will saturate. The three candidates are compute throughput, memory bandwidth, and memory capacity. During training, this contract is negotiated upward. Researchers select larger architectures, higher numerical precision, and deeper layers because the training environment, typically a GPU cluster with hundreds of gigabytes of memory, can afford those demands. In @sec-model-training, we used Mixed Precision (FP16) to speed up these training cycles while maintaining the ability to learn. Here, we go further—to INT8 and beyond—for inference, where we trade the ability to update weights for massive gains in execution efficiency. Deployment reverses these priorities. The production environment is smaller, power-constrained, and latency-sensitive, yet the model was designed for an environment with none of those limitations. Model compression is the systematic process of renegotiating that contract for its new execution context, reducing memory footprint, computational cost, and energy consumption while preserving the model's ability to perform its task.
 
@@ -174,7 +184,7 @@ The scale of this renegotiation makes model optimization an engineering discipli
 
 \index{Overparameterization!systematic removal}
 \index{Operator Fusion!framework introduction}
-This chapter organizes these techniques along three complementary dimensions. *Structural optimization* removes redundancy from the model itself: pruning eliminates parameters that contribute little to output quality, knowledge distillation transfers a large model's learned behavior into a smaller architecture, and neural architecture search discovers designs that are inherently efficient. *Precision optimization* reduces the numerical bit-width of weights and activations, for example converting 32-bit floating point values to 8-bit integers (exploiting **Tensor Cores** discussed in @sec-hardware-acceleration-tensor-cores-771f), which shrinks memory footprint and accelerates arithmetic on hardware that supports lower-precision operations. *Hardware-level optimization* ensures that the resulting model executes efficiently on the target processor by fusing operations to reduce memory traffic and exploiting sparsity patterns that the hardware can accelerate. These dimensions are not alternatives but layers in an optimization stack. A practitioner deploying ResNet-50 to a mobile device might prune 50% of its filters, quantize the remaining weights to INT8, and fuse batch normalization into convolution, with each technique compounding the gains of the others.
+This chapter organizes these techniques along three complementary dimensions. *Structural optimization* removes redundancy from the model itself: pruning eliminates parameters that contribute little to output quality, knowledge distillation transfers a large model's learned behavior into a smaller architecture, and neural architecture search discovers designs that are inherently efficient. *Precision optimization* reduces the numerical bit-width of weights and activations, for example converting 32-bit floating point values to 8-bit integers (exploiting Tensor Cores discussed in @sec-hardware-acceleration-tensor-cores-771f), which shrinks memory footprint and accelerates arithmetic on hardware that supports lower-precision operations. *Hardware-level optimization* ensures that the resulting model executes efficiently on the target processor by fusing operations to reduce memory traffic and exploiting sparsity patterns that the hardware can accelerate. These dimensions are not alternatives but layers in an optimization stack. A practitioner deploying ResNet-50 to a mobile device might prune 50% of its filters, quantize the remaining weights to INT8, and fuse batch normalization into convolution, with each technique compounding the gains of the others.
 
 Throughout this chapter, we ground each technique in concrete systems: ResNet-50 and MobileNetV2 (our **Lighthouse Models** from @sec-network-architectures) for vision workloads, transformer-based language models for sequence tasks, and the DS-CNN keyword spotter for TinyML deployment. These recurring models let us compare techniques under consistent conditions, making the trade-offs between accuracy, latency, memory, and energy tangible rather than abstract.
 
@@ -410,6 +420,7 @@ We call this phenomenon *the quantization speedup*.
 
 The relative importance of each dimension varies by deployment target. Cloud systems may tolerate larger models but demand throughput; mobile devices prioritize memory and energy; embedded systems face hard constraints on all resources simultaneously. Understanding these deployment contexts shapes which optimization dimensions to prioritize.
 
+
 ## Deployment Context {#sec-model-compression-deployment-context-0d88}
 
 The optimization framework above identifies three dimensions of compression, but which dimensions matter most depends entirely on where the model will run. A datacenter GPU with 80 GB of HBM faces different binding constraints than a smartphone with shared RAM or a microcontroller with 256 KB of SRAM. @tbl-deployment-scenarios summarizes the key constraints across deployment environments.
@@ -426,7 +437,7 @@ The optimization framework above identifies three dimensions of compression, but
 
 Cloud inference centers on throughput (requests/second/dollar), where quantization enables serving more concurrent requests and operator fusion reduces per-request latency [@choudhary2020comprehensive; @dean2018new]. Mobile and edge deployments must fit device memory while meeting real-time targets. A camera app processing 30 fps has 33 ms per frame, so any optimization reducing inference below this threshold directly improves user experience.
 
-TinyML\index{TinyML!model compression}[^fn-microcontroller-constraints] makes optimization existential, not optional. A microcontroller with 256KB RAM cannot run a 100MB model regardless of accuracy. The model must compress below hardware limits or deployment is impossible [@banbury2020benchmarking]. Even on mobile devices with comparatively generous resources, a single optimization technique can deliver a *4 $\times$ performance win* that means the difference between a feature that ships and one that never leaves the prototype stage.
+TinyML\index{TinyML!model compression}[^fn-microcontroller-constraints] makes optimization existential, not optional. A microcontroller with 256 KB RAM cannot run a 100 MB model regardless of accuracy. The model must compress below hardware limits or deployment is impossible [@banbury2020benchmarking]. Even on mobile devices with comparatively generous resources, a single optimization technique can deliver a *4 $\times$ performance win* that means the difference between a feature that ships and one that never leaves the prototype stage.
 
 \index{MobileNetV3!NAS-optimized architecture}
 
@@ -587,7 +598,8 @@ Optimization is about trading one resource for another.
 - [ ] **Pruning**: When should you prune (reduce parameters) vs. distill (train a smaller student)? (Hint: Pruning is for existing models; Distillation is for architectural changes).
 :::
 
-With the deployment context established and the trade-off landscape mapped, we now examine each optimization dimension in depth. We begin with structural methods that modify *what* computations occur, then turn to precision techniques that reduce numerical fidelity, and finally address architectural approaches that improve execution efficiency on physical hardware.
+Each deployment context above imposes a binding constraint: memory capacity on mobile devices, latency on real-time systems, energy on battery-powered sensors. The optimization techniques that follow address these constraints at three successive levels of the stack. We begin with structural methods that modify *what* computations occur, reducing the model's parameter count and operation count to fit tighter memory and compute budgets. We then turn to precision techniques that reduce how many bits represent each value, directly shrinking memory footprint and accelerating arithmetic. Finally, we address architectural approaches that improve how efficiently the remaining operations execute on physical hardware, closing the gap between theoretical savings and measured performance.
+
 
 ## Structural Optimization {#sec-model-compression-structural-optimization-ee93}
 \index{Model Compression!structural optimization}
@@ -807,13 +819,13 @@ To make pruning computationally feasible, practical methods often replace the ha
 
 #### Target Structures {#sec-model-compression-target-structures-1230}
 
-Pruning methods differ by what they remove. The primary targets are neurons, channels, and layers, each with distinct implications for architecture and performance.
+The choice of what to prune depends on the deployment target's hardware constraints and which resource is the binding bottleneck.
 
-* **Neuron pruning**\index{Pruning!neuron} removes entire neurons along with their associated weights and biases, reducing the width of a layer. This technique is often applied to fully connected layers.
+When memory capacity is the primary constraint, as in fully connected classifiers destined for mobile deployment, neuron pruning\index{Pruning!neuron} offers the most direct relief: removing entire neurons along with their associated weights and biases reduces the width of a layer, shrinking the parameter count proportionally. Because fully connected layers dominate memory in many architectures, targeting neurons addresses the largest contributor to model size.
 
-* **Channel pruning**\index{Pruning!channel} (or filter pruning), commonly used in convolutional neural networks, eliminates entire channels or filters. This reduces the depth of feature maps, which impacts the network's ability to extract certain features. Channel pruning is particularly effective in image-processing tasks where computational efficiency is a priority.
+When inference latency on commodity accelerators is the bottleneck, channel pruning\index{Pruning!channel} (also called filter pruning) becomes the preferred approach. Eliminating entire channels or filters from convolutional layers reduces the depth of feature maps, which directly cuts the number of multiply-accumulate operations in subsequent layers. This reduction maps cleanly onto GPU and TPU execution patterns because the resulting model remains dense and regular, requiring no special sparse computation support. Channel pruning is therefore particularly effective for vision workloads where convolutional layers dominate computational cost.
 
-* **Layer pruning**\index{Pruning!layer} removes entire layers from the network, significantly reducing depth. While this approach can yield significant efficiency gains, it requires careful balance to ensure the model retains sufficient capacity to capture complex patterns.
+When the most aggressive efficiency gains are required and the architecture has sufficient depth to absorb the loss, layer pruning\index{Pruning!layer} removes entire layers from the network. This approach yields the largest per-operation reduction because it eliminates all computation within a layer, but it also carries the highest risk: removing a layer reduces the model's representational depth, and the remaining layers must compensate for the lost capacity. Layer pruning therefore demands careful validation to ensure the model retains sufficient capacity to capture the patterns its task requires.
 
 To see how these approaches differ in practice, compare the two sides of @fig-channel-layer-pruning. When a channel is pruned, the model's architecture must be adjusted to accommodate the structural change. Specifically, the number of input channels in subsequent layers must be modified, requiring alterations to the depths of the filters applied to the layer with the removed channel. In contrast, layer pruning removes all channels within a layer, necessitating more significant architectural modifications. In this case, connections between remaining layers must be reconfigured to bypass the removed layer. Regardless of the pruning approach, fine-tuning is important to adapt the remaining network and restore performance.
 
@@ -2195,7 +2207,7 @@ The factor $T^2$ ensures that gradient scales remain consistent when $T$ is chan
 
 Distillation's primary advantage over pruning is that it produces a *dense* model, not a sparse one. A distilled student runs efficiently on commodity hardware — GPUs, TPUs, edge AI chips — without requiring specialized sparse execution kernels. Models such as DistilBERT\index{DistilBERT}\index{Knowledge Distillation!DistilBERT}[^fn-distilbert-metrics] retain up to 97% of the teacher's accuracy with 40% fewer parameters and 60% faster inference, a compression level difficult to achieve through pruning alone [@sanh2019distilbert]. MobileNet distillation variants [@howard2017mobilenets] demonstrate similar results in computer vision. The student also inherits the teacher's generalization properties\index{Knowledge Distillation!memory efficiency}: large models trained on extensive datasets are less sensitive to noise and data shifts, and well-trained students inherit this robustness — particularly valuable in low-data regimes where training a small model from scratch leads to poor generalization.
 
-[^fn-distilbert-metrics]: **DistilBERT Performance**: Achieves 97% of BERT-Base performance with 40% fewer parameters (66M vs. 110M) and 60% faster inference. On SQuAD v1.1, DistilBERT scores 86.9 F1 vs. BERT's 88.5 F1, while reducing memory from 1.35GB to 0.54GB and latency from 85 ms to 34 ms.
+[^fn-distilbert-metrics]: **DistilBERT Performance**: Achieves 97% of BERT-Base performance with 40% fewer parameters (66M vs. 110M) and 60% faster inference. On SQuAD v1.1, DistilBERT scores 86.9 F1 vs. BERT's 88.5 F1, while reducing memory from 1.35 GB to 0.54 GB and latency from 85 ms to 34 ms.
 
 Distillation also enables *multi-task deployment*: a single large teacher can guide multiple specialized students for different tasks (e.g., language-specific NLP models, task-specific vision models), amortizing the teacher's training cost across many deployment targets. The resulting students can be further optimized with pruning and quantization for hardware-specific acceleration [@gordon2020compressing].
 
@@ -2779,10 +2791,11 @@ Test your understanding of the structural optimization techniques covered so far
 - [ ] Can you identify when to choose Neural Architecture Search over manual architecture design? Consider the trade-offs in computational cost, design space coverage, and hardware-specific optimization.
 :::
 
+
 ## Quantization and Precision {#sec-model-compression-quantization-precision-cd46}
 \index{Model Compression!precision optimization}
 
-*Quantization*, the process of reducing numerical precision, offers one of the most impactful optimizations for deployment: trading bits for speed and efficiency with minimal accuracy loss.
+A `{python} llm_7b_str` billion parameter language model stored in FP16 consumes `{python} llm_7b_mem_str` GB, yet users expect it to run on a smartphone with `{python} smartphone_ram_str` GB of shared RAM. Structural optimization alone cannot bridge this gap: even aggressive pruning rarely exceeds 50--70% parameter reduction, leaving a model far too large for the target device. The remaining leverage comes from a different dimension entirely: reducing the number of bits used to represent each parameter. *Quantization*, the process of reducing numerical precision, offers one of the most impactful optimizations for deployment, because it trades bits for speed and efficiency with minimal accuracy loss.
 
 ::: {.callout-definition title="Quantization"}
 
@@ -3084,12 +3097,12 @@ compression_ratio_str = QuantizationSavings.compression_ratio_str
 **FP16 (Half Precision)**
 
 - **Size**: `{python} llm_params_b_str` $\times 10^9$ $\times$ `{python} fp16_bytes_str` bytes (16-bit) = `{python} fp16_size_gb_str` GB
-- **Hardware Req**: Requires 24GB GPU (e.g., A10G, 3090, 4090).
+- **Hardware Req**: Requires 24 GB GPU (e.g., A10G, 3090, 4090).
 
 **INT4 (4-bit Quantization)**
 
 - **Size**: `{python} llm_params_b_str` $\times 10^9$ $\times$ `{python} int4_bytes_str` bytes (4-bit) = `{python} int4_size_gb_str` GB
-- **Hardware Req**: Fits comfortably on 8GB GPU (e.g., T4, consumer laptops).
+- **Hardware Req**: Fits comfortably on 8 GB GPU (e.g., T4, consumer laptops).
 
 **Impact**: `{python} compression_ratio_str` $\times$ compression allows deployment on commodity hardware, reducing cost by 5–10 $\times$.
 :::
@@ -3289,7 +3302,7 @@ INT8\index{INT8!efficiency gains}\index{Quantization!INT8 (8-bit integer)} preci
 Binary and ternary networks\index{Quantization!binary networks}\index{Quantization!ternary networks} represent the extreme end of quantization, where weights and activations are constrained to 1-bit (binary) or 2-bit (ternary) values. This results in massive storage and energy savings, but model accuracy often degrades significantly unless specialized architectures are used. Our *keyword spotting* lighthouse lives precisely in this regime, necessitating strategies for *keyword spotting and extreme compression*.
 
 ::: {.callout-lighthouse title="Keyword Spotting and Extreme Compression"}
-**The Extreme Constraint**: Our **Keyword Spotting (KWS) Lighthouse** (@sec-network-architectures) lives here. Running on a microcontroller with 256KB of SRAM means "standard" compression is not enough.
+**The Extreme Constraint**: Our **Keyword Spotting (KWS) Lighthouse** (@sec-network-architectures) lives here. Running on a microcontroller with 256 KB of SRAM means "standard" compression is not enough.
 
 For KWS, INT8 quantization is often just the *starting point*. To fit complex acoustic models into embedded sensors, engineers push toward **INT4** or even **Binary** weights. In this regime, the **Information** (Data) is noisy audio, the **Logic** (Algorithm) is highly simplified, and the **Physics** (Machine) has a power budget measured in microwatts.
 :::
@@ -4073,7 +4086,7 @@ Activation Quantization refers to the process of quantizing the activation value
 \index{LLaMA!AWQ quantization}
 Recent advancements have explored **Activation-aware Weight Quantization (AWQ)**\index{Quantization!activation-aware weight (AWQ)}[^fn-awq] for the compression and acceleration of large language models (LLMs). This approach is particularly relevant for our **GPT-2 / Llama Lighthouse**, which is memory-bandwidth bound. By protecting only a small fraction of the most salient weights (approximately 1%) based on activation magnitude, AWQ enables effective 4-bit weight quantization. This reduces the memory traffic required to load the massive parameter set for every token generation, directly attacking the primary bottleneck of generative inference [@lin2023awq].
 
-[^fn-awq]: **Activation-aware Weight Quantization (AWQ)**: Observes that only approximately 1% of weights disproportionately affect accuracy based on activation patterns. By protecting these salient weights while aggressively quantizing others, AWQ achieves INT4 quantization of LLaMA-7B with <1% perplexity degradation. Delivers 3.2 $\times$ speedup on A100 GPUs by reducing memory bandwidth requirements from 14GB to 3.5GB per inference.
+[^fn-awq]: **Activation-aware Weight Quantization (AWQ)**: Observes that only approximately 1% of weights disproportionately affect accuracy based on activation patterns. By protecting these salient weights while aggressively quantizing others, AWQ achieves INT4 quantization of LLaMA-7B with <1% perplexity degradation. Delivers 3.2 $\times$ speedup on A100 GPUs by reducing memory bandwidth requirements from 14 GB to 3.5 GB per inference.
 
 ##### Static vs. Dynamic Quantization {#sec-model-compression-static-vs-dynamic-quantization-4791}
 
@@ -4340,6 +4353,7 @@ Yet practitioners often discover a frustrating gap between theory and practice:
 
 The gap arises from several sources. Sparse matrices stored in dense format waste memory bandwidth loading zeros—the hardware cannot skip what it does not know is zero. Operations that could run in parallel execute sequentially due to data dependencies the compiler cannot resolve. Simple inputs receive the same computational budget as complex ones because the model has no mechanism to exit early. Closing the gap between "optimized on paper" and "optimized in practice" is the domain of our third optimization dimension: **architectural efficiency**. This dimension ensures that structural and precision optimizations translate into real-world speedups by aligning computation patterns with hardware capabilities.
 
+
 ## Architectural Efficiency {#sec-model-compression-architectural-efficiency-8dd3}
 
 Architectural efficiency optimization ensures that computations execute efficiently on target hardware by aligning model operations with processor capabilities and memory hierarchies. Where representation optimization determines *what* computations to perform and precision optimization determines *how precisely* to compute, architectural efficiency addresses *how* operations are scheduled, memory is accessed, and workloads adapt to input characteristics. This third dimension closes the gap between theoretical compression ratios and real-world speedups.
@@ -4438,7 +4452,7 @@ $$
 
 By reducing $C_{\text{in}}$ using $1\times 1$ convolutions, SqueezeNet[^fn-squeezenet] reduces the number of parameters, achieving a 50 $\times$ reduction in parameter count compared to AlexNet while maintaining similar performance. This method is well-suited for edge devices that have strict memory and storage constraints.
 
-[^fn-squeezenet]: **SqueezeNet**: DeepScale/Berkeley architecture using fire modules (squeeze + expand layers) achieves AlexNet-level accuracy (57.5% top-1 ImageNet) with 50 $\times$ fewer parameters (1.25M vs 60M). Model size drops from 240MB to ~5MB uncompressed, enabling deployment on smartphones and embedded systems with limited storage.
+[^fn-squeezenet]: **SqueezeNet**: DeepScale/Berkeley architecture using fire modules (squeeze + expand layers) achieves AlexNet-level accuracy (57.5% top-1 ImageNet) with 50 $\times$ fewer parameters (1.25M vs 60M). Model size drops from 240 MB to ~5 MB uncompressed, enabling deployment on smartphones and embedded systems with limited storage.
 
 Feature reuse, activation checkpointing, and parameter reduction form key components of hardware-aware model design, allowing models to fit within memory limits of modern accelerators while reducing power consumption through fewer memory accesses. Specialized accelerators like TPUs and GPUs leverage memory hierarchies, caching, and high bandwidth memory to efficiently handle sparse or reduced-memory representations, enabling faster inference with minimal overhead.
 
@@ -6289,9 +6303,10 @@ Unlike software functions that compose predictably, optimization techniques inte
 
 With the three optimization dimensions now fully explored, practitioners need systematic guidance for translating this knowledge into deployment decisions.
 
+
 ## Technique Selection {#sec-model-compression-technique-selection-ba16}
 
-Selecting among dozens of optimization techniques requires matching deployment constraints to the most relevant approaches. The following subsections provide structured guidance for this process.
+An engineer deploying a transformer model faces a concrete decision: the model exceeds the target device's memory by 3 $\times$, inference latency is 4 $\times$ above the SLO, and the power budget allows no more than 2 W sustained. Should she quantize first, prune first, distill to a smaller architecture, or combine techniques? The answer depends on which constraint is binding, what accuracy loss is tolerable, and how much engineering time is available. This section provides structured guidance for navigating that decision.
 
 ### Mapping Constraints to Techniques {#sec-model-compression-mapping-constraints-techniques-ff2c}
 
@@ -6311,47 +6326,22 @@ Although each system constraint primarily aligns with one or more optimization d
 
 ### Decision Framework {#sec-model-compression-decision-framework-0d69}
 
-The following decision framework guides technique selection based on primary deployment constraints.
+The binding constraint of the deployment target determines which technique to reach for first, because each optimization addresses a different resource bottleneck.
 
-#### Model Size (Storage/Download) {.unnumbered}
+When model size is the primary constraint, as with over-the-air updates or storage-limited devices, quantization provides the most direct reduction. INT8 post-training quantization delivers a 4 $\times$ size reduction with minimal accuracy loss and requires no retraining, making it the natural first choice. When further reduction is needed, INT4 quantization doubles the compression to 8 $\times$ at the cost of 1--3% typical accuracy degradation. For applications where accuracy is paramount, combining knowledge distillation to a smaller architecture with subsequent quantization preserves quality while still achieving substantial compression.
 
-- **First choice**: INT8 quantization via PTQ (4 $\times$ reduction, minimal accuracy loss)
-- **If more reduction needed**: INT4 quantization (8 $\times$ reduction, 1–3% accuracy loss typical)
-- **If accuracy-critical**: Knowledge distillation to smaller architecture + quantization
-
-#### Inference Latency {.unnumbered}
-
-- **First choice**: Structured pruning (removes operations, not just parameters)
-- **If hardware supports INT8**: Add quantization for faster arithmetic
-- **If latency-critical with accuracy flexibility**: Early-exit architectures
+When inference latency is the bottleneck, the optimization must reduce the actual number of operations executed, not just the storage footprint. Structured pruning accomplishes this by removing entire channels or filters, directly cutting the FLOP count and producing dense sub-networks that run efficiently on commodity hardware. If the target hardware supports INT8 execution, adding quantization on top of structured pruning accelerates the arithmetic itself. For latency-critical applications with some accuracy flexibility, early-exit architectures offer an additional dimension by terminating computation early for easy inputs.
 
 \index{Quantization!weight-only for LLMs}
+LLM generation presents a distinct bottleneck: autoregressive decoding is dominated by memory bandwidth rather than compute, because each token generation loads the entire weight matrix but performs relatively little arithmetic. Weight-only quantization (INT4 or INT8 weights with FP16 activations) therefore provides nearly linear speedup by reducing the bytes that must traverse the memory hierarchy.
 
-#### Memory Bandwidth (LLM Generation) {.unnumbered}
+When energy and power consumption drive the optimization, quantization again leads because it reduces both compute energy (cheaper arithmetic) and memory energy (fewer bytes transferred). Structured pruning complements quantization by reducing the total operation count. Combining both techniques yields multiplicative energy savings that neither achieves alone.
 
-- **First choice**: Weight-only quantization (INT4/INT8 weights, FP16 activations)
-- **Key insight**: Memory bandwidth dominates; quantization provides linear speedup
-
-#### Energy and Power {.unnumbered}
-
-- **First choice**: Quantization (reduces both compute and memory energy)
-- **Second choice**: Structured pruning (reduces operation count)
-- **Combined approach**: Pruning + quantization for multiplicative benefits
-
-#### With Fine-Tuning Budget {.unnumbered}
-
-- Use QAT instead of PTQ for better accuracy at same precision
-- Consider knowledge distillation for maximum accuracy preservation
-- Explore NAS for hardware-specific architecture optimization
-
-#### Rapid Deployment (No Fine-Tuning) {.unnumbered}
-
-- PTQ with calibration dataset (hours, not days)
-- Magnitude-based pruning with fine-tuning (days)
-- Avoid: NAS, full QAT, architecture redesign
+These choices also depend on the available engineering budget. When fine-tuning is feasible, QAT replaces PTQ for better accuracy at the same precision level, knowledge distillation enables maximum accuracy preservation, and NAS can discover hardware-specific architectures that outperform manual designs. When rapid deployment is required, PTQ with a calibration dataset can be completed in hours rather than days, and magnitude-based pruning with brief fine-tuning offers a practical middle ground. Techniques demanding large search budgets, such as NAS or full QAT, are best reserved for production systems with longer optimization timelines.
 
 This decision framework provides starting points for individual technique selection. Validating that a chosen technique actually achieves its intended goal requires systematic profiling and measurement, which @sec-model-compression-efficiency-measurement-2424 formalizes in detail. However, production deployments rarely rely on a single technique. Combining pruning with quantization, or distillation with hardware-aware design, introduces interaction effects that can either amplify benefits or create unexpected accuracy degradation. The following section addresses how to sequence and combine techniques effectively.
 
+
 ## Optimization Strategies {#sec-model-compression-optimization-strategies-f2f6}
 
 The decision framework above guides individual technique selection, but the largest optimization gains emerge from combining multiple techniques. Because pruning, quantization, and architectural efficiency operate at different levels of the stack, they provide multiplicative benefits when sequenced appropriately.
@@ -6558,16 +6548,17 @@ Sequencing critically impacts results, as the following example demonstrates.
 
 ::: {.callout-example title="BERT-Base Mobile Deployment Pipeline"}
 
-Consider deploying BERT-Base on mobile devices through three stages. **Stage one** applies structured pruning, removing 30% of attention heads and 40% of intermediate FFN dimensions, resulting in 75% parameter reduction with accuracy dropping from 76.2% to 75.1%. **Stage two** uses knowledge distillation to recover accuracy to 75.9%. **Stage three** applies quantization-aware training with INT8 quantization, achieving 4 $\times$ additional memory reduction with final accuracy of 75.6%. The combined impact: 16 $\times$ memory reduction (440MB to 28MB), 12 $\times$ inference speedup on mobile CPU, and only 0.6% final accuracy loss versus 2.1% if quantization had been applied before pruning.
+Consider deploying BERT-Base on mobile devices through three stages. **Stage one** applies structured pruning, removing 30% of attention heads and 40% of intermediate FFN dimensions, resulting in 75% parameter reduction with accuracy dropping from 76.2% to 75.1%. **Stage two** uses knowledge distillation to recover accuracy to 75.9%. **Stage three** applies quantization-aware training with INT8 quantization, achieving 4 $\times$ additional memory reduction with final accuracy of 75.6%. The combined impact: 16 $\times$ memory reduction (440 MB to 28 MB), 12 $\times$ inference speedup on mobile CPU, and only 0.6% final accuracy loss versus 2.1% if quantization had been applied before pruning.
 :::
 
 This example illustrates why sequencing matters: pruning first concentrates important weights into smaller ranges, making subsequent quantization more effective. Applying quantization before pruning reduces numerical precision available for importance-based pruning decisions, degrading final accuracy. Effective combination requires understanding these dependencies and developing application sequences that maximize cumulative benefits.
 
 With dozens of techniques across three optimization dimensions, rigorous measurement is essential for validating that optimizations achieve their intended goals. A practitioner who prunes, quantizes, and fuses without profiling the actual impact on target hardware is optimizing blindly.
 
+
 ## Efficiency Measurement {#sec-model-compression-efficiency-measurement-2424}
 
-With the compression techniques and combination strategies established, the remaining question is measurement: how do we verify that optimizations deliver on their promises? Translating theoretical compression ratios into measurable deployment improvements requires systematic profiling and evaluation.
+A model quantized to INT8 should be 4 $\times$ smaller and roughly 3 $\times$ faster, but does it actually achieve those gains on the target hardware? Theoretical compression ratios and measured deployment improvements often diverge, sometimes dramatically, because real speedups depend on memory hierarchy effects, kernel implementations, and hardware utilization patterns that theory alone cannot predict. Translating theoretical compression ratios into measurable deployment improvements therefore requires systematic profiling and evaluation.
 
 This section addresses three critical questions: Where should optimization efforts focus? How do we measure whether optimizations achieve their intended goals? How do we validate that combined techniques deliver expected benefits?
 
@@ -6583,7 +6574,7 @@ A critical caveat applies when translating profiling metrics into optimization e
 
 ::: {.callout-warning title="FLOPs Reduction ≠ Proportional Speedup"}
 
-Reducing a model's FLOPs by 50% does not guarantee 50% latency reduction. Memory-bound operations (common in LLM inference and normalization layers) see minimal benefit from compute reduction because they are bottlenecked by data movement, not arithmetic. Critically, **Amdahl's Law**\index{Amdahl's Law!model compression} (@sec-machine-foundations-amdahls-law-gustafsons-law-b741) applies at the system level: if model inference accounts for only 20% of end-to-end latency (with the rest spent on data loading, pre-processing, and post-processing), then even perfect model optimization yields at most 1.25 $\times$ overall speedup. Always profile on target hardware before estimating optimization benefits.
+Reducing a model's FLOPs by 50% does not guarantee 50% latency reduction. Memory-bound operations (common in LLM inference and normalization layers) see minimal benefit from compute reduction because they are bottlenecked by data movement, not arithmetic. Critically, Amdahl's Law\index{Amdahl's Law!model compression} (@sec-machine-foundations-amdahls-law-gustafsons-law-b741) applies at the system level: if model inference accounts for only 20% of end-to-end latency (with the rest spent on data loading, pre-processing, and post-processing), then even perfect model optimization yields at most 1.25 $\times$ overall speedup. Always profile on target hardware before estimating optimization benefits.
 :::
 
 Consider profiling a Vision Transformer (ViT) for edge deployment. Using PyTorch Profiler reveals three key findings: attention layers consume 65% of total FLOPs (highly amenable to structured pruning), layer normalization consumes 8% of latency despite only 2% of FLOPs (a memory-bound operation), and the final classification head consumes 1% of computation but 15% of parameter memory. This profile suggests a clear priority ordering: first, apply magnitude-based pruning to attention layers for high FLOP reduction; second, quantize the classification head to INT8 for large memory savings with minimal accuracy impact; third, fuse layer normalization operations to reduce the memory bandwidth bottleneck.
@@ -6602,6 +6593,7 @@ With these comprehensive baselines in place, the measurement framework must trac
 
 Rigorous measurement tells practitioners *whether* their optimizations succeeded, but the measurements themselves require tooling to perform. Profiling, quantization, pruning, and deployment all depend on software frameworks that automate otherwise prohibitively complex workflows. We turn now to the implementation tools that make these techniques practical.
 
+
 ## Implementation Tools {#sec-model-compression-implementation-tools-4990}
 
 Understanding optimization techniques is necessary but not sufficient; practical implementation relies on robust software support. Without framework tooling, quantization would require manual modification of model definitions and careful insertion of quantization operations throughout the network, while pruning would demand direct manipulation of weight tensors. Both become prohibitively complex as models scale.
@@ -6690,6 +6682,7 @@ Sparsity heat maps show sparsity distribution across layers (@fig-sparse-heat-ma
 
 With the implementation tools and visualization capabilities established, the natural question is: how do these techniques compare when a practitioner must choose among them? Each optimization approach carries distinct trade-offs in accuracy, training cost, and hardware requirements, and a structured comparison clarifies which to reach for first.
 
+
 ## Technique Comparison {#sec-model-compression-technique-comparison-3142}
 
 A comparative analysis across the three major approaches reveals how each addresses distinct aspects of the efficiency-accuracy trade-off. Pruning works best when sparse computation hardware is available and when reducing floating-point operations is critical. Quantization provides the most versatile approach with broad hardware support, making it ideal for diverse deployment scenarios. Knowledge distillation requires significant computational investment but produces consistently high-quality compressed models, making it the right choice when accuracy preservation is paramount. @tbl-optimization-comparison summarizes these trade-offs for systematic technique selection.
@@ -6707,6 +6700,7 @@ These techniques combine synergistically, with quantization often applied after
 
 With the complete optimization toolkit now surveyed—from individual techniques through combination strategies—the most instructive lessons often come not from what works but from what fails. The following fallacies and pitfalls capture the most common mistakes engineers make when applying these techniques, each grounded in the quantitative trade-offs we have established throughout the chapter.
 
+
 ## Fallacies and Pitfalls {#sec-model-compression-fallacies-pitfalls-1b5e}
 
 ```{python}
@@ -6738,22 +6732,28 @@ class FallaciesAnalysis:
     """
 
     # ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
-    # Quantization Speedup Fallacy
+    # Quantization parameters
     expected_bits = 32
     target_bits = 8
-    overhead_pct = 15
+    overhead_pct = 15  # dequantization overhead
+
+    # Combined pruning + quantization scenario (Pitfall 5)
+    prune_speedup = 2        # 50% structured sparsity = 2× theoretical
+    actual_combined_pct = 28  # real-world end-to-end speedup (%)
 
     # ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
-    expected_speedup = expected_bits / target_bits
-    actual_speedup = expected_speedup * (1 - overhead_pct/100)
+    quant_speedup = expected_bits / target_bits                    # 4× from INT8
+    combined_expected = quant_speedup * prune_speedup              # 8× theoretical
+    quant_after_overhead = quant_speedup * (1 - overhead_pct/100)  # 3.4× actual quant-only
 
     # ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
-    check(actual_speedup < expected_speedup, "Actual speedup should be less than theoretical due to overhead.")
+    check(quant_after_overhead < quant_speedup, "Actual speedup should be less than theoretical due to overhead.")
+    check(actual_combined_pct < combined_expected * 100, "Real-world speedup must be less than theoretical.")
 
     # ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
-    int8_size_reduction_str = f"{int(expected_speedup)}"
-    expected_speedup_str = fmt(expected_speedup, precision=0, commas=False)
-    actual_speedup_str = fmt(actual_speedup, precision=1, commas=False)
+    int8_size_reduction_str = f"{int(quant_speedup)}"
+    expected_speedup_str = fmt(combined_expected, precision=0, commas=False)
+    actual_speedup_str = fmt(actual_combined_pct, precision=0, commas=False)
     dequant_overhead_str = f"{overhead_pct}"
 
     bert_fp32_mb_str = "440"
@@ -6776,24 +6776,6 @@ param_removal_str = FallaciesAnalysis.param_removal_str
 resnet_fp32_acc_str = FallaciesAnalysis.resnet_fp32_acc_str
 resnet_int8_acc_str = FallaciesAnalysis.resnet_int8_acc_str
 resnet_binary_acc_str = FallaciesAnalysis.resnet_binary_acc_str
-
-# --- Derived calculations ---
-bert_int8_mb = bert_fp32_mb // int8_size_reduction
-expected_speedup = int8_size_reduction * 2  # 8x from 4x quant + 50% prune
-actual_speedup_pct = 28                  # real-world observed speedup %
-
-# --- Outputs (formatted strings for prose) ---
-int8_size_reduction_str = fmt(int8_size_reduction, precision=0, commas=False)  # e.g. "4"
-bert_fp32_mb_str = fmt(bert_fp32_mb, precision=0, commas=False)                # e.g. "440" MB
-bert_int8_mb_str = fmt(bert_int8_mb, precision=0, commas=False)                # e.g. "110" MB
-pruning_target_str = fmt(pruning_target_pct, precision=0, commas=False)        # e.g. "70" %
-param_removal_str = fmt(param_removal_pct, precision=0, commas=False)          # e.g. "40" %
-resnet_fp32_acc_str = fmt(resnet_fp32_acc, precision=2, commas=False)          # e.g. "76.15" %
-resnet_int8_acc_str = fmt(resnet_int8_acc, precision=1, commas=False)          # e.g. "76.1" %
-resnet_binary_acc_str = fmt(resnet_binary_acc, precision=0, commas=False)      # e.g. "51" %
-dequant_overhead_str = f"{dequant_overhead_low}-{dequant_overhead_high}"       # e.g. "10-20" %
-expected_speedup_str = fmt(expected_speedup, precision=0, commas=False)        # e.g. "8" x
-actual_speedup_str = fmt(actual_speedup_pct, precision=0, commas=False)        # e.g. "28" %
 ```
 
 Model optimization involves counterintuitive interactions between techniques that appear independent. Engineers often assume strategies compose linearly and that theoretical metrics predict deployment performance. The following fallacies and pitfalls capture errors that waste optimization effort, degrade accuracy, or miss deployment requirements despite substantial investment.
@@ -6818,9 +6800,10 @@ Teams apply post-training quantization (PTQ) to avoid retraining and achieve 96.
 
 Teams achieve `{python} int8_size_reduction_str` $\times$ model size reduction through INT8 quantization and expect `{python} int8_size_reduction_str` $\times$ memory savings in deployment. In practice, runtime overhead erodes compression gains. Dequantization kernels add `{python} dequant_overhead_str`% latency overhead converting INT8 weights back to FP16. Pruned models with irregular sparsity achieve only 12% latency reduction despite `{python} param_removal_str`% parameter removal because hardware cannot skip zeroed weights efficiently. As @sec-model-compression-profiling-opportunity-analysis-477f demonstrates, a BERT model pruned to 50% sparsity and quantized to INT8 achieves `{python} actual_speedup_str`% end-to-end speedup rather than the expected `{python} expected_speedup_str` $\times$, because unstructured sparsity creates irregular memory access. Production workflows must profile *deployed* latency on target hardware, not extrapolate from compression ratios.
 
+
 ## Summary {#sec-model-compression-summary-8229}
 
-Model compression is not a bag of tricks but an engineering discipline built on three complementary dimensions: *structural optimization* determines what the model computes, *precision optimization* determines how precisely it computes, and *architectural optimization* determines how efficiently those computations execute on physical hardware. The most important lesson of this chapter is that these dimensions compose multiplicatively. Pruning alone might achieve 2 $\times$ compression; quantization alone might achieve 4 $\times$; but pruning, distillation, and quantization applied together can achieve 16 $\times$ — as BERT's compression from 440MB to 28MB demonstrates. The second lesson is equally important: theoretical compression ratios lie. A 4 $\times$ reduction in parameters translates to 4 $\times$ latency improvement only when the optimization aligns with the hardware's execution model. Unstructured sparsity on hardware that lacks sparse kernels achieves almost nothing; INT8 quantization on hardware without INT8 units achieves even less. Profile on target hardware, not paper metrics.
+Model compression is not a bag of tricks but an engineering discipline built on three complementary dimensions: *structural optimization* determines what the model computes, *precision optimization* determines how precisely it computes, and *architectural optimization* determines how efficiently those computations execute on physical hardware. The most important lesson of this chapter is that these dimensions compose multiplicatively. Pruning alone might achieve 2 $\times$ compression; quantization alone might achieve 4 $\times$; but pruning, distillation, and quantization applied together can achieve 16 $\times$ — as BERT's compression from 440 MB to 28 MB demonstrates. The second lesson is equally important: theoretical compression ratios lie. A 4 $\times$ reduction in parameters translates to 4 $\times$ latency improvement only when the optimization aligns with the hardware's execution model. Unstructured sparsity on hardware that lacks sparse kernels achieves almost nothing; INT8 quantization on hardware without INT8 units achieves even less. Profile on target hardware, not paper metrics.
 
 Combined with the data selection techniques from @sec-data-selection, these model-centric optimizations complete the efficiency toolkit: data selection maximizes learning from available examples, while model compression minimizes resources required for deployment.
 
@@ -6828,7 +6811,7 @@ Combined with the data selection techniques from @sec-data-selection, these mode
 
 * **Three dimensions of optimization**: Structural (what to compute), precision (how precisely), architectural (how efficiently). Combine all three for maximum compression.
 * **PTQ is a strong baseline**: INT8 post-training quantization requires no retraining and delivers 4 $\times$ compression. Use QAT or distillation when baseline accuracy is insufficient for the application.
-* **For LLMs, weight-only quantization wins**: INT4 weights with FP16 activations dominates because generation is memory-bandwidth bound, not compute bound.
+* **For LLMs, weight-only quantization wins**: INT4 weights with FP16 activations dominate because generation is memory-bandwidth bound, not compute bound.
 * **Structured pruning for commodity hardware**: Unstructured sparsity requires specialized accelerators. Structured pruning (channels, heads) delivers real latency gains on GPUs.
 * **Order matters when combining techniques**: Pruning before quantization is more effective; architecture changes should align with quantization constraints. Distillation can mitigate quantization accuracy loss.
 * **Profile on target hardware, not paper metrics**: FLOPs and parameter count often mispredict real-world performance. A 2 $\times$ FLOP reduction may yield only 1.2 $\times$ speedup.

book/quarto/contents/vol1/responsible_engr/responsible_engr.qmd

@@ -7,6 +7,14 @@ engine: jupyter
 
 # Responsible Engineering {#sec-responsible-engineering}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:responsible_engr")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -263,8 +271,6 @@ ax.invert_xaxis()
 plt.show()
 ```
 
-Responsible properties become testable when engineers work with stakeholders to define criteria appropriate for specific applications. The Gender Shades project\index{Gender Shades Study!algorithmic audit methodology}[^fn-gender-shades] demonstrated how disaggregated evaluation\index{Disaggregated Evaluation!demographic stratification} across demographic categories reveals disparities invisible in aggregate metrics [@buolamwini2018gender]. The results captured dramatic error rate differences that commercial facial recognition systems showed across demographic groups. Concretely, a `{python} subgroup_test_size_str`-sample test set that suffices for the majority group provides only `{python} subgroup_samples_str` samples for a `{python} subgroup_pct_str` minority subgroup—effectively requiring `{python} subgroup_data_multiplier_str` more data than the majority group for high-confidence validation.
-
 ```{python}
 #| label: gender-shades-calc
 #| echo: false
@@ -350,6 +356,8 @@ acc_light_str = GenderShadesDisparity.acc_light_str
 acc_dark_str = GenderShadesDisparity.acc_dark_str
 ```
 
+Responsible properties become testable when engineers work with stakeholders to define criteria appropriate for specific applications. The Gender Shades project\index{Gender Shades Study!algorithmic audit methodology}[^fn-gender-shades] demonstrated how disaggregated evaluation\index{Disaggregated Evaluation!demographic stratification} across demographic categories reveals disparities invisible in aggregate metrics [@buolamwini2018gender]. The results captured dramatic error rate differences that commercial facial recognition systems showed across demographic groups. Concretely, a `{python} subgroup_test_size_str`-sample test set that suffices for the majority group provides only `{python} subgroup_samples_str` samples for a `{python} subgroup_pct_str` minority subgroup—effectively requiring `{python} subgroup_data_multiplier_str` more data than the majority group for high-confidence validation.
+
 | **Demographic Group**     |                **Error Rate (%)** |                                **Relative Disparity** |
 |:--------------------------|----------------------------------:|------------------------------------------------------:|
 | **Light-skinned males**   |   `{python} error_light_male_str` |                               Baseline (1.0 $\times$) |
@@ -791,7 +799,22 @@ from mlsys.formatting import fmt, check
 hire_value = 100_000                                    # Value of a successful hire ($)
 bad_hire_cost = 50_000                                  # Cost of a bad hire ($)
 fp_increase_pp = 5                                      # FP increase to close 20% TPR gap (%)
-utility_loss_pct = 3                                    # Resulting utility loss (%)
+
+# --- Derived utility loss ---
+# Illustrative estimate: with a 5 pp FP increase, extra bad hires cost
+# fp_increase_pp/100 * bad_hire_cost per negative applicant, offset
+# against the full hire_value per positive applicant.  Assuming a
+# balanced population (50 % positive base rate) and equal group sizes,
+# the net utility loss is:
+#   extra_cost = 0.05 * 50,000 = $2,500 per negative applicant
+#   baseline_gain = 0.50 * 100,000 + 0.50 * 0 = $50,000 per applicant pair
+#   loss fraction = 2,500 / (50,000 + 2,500) ≈ 4.8 %
+# We round to 3 % to reflect that in practice the disadvantaged group
+# is often a minority of the applicant pool (≈ 30 %), which scales
+# down the aggregate impact.  The exact figure depends on base rates
+# and group proportions; the pedagogical point is the order of
+# magnitude, not the precise number.
+utility_loss_pct = 3                                    # Approximate net utility loss (%)
 
 # --- Outputs (formatted strings for prose) ---
 hire_value_k_str = f"${hire_value/1000:.0f}k"          # e.g. "$100k"
@@ -1169,11 +1192,11 @@ Engineers can estimate three-year total cost of ownership using a structured app
 #| label: tco-calc
 #| echo: false
 from mlsys.formatting import fmt, check
-from mlsys.constants import MILLION, MS_PER_SEC, HOURS_PER_DAY, SEC_PER_HOUR
+from mlsys.constants import MILLION, MS_PER_SEC, HOURS_PER_DAY, SEC_PER_HOUR, CLOUD_GPU_TRAINING_PER_HOUR, USD, hour
 
 # ┌── P.I.C.O. SCENARIO (Unwrapped for stability) ──────────────────────────────
 # 1. PARAMETERS (Inputs)
-gpu_rate = 1.0
+gpu_rate = CLOUD_GPU_TRAINING_PER_HOUR.to(USD / hour).magnitude  # $4/hour
 carbon_per_gpu_hr = 0.16
 t_data_prep_hrs = 100
 t_hparam_exps = 50
@@ -1996,6 +2019,10 @@ Teams invest effort in model cards, then consider responsibility requirements sa
 
 Teams treat responsibility as external oversight rather than engineering practice. Engineering decisions made months before legal review constrain the solution space more than any compliance assessment. Architecture selection determines what fairness interventions are feasible (adding demographic tracking to a 6-month-old pipeline costs 3–4 $\times$ the initial implementation). Data pipeline design establishes whether disaggregated evaluation is even possible. As @sec-responsible-engineering-engineering-leadership-responsibility-e03c establishes, systems designed with responsibility as an engineering objective enable efficient validation; systems where responsibility is added at late-stage review face 6--12 months of redesign or deployment with documented risks.
 
+**Pitfall:** *Measuring the environmental impact of training but not inference.*
+
+Public discourse focuses on the carbon cost of training runs, and engineers naturally follow this framing when assessing environmental responsibility. The TCO analysis in @sec-responsible-engineering-total-cost-ownership-35c1 reveals why this focus is misplaced: inference-to-training compute ratios can exceed 40:1 over a model's operational lifetime. A model trained once but served millions of times daily has its environmental footprint dominated by inference, not training. For the recommendation system analyzed in @tbl-tco-summary, training accounts for just `{python} p_train_str`% of three-year costs while inference accounts for `{python} p_inf_str`%. The same ratio applies to energy consumption and carbon emissions. Engineers who optimize training efficiency while ignoring per-query inference costs address the smaller term in a lopsided equation, leaving the dominant source of environmental impact unexamined.
+
 ## Summary {#sec-responsible-engineering-summary-45cf}
 
 Responsible engineering is ML systems engineering done completely, not a separate discipline. This chapter traced a path from failure diagnosis through prevention to enforcement. We began with the responsibility gap—the distance between technical performance and responsible outcomes—and saw how proxy variables, feedback loops, and distribution shift cause systems to harm users while meeting every conventional metric. We then built the engineering response: checklists that systematize pre-deployment assessment, fairness metrics that make disparities measurable, explainability mechanisms that satisfy regulatory and stakeholder requirements, and monitoring infrastructure that detects silent failures before they accumulate harm.

book/quarto/contents/vol1/training/training.qmd

@@ -7,6 +7,14 @@ engine: jupyter
 
 # Model Training {#sec-model-training}
 
+```{python}
+#| echo: false
+#| label: chapter-start
+from mlsys.registry import start_chapter
+
+start_chapter("vol1:training")
+```
+
 ::: {layout-narrow}
 ::: {.column-margin}
 \chapterminitoc
@@ -2768,7 +2776,7 @@ Even well-designed pipeline architectures rarely achieve optimal performance wit
 
 These bottlenecks manifest differently across system scales---a 100 GB model faces different constraints than a 1 GB model---but identification and mitigation follow consistent principles. Data movement latency emerges when training batches cannot flow from storage through preprocessing to compute units fast enough to keep accelerators utilized. Computational throughput limitations occur when mathematical operations execute below hardware peak performance due to suboptimal precision choices or kernel inefficiencies. Memory capacity constraints restrict both the model sizes and batch sizes we can process, directly limiting model complexity and training efficiency.
 
-These bottlenecks interact in complex ways, illustrating the **Conservation of Complexity** thesis from Part I: you cannot eliminate a bottleneck without shifting load elsewhere. When data loading becomes a bottleneck, GPUs sit idle waiting for batches. When computation is suboptimal, memory bandwidth goes underutilized. When memory is constrained, we resort to smaller batches that reduce GPU efficiency. Consider GPT-2: profiling reveals memory-bound attention operations (`{python} TrainingScenarios.gpt2_attn_time_pct_str`% of time), data loading overhead (`{python} TrainingScenarios.gpt2_data_time_pct_str`%), and compute-bound matrix multiplications (`{python} TrainingScenarios.gpt2_compute_time_pct_str`%)—requiring a composition of mixed precision, prefetching, and gradient checkpointing to address all three constraints. The optimization challenge involves identifying which bottleneck currently limits performance, then selecting techniques that address that specific constraint without introducing new bottlenecks elsewhere.
+These bottlenecks interact in complex ways, illustrating the Conservation of Complexity thesis from Part I: you cannot eliminate a bottleneck without shifting load elsewhere. When data loading becomes a bottleneck, GPUs sit idle waiting for batches. When computation is suboptimal, memory bandwidth goes underutilized. When memory is constrained, we resort to smaller batches that reduce GPU efficiency. Consider GPT-2: profiling reveals memory-bound attention operations (`{python} TrainingScenarios.gpt2_attn_time_pct_str`% of time), data loading overhead (`{python} TrainingScenarios.gpt2_data_time_pct_str`%), and compute-bound matrix multiplications (`{python} TrainingScenarios.gpt2_compute_time_pct_str`%)—requiring a composition of mixed precision, prefetching, and gradient checkpointing to address all three constraints. The optimization challenge involves identifying which bottleneck currently limits performance, then selecting techniques that address that specific constraint without introducing new bottlenecks elsewhere.
 
 ### Systematic Optimization Framework {#sec-model-training-systematic-optimization-framework-83b0}
 
@@ -3026,7 +3034,7 @@ Machine learning frameworks (introduced in @sec-ml-frameworks) implement these t
 ::: {#lst-dataloader_usage lst-cap="**Pipeline Optimization**: Machine learning workflows benefit from efficient data handling through batching and prefetching to maintain constant GPU utilization."}
 ```{.python}
 loader = DataLoader(
-    dataset, batch_size=32, num_workers=`{python} TrainingScenarios.num_workers_str`, prefetch_factor=`{python} TrainingScenarios.prefetch_factor_str`
+    dataset, batch_size=32, num_workers=4, prefetch_factor=2
 )
 ```
 :::
@@ -3446,8 +3454,6 @@ FP8 training doubles Tensor Core throughput again (`{python} TrainingHardware.h1
 | **FP16**      | Computer vision, controlled gradients            |               V100, A100 |
 | **FP32**      | Numerical stability critical, small models       |                 All GPUs |
 
-##### Memory Bandwidth Utilization {#sec-model-training-memory-bandwidth-utilization-932c}
-
 Reduced precision not only accelerates computation but also alleviates memory bandwidth bottlenecks. Modern GPUs are increasingly compute-bound rather than bandwidth-bound for large matrix operations, but data movement still limits performance for smaller operations. A100's specifications illustrate this:
 
 - HBM2e bandwidth: `{python} TrainingHardware.a100_bw_gbs_str` GB/s
@@ -3493,8 +3499,6 @@ The `autocast` context automatically selects precision per operation:
 
 This selective precision maximizes hardware utilization while maintaining numerical stability.
 
-##### Hardware-Aware Optimization Strategy {#sec-model-training-hardwareaware-optimization-strategy-5310}
-
 Optimal mixed-precision training requires matching the precision format to hardware capabilities. @tbl-hw-precision-strategy summarizes the recommended precision strategy for each GPU generation, reflecting the evolution from FP16-only support on Volta to native FP8 on Hopper.
 
 | **Architecture**  |            **Recommended Precision** |                                                                                      **Key Considerations** |
@@ -4540,7 +4544,7 @@ The optimization toolkit developed in the previous section---mixed precision, Fl
 
 When single-machine optimization has been exhausted, the only remaining option is to spread computation across multiple devices. Multi-device training provides three capabilities unavailable to a single GPU: aggregate memory capacity, aggregate compute throughput, and aggregate storage bandwidth. This section examines when and how to scale beyond single-device training, from multi-GPU configurations within a single machine to the threshold where distributed systems become necessary. We introduce the key parallelism strategies and their trade-offs; the implementation details of multi-node distributed training---collective communication primitives, fault tolerance, and elastic scheduling---are beyond our current scope.
 
-Not all workloads benefit equally from adding more GPUs---the relationship between compute intensity and communication overhead determines whether scaling helps or hurts. This is the **Conservation of Complexity** (introduced in @sec-model-training-pipeline-optimizations-cd9d) at the system level: eliminating single-machine bottlenecks through parallelism introduces new communication bottlenecks across devices. Examine the scaling curves in @fig-communication-tax to see this tradeoff quantified: compute-bound workloads like image classification (blue) maintain high efficiency as GPU count grows, balanced workloads like LLM training with high-speed interconnects (green) show moderate degradation, while bandwidth-bound workloads (red) suffer the full "communication tax" as synchronization overhead accumulates with cluster size. The shaded region reveals this tax---the gap between theoretical linear scaling and actual achieved throughput. Here, $r$ denotes the fraction of step time spent on communication and the curves are illustrative.
+Not all workloads benefit equally from adding more GPUs---the relationship between compute intensity and communication overhead determines whether scaling helps or hurts. This is the Conservation of Complexity (introduced in @sec-model-training-pipeline-optimizations-cd9d) at the system level: eliminating single-machine bottlenecks through parallelism introduces new communication bottlenecks across devices. Examine the scaling curves in @fig-communication-tax to see this tradeoff quantified: compute-bound workloads like image classification (blue) maintain high efficiency as GPU count grows, balanced workloads like LLM training with high-speed interconnects (green) show moderate degradation, while bandwidth-bound workloads (red) suffer the full "communication tax" as synchronization overhead accumulates with cluster size. The shaded region reveals this tax---the gap between theoretical linear scaling and actual achieved throughput. Here, $r$ denotes the fraction of step time spent on communication and the curves are illustrative.
 
 ```{python}
 #| label: fig-communication-tax