github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-21 00:23:30 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

Unify the C3 axis name on Compute

Browse Source 
The book split on the first axis's name: the vol2 introduction's
formal definition list, the C3 diagnostic appendix, the DAM
projection, the fleet quick-reference, and quizzes said 'Computation
(C1)' while the c3-taxonomy figure, the part principles, the
displacement-of-overhead invariant, and all prose triads (4:0) said
'Compute'. Unified on Compute everywhere the word names the axis
(~30 sites), including index subentries (!computation -> !compute),
the Compute-bound regime label, and the MFU underbrace. Generic uses
(prefill computation, Communication-Computation overlap/ratio, SMPC,
MoE stage labels, code identifiers) untouched; anchors unchanged.
Both heading-case checkers' axis allowlists learn 'Compute' so the
'(Compute)' case-study heading label passes as a framework label.
This commit is contained in:
Vijay Janapa Reddi
2026-06-07 20:27:15 -04:00
parent 21702477b1
commit f24bd48672
9 changed files with 45 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
book/cli/commands/headings.py
+3 -1
View File
@@ -95,7 +95,9 @@ SKIP_HEADINGS = {
"1-bit Adam: Compression-aware optimization",
}
DAM_AXES = {"Data", "Algorithm", "Machine", "Computation", "Communication", "Coordination"}
# "Compute" added 2026-06: the C³ first axis was unified from "Computation"
# to "Compute" (the c3-taxonomy figure / part-principles / prose-triad form).
DAM_AXES = {"Data", "Algorithm", "Machine", "Compute", "Computation", "Communication", "Coordination"}
# Concept terms from §10.3 that stay lowercase even at heading sentence start.
CONCEPT_TERMS_LOWER = {
book/quarto/contents/vol2/backmatter/appendix_c3.qmd
+22 -22
View File
@@ -6,19 +6,19 @@ engine: jupyter
## Purpose {.unnumbered}
_When the fleet is slow, where do you look first: Computation, Communication, or Coordination?_
_When the fleet is slow, where do you look first: Compute, Communication, or Coordination?_
In a distributed training cluster, "it is slow" is even less informative than on a single machine. A 4,096-GPU job can miss its throughput target because individual accelerators are underutilized (computation), because gradient synchronization saturates the network fabric (communication), or because checkpoint overhead and failure recovery consume too much wall-clock time (coordination). Without a taxonomy, teams buy more GPUs when they should be upgrading interconnects, or optimize kernels when the real problem is pipeline bubble overhead.
This appendix provides a compact diagnostic framework---**Computation, Communication, Coordination (C$^3$)**\index{C³ Taxonomy!appendix refresher}---and shows how to map fleet-scale symptoms and measurements to the term of the Fleet Law that dominates. It is the fleet-scale extension of the Single-Machine Foundations (@sec-appdx-dam-taxonomy), projecting the same diagnostic philosophy from a single machine to the distributed fleet. It serves as a "first response" checklist before committing to deeper fleet-scale optimization.
This appendix provides a compact diagnostic framework---**Compute, Communication, Coordination (C$^3$)**\index{C³ Taxonomy!appendix refresher}---and shows how to map fleet-scale symptoms and measurements to the term of the Fleet Law that dominates. It is the fleet-scale extension of the Single-Machine Foundations (@sec-appdx-dam-taxonomy), projecting the same diagnostic philosophy from a single machine to the distributed fleet. It serves as a "first response" checklist before committing to deeper fleet-scale optimization.
## How to Use This Appendix {.unnumbered}
This appendix is designed as a reference. Start with the diagnostic summary table, form a hypothesis about which C$^3$ axis dominates, and then pick the tool that can confirm (or falsify) that hypothesis.
When training throughput is low, check MFU, communication fraction, and goodput ratio, then map each to its Computation, Communication, or Coordination axis. When scaling efficiency drops below expectations, use the Fleet Law decomposition to identify which term grew. When cost is exploding, use the C$^3$ scorecard to ensure that effort targets the dominant term, not a nonbottleneck.
When training throughput is low, check MFU, communication fraction, and goodput ratio, then map each to its Compute, Communication, or Coordination axis. When scaling efficiency drops below expectations, use the Fleet Law decomposition to identify which term grew. When cost is exploding, use the C$^3$ scorecard to ensure that effort targets the dominant term, not a nonbottleneck.
The **C$^3$ Taxonomy** is the diagnostic framework for fleet-scale ML systems engineering. Where the Single-Machine Foundations (@sec-appdx-dam-taxonomy) diagnose bottlenecks within a single node---data starvation, algorithmic overhead, or hardware saturation---the C$^3$ taxonomy diagnoses bottlenecks across the distributed fleet. Most fleet-scale performance problems can be diagnosed by identifying the dominant axis: Computation (are the accelerators doing useful math?), Communication (is the network moving data fast enough?), or Coordination (is the system spending too much time on synchronization, failure recovery, and scheduling?). Many production bottlenecks sit at intersections between these axes.
The **C$^3$ Taxonomy** is the diagnostic framework for fleet-scale ML systems engineering. Where the Single-Machine Foundations (@sec-appdx-dam-taxonomy) diagnose bottlenecks within a single node---data starvation, algorithmic overhead, or hardware saturation---the C$^3$ taxonomy diagnoses bottlenecks across the distributed fleet. Most fleet-scale performance problems can be diagnosed by identifying the dominant axis: Compute (are the accelerators doing useful math?), Communication (is the network moving data fast enough?), or Coordination (is the system spending too much time on synchronization, failure recovery, and scheduling?). Many production bottlenecks sit at intersections between these axes.
## From D·A·M to C$^3$ {#sec-appdx-c3-taxonomy-from-dam}
@@ -27,8 +27,8 @@ The C$^3$ taxonomy does not replace D·A·M---it extends it. When a workload mov
| **D·A·M Axis** | **Single-Machine Concern** | **C$^3$ Extension** | **What Changes at Fleet Scale** |
|:----------------------|:----------------------------------:|:-------------------------:|:-----------------------------------------------------------------------------------------------------------------|
| **Data (D)** | I/O bandwidth, disk to GPU | **Communication ($C_2$)** | Data moves across network, not just memory hierarchy |
| **Algorithm (A)** | FLOPs, model depth, ops count | **Computation ($C_1$)** | Per-GPU utilization (MFU) still matters, but scaling efficiency erodes it |
| **Machine (M)** | Peak FLOP/s, hardware limits | **Computation ($C_1$)** | Fleet peak = $N \times$ single-GPU peak, but compound losses reduce effective FLOP/s |
| **Algorithm (A)** | FLOPs, model depth, ops count | **Compute ($C_1$)** | Per-GPU utilization (MFU) still matters, but scaling efficiency erodes it |
| **Machine (M)** | Peak FLOP/s, hardware limits | **Compute ($C_1$)** | Fleet peak = $N \times$ single-GPU peak, but compound losses reduce effective FLOP/s |
| ***(no equivalent)*** | *(overhead term $L_{\text{lat}}$)* | **Coordination ($C_3$)** | New axis: barriers, checkpoints, failure recovery, scheduling---negligible on one machine, dominant at 10K+ GPUs |
: **D·A·M to C$^3$ Mapping**: Each D·A·M axis maps to a C$^3$ counterpart, but Coordination ($C_3$) is genuinely new---it captures overhead that is negligible on a single machine but can consume 40 percent of wall-clock time at fleet scale. {#tbl-c3-dam-mapping}
@@ -43,7 +43,7 @@ With the mapping in place, diagnosis becomes a matter of matching symptoms to th
| **C$^3$ Axis** | **Physical Constraint** | **Symptoms** | **Key Metric** | **High-Leverage Optimization** |
|:--------------------------|:------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------|
| **Computation ($C_1$)** | Arithmetic throughput ($R_{\text{peak}} \times \eta_{\text{hw}}$) | Low MFU, GPU utilization below 80 percent, poor per-GPU performance | MFU (Model FLOPs Utilization) | Kernel optimization, mixed precision, operator fusion (@sec-performance-engineering) |
| **Compute ($C_1$)** | Arithmetic throughput ($R_{\text{peak}} \times \eta_{\text{hw}}$) | Low MFU, GPU utilization below 80 percent, poor per-GPU performance | MFU (Model FLOPs Utilization) | Kernel optimization, mixed precision, operator fusion (@sec-performance-engineering) |
| **Communication ($C_2$)** | Network bandwidth ($\text{BW}_{\text{net}}$) | High AllReduce time, low scaling efficiency, communication > 30 percent of step | Scaling efficiency ($\eta_{\text{scaling}}$), communication fraction ($T_{\text{comm}}(N)/T_{\text{step}}(N)$) | Gradient compression, overlap compute/communication, topology optimization (@sec-collective-communication) |
| **Coordination ($C_3$)** | Synchronization overhead and failure recovery | Low goodput ratio, frequent restarts, large pipeline bubbles, scheduler churn | Goodput ratio ($T_{\text{useful}}/T_{\text{wall}}$) | Async checkpointing, elastic training, faster failure detection (@sec-fault-tolerance-reliability) |
@@ -78,7 +78,7 @@ When $f_{\text{compute}}$ drops below 0.5, the fleet spends more time on communi
Like D·A·M, the C$^3$ axes interact at their boundaries. Production bottlenecks often sit at an intersection where two axes compound.
### Computation $\cap$ communication {#sec-appdx-c3-taxonomy-comp-comm}
### Compute $\cap$ communication {#sec-appdx-c3-taxonomy-comp-comm}
This intersection governs whether the system can hide communication behind computation. The **communication-computation ratio** ($\rho = T_{\text{comm}}(N) / (T_{\text{compute}}/N)$) is the key metric (@sec-appdx-fleet-foundations). When $\rho < 1$, computation takes longer than communication and the network transfer can be overlapped---the system is compute-bound and healthy. When $\rho > 1$, GPUs finish their local work before the network delivers the next round of data, and the system is communication-bound.
@@ -90,7 +90,7 @@ This intersection captures the synchronization cost embedded in communication. E
Engineering at this intersection focuses on reducing barrier sensitivity: asynchronous gradient methods that decouple communication from synchronization, hierarchical AllReduce that limits the blast radius of stragglers, and straggler detection with proactive mitigation. @Sec-fault-tolerance-reliability addresses straggler management.
### Computation $\cap$ coordination {#sec-appdx-c3-taxonomy-comp-coord}
### Compute $\cap$ coordination {#sec-appdx-c3-taxonomy-comp-coord}
This intersection captures the idle compute caused by coordination overhead. Pipeline bubbles are the canonical example: during warmup and cooldown phases of pipeline parallelism, some stages are idle while others compute. Checkpoint writes that block the training loop convert coordination overhead into wasted compute capacity. Failure recovery that requires rolling back and recomputing work transforms a coordination event into a computation penalty.
@@ -122,7 +122,7 @@ In the middle of a production incident, fast heuristics narrow the search space
| **C$^3$ Axis** | **Green (Healthy)** | **Yellow (Investigate)** | **Red (Bottleneck)** |
|:------------------|:-----------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------|
| **Computation** | MFU $>$ `{python} C3TrafficLight.mfu_high_str` | MFU `{python} C3TrafficLight.mfu_low_str`--`{python} C3TrafficLight.mfu_high_str` | MFU $<$ `{python} C3TrafficLight.mfu_low_str` |
| **Compute** | MFU $>$ `{python} C3TrafficLight.mfu_high_str` | MFU `{python} C3TrafficLight.mfu_low_str`--`{python} C3TrafficLight.mfu_high_str` | MFU $<$ `{python} C3TrafficLight.mfu_low_str` |
| **Communication** | Comm fraction $<$ 20 percent | Comm fraction 20--40 percent | Comm fraction $>$ 40 percent |
| **Coordination** | Goodput ratio $>$ 90 percent | Goodput ratio 75--90 percent | Goodput ratio $<$ 75 percent |
@@ -134,7 +134,7 @@ Once the bottleneck axis is identified, @tbl-c3-bottleneck-actions shows which o
| **If the fleet is...** | **Dominant Term** | **Optimization That Works** | **Optimization That is Wasted** |
|:------------------------|:----------------------:|:-----------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------|
| **Computation-bound** | $T_{\text{compute}}/N$ | Better kernels, mixed precision, operator fusion, next-gen accelerators | More network bandwidth (GPUs are not waiting on the network) |
| **Compute-bound** | $T_{\text{compute}}/N$ | Better kernels, mixed precision, operator fusion, next-gen accelerators | More network bandwidth (GPUs are not waiting on the network) |
| **Communication-bound** | $T_{\text{comm}}(N)$ | Gradient compression, compute-comm overlap, hierarchical collectives, InfiniBand upgrade | Faster GPUs (they will just idle faster while waiting for the network) |
| **Coordination-bound** | $T_{\text{sync}}(N)$ | Async checkpointing, elastic training, faster failure detection, fewer pipeline stages | Neither faster GPUs nor faster network (the time is lost to overhead, not to data movement or arithmetic) |
@@ -211,7 +211,7 @@ Theoretical constraints manifest as confusing symptoms in production. These scen
Each case isolates one dominant axis before showing which optimization levers follow from that diagnosis.
### Case 1: The underutilized fleet (Computation) {#sec-appdx-c3-taxonomy-case-1}
### Case 1: The underutilized fleet (Compute) {#sec-appdx-c3-taxonomy-case-1}
#### Symptom {.unnumbered}
@@ -219,11 +219,11 @@ You provision `{python} C3CaseStudies.case1_n_gpus_str` H100 GPUs for a large la
#### Diagnosis {.unnumbered}
The **Computation**\index{C³ Taxonomy!computation} axis is the bottleneck. With `{python} C3CaseStudies.case1_mfu_pct_str` MFU, `{python} C3CaseStudies.case1_wasted_pct_str` of the fleet's arithmetic capacity sits idle on every step. This is not a communication or coordination problem---the network is fast enough and the system is stable. The system is not feeding work to the GPUs efficiently.
The **Compute**\index{C³ Taxonomy!computation} axis is the bottleneck. With `{python} C3CaseStudies.case1_mfu_pct_str` MFU, `{python} C3CaseStudies.case1_wasted_pct_str` of the fleet's arithmetic capacity sits idle on every step. This is not a communication or coordination problem---the network is fast enough and the system is stable. The system is not feeding work to the GPUs efficiently.
#### The fix {.unnumbered}
This is a per-GPU efficiency problem that happens to be multiplied across `{python} C3CaseStudies.case1_n_gpus_str` accelerators. Target the Computation axis:
This is a per-GPU efficiency problem that happens to be multiplied across `{python} C3CaseStudies.case1_n_gpus_str` accelerators. Target the Compute axis:
- **Mixed precision**: Ensure BF16/FP8 Tensor Cores are engaged. A common culprit is FP32 fallback in normalization layers or loss computation.
- **Operator fusion**: Use `torch.compile` or similar just-in-time (JIT) compilation to fuse element-wise operations and reduce kernel launch overhead.
@@ -276,7 +276,7 @@ Target the Coordination axis:
| **Symptom** | **C$^3$ Axis** | **Diagnostic Question** | **Measurement** | **Action** |
|:-----------------------------------------------|:---------------------:|:-----------------------------------------------------------------------------|:--------------------------------------------------------|:-----------------------------------------------------------------------|
| **Low MFU despite fast network** | **Computation** | Are Tensor Cores engaged? Is batch size sufficient for arithmetic intensity? | Per-GPU kernel trace (Nsight/PyTorch Profiler) | Enable mixed precision, increase per-GPU batch size |
| **Low MFU despite fast network** | **Compute** | Are Tensor Cores engaged? Is batch size sufficient for arithmetic intensity? | Per-GPU kernel trace (Nsight/PyTorch Profiler) | Enable mixed precision, increase per-GPU batch size |
| **Throughput plateaus when adding GPUs** | **Communication** | Does AllReduce time grow faster than computation shrinks? | NCCL trace, $\rho$ ratio | Gradient compression, hierarchical collectives, overlap |
| **Frequent job restarts** | **Coordination** | What is the cluster MTBF? Is detection fast enough? | Failure logs, MTBF calculation | Elastic training, faster detection, smaller blast radius |
| **High GPU-hours but slow progress** | **Coordination** | What fraction of GPU-hours produce useful training steps? | Goodput ratio ($T_{\text{useful}}/T_{\text{wall}}$) | Async checkpointing, reduce pipeline stages, eliminate scheduler churn |
@@ -291,11 +291,11 @@ Engineers must measure abstract C$^3$ axes with concrete profiling tools. @Tbl-c
| **C$^3$ Axis** | **Key Metric** | **Primary Tool** | **Secondary Tool** |
|:------------------|:----------------------------:|:-----------------------------------------------|:-----------------------------------------------------------|
| **Computation** | MFU, kernel utilization | PyTorch Profiler (TensorBoard plugin) | Nsight Compute (per-kernel roofline analysis) |
| **Compute** | MFU, kernel utilization | PyTorch Profiler (TensorBoard plugin) | Nsight Compute (per-kernel roofline analysis) |
| **Communication** | AllReduce time, $\rho$ ratio | NCCL debug logs (`NCCL_DEBUG=INFO`) | Nsight Systems (timeline), `ibstat`/`perfquery` (IB) |
| **Coordination** | Goodput ratio, restart count | Cluster scheduler logs (Slurm, K8s event logs) | Custom goodput dashboards (for example, Google ML Goodput) |
: **C$^3$ Tooling Map**: Profiling utilities for diagnosing fleet-scale bottlenecks. Start with the primary tool for quick triage; use secondary tools for deep-dive analysis. Computation tools operate per-GPU; Communication tools operate at the network layer; Coordination tools operate at the cluster/job level. {#tbl-c3-tooling}
: **C$^3$ Tooling Map**: Profiling utilities for diagnosing fleet-scale bottlenecks. Start with the primary tool for quick triage; use secondary tools for deep-dive analysis. Compute tools operate per-GPU; Communication tools operate at the network layer; Coordination tools operate at the cluster/job level. {#tbl-c3-tooling}
```{python}
#| echo: false
@@ -317,7 +317,7 @@ The C$^3$ Scorecard grades fleet efficiency against known thresholds, extending
| **C$^3$ Axis** | **Metric** | **Definition** | **Failing Grade** | **Passing Grade** |
|:------------------|:------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------:|:---------------------------------------:|:---------------------------------------:|
| **Computation** | **MFU** | $\frac{O_{\text{step}}}{R_{\text{peak}} \times T_{\text{step}}}$ | $<$ `{python} C3Scorecard.mfu_low_str` | $>$ `{python} C3Scorecard.mfu_high_str` |
| **Compute** | **MFU** | $\frac{O_{\text{step}}}{R_{\text{peak}} \times T_{\text{step}}}$ | $<$ `{python} C3Scorecard.mfu_low_str` | $>$ `{python} C3Scorecard.mfu_high_str` |
| **Communication** | **Scaling Efficiency** ($\eta_{\text{scaling}}$) | $\frac{T_1}{N \times T_N}$ | $<$ `{python} C3Scorecard.eff_8192_str` | $>$ `{python} C3Scorecard.eff_256_str` |
| **Coordination** | **Goodput Ratio** | $\frac{T_{\text{useful}}}{T_{\text{wall}}}$ or $\frac{\text{useful steps/sec}}{\text{ideal or allocated steps/sec}}$ | $<$ 75 percent | $>$ 90 percent |
@@ -379,7 +379,7 @@ class C3EffectiveFlops:
A fleet's **Effective FLOP/s**\index{Effective FLOP/s} is the usable throughput after compounding three independent C$^3$ losses:
$$\text{Effective} = \text{Peak} \times \underbrace{\text{MFU}}_{\text{Computation}} \times \underbrace{\eta_{\text{scaling}}}_{\text{Communication}} \times \underbrace{\text{Goodput Ratio}}_{\text{Coordination}}$$
$$\text{Effective} = \text{Peak} \times \underbrace{\text{MFU}}_{\text{Compute}} \times \underbrace{\eta_{\text{scaling}}}_{\text{Communication}} \times \underbrace{\text{Goodput Ratio}}_{\text{Coordination}}$$
Each factor maps to one C$^3$ axis. MFU captures per-GPU computation efficiency. Scaling efficiency captures communication overhead as GPUs are added. Goodput ratio captures coordination losses from checkpoints, failures, pipeline bubbles, and maintenance.
@@ -391,7 +391,7 @@ Consider a `{python} C3EffectiveFlops.n_gpus_str`-GPU H100 cluster with `{python
`{python} C3EffectiveFlops.effective_eq_math`
The fleet delivers `{python} C3EffectiveFlops.eff_fraction_pct_str` of its peak capacity as useful training work. The **C$^3$ tax**\index{C³ Taxonomy!tax}---the ratio of peak to effective---is `{python} C3EffectiveFlops.c3_tax_mult_str`: achieving a given effective compute budget requires `{python} C3EffectiveFlops.c3_tax_mult_str` the raw hardware. Broken down by axis: Computation consumes a `{python} C3EffectiveFlops.mfu_pct_str` factor (MFU), Communication consumes a `{python} C3EffectiveFlops.scaling_pct_str` factor (using the `{python} C3EffectiveFlops.scaling_ref_gpus_str`-GPU scaling-efficiency reference as an illustrative proxy), and Coordination consumes a `{python} C3EffectiveFlops.goodput_pct_str` factor (goodput ratio after pipeline bubbles, checkpoints, failures, scheduler preemptions, and maintenance).
The fleet delivers `{python} C3EffectiveFlops.eff_fraction_pct_str` of its peak capacity as useful training work. The **C$^3$ tax**\index{C³ Taxonomy!tax}---the ratio of peak to effective---is `{python} C3EffectiveFlops.c3_tax_mult_str`: achieving a given effective compute budget requires `{python} C3EffectiveFlops.c3_tax_mult_str` the raw hardware. Broken down by axis: Compute consumes a `{python} C3EffectiveFlops.mfu_pct_str` factor (MFU), Communication consumes a `{python} C3EffectiveFlops.scaling_pct_str` factor (using the `{python} C3EffectiveFlops.scaling_ref_gpus_str`-GPU scaling-efficiency reference as an illustrative proxy), and Coordination consumes a `{python} C3EffectiveFlops.goodput_pct_str` factor (goodput ratio after pipeline bubbles, checkpoints, failures, scheduler preemptions, and maintenance).
This is not a failure of engineering---it is the physics of fleet-scale computation. The C$^3$ taxonomy quantifies where the losses occur so that optimization effort targets the dominant term.
@@ -429,11 +429,11 @@ class C3SummaryRecap:
## Summary {#sec-appdx-c3-taxonomy-summary}
The C$^3$ taxonomy provides a systematic framework for diagnosing fleet-scale bottlenecks. Each axis maps to a distinct physical constraint: arithmetic throughput and MFU bound Computation; network bandwidth and collective algorithm efficiency bound Communication; and synchronization overhead, failure recovery, and operational losses bound Coordination. The Fleet Law quantifies these constraints, enabling systematic diagnosis. Use the C$^3$ Traffic Light for quick triage, the Bottleneck Diagnostic Table to choose the right lever, and the C$^3$ Scorecard to grade fleet health.
The C$^3$ taxonomy provides a systematic framework for diagnosing fleet-scale bottlenecks. Each axis maps to a distinct physical constraint: arithmetic throughput and MFU bound Compute; network bandwidth and collective algorithm efficiency bound Communication; and synchronization overhead, failure recovery, and operational losses bound Coordination. The Fleet Law quantifies these constraints, enabling systematic diagnosis. Use the C$^3$ Traffic Light for quick triage, the Bottleneck Diagnostic Table to choose the right lever, and the C$^3$ Scorecard to grade fleet health.
::: {.callout-takeaways title="Where to look first at fleet scale"}
* **Every fleet-scale bottleneck has a dominant C$^3$ axis**: Computation, Communication, or Coordination, with many real bottlenecks spanning intersections. Identify the dominant axis before optimizing.
* **Every fleet-scale bottleneck has a dominant C$^3$ axis**: Compute, Communication, or Coordination, with many real bottlenecks spanning intersections. Identify the dominant axis before optimizing.
* **Measure the C$^3$ Scorecard**: Use MFU $>$ `{python} C3SummaryRecap.mfu_high_str`, Scaling Efficiency $>$ `{python} C3SummaryRecap.eff_256_str`, and Goodput Ratio $>$ 90 percent before investing in optimizations.
* **The C$^3$ tax is multiplicative**: Peak FLOP/s $\times$ MFU $\times$ Scaling Efficiency $\times$ Goodput Ratio = Effective FLOP/s. At `{python} C3SummaryRecap.fleet_gpu_count_str`, expect only ~`{python} C3SummaryRecap.eff_fraction_pct_str` of peak.
* **Coordination is the new axis**: On a single machine, overhead is negligible. At fleet scale, checkpoints, failures, pipeline bubbles, and scheduling consume `{python} C3SummaryRecap.case3_coord_pct_str` or more of wall time.
book/quarto/contents/vol2/backmatter/appendix_dam.qmd
+1 -1
View File
@@ -207,7 +207,7 @@ These single-node grades become most useful when they are carried forward into d
While D·A·M diagnoses the *components* of a single node, the $C^3$ Taxonomy (@sec-appdx-c3-taxonomy) diagnoses the *interactions* of the fleet.
1. **Computation ($C_1$)** inherits the Algorithm and Machine axes, but adds the loss of scaling efficiency.
1. **Compute ($C_1$)** inherits the Algorithm and Machine axes, but adds the loss of scaling efficiency.
2. **Communication ($C_2$)** inherits the **Data** axis, but is governed by the speed of light and network topology rather than just local I/O.
3. **Coordination ($C_3$)** is the "at scale" tie-in that has no single-node equivalent. It represents the coordination tax—the time spent on synchronization, checkpoints, and failure recovery that only emerges at fleet scale.
book/quarto/contents/vol2/backmatter/appendix_fleet.qmd
+4 -4
View File
@@ -10,7 +10,7 @@ _What reference numbers and physical laws should every fleet-scale ML engineer c
Designing a system for a single machine requires knowledge of memory hierarchy latencies, roofline ridge points, and precision trade-offs. The moment a training job spans two machines, however, a new set of numbers takes over. Network latency replaces cache latency as the dominant concern. Component failure rates compound from negligible to inevitable. Communication overhead erodes the scaling efficiency that justifies buying more accelerators in the first place.
This appendix collects the reference numbers and compact models for fleet-scale reasoning. It begins with a comparison of the three system paradigms that underpin this book, then provides the "numbers every fleet engineer should know"---organized around the three axes of Communication, Computation, and Coordination (C$^3$, @sec-appdx-c3-taxonomy). It concludes with the scaling physics and thermal constraints that govern cluster design.
This appendix collects the reference numbers and compact models for fleet-scale reasoning. It begins with a comparison of the three system paradigms that underpin this book, then provides the "numbers every fleet engineer should know"---organized around the three axes of Compute, Communication, and Coordination (C$^3$, @sec-appdx-c3-taxonomy). It concludes with the scaling physics and thermal constraints that govern cluster design.
## How to Use This Appendix {.unnumbered}
@@ -183,14 +183,14 @@ class FleetQuickRef:
|:---------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------|
| **Communication** | NVLink ~`{python} FleetQuickRef.nvlink_ib_ratio_mult_str` IB NDR | Parallelism boundary (in-node vs. cross-node) |
| **Communication** | IB NDR ~`{python} FleetQuickRef.ib_ndr_gb_per_s_str`, ~`{python} FleetQuickRef.ib_ndr_us_str` one-way | Inter-node bandwidth and latency |
| **Computation** | MFU `{python} FleetQuickRef.mfu_range_str`, $\eta_{\text{scaling}}$ ~`{python} FleetQuickRef.eff_1024_str` @ 1K, ~`{python} FleetQuickRef.eff_8192_str` @ 8K | Effective FLOP/s and scaling sanity checks |
| **Compute** | MFU `{python} FleetQuickRef.mfu_range_str`, $\eta_{\text{scaling}}$ ~`{python} FleetQuickRef.eff_1024_str` @ 1K, ~`{python} FleetQuickRef.eff_8192_str` @ 8K | Effective FLOP/s and scaling sanity checks |
| **Coordination** | MTBF 8K: ~`{python} FleetQuickRef.mtbf_8k_min_str`; 100K: ~`{python} FleetQuickRef.mtbf_100k_min_str` | Failure expectation and checkpoint cadence |
| **Coordination** | 175B checkpoint ~`{python} FleetQuickRef.ckpt_175_gb_str` (14 B/param common Adam state) | Recovery and storage sizing |
| **Coordination** | Goodput ~`{python} FleetQuickRef.goodput_str` after overheads | Wall-clock utilization |
| **Power & sustainability** | AI rack ~`{python} FleetQuickRef.rack_ai_kw_str`; air limit ~`{python} FleetQuickRef.air_limit_kw_str` | Cooling feasibility (liquid required above air limit) |
| **Power & sustainability** | PUE liquid ~`{python} FleetQuickRef.pue_liq_str`, typical ~`{python} FleetQuickRef.pue_typ_str`; H100 `{python} FleetQuickRef.h100_tdp_w_str` -> `{python} FleetQuickRef.tenk_gpus_str` ≈ `{python} FleetQuickRef.tenk_it_mw_str` IT $\times$ PUE | Facility load and carbon (see @sec-appdx-fleet-foundations-thermal-power) |
: **Numbers Every Fleet Engineer Should Know (Quick Reference)**: One-page summary of the fleet-scale reference numbers in this section. See @tbl-fleet-bandwidth-hierarchy and @tbl-fleet-latency-hierarchy for Communication, @tbl-fleet-scaling-efficiency for Computation, and @tbl-fleet-mtbf for Coordination. {#tbl-fleet-numbers-quick-ref}
: **Numbers Every Fleet Engineer Should Know (Quick Reference)**: One-page summary of the fleet-scale reference numbers in this section. See @tbl-fleet-bandwidth-hierarchy and @tbl-fleet-latency-hierarchy for Communication, @tbl-fleet-scaling-efficiency for Compute, and @tbl-fleet-mtbf for Coordination. {#tbl-fleet-numbers-quick-ref}
```{python}
#| echo: false
@@ -312,7 +312,7 @@ class FleetComputationNumbers:
gpu_mttf_hours_str = fmt_time(Systems.Reliability.Gpu.mttf_hours, "hour", precision=0, commas=True, style="word")
```
### Computation numbers {.unnumbered}
### Compute numbers {.unnumbered}
Raw peak FLOP/s is a necessary but misleading metric for fleet capacity planning. Two multiplicative losses---Model FLOPs Utilization (MFU) and scaling efficiency---reduce effective throughput dramatically. Understanding these losses transforms fleet sizing from guesswork into engineering.
book/quarto/contents/vol2/conclusion/conclusion_concepts.yml
+1 -1
View File
@@ -67,7 +67,7 @@ concept_map:
technical_terms:
- Fleet Stack
- Machine Learning Fleet
- Computation
- Compute
- Communication
- Coordination
- Ring AllReduce
book/quarto/contents/vol2/distributed_training/distributed_training.qmd
+2 -2
View File
@@ -666,7 +666,7 @@ class Scaling8GPU:
Performance results:
- Computation: `{python} Scaling8GPU.compute_8gpu_ms_str` per step
- Compute: `{python} Scaling8GPU.compute_8gpu_ms_str` per step
- Communication: `{python} Scaling8GPU.comm_8gpu_ms_str` per step
- Total: `{python} Scaling8GPU.total_8gpu_ms_str` per step
- Speedup (throughput): `{python} Scaling8GPU.speedup_8gpu_mult_str`
@@ -744,7 +744,7 @@ class Scaling32GPU:
Performance results:
- Computation: `{python} Scaling32GPU.compute_32gpu_ms_str` (`{python} Scaling32GPU.compute_pct_str` of time)
- Compute: `{python} Scaling32GPU.compute_32gpu_ms_str` (`{python} Scaling32GPU.compute_pct_str` of time)
- Communication: `{python} Scaling32GPU.comm_32gpu_ms_str` (`{python} Scaling32GPU.comm_pct_str` of time)
- Total: `{python} Scaling32GPU.total_32gpu_ms_str` per step
- Speedup (throughput): `{python} Scaling32GPU.speedup_32gpu_mult_str` faster → `{python} Scaling32GPU.training_32gpu_hours_str`
book/quarto/contents/vol2/introduction/introduction.qmd
+2 -2
View File
@@ -1547,11 +1547,11 @@ Beyond legal compliance, the Machine Learning Fleet carries ethical obligations.
Scaling from one machine to a fleet introduces three new fundamental resources that compete for wall-clock time. The **C$^3$ Taxonomy**\index{C³ Taxonomy} extends the D·A·M lens to the fleet, identifying the physical and logical boundaries of the Machine Learning Fleet. Every fleet-scale engineering decision revolves around balancing these three anchor points:
1. **Computation ($C_1$ - The Math)**\index{C³ Taxonomy!computation}: The local execution of matrix operations on individual accelerators. This is the domain of TFLOP/s, memory bandwidth, and arithmetic intensity. At scale, the goal is to keep the math engine running at peak utilization.
1. **Compute ($C_1$ - The Math)**\index{C³ Taxonomy!compute}: The local execution of matrix operations on individual accelerators. This is the domain of TFLOP/s, memory bandwidth, and arithmetic intensity. At scale, the goal is to keep the math engine running at peak utilization.
2. **Communication ($C_2$ - The Wire)**\index{C³ Taxonomy!communication}: The movement of data across the network fabric. This is governed by bisection bandwidth and the speed of light. At scale, communication becomes the primary system bottleneck, often taking more time than the math itself.
3. **Coordination ($C_3$ - The Logic)**\index{C³ Taxonomy!coordination}: The synchronous management of state across thousands of nodes. This is the domain of collective algorithms (All-Reduce), fault tolerance, and distributed consensus. Coordination is the "Software Tax" that determines how efficiently $N$ independent nodes can act as a single computer.
These three dimensions form the **Triad of Distributed Efficiency**\index{Triad of Distributed Efficiency}. If the fleet spends too much time on *Communication* or *Coordination*, the expensive *Computation* capacity sits idle. The central challenge is engineering the fleet to minimize the C$^3$ Gap: the difference between theoretical hardware peak and actual distributed throughput. We introduce the three dimensions here as a framing device; @sec-appdx-c3-taxonomy develops the complete taxonomy and the diagnostic structure that lets a reader localize a bottleneck to one of the three C's.
These three dimensions form the **Triad of Distributed Efficiency**\index{Triad of Distributed Efficiency}. If the fleet spends too much time on *Communication* or *Coordination*, the expensive *Compute* capacity sits idle. The central challenge is engineering the fleet to minimize the C$^3$ Gap: the difference between theoretical hardware peak and actual distributed throughput. We introduce the three dimensions here as a framing device; @sec-appdx-c3-taxonomy develops the complete taxonomy and the diagnostic structure that lets a reader localize a bottleneck to one of the three C's.
### The projection: From D·A·M to C$^3$ {#sec-vol2-introduction-projection}
book/quarto/contents/vol2/introduction/introduction_quizzes.json
+5 -5
View File
@@ -303,11 +303,11 @@
"choices": [
"Both are Communication, because they both involve messages on the network.",
"Both are Coordination, because the Fleet Law does not distinguish network time from barrier time.",
"The 12 percent of NCCL transfer time is Communication ($C_2$ — the wire), and the 18 percent of barrier and fault-recovery waiting is Coordination ($C_3$ — the logic of synchronizing state across ranks); Computation ($C_1$) is the remaining useful arithmetic.",
"The 12 percent of NCCL transfer time is Communication ($C_2$ — the wire), and the 18 percent of barrier and fault-recovery waiting is Coordination ($C_3$ — the logic of synchronizing state across ranks); Compute ($C_1$) is the remaining useful arithmetic.",
"The 18 percent of barrier waiting is Compute, because every rank is sitting on a GPU that is technically powered on."
],
"answer": "The correct answer is C. The section defines Communication as the movement of bytes across the fabric and Coordination as the synchronous management of state — barriers, fault recovery, and collective orchestration. NCCL transfer time is data on the wire; barrier-and-recovery time is state-management logic. Conflating them as 'both Communication' or 'both Coordination' collapses a distinction the Fleet Law explicitly uses to diagnose which intervention applies — faster interconnects address Communication, while asynchronous methods or reduced barrier frequency address Coordination. Counting idle barrier time as Compute confuses wall-clock billing with useful arithmetic.",
"learning_objective": "Classify specific wall-clock time shares into the Computation, Communication, and Coordination categories of the C$^3$ taxonomy."
"learning_objective": "Classify specific wall-clock time shares into the Compute, Communication, and Coordination categories of the C$^3$ taxonomy."
},
{
"question_type": "SHORT",
@@ -395,11 +395,11 @@
"question": "A fleet engineer is asked to match each of the three Lighthouse Archetypes to its dominant $C^3$ stress regime as the section defines it. Which matching is correct?",
"choices": [
"Archetype A (GPT-4/Llama-3) stresses Communication through gradient and activation transfers; Archetype B (DLRM at scale) stresses Coordination through all-to-all contention between embedding shards and dense layers under tight latency; Archetype C (Federated MobileNet) stresses the Compute envelope on each unreliable, power-constrained edge device.",
"All three archetypes stress Computation equally, since they are chosen to maximize per-device arithmetic.",
"Archetype A stresses Coordination through fault recovery; Archetype B stresses Computation through dense embeddings; Archetype C stresses Communication through centralized aggregation.",
"All three archetypes stress Compute equally, since they are chosen to maximize per-device arithmetic.",
"Archetype A stresses Coordination through fault recovery; Archetype B stresses Compute through dense embeddings; Archetype C stresses Communication through centralized aggregation.",
"The archetypes are interchangeable because each one is a proxy for any fleet-scale workload regardless of constraint."
],
"answer": "The correct answer is A. The section is explicit: language-model training is Communication-dominant because gradient and activation volumes across large accelerator fleets exceed the bisection budget; DLRM is Coordination-dominant because its all-to-all traffic between embedding shards and dense layers creates routing and placement pressure under tight latency; Federated MobileNet is Compute-dominant at the per-device envelope because each local training step is bound by milliwatt-scale silicon, even though the fleet still must coordinate updates across unreliable devices. The 'all stress Computation' answer ignores the section's point that the archetypes span distinct $C^3$ corners. The swapped mapping inverts the LLM and federated cases. The 'interchangeable' answer erases the pedagogical rationale for having three archetypes in the first place.",
"answer": "The correct answer is A. The section is explicit: language-model training is Communication-dominant because gradient and activation volumes across large accelerator fleets exceed the bisection budget; DLRM is Coordination-dominant because its all-to-all traffic between embedding shards and dense layers creates routing and placement pressure under tight latency; Federated MobileNet is Compute-dominant at the per-device envelope because each local training step is bound by milliwatt-scale silicon, even though the fleet still must coordinate updates across unreliable devices. The 'all stress Compute' answer ignores the section's point that the archetypes span distinct $C^3$ corners. The swapped mapping inverts the LLM and federated cases. The 'interchangeable' answer erases the pedagogical rationale for having three archetypes in the first place.",
"learning_objective": "Classify the three Lighthouse Archetypes by their dominant $C^3$ stress regime as defined in the section."
},
{
book/tools/audit/checks/h3_titlecase.py
+5 -1
View File
@@ -174,7 +174,11 @@ _NAMED_PHRASES = (
# "Machine" or "Data" in body prose would be ambiguous.
_DAM_C3_AXIS_WORDS = (
"Data", "Algorithm", "Machine",
"Computation", "Communication", "Coordination",
# "Compute" added 2026-06 when the C³ first axis was unified from
# "Computation" to "Compute" (the form used by the c3-taxonomy figure,
# part principles, and all prose triads). "Computation" retained for
# any historical headings.
"Compute", "Computation", "Communication", "Coordination",
)
# Sentinels used by the preprocessors. Each sentinel matches `_WORD_RE` as