diff --git a/.codespell-ignore-words.txt b/.codespell-ignore-words.txt index 2e15986d4c..58e2396357 100644 --- a/.codespell-ignore-words.txt +++ b/.codespell-ignore-words.txt @@ -44,3 +44,6 @@ coo trough ehr dout +UserA +UserB +clientA diff --git a/book/quarto/_quarto.yml b/book/quarto/_quarto.yml index bf839f351c..cec4f41155 120000 --- a/book/quarto/_quarto.yml +++ b/book/quarto/_quarto.yml @@ -1 +1 @@ -config/_quarto-pdf-vol2.yml \ No newline at end of file +config/_quarto-pdf-vol1.yml \ No newline at end of file diff --git a/book/quarto/contents/vol1/backmatter/appendix_algorithm.qmd b/book/quarto/contents/vol1/backmatter/appendix_algorithm.qmd index 1234441c33..6b0756496d 100644 --- a/book/quarto/contents/vol1/backmatter/appendix_algorithm.qmd +++ b/book/quarto/contents/vol1/backmatter/appendix_algorithm.qmd @@ -411,20 +411,20 @@ class GPT2TrainingMem: ::: {.callout-notebook title="Worked Example: GPT-2 (1.5B) Training Memory"} -**The model**: GPT-2 XL has $P = `{python} GPT2TrainingMem.P_str`$ parameters, `{python} GPT2TrainingMem.layers_str` layers, hidden dimension $d = `{python} GPT2TrainingMem.d_str`$. +**The model**: GPT-2 XL has $P =$ `{python} GPT2TrainingMem.P_str` parameters, `{python} GPT2TrainingMem.layers_str` layers, hidden dimension $d =$ `{python} GPT2TrainingMem.d_str`. **Model state (fixed per step)**: -- Weights (BF16): $`{python} GPT2TrainingMem.P_str` \times `{python} GPT2TrainingMem.bf16_str`$ bytes = `{python} GPT2TrainingMem.weights_gb_str` GB -- Gradients (BF16): $`{python} GPT2TrainingMem.P_str` \times `{python} GPT2TrainingMem.bf16_str`$ bytes = `{python} GPT2TrainingMem.grads_gb_str` GB -- Optimizer (FP32 master + momentum + variance): $`{python} GPT2TrainingMem.P_str` \times `{python} GPT2TrainingMem.optim_bytes_str`$ bytes = `{python} GPT2TrainingMem.optim_gb_str` GB +- Weights (BF16): `{python} GPT2TrainingMem.P_str` $\times$ `{python} GPT2TrainingMem.bf16_str` bytes = `{python} GPT2TrainingMem.weights_gb_str` GB +- Gradients (BF16): `{python} GPT2TrainingMem.P_str` $\times$ `{python} GPT2TrainingMem.bf16_str` bytes = `{python} GPT2TrainingMem.grads_gb_str` GB +- Optimizer (FP32 master + momentum + variance): `{python} GPT2TrainingMem.P_str` $\times$ `{python} GPT2TrainingMem.optim_bytes_str` bytes = `{python} GPT2TrainingMem.optim_gb_str` GB - **Total model state: `{python} GPT2TrainingMem.model_state_gb_str` GB** — fits on an A100/H100 (`{python} GPT2TrainingMem.accel_gb_str` GB), but leaves only `{python} GPT2TrainingMem.remaining_gb_str` GB for activations. **Activations (scale with batch)**: Per-layer activation memory for a Transformer is approximately $12 \times B \times S \times d$ bytes (in BF16), where $B$ is batch size and $S$ is sequence length. The factor 12 accounts for the major intermediate tensors retained for backpropagation: input activations, QKV projections ($3d$), attention output, FFN intermediate ($4d$), and layer norm/dropout masks. With `{python} GPT2TrainingMem.layers_str` layers, batch size `{python} GPT2TrainingMem.batch_small_str`, and sequence length `{python} GPT2TrainingMem.seq_len_str`: -$`{python} GPT2TrainingMem.layers_str` \times 12 \times `{python} GPT2TrainingMem.batch_small_str` \times `{python} GPT2TrainingMem.seq_len_str` \times `{python} GPT2TrainingMem.d_str` \times `{python} GPT2TrainingMem.bf16_str`$ bytes $\approx$ `{python} GPT2TrainingMem.act_small_gb_str` GB +`{python} GPT2TrainingMem.layers_str` $\times 12 \times$ `{python} GPT2TrainingMem.batch_small_str` $\times$ `{python} GPT2TrainingMem.seq_len_str` $\times$ `{python} GPT2TrainingMem.d_str` $\times$ `{python} GPT2TrainingMem.bf16_str` bytes $\approx$ `{python} GPT2TrainingMem.act_small_gb_str` GB **Total: `{python} GPT2TrainingMem.total_small_gb_str` GB** — fits on one `{python} GPT2TrainingMem.accel_gb_str` GB accelerator. However, increase the batch to `{python} GPT2TrainingMem.batch_large_str` and activations grow to `{python} GPT2TrainingMem.act_large_gb_str` GB, exceeding the remaining capacity. This is the threshold where gradient checkpointing (trading ~33% more compute for $O(\sqrt{L})$ activation memory) becomes necessary. diff --git a/book/quarto/contents/vol1/backmatter/appendix_assumptions.qmd b/book/quarto/contents/vol1/backmatter/appendix_assumptions.qmd index bf2039bcd1..bd401f3ad7 100644 --- a/book/quarto/contents/vol1/backmatter/appendix_assumptions.qmd +++ b/book/quarto/contents/vol1/backmatter/appendix_assumptions.qmd @@ -54,7 +54,7 @@ from mlsys.constants import ( CLOUD_ELECTRICITY_PER_KWH, TFLOPs, second, TB, GB, watt, ) -from mlsys.formatting import fmt, check, sci_latex +from mlsys.formatting import fmt, check, sci_latex, md_math class NapkinMath: # ┌── 1. LOAD ────────────────────────────────────────── @@ -93,7 +93,7 @@ class NapkinMath: train_mem_gb_str = fmt(train_mem_gb, precision=0, commas=False) accel_mem_str = fmt(h100_mem_gb, precision=0, commas=False) a100_tdp_str = fmt(a100_tdp_w, precision=0, commas=False) - gpt3_ops_str = sci_latex(gpt3_ops, precision=2) + gpt3_ops_str = md_math(sci_latex(gpt3_ops, precision=2)) gpt3_days_str = f"~{gpt3_days:,.0f}" elec_rate_str = f"{elec_per_kwh:.2f}" a100_kw_str = f"{a100_tdp_kw:.1f}" @@ -105,11 +105,11 @@ class NapkinMath: The constants in this appendix are not just for auditing—they are designed for quick calculations. Three examples illustrate the pattern. -**Is this workload compute-bound or memory-bound?** Divide peak FLOPS by memory bandwidth to get the *ridge point* (@sec-machine-foundations-roofline-model-2529). For an H100: `{python} NapkinMath.h100_tflops_str` TFLOPS / `{python} NapkinMath.h100_bw_str` TB/s $\approx$ `{python} NapkinMath.ridge_str` FLOP/byte. Any operation with arithmetic intensity above `{python} NapkinMath.ridge_str` FLOP/byte is compute-bound; below it, memory-bound. A large GEMM with $n=`{python} NapkinMath.gemm_n_str`$ has intensity $n/3 \approx `{python} NapkinMath.gemm_intensity_str`$ FLOP/byte (compute-bound). A single-token autoregressive decode has intensity $\approx 1$ FLOP/byte (deeply memory-bound). +**Is this workload compute-bound or memory-bound?** Divide peak FLOPS by memory bandwidth to get the *ridge point* (@sec-machine-foundations-roofline-model-2529). For an H100: `{python} NapkinMath.h100_tflops_str` TFLOPS / `{python} NapkinMath.h100_bw_str` TB/s $\approx$ `{python} NapkinMath.ridge_str` FLOP/byte. Any operation with arithmetic intensity above `{python} NapkinMath.ridge_str` FLOP/byte is compute-bound; below it, memory-bound. A large GEMM with $n=$ `{python} NapkinMath.gemm_n_str` has intensity $n/3 \approx$ `{python} NapkinMath.gemm_intensity_str` FLOP/byte (compute-bound). A single-token autoregressive decode has intensity $\approx 1$ FLOP/byte (deeply memory-bound). -**How much memory does training a `{python} NapkinMath.model_b_str` model require?** Mixed-precision Adam stores 2 bytes (BF16 weights) + 2 bytes (gradients) + 12 bytes (FP32 master weights + momentum + variance) = `{python} NapkinMath.bytes_per_param_str` bytes per parameter. For `{python} NapkinMath.model_b_str` parameters: $`{python} NapkinMath.napkin_model_b_digit_str` \times 10^9 \times `{python} NapkinMath.bytes_per_param_str`$ bytes = `{python} NapkinMath.train_mem_gb_str` GB. An A100 or H100 has `{python} NapkinMath.accel_mem_str` GB of HBM, so the model state alone exceeds a single accelerator—before accounting for activations. +**How much memory does training a `{python} NapkinMath.model_b_str` model require?** Mixed-precision Adam stores 2 bytes (BF16 weights) + 2 bytes (gradients) + 12 bytes (FP32 master weights + momentum + variance) = `{python} NapkinMath.bytes_per_param_str` bytes per parameter. For `{python} NapkinMath.model_b_str` parameters: `{python} NapkinMath.napkin_model_b_digit_str` $\times 10^9 \times$ `{python} NapkinMath.bytes_per_param_str` bytes = `{python} NapkinMath.train_mem_gb_str` GB. An A100 or H100 has `{python} NapkinMath.accel_mem_str` GB of HBM, so the model state alone exceeds a single accelerator—before accounting for activations. -**How much energy does one GPT-3 training run cost?** An A100 draws `{python} NapkinMath.a100_tdp_str` W at TDP. GPT-3 training used roughly $`{python} NapkinMath.gpt3_ops_str`$ FLOPS across `{python} NapkinMath.gpt3_days_str` accelerator-days. At \$`{python} NapkinMath.elec_rate_str`/kWh: `{python} NapkinMath.gpt3_days_str` accelerator-days$\times$ 24 h/day$\times$ `{python} NapkinMath.a100_kw_str` kW$\times$ \$`{python} NapkinMath.elec_rate_str`/kWh $\approx$ `{python} NapkinMath.elec_cost_str` in electricity alone—a small fraction of the total cost, which is dominated by accelerator amortization. +**How much energy does one GPT-3 training run cost?** An A100 draws `{python} NapkinMath.a100_tdp_str` W at TDP. GPT-3 training used roughly `{python} NapkinMath.gpt3_ops_str` FLOPS across `{python} NapkinMath.gpt3_days_str` accelerator-days. At \$`{python} NapkinMath.elec_rate_str`/kWh: `{python} NapkinMath.gpt3_days_str` accelerator-days $\times$ 24 h/day $\times$ `{python} NapkinMath.a100_kw_str` kW $\times$ \$`{python} NapkinMath.elec_rate_str`/kWh $\approx$ `{python} NapkinMath.elec_cost_str` in electricity alone—a small fraction of the total cost, which is dominated by accelerator amortization. ::: @@ -387,8 +387,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:---------------------------|--------------------------------------:|:---------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------|------------------------------------------------:|:-------------------------------------------------| | **V100_FLOPS_FP16_TENSOR** | `{python} Constants.V100_FLOPS_FP16_TENSOR_val` | `{python} Constants.V100_FLOPS_FP16_TENSOR_unit` | | **V100_FLOPS_FP32** | `{python} Constants.V100_FLOPS_FP32_val` | `{python} Constants.V100_FLOPS_FP32_unit` | | **V100_MEM_BW** | `{python} Constants.V100_MEM_BW_val` | `{python} Constants.V100_MEM_BW_unit` | @@ -403,8 +403,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:-------------------------|------------------------------------:|:-------------------------------------| +| **Constant** | **Value** | **Unit** | +|:-------------------------|----------------------------------------------:|:-----------------------------------------------| | **T4_FLOPS_FP16_TENSOR** | `{python} Constants.T4_FLOPS_FP16_TENSOR_val` | `{python} Constants.T4_FLOPS_FP16_TENSOR_unit` | | **T4_FLOPS_INT8** | `{python} Constants.T4_FLOPS_INT8_val` | `{python} Constants.T4_FLOPS_INT8_unit` | | **T4_MEM_BW** | `{python} Constants.T4_MEM_BW_val` | `{python} Constants.T4_MEM_BW_unit` | @@ -420,8 +420,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:---------------------------|--------------------------------------:|:---------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------|------------------------------------------------:|:-------------------------------------------------| | **A100_FLOPS_FP16_TENSOR** | `{python} Constants.A100_FLOPS_FP16_TENSOR_val` | `{python} Constants.A100_FLOPS_FP16_TENSOR_unit` | | **A100_FLOPS_FP32** | `{python} Constants.A100_FLOPS_FP32_val` | `{python} Constants.A100_FLOPS_FP32_unit` | | **A100_FLOPS_INT8** | `{python} Constants.A100_FLOPS_INT8_val` | `{python} Constants.A100_FLOPS_INT8_unit` | @@ -440,8 +440,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:---------------------------|--------------------------------------:|:---------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------|------------------------------------------------:|:-------------------------------------------------| | **H100_FLOPS_FP16_TENSOR** | `{python} Constants.H100_FLOPS_FP16_TENSOR_val` | `{python} Constants.H100_FLOPS_FP16_TENSOR_unit` | | **H100_FLOPS_FP8_TENSOR** | `{python} Constants.H100_FLOPS_FP8_TENSOR_val` | `{python} Constants.H100_FLOPS_FP8_TENSOR_unit` | | **H100_FLOPS_INT8** | `{python} Constants.H100_FLOPS_INT8_val` | `{python} Constants.H100_FLOPS_INT8_unit` | @@ -460,8 +460,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:---------------------------|--------------------------------------:|:---------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------|------------------------------------------------:|:-------------------------------------------------| | **B200_FLOPS_FP16_TENSOR** | `{python} Constants.B200_FLOPS_FP16_TENSOR_val` | `{python} Constants.B200_FLOPS_FP16_TENSOR_unit` | | **B200_FLOPS_FP8_TENSOR** | `{python} Constants.B200_FLOPS_FP8_TENSOR_val` | `{python} Constants.B200_FLOPS_FP8_TENSOR_unit` | | **B200_MEM_BW** | `{python} Constants.B200_MEM_BW_val` | `{python} Constants.B200_MEM_BW_unit` | @@ -478,8 +478,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:---------------------|--------------------------------:|:---------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------|------------------------------------------:|:-------------------------------------------| | **TPUV4_FLOPS_BF16** | `{python} Constants.TPUV4_FLOPS_BF16_val` | `{python} Constants.TPUV4_FLOPS_BF16_unit` | | **TPUV4_MEM_BW** | `{python} Constants.TPUV4_MEM_BW_val` | `{python} Constants.TPUV4_MEM_BW_unit` | @@ -493,8 +493,8 @@ The tables in this section capture the peak compute throughput, memory bandwidth ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:----------------------------|---------------------------------------:|:----------------------------------------| +| **Constant** | **Value** | **Unit** | +|:----------------------------|-------------------------------------------------:|:--------------------------------------------------| | **CPU_FLOPS_FP32** | `{python} Constants.CPU_FLOPS_FP32_val` | `{python} Constants.CPU_FLOPS_FP32_unit` | | **SYSTEM_MEMORY_BW** | `{python} Constants.SYSTEM_MEMORY_BW_val` | `{python} Constants.SYSTEM_MEMORY_BW_unit` | | **MOBILE_NPU_TOPS_INT8** | `{python} Constants.MOBILE_NPU_TOPS_INT8_val` | `{python} Constants.MOBILE_NPU_TOPS_INT8_unit` | @@ -515,22 +515,22 @@ These constants define the parameter counts, per-inference FLOP budgets, and (wh ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:---------------------------|--------------------------------------:|:---------------------------------------| -| **BERT_BASE_FLOPs** | `{python} Constants.BERT_BASE_FLOPs_val` | `{python} Constants.BERT_BASE_FLOPs_unit` | -| **BERT_BASE_PARAMS** | `{python} Constants.BERT_BASE_PARAMS_val` | `{python} Constants.BERT_BASE_PARAMS_unit` | -| **GPT2_HIDDEN_DIM** | `{python} Constants.GPT2_HIDDEN_DIM_val` | `{python} Constants.GPT2_HIDDEN_DIM_unit` | -| **GPT2_LAYERS** | `{python} Constants.GPT2_LAYERS_val` | `{python} Constants.GPT2_LAYERS_unit` | -| **GPT2_PARAMS** | `{python} Constants.GPT2_PARAMS_val` | `{python} Constants.GPT2_PARAMS_unit` | -| **GPT3_PARAMS** | `{python} Constants.GPT3_PARAMS_val` | `{python} Constants.GPT3_PARAMS_unit` | -| **GPT3_TRAINING_DAYS_REF** | `{python} Constants.GPT3_TRAINING_DAYS_REF_val` | `{python} Constants.GPT3_TRAINING_DAYS_REF_unit` | -| **GPT3_TRAINING_OPS** | `{python} Constants.GPT3_TRAINING_OPS_val` | `{python} Constants.GPT3_TRAINING_OPS_unit` | +| **Constant** | **Value** | **Unit** | +|:-----------------------------------|------------------------------------------------:|:-------------------------------------------------| +| **BERT_BASE_FLOPs** | `{python} Constants.BERT_BASE_FLOPs_val` | `{python} Constants.BERT_BASE_FLOPs_unit` | +| **BERT_BASE_PARAMS** | `{python} Constants.BERT_BASE_PARAMS_val` | `{python} Constants.BERT_BASE_PARAMS_unit` | +| **GPT2_HIDDEN_DIM** | `{python} Constants.GPT2_HIDDEN_DIM_val` | `{python} Constants.GPT2_HIDDEN_DIM_unit` | +| **GPT2_LAYERS** | `{python} Constants.GPT2_LAYERS_val` | `{python} Constants.GPT2_LAYERS_unit` | +| **GPT2_PARAMS** | `{python} Constants.GPT2_PARAMS_val` | `{python} Constants.GPT2_PARAMS_unit` | +| **GPT3_PARAMS** | `{python} Constants.GPT3_PARAMS_val` | `{python} Constants.GPT3_PARAMS_unit` | +| **GPT3_TRAINING_DAYS_REF** | `{python} Constants.GPT3_TRAINING_DAYS_REF_val` | `{python} Constants.GPT3_TRAINING_DAYS_REF_unit` | +| **GPT3_TRAINING_OPS** | `{python} Constants.GPT3_TRAINING_OPS_val` | `{python} Constants.GPT3_TRAINING_OPS_unit` | | **GPT4_TRAINING_ACCELERATOR_DAYS** | `{python} Constants.GPT4_TRAINING_GPU_DAYS_val` | `{python} Constants.GPT4_TRAINING_GPU_DAYS_unit` | -| **RESNET50_FLOPs** | `{python} Constants.RESNET50_FLOPs_val` | `{python} Constants.RESNET50_FLOPs_unit` | -| **RESNET50_PARAMS** | `{python} Constants.RESNET50_PARAMS_val` | `{python} Constants.RESNET50_PARAMS_unit` | -| **MOBILENETV2_FLOPs** | `{python} Constants.MOBILENETV2_FLOPs_val` | `{python} Constants.MOBILENETV2_FLOPs_unit` | -| **MOBILENETV2_PARAMS** | `{python} Constants.MOBILENETV2_PARAMS_val` | `{python} Constants.MOBILENETV2_PARAMS_unit` | -| **YOLOV8_NANO_FLOPs** | `{python} Constants.YOLOV8_NANO_FLOPs_val` | `{python} Constants.YOLOV8_NANO_FLOPs_unit` | +| **RESNET50_FLOPs** | `{python} Constants.RESNET50_FLOPs_val` | `{python} Constants.RESNET50_FLOPs_unit` | +| **RESNET50_PARAMS** | `{python} Constants.RESNET50_PARAMS_val` | `{python} Constants.RESNET50_PARAMS_unit` | +| **MOBILENETV2_FLOPs** | `{python} Constants.MOBILENETV2_FLOPs_val` | `{python} Constants.MOBILENETV2_FLOPs_unit` | +| **MOBILENETV2_PARAMS** | `{python} Constants.MOBILENETV2_PARAMS_val` | `{python} Constants.MOBILENETV2_PARAMS_unit` | +| **YOLOV8_NANO_FLOPs** | `{python} Constants.YOLOV8_NANO_FLOPs_val` | `{python} Constants.YOLOV8_NANO_FLOPs_unit` | : **Reference Model Specifications**: Parameter counts and FLOP budgets for the models used in worked examples. These span four orders of magnitude — from MobileNetV2 on a phone to GPT-4 across thousands of accelerators — illustrating how model scale drives every systems decision from memory planning to cluster sizing. {#tbl-assumptions-models} @@ -544,8 +544,8 @@ Energy-per-operation and energy-per-access constants come from semiconductor pro ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:--------------------------------|-------------------------------------------:|:--------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:--------------------------------|-----------------------------------------------------:|:------------------------------------------------------| | **ENERGY_REG_PJ** | `{python} Constants.ENERGY_REG_PJ_val` | `{python} Constants.ENERGY_REG_PJ_unit` | | **ENERGY_SRAM_L1_PJ** | `{python} Constants.ENERGY_SRAM_L1_PJ_val` | `{python} Constants.ENERGY_SRAM_L1_PJ_unit` | | **ENERGY_SRAM_L2_PJ** | `{python} Constants.ENERGY_SRAM_L2_PJ_val` | `{python} Constants.ENERGY_SRAM_L2_PJ_unit` | @@ -570,8 +570,8 @@ Distributed training and multi-accelerator serving are bottlenecked by interconn ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:------------------------------|-----------------------------------------:|:------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------------|---------------------------------------------------:|:----------------------------------------------------| | **NVLINK_V100_BW** | `{python} Constants.NVLINK_V100_BW_val` | `{python} Constants.NVLINK_V100_BW_unit` | | **NVLINK_A100_BW** | `{python} Constants.NVLINK_A100_BW_val` | `{python} Constants.NVLINK_A100_BW_unit` | | **NVLINK_H100_BW** | `{python} Constants.NVLINK_H100_BW_val` | `{python} Constants.NVLINK_H100_BW_unit` | @@ -596,8 +596,8 @@ Cost estimates throughout the book depend on the electricity and cloud pricing a ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:------------------------------|-----------------------------------------:|:------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------------|---------------------------------------------------:|:----------------------------------------------------| | **CLOUD_ELECTRICITY_PER_KWH** | `{python} Constants.CLOUD_ELECTRICITY_PER_KWH_val` | `{python} Constants.CLOUD_ELECTRICITY_PER_KWH_unit` | | **CLOUD_EGRESS_PER_GB** | `{python} Constants.CLOUD_EGRESS_PER_GB_val` | `{python} Constants.CLOUD_EGRESS_PER_GB_unit` | @@ -613,8 +613,8 @@ When reasoning about production ML systems, it helps to have concrete scale anch ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:------------------------------|-----------------------------------------:|:------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------------|---------------------------------------------------:|:----------------------------------------------------| | **GMAIL_EMAILS_PER_DAY** | `{python} Constants.GMAIL_EMAILS_PER_DAY_val` | `{python} Constants.GMAIL_EMAILS_PER_DAY_unit` | | **GOOGLE_SEARCHES_PER_DAY** | `{python} Constants.GOOGLE_SEARCHES_PER_DAY_val` | `{python} Constants.GOOGLE_SEARCHES_PER_DAY_unit` | | **WAYMO_DATA_PER_HOUR_LOW** | `{python} Constants.WAYMO_DATA_PER_HOUR_LOW_val` | `{python} Constants.WAYMO_DATA_PER_HOUR_LOW_unit` | @@ -636,8 +636,8 @@ The constants file defines the base and derived units listed in @tbl-assumptions ::: {.column-page} -| **Constant** | **Value** | **Unit** | -|:-------------|----------------------:|:-----------------------| +| **Constant** | **Value** | **Unit** | +|:-------------|--------------------------------:|:---------------------------------| | **byte** | `{python} Constants.byte_val` | `{python} Constants.byte_unit` | | **KB** | `{python} Constants.KB_val` | `{python} Constants.KB_unit` | | **MB** | `{python} Constants.MB_val` | `{python} Constants.MB_unit` | diff --git a/book/quarto/contents/vol1/backmatter/appendix_data.qmd b/book/quarto/contents/vol1/backmatter/appendix_data.qmd index 6cc46aadbf..be1148b242 100644 --- a/book/quarto/contents/vol1/backmatter/appendix_data.qmd +++ b/book/quarto/contents/vol1/backmatter/appendix_data.qmd @@ -109,11 +109,11 @@ Data gravity[^fn-data-gravity] is not a metaphor; it is a calculation of transfe The transfer time for moving a dataset is simply $T = \text{Data Volume} / \text{Bandwidth}$ (for the large volumes here, latency is negligible; the full equation appears in @sec-machine-foundations-bandwidth-vs-latency-5320). @tbl-data-gravity illustrates the sobering reality: -| **Data Volume** | **1 Gbps (Standard WAN)** | **10 Gbps (High-End WAN)** | **100 Gbps (Direct Connect)** | **Snowmobile (Truck)** | -|:----------------|--------------------------:|---------------------------:|------------------------------:|:-----------------------| -| **1 TB** | `{python} DataGravity.t_1tb_1g_str` | `{python} DataGravity.t_1tb_10g_str` | `{python} DataGravity.t_1tb_100g_str` | N/A | -| **100 TB** | `{python} DataGravity.t_100tb_1g_str` | `{python} DataGravity.t_100tb_10g_str` | `{python} DataGravity.t_100tb_100g_str` | N/A | -| **1 PB** | **`{python} DataGravity.t_1pb_1g_str`** | **`{python} DataGravity.t_1pb_10g_str`** | `{python} DataGravity.t_1pb_100g_str` | **2 Days** | +| **Data Volume** | **1 Gbps (Standard WAN)** | **10 Gbps (High-End WAN)** | **100 Gbps (Direct Connect)** | **Snowmobile (Truck)** | +|:----------------|----------------------------------------:|-----------------------------------------:|----------------------------------------:|:-----------------------| +| **1 TB** | `{python} DataGravity.t_1tb_1g_str` | `{python} DataGravity.t_1tb_10g_str` | `{python} DataGravity.t_1tb_100g_str` | N/A | +| **100 TB** | `{python} DataGravity.t_100tb_1g_str` | `{python} DataGravity.t_100tb_10g_str` | `{python} DataGravity.t_100tb_100g_str` | N/A | +| **1 PB** | **`{python} DataGravity.t_1pb_1g_str`** | **`{python} DataGravity.t_1pb_10g_str`** | `{python} DataGravity.t_1pb_100g_str` | **2 Days** | : **The Cost of Inertia.** Why we "ship compute to data" rather than "ship data to compute." If you have 1 PB, the network is often not a viable option. {#tbl-data-gravity} @@ -163,8 +163,8 @@ Even after data arrives at the machine, we face one final hurdle: the serializat [^fn-serialization]: **Serialization**: From Latin *serialis* (forming a series). The process of converting in-memory data structures into a byte stream for storage or transmission, and the reverse (*deserialization*). In ML pipelines, the choice of serialization format can dominate end-to-end training time; @sec-data-engineering covers pipeline design strategies that minimize this overhead. -| **Format** | **Decoding Speed (MB/s)** | **CPU Cycles per Byte** | **Suitability** | -|:--------------------|--------------------------------------:|-----------------------------------------:|:------------------------| +| **Format** | **Decoding Speed (MB/s)** | **CPU Cycles per Byte** | **Suitability** | +|:--------------------|--------------------------------------------------------:|-----------------------------------------------------------:|:------------------------| | **CSV / JSON** | `{python} SerializationCost.csv_speed_str` MB/s | `{python} SerializationCost.csv_cycles_str` cycles | Debugging only | | **Protobuf** | `{python} SerializationCost.proto_speed_str` MB/s | `{python} SerializationCost.proto_cycles_str` cycles | RPC / Messages | | **Parquet / Arrow** | **`{python} SerializationCost.parquet_speed_str` MB/s** | **`{python} SerializationCost.parquet_cycles_str` cycles** | **High-Scale Training** | @@ -188,7 +188,7 @@ The choice of file format determines the "physics" of how your data is read. Label/.style={text=TextBlack, font=\small\bfseries\usefont{T1}{phv}{m}{n}} } -\tikzset{mycylinder/.style={cylinder, shape border rotate=90, aspect=1.3, draw, fill=white, +\tikzset{mycylinder/.style={cylinder, shape border rotate=90, aspect=1.3, draw, fill=white, minimum width=25mm,minimum height=11mm,line width=\Linewidth,node distance=-0.15}, pics/data/.style = { code = { @@ -219,14 +219,14 @@ fill=\filllcirclecolor,minimum height=6mm,anchor=north]at(PO){}; /channel/.cd, Depth/.store in=\Depth, Height/.store in=\Height, - Width/.store in=\Width, + Width/.store in=\Width, filllcirclecolor/.store in=\filllcirclecolor, filllcolor/.store in=\filllcolor, drawcolor/.store in=\drawcolor, drawcircle/.store in=\drawcircle, scalefac/.store in=\scalefac, Linewidth/.store in=\Linewidth, - picname/.store in=\picname, + picname/.store in=\picname, filllcolor=BrownLine, filllcirclecolor=violet!20, drawcolor=black, @@ -235,7 +235,7 @@ fill=\filllcirclecolor,minimum height=6mm,anchor=north]at(PO){}; Linewidth=0.5pt, Depth=1.3, Height=0.8, - Width=1.1, + Width=1.1, picname=C } @@ -680,8 +680,8 @@ class LogSumExp: check(all(0 <= s <= 1 for s in softmax), "Softmax values must be in [0,1]") # ┌── 4. OUTPUT ──────────────────────────────────────── - # Use \lbrack/\rbrack for LaTeX math (raw [ ] confuses display-math delimiter) - z_str = f"\\lbrack {z[0]}, {z[1]}, {z[2]} \\rbrack" + # Use \lbrack/\rbrack for LaTeX math; md_math wraps in $...$ so prose needs no extra delimiters + z_str = md_math(f"\\lbrack {z[0]}, {z[1]}, {z[2]} \\rbrack") a_str = f"{a}" # Naive exponentials in scientific notation @@ -690,12 +690,12 @@ class LogSumExp: mantissa = v / 10**exp return f"{mantissa:.1f} \\times 10^{{{exp}}}" - naive0_str = _sci(naive_exp[0]) - naive1_str = _sci(naive_exp[1]) - naive2_str = _sci(naive_exp[2]) + naive0_str = md_math(_sci(naive_exp[0])) + naive1_str = md_math(_sci(naive_exp[1])) + naive2_str = md_math(_sci(naive_exp[2])) - # Shifted logits (use \lbrack/\rbrack for LaTeX math) - shifted_str = f"\\lbrack {shifted[0]}, {shifted[1]}, {shifted[2]} \\rbrack" + # Shifted logits (use \lbrack/\rbrack for LaTeX math); md_math wraps in $...$ + shifted_str = md_math(f"\\lbrack {shifted[0]}, {shifted[1]}, {shifted[2]} \\rbrack") # Stable exponentials sexp0_str = f"{stable_exp[0]:.3f}" @@ -720,23 +720,23 @@ lse_shifted2 = LogSumExp.shifted[2] ::: {.callout-notebook title="Worked Example: Log-Sum-Exp in Action"} -**The setup**: A 3-class classifier outputs logits $z = `{python} LogSumExp.z_str`$. +**The setup**: A 3-class classifier outputs logits $z =$ `{python} LogSumExp.z_str`. **Without the trick (naive softmax)**: -$e^{`{python} lse_z0`} \approx `{python} LogSumExp.naive0_str`$, $e^{`{python} lse_z1`} \approx `{python} LogSumExp.naive1_str`$, $e^{`{python} lse_z2`} \approx `{python} LogSumExp.naive2_str`$ +$\exp($`{python} lse_z0`$) \approx$ `{python} LogSumExp.naive0_str`, $\exp($`{python} lse_z1`$) \approx$ `{python} LogSumExp.naive1_str`, $\exp($`{python} lse_z2`$) \approx$ `{python} LogSumExp.naive2_str` These numbers are representable in FP64 but overflow FP32 (max $\approx 3.4 \times 10^{38}$). With FP16 (max $\approx 65{,}504$), even $e^{12}$ overflows. In practice, logits of magnitude 100 are not unusual in deep networks, and FP16/BF16 is the standard training precision—so naive softmax fails routinely. -**With the trick**: Subtract $a = \max(z) = `{python} LogSumExp.a_str`$: +**With the trick**: Subtract $a = \max(z) =$ `{python} LogSumExp.a_str`: -$z - a = `{python} LogSumExp.shifted_str`$ +$z - a =$ `{python} LogSumExp.shifted_str` -$e^{`{python} lse_shifted0`} \approx `{python} LogSumExp.sexp0_str`$, $e^{`{python} lse_shifted1`} \approx `{python} LogSumExp.sexp1_str`$, $e^{`{python} lse_shifted2`} = `{python} LogSumExp.sexp2_str`$ +$\exp($`{python} lse_shifted0`$) \approx$ `{python} LogSumExp.sexp0_str`, $\exp($`{python} lse_shifted1`$) \approx$ `{python} LogSumExp.sexp1_str`, $\exp($`{python} lse_shifted2`$) =$ `{python} LogSumExp.sexp2_str` -Sum = `{python} LogSumExp.sum_str`. LogSumExp = $`{python} LogSumExp.a_str` + \log(`{python} LogSumExp.sum_str`) = `{python} LogSumExp.lse_str`$. +Sum = `{python} LogSumExp.sum_str`. LogSumExp = `{python} LogSumExp.a_str` $+ \log($`{python} LogSumExp.sum_str`$) =$ `{python} LogSumExp.lse_str`. -Softmax: $[`{python} LogSumExp.sexp0_str`/`{python} LogSumExp.sum_str`,\; `{python} LogSumExp.sexp1_str`/`{python} LogSumExp.sum_str`,\; `{python} LogSumExp.sexp2_str`/`{python} LogSumExp.sum_str`] = [`{python} LogSumExp.sm0_str`,\; `{python} LogSumExp.sm1_str`,\; `{python} LogSumExp.sm2_str`]$ +Softmax: $[$`{python} LogSumExp.sexp0_str`/`{python} LogSumExp.sum_str`$,\;$ `{python} LogSumExp.sexp1_str`/`{python} LogSumExp.sum_str`$,\;$ `{python} LogSumExp.sexp2_str`/`{python} LogSumExp.sum_str`$] = [$`{python} LogSumExp.sm0_str`$,\;$ `{python} LogSumExp.sm1_str`$,\;$ `{python} LogSumExp.sm2_str`$]$ All intermediate values are in $[0, 1]$—no overflow risk, even in FP16. diff --git a/book/quarto/contents/vol1/hw_acceleration/hw_acceleration.qmd b/book/quarto/contents/vol1/hw_acceleration/hw_acceleration.qmd index 63c25dc4dc..d0b6c07dd7 100644 --- a/book/quarto/contents/vol1/hw_acceleration/hw_acceleration.qmd +++ b/book/quarto/contents/vol1/hw_acceleration/hw_acceleration.qmd @@ -54,7 +54,7 @@ start_chapter("vol1:hw_acceleration") ## Acceleration Fundamentals {#sec-hardware-acceleration-ai-hardware-acceleration-fundamentals-9b28} \index{D·A·M Taxonomy!machine axis} -@sec-data-selection optimized the Data and @sec-model-compression compressed the Algorithm (Model). The final axis of the D·A·M taxonomy (@sec-introduction) is the Machine. Hardware acceleration exists because of a striking asymmetry in modern computing: arithmetic is *cheap*, but moving data is *expensive*. In the time a modern accelerator computes a thousand floating-point operations, a single value travels from main memory. This inversion, where computation is the abundant resource and bandwidth is the scarce one, is the reason specialized hardware matters for machine learning. +Data was optimized in @sec-data-selection and the Algorithm (Model) was compressed in @sec-model-compression. The final axis of the D·A·M taxonomy (@sec-introduction) is the Machine. Hardware acceleration exists because of a striking asymmetry in modern computing: arithmetic is *cheap*, but moving data is *expensive*. In the time a modern accelerator computes a thousand floating-point operations, a single value travels from main memory. This inversion, where computation is the abundant resource and bandwidth is the scarce one, is the reason specialized hardware matters for machine learning. ::: {.callout-definition title="Hardware Acceleration"} @@ -2059,8 +2059,8 @@ class AcceleratorEconomics: @tbl-accelerator-economics provides representative cost-performance data for common accelerators. Note that these figures are approximate and vary by vendor, region, and purchase volume; the key insight is the trend rather than the absolute numbers. Observe how the cost per TFLOP has collapsed with each generation, even as the absolute power requirement (TDP)\index{TDP!Thermal Design Power} has climbed to nearly 1,000 Watts for flagship units, reflecting the industry's shift toward density over raw unit cost. -| **Accelerator** | **List Price (USD)** | **Peak FLOPS (FP16)** | **Memory Bandwidth** | **Price/Performance** | -|:-----------------------------------|:---------------------------|------------------------------------------:|-------------------------:|:------------------------| +| **Accelerator** | **List Price (USD)** | **Peak FLOPS (FP16)** | **Memory Bandwidth** | **Price/Performance** | +|:-----------------------------------|:------------------------------------------------|---------------------------------------------------------------:|----------------------------------------------:|:---------------------------------------------| | **NVIDIA V100**\index{NVIDIA!V100} | `{python} AcceleratorEconomics.v100_price_str` | `{python} AcceleratorEconomics.v100_tflops` TFLOPS | `{python} AcceleratorEconomics.v100_bw` GB/s | `{python} AcceleratorEconomics.v100_pp_str` | | **NVIDIA A100**\index{NVIDIA!A100} | `{python} AcceleratorEconomics.a100_price_str` | `{python} AcceleratorEconomics.a100_tflops_fp16` TFLOPS | `{python} AcceleratorEconomics.a100_bw` GB/s | `{python} AcceleratorEconomics.a100_pp_str` | | **NVIDIA H100** | `{python} AcceleratorEconomics.h100_price_str` | `{python} AcceleratorEconomics.h100_tflops_tf32` TFLOPS (TF32) | `{python} AcceleratorEconomics.h100_bw` GB/s | `{python} AcceleratorEconomics.h100_pp_str` | @@ -3944,13 +3944,13 @@ Y = 2.0 * X2 + 1.0 # Final result Each operation produces an intermediate tensor that must be written to memory and retrieved for the next operation. On large tensors, this overhead of moving data can outweigh the computational cost of the operations [@shazeer2018mesh]. @tbl-memory-footprint illustrates the memory overhead in a naïve execution model. While only the final result $Y$ is needed, storing multiple intermediate tensors creates unnecessary memory traffic and inefficient memory usage. -| **Tensor** | **Size (MB) for $1024\times1024$ Tensor** | -|:-----------------|:------------------------------------------| -| **X** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | -| **X'** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | -| **X''** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | -| **Y** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | -| **Total Memory** | **`{python} MemoryFootprintCalc.total_mb_str` MB** | +| **Tensor** | **Size (MB) for $1024\times1024$ Tensor** | +|:-----------------|:---------------------------------------------------| +| **X** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | +| **X'** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | +| **X''** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | +| **Y** | `{python} MemoryFootprintCalc.tensor_mb_str` MB | +| **Total Memory** | **`{python} MemoryFootprintCalc.total_mb_str` MB** | : **Intermediate Tensor Storage.** Naive execution models require substantial memory to store intermediate tensors generated by each operation. For a $1024\times1024$ tensor, storing intermediate results (even when only the final output is needed) quadruples the total memory footprint from 4 MB to 16 MB. Minimizing intermediate data storage is essential for improving memory efficiency. {#tbl-memory-footprint} @@ -4050,10 +4050,10 @@ class FusionBenefitsCalc: @tbl-fusion-benefits highlights the impact of operation fusion on memory efficiency. By keeping intermediate results in registers or local memory rather than writing them to main memory, fusion significantly reduces memory traffic. This optimization is especially beneficial on highly parallel architectures like GPUs and TPUs, where minimizing memory accesses translates directly into improved execution throughput. Compared to the naïve execution model, fused execution eliminates the need for storing intermediate tensors, dramatically lowering the total memory footprint and improving overall efficiency. -| **Execution Model** | **Intermediate Tensors Stored** | **Total Memory Usage (MB)** | -|:--------------------|:--------------------------------|----------------------------:| -| **Naïve Execution** | X', X'' | `{python} FusionBenefitsCalc.naive_mb_str` | -| **Fused Execution** | None | `{python} FusionBenefitsCalc.fused_mb_str` | +| **Execution Model** | **Intermediate Tensors Stored** | **Total Memory Usage (MB)** | +|:--------------------|:--------------------------------|-------------------------------------------:| +| **Naïve Execution** | X', X'' | `{python} FusionBenefitsCalc.naive_mb_str` | +| **Fused Execution** | None | `{python} FusionBenefitsCalc.fused_mb_str` | : **Operation Fusion Benefits.** Fused execution reduces memory usage by eliminating the need to store intermediate tensors, directly improving efficiency on memory-bound hardware like GPUs and TPUs. Memory consumption drops from 16 MB in naive execution to 4 MB with fused operations. {#tbl-fusion-benefits} diff --git a/book/quarto/contents/vol1/ml_systems/ml_systems.qmd b/book/quarto/contents/vol1/ml_systems/ml_systems.qmd index 6f6a39ec94..c0011cd6e3 100644 --- a/book/quarto/contents/vol1/ml_systems/ml_systems.qmd +++ b/book/quarto/contents/vol1/ml_systems/ml_systems.qmd @@ -296,12 +296,12 @@ pics/mobile/.style = { @tbl-deployment-paradigms-overview compares the quantitative trade-offs across these four paradigms: -| **Paradigm** | **Where** | **Latency** | **Power** | **Memory** | **Best For** | -|:--------------|:-----------------|---------------------------------------:|:----------------------------------|:-----------|:-----------------------------| -| **Cloud ML** | Data centers | `{python} MLSystemsSetup.cloud_latency_range_str` ms | MW | TB | Training, complex inference | -| **Edge ML** | Local servers | `{python} MLSystemsSetup.edge_latency_range_str` ms | 100s W | GB | Real-time inference, privacy | +| **Paradigm** | **Where** | **Latency** | **Power** | **Memory** | **Best For** | +|:--------------|:-----------------|------------------------------------------------------:|:-------------------------------------------------|:-----------|:-----------------------------| +| **Cloud ML** | Data centers | `{python} MLSystemsSetup.cloud_latency_range_str` ms | MW | TB | Training, complex inference | +| **Edge ML** | Local servers | `{python} MLSystemsSetup.edge_latency_range_str` ms | 100s W | GB | Real-time inference, privacy | | **Mobile ML** | Smartphones | `{python} MLSystemsSetup.mobile_latency_range_str` ms | `{python} MLSystemsSetup.mobile_tdp_range_str` W | GB | Personal AI, offline | -| **TinyML** | Microcontrollers | `{python} MLSystemsSetup.tiny_latency_range_str` ms | mW | KB | Always-on sensing | +| **TinyML** | Microcontrollers | `{python} MLSystemsSetup.tiny_latency_range_str` ms | mW | KB | Always-on sensing | : **The Deployment Spectrum (Conceptual)**: Four paradigms span nine orders of magnitude in power (MW to mW) and memory (TB to KB). This conceptual overview defines each paradigm by its operating regime; @tbl-representative-systems later grounds these categories in specific hardware platforms and quantitative decision thresholds. The hardware specifications and physical constants underpinning these numbers are catalogued in the System Assumptions appendix. {#tbl-deployment-paradigms-overview} @@ -826,21 +826,21 @@ class LatencyConstants: These latencies, organized by category in @tbl-latency-numbers, span eight orders of magnitude: -| **Operation** | **Latency** | **Deployment Implication** | -|:---------------------------------|:------------------------------|:---------------------------------| -| **Compute** | | | +| **Operation** | **Latency** | **Deployment Implication** | +|:---------------------------------|:-----------------------------------------------|:---------------------------------| +| **Compute** | | | | **GPU matrix multiply (per op)** | `{python} LatencyConstants.lat_compute_str` | Compute is rarely the bottleneck | | **NPU inference (MobileNet)** | `{python} LatencyConstants.lat_npu_str` | Mobile can do real-time vision | | **LLM token generation** | `{python} LatencyConstants.lat_llm_str` | Perceived as "typing speed" | -| **Memory** | | | +| **Memory** | | | | **L1 cache hit** | `{python} LatencyConstants.lat_l1_str` | Keep hot data in registers | | **HBM read (GPU)** | `{python} LatencyConstants.lat_hbm_str` | 100$\times$ slower than compute | | **DRAM read (mobile)** | `{python} LatencyConstants.lat_dram_str` | Memory-bound on most devices | -| **Network** | | | +| **Network** | | | | **Same datacenter** | `{python} LatencyConstants.lat_net_dc_str` | Microservices feasible | | **Same region** | `{python} LatencyConstants.lat_net_region_str` | Edge servers viable | | **Cross-region** | `{python} LatencyConstants.lat_net_cross_str` | Batch processing only | -| **ML Operations** | | | +| **ML Operations** | | | | **Wake-word detection (TinyML)** | `{python} LatencyConstants.lat_kws_str` | Always-on feasible at <1 mW | | **Face detection (mobile)** | `{python} LatencyConstants.lat_face_str` | Real-time at 30 FPS | | **GPT-4 first token** | `{python} LatencyConstants.lat_gpt4_str` | User notices delay | @@ -862,13 +862,13 @@ Here, $O$ represents total operations, $R_{peak}$ is peak compute rate, $\eta$ i The **dominant term varies by paradigm and workload**, changing the optimization strategy entirely: -| **Paradigm** | **Dominant Constraint** | **Why** | **Optimization Focus** | -|:------------------------|:--------------------------|:-----------------------------------------------------------|:-------------------------------------| +| **Paradigm** | **Dominant Constraint** | **Why** | **Optimization Focus** | +|:------------------------|:--------------------------|:-----------------------------------------------------------|:---------------------------------------------| | **Cloud Training** | $O/R_{peak}$ (Compute) | Abundant memory/network; FLOPS limit throughput | Maximize accelerator utilization, batch size | -| **Cloud LLM Inference** | $D_{vol}/BW$ (Memory BW) | Autoregressive: ~1 FLOP/byte, memory-bound | KV-caching, quantization, batching | -| **Edge Inference** | $D_{vol}/BW$ (Memory BW) | Limited HBM; models often memory-bound | Model compression, operator fusion | -| **Mobile** | Energy (implicit) | Battery = $\int \text{Power} \cdot dt$; thermal throttling | Reduced precision, duty cycling | -| **TinyML** | $D_{vol}/\text{Capacity}$ | 256 KB total; model must fit on-chip | Extreme compression, binary networks | +| **Cloud LLM Inference** | $D_{vol}/BW$ (Memory BW) | Autoregressive: ~1 FLOP/byte, memory-bound | KV-caching, quantization, batching | +| **Edge Inference** | $D_{vol}/BW$ (Memory BW) | Limited HBM; models often memory-bound | Model compression, operator fusion | +| **Mobile** | Energy (implicit) | Battery = $\int \text{Power} \cdot dt$; thermal throttling | Reduced precision, duty cycling | +| **TinyML** | $D_{vol}/\text{Capacity}$ | 256 KB total; model must fit on-chip | Extreme compression, binary networks | The same ResNet-50 model is **compute-bound**\index{compute-bound vs memory-bound!training vs inference}\index{roofline model!bottleneck analysis} during cloud training (high batch size, high arithmetic intensity) but **memory-bound** during single-image inference (batch=1, low arithmetic intensity) [@williams2009roofline]. Deployment paradigm selection must account for this shift. ::: @@ -1236,23 +1236,23 @@ class HardwareSpectrumSetup: \begingroup\small -| **Category** | **Example Device** | **Processor** | **Memory** | **Storage** | **Power** | **Price Range** | -|:--------------|:--------------------|------------------------------------------------:|---------------------------------------:|:--------------------------------------|----------------------------------------------------------:|:---------------------------------------------------------------| -| **Cloud ML** | Google TPU v4 Pod | `{python} HardwareSpectrumSetup.tpu_chips_str` TPU v4 chips, >1 EFLOP | `{python} HardwareSpectrumSetup.cloud_mem_tb_str` TB HBM2 | Cloud-scale (PB) | ~`{python} HardwareSpectrumSetup.cloud_pwr_mw_str` MW | Cloud service (rental) | -| **Edge ML** | NVIDIA DGX Spark | GB10 Grace Blackwell, 1 PFLOPS AI | `{python} HardwareSpectrumSetup.edge_mem_gb_str` GB LPDDR5x | `{python} HardwareSpectrumSetup.edge_stor_tb_str` TB NVMe | ~`{python} HardwareSpectrumSetup.edge_pwr_w_str` W | ~\$`{python} HardwareSpectrumSetup.edge_price_min_str`–`{python} HardwareSpectrumSetup.edge_price_max_str` | -| **Mobile ML** | Flagship Smartphone | Mobile SoC (CPU + GPU + NPU) | `{python} MobileHardwareSpecs.mobile_ram_range_str` GB RAM | `{python} MobileHardwareSpecs.mobile_storage_range_str` | `{python} LighthouseModels.mobile_tdp_range_str` W | USD 999+ | -| **TinyML** | ESP32-CAM | Dual-core @ 240 MHz | `{python} HardwareSpectrumSetup.tiny_ram_kb_str` KB RAM | `{python} HardwareSpectrumSetup.tiny_flash_mb_str` MB Flash | `{python} HardwareSpectrumSetup.tiny_pwr_min_str`–`{python} HardwareSpectrumSetup.tiny_pwr_max_str` W | \$`{python} HardwareSpectrumSetup.tiny_price_str` | +| **Category** | **Example Device** | **Processor** | **Memory** | **Storage** | **Power** | **Price Range** | +|:--------------|:--------------------|----------------------------------------------------------------------:|------------------------------------------------------------:|:------------------------------------------------------------|------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------| +| **Cloud ML** | Google TPU v4 Pod | `{python} HardwareSpectrumSetup.tpu_chips_str` TPU v4 chips, >1 EFLOP | `{python} HardwareSpectrumSetup.cloud_mem_tb_str` TB HBM2 | Cloud-scale (PB) | ~`{python} HardwareSpectrumSetup.cloud_pwr_mw_str` MW | Cloud service (rental) | +| **Edge ML** | NVIDIA DGX Spark | GB10 Grace Blackwell, 1 PFLOPS AI | `{python} HardwareSpectrumSetup.edge_mem_gb_str` GB LPDDR5x | `{python} HardwareSpectrumSetup.edge_stor_tb_str` TB NVMe | ~`{python} HardwareSpectrumSetup.edge_pwr_w_str` W | ~\$`{python} HardwareSpectrumSetup.edge_price_min_str`–`{python} HardwareSpectrumSetup.edge_price_max_str` | +| **Mobile ML** | Flagship Smartphone | Mobile SoC (CPU + GPU + NPU) | `{python} MobileHardwareSpecs.mobile_ram_range_str` GB RAM | `{python} MobileHardwareSpecs.mobile_storage_range_str` | `{python} LighthouseModels.mobile_tdp_range_str` W | USD 999+ | +| **TinyML** | ESP32-CAM | Dual-core @ 240 MHz | `{python} HardwareSpectrumSetup.tiny_ram_kb_str` KB RAM | `{python} HardwareSpectrumSetup.tiny_flash_mb_str` MB Flash | `{python} HardwareSpectrumSetup.tiny_pwr_min_str`–`{python} HardwareSpectrumSetup.tiny_pwr_max_str` W | \$`{python} HardwareSpectrumSetup.tiny_price_str` | : **Hardware Spectrum (Concrete Platforms)**\index{hardware spectrum!deployment platforms}\index{domain-specific accelerators!datacenter scale}\index{workstation-class accelerators!edge deployment}: Representative devices that instantiate each deployment paradigm from @tbl-deployment-paradigms-overview. Where the conceptual table defines operating regimes, this table provides the specific processors, memory capacities, power envelopes, and price points that practitioners use to match workloads to hardware. The DGX Spark sits at the high end of the edge spectrum; most edge deployments use far smaller devices (e.g., Jetson Orin Nano). We include it to illustrate the *ceiling* of non-cloud deployment. {#tbl-representative-systems} \endgroup -| **Paradigm** | **Compute** | **Memory BW** | **Power** | **Latency** | -|:--------------|:---------------------------------------------|:-------------------------------------|:----------------------------------|:----------------------------------------| -| **Cloud ML** | >`{python} HardwareSpectrumSetup.cloud_thresh_tflops_str` TFLOPS | >`{python} HardwareSpectrumSetup.cloud_thresh_bw_str` GB/s | PUE 1.1–1.3 | 100–500 ms | -| **Edge ML** | ~`{python} HardwareSpectrumSetup.edge_thresh_pflops_str` PFLOPS AI | >`{python} HardwareSpectrumSetup.edge_thresh_bw_str` GB/s | 100s W | `{python} MLSystemsSetup.edge_latency_range_str` ms | -| **Mobile ML** | `{python} MobileHardwareSpecs.mobile_npu_range_str` TOPS | `{python} MobileHardwareSpecs.mobile_bw_range_str` GB/s | <2 W | <`{python} MLSystemsSetup.mobile_latency_range_str` ms | -| **TinyML** | <`{python} HardwareSpectrumSetup.tiny_thresh_tops_str` TOPS | — | <`{python} HardwareSpectrumSetup.tiny_thresh_mw_str` mW | µs | +| **Paradigm** | **Compute** | **Memory BW** | **Power** | **Latency** | +|:--------------|:-------------------------------------------------------------------|:-----------------------------------------------------------|:--------------------------------------------------------|:-------------------------------------------------------| +| **Cloud ML** | >`{python} HardwareSpectrumSetup.cloud_thresh_tflops_str` TFLOPS | >`{python} HardwareSpectrumSetup.cloud_thresh_bw_str` GB/s | PUE 1.1–1.3 | 100–500 ms | +| **Edge ML** | ~`{python} HardwareSpectrumSetup.edge_thresh_pflops_str` PFLOPS AI | >`{python} HardwareSpectrumSetup.edge_thresh_bw_str` GB/s | 100s W | `{python} MLSystemsSetup.edge_latency_range_str` ms | +| **Mobile ML** | `{python} MobileHardwareSpecs.mobile_npu_range_str` TOPS | `{python} MobileHardwareSpecs.mobile_bw_range_str` GB/s | <2 W | <`{python} MLSystemsSetup.mobile_latency_range_str` ms | +| **TinyML** | <`{python} HardwareSpectrumSetup.tiny_thresh_tops_str` TOPS | — | <`{python} HardwareSpectrumSetup.tiny_thresh_mw_str` mW | µs | : **Deployment Decision Thresholds**: Quantitative thresholds that practitioners use to determine deployment feasibility for each paradigm in @tbl-representative-systems. These numbers answer the practical question "can my workload run here?" by specifying the compute, memory bandwidth, and power envelope that each paradigm provides. {#tbl-deployment-thresholds} @@ -1270,8 +1270,8 @@ Each section follows a consistent structure: definition, key characteristics, be # ┌───────────────────────────────────────────────────────────────────────────── # │ GPT-3 TRAINING SCALE: CLOUD ML OPENING EXAMPLE # ├───────────────────────────────────────────────────────────────────────────── -# │ Context: Opening paragraph of @sec-ml-systems-cloud-ml-maximizing- -# │ computational-power-a338 (immediately below). +# │ Context: Opening paragraph of the Cloud ML section +# │ (immediately below). # │ # │ Goal: Compute GPT-3 petaflop-days and provide training cost/scale stats. # │ Show: 3,640 PF-days, 10,000 V100s, 15 days, ~$4.6M cost. @@ -1625,24 +1625,24 @@ class CloudEdgeTCO: **Cloud Implementation** (AWS/GCP pricing, 2024) -| **Cost Component** | **Calculation** | **Annual Cost** | -|:-------------------------|---------------------------------------------------------------------------------------------:|---------------------------:| +| **Cost Component** | **Calculation** | **Annual Cost** | +|:-------------------------|-----------------------------------------------------------------------------------------------------------------------:|----------------------------------------:| | **GPU inference (A10G)** | `{python} CloudEdgeTCO.gpu_instances_str` instances$\times$ 8,760 hrs$\times$ `{python} CloudEdgeTCO.gpu_price_str`/hr | `{python} CloudEdgeTCO.c_gpu_str` | -| **Network egress** | `{python} CloudEdgeTCO.egress_gb_str` GB/day$\times$ 365$\times$ USD 0.09/GB | `{python} CloudEdgeTCO.c_egress_str` | -| **Load balancer** | USD 0.025/hr + LCU charges | `{python} CloudEdgeTCO.c_lb_str` | -| **CloudWatch/logging** | Monitoring, alerts | `{python} CloudEdgeTCO.c_logs_str` | -| **Total Cloud** | | **`{python} CloudEdgeTCO.c_total_str`** | +| **Network egress** | `{python} CloudEdgeTCO.egress_gb_str` GB/day$\times$ 365$\times$ USD 0.09/GB | `{python} CloudEdgeTCO.c_egress_str` | +| **Load balancer** | USD 0.025/hr + LCU charges | `{python} CloudEdgeTCO.c_lb_str` | +| **CloudWatch/logging** | Monitoring, alerts | `{python} CloudEdgeTCO.c_logs_str` | +| **Total Cloud** | | **`{python} CloudEdgeTCO.c_total_str`** | **Edge Implementation** (On-premise NVIDIA T4 server) -| **Cost Component** | **Calculation** | **Annual Cost** | -|:---------------------|------------------------------------------------------------------------------:|---------------------------:| +| **Cost Component** | **Calculation** | **Annual Cost** | +|:---------------------|--------------------------------------------------------------------------------------------------------:|----------------------------------------:| | **Hardware CAPEX** | `{python} CloudEdgeTCO.server_cost_str` server ÷ `{python} CloudEdgeTCO.server_life_str`-year life | `{python} CloudEdgeTCO.e_capex_str` | | **Power (24/7)** | `{python} CloudEdgeTCO.power_str`$\times$ 8,760 hrs$\times$ `{python} CloudEdgeTCO.electricity_str`/kWh | `{python} CloudEdgeTCO.e_power_str` | -| **Cooling overhead** | ~30% of power | `{python} CloudEdgeTCO.e_cool_str` | -| **Network (fiber)** | Fixed line for remote management | `{python} CloudEdgeTCO.e_net_str` | +| **Cooling overhead** | ~30% of power | `{python} CloudEdgeTCO.e_cool_str` | +| **Network (fiber)** | Fixed line for remote management | `{python} CloudEdgeTCO.e_net_str` | | **DevOps labor** | `{python} CloudEdgeTCO.devops_fte_str` FTE$\times$ `{python} CloudEdgeTCO.devops_salary_str` salary | `{python} CloudEdgeTCO.e_labor_str` | -| **Total Edge** | | **`{python} CloudEdgeTCO.e_total_str`** | +| **Total Edge** | | **`{python} CloudEdgeTCO.e_total_str`** | **Break-even Analysis**: @eq-edge-breakeven determines when edge deployment becomes cost-effective. **Edge Fixed Costs** include hardware amortization and maintenance, **Cloud Variable Cost per Unit** is the per-inference cloud pricing, and **Capacity** is the maximum inference rate of the edge system: @@ -2236,18 +2236,18 @@ class EdgeSizing: **Requirements Analysis** -| **Metric** | **Calculation** | **Result** | -|:-------------------------|:------------------------------------------------------------------------------|:------------------------------------------| +| **Metric** | **Calculation** | **Result** | +|:-------------------------|:----------------------------------------------------------------------------------------------------|:-----------------------------------------------------| | **Inferences per store** | `{python} EdgeSizing.cameras_per_store_str` cameras$\times$ `{python} EdgeSizing.fps_str` FPS | `{python} EdgeSizing.inf_per_sec_str` inferences/sec | -| **Model compute** | YOLOv8-nano: `{python} EdgeSizing.yolo_gflops_str` GFLOPs/inference | `{python} EdgeSizing.sustained_gf_str` GFLOPs/sec | +| **Model compute** | YOLOv8-nano: `{python} EdgeSizing.yolo_gflops_str` GFLOPs/inference | `{python} EdgeSizing.sustained_gf_str` GFLOPs/sec | | **Required throughput** | `{python} EdgeSizing.sustained_gf_str` GFLOPs$\times$ `{python} EdgeSizing.headroom_str` (headroom) | ~`{python} EdgeSizing.req_tflops_str` TFLOPS | \index{edge accelerators!deployment selection} \index{embedded GPU accelerators!edge deployment} **Hardware Selection** -| **Edge Device** | **INT8 TOPS** | **Power** | **Unit Cost** | **Fleet Cost** | -|:--------------------------|--------------------------------:|:------------------------------------|:-------------------------------|--------------------------------------:| +| **Edge Device** | **INT8 TOPS** | **Power** | **Unit Cost** | **Fleet Cost** | +|:--------------------------|-------------------------------------------:|:-----------------------------------------------|:------------------------------------------|-------------------------------------------------:| | **NVIDIA Jetson Orin NX** | `{python} EdgeSizing.jetson_tops_str` TOPS | `{python} EdgeSizing.jetson_power_range_str` W | USD `{python} EdgeSizing.jetson_cost_str` | USD `{python} EdgeSizing.jetson_fleet_k_str`,000 | | **Intel NUC + Movidius** | `{python} EdgeSizing.nuc_tops_str` TOPS | `{python} EdgeSizing.nuc_power_w_str` W | USD `{python} EdgeSizing.nuc_cost_str` | USD `{python} EdgeSizing.nuc_fleet_k_str`,000 | | **Google Coral Dev** | `{python} EdgeSizing.coral_tops_str` TOPS | `{python} EdgeSizing.coral_power_w_str` W | USD `{python} EdgeSizing.coral_cost_str` | USD `{python} EdgeSizing.coral_fleet_k_str`,000 | @@ -2667,13 +2667,13 @@ class EnergyInference: \index{energy per inference!paradigm comparison} \index{TinyML!energy efficiency}Energy consumption spans eight orders of magnitude across deployment paradigms: -| **Paradigm** | **Example Workload** | **Energy/Inference** | **Battery Life (`{python} EnergyInference.batt_volt_str`V, `{python} EnergyInference.batt_cap_mah_str`mAh)** | -|:-------------|:---------------------|------------------------------:|-----------------------------------------------------------------------------:| -| **Cloud** | GPT-4 query | `{python} EnergyInference.e_gpt4_str` | ~`{python} EnergyInference.q_gpt4_str` queries | -| **Cloud** | ResNet-50 (A100) | `{python} EnergyInference.e_resnet_cloud_str` | ~`{python} EnergyInference.q_resnet_cloud_str` queries | -| **Edge** | ResNet-50 (Jetson) | `{python} EnergyInference.e_resnet_edge_str` | ~`{python} EnergyInference.q_resnet_edge_str` queries | -| **Mobile** | MobileNet (NPU) | `{python} EnergyInference.e_mobilenet_str` | ~`{python} EnergyInference.q_mobilenet_str` queries | -| **TinyML** | Keyword spotting | `{python} EnergyInference.e_kws_str` | ~`{python} EnergyInference.q_kws_str` queries | +| **Paradigm** | **Example Workload** | **Energy/Inference** | **Battery Life (`{python} EnergyInference.batt_volt_str`V, `{python} EnergyInference.batt_cap_mah_str`mAh)** | +|:-------------|:---------------------|----------------------------------------------:|-------------------------------------------------------------------------------------------------------------:| +| **Cloud** | GPT-4 query | `{python} EnergyInference.e_gpt4_str` | ~`{python} EnergyInference.q_gpt4_str` queries | +| **Cloud** | ResNet-50 (A100) | `{python} EnergyInference.e_resnet_cloud_str` | ~`{python} EnergyInference.q_resnet_cloud_str` queries | +| **Edge** | ResNet-50 (Jetson) | `{python} EnergyInference.e_resnet_edge_str` | ~`{python} EnergyInference.q_resnet_edge_str` queries | +| **Mobile** | MobileNet (NPU) | `{python} EnergyInference.e_mobilenet_str` | ~`{python} EnergyInference.q_mobilenet_str` queries | +| **TinyML** | Keyword spotting | `{python} EnergyInference.e_kws_str` | ~`{python} EnergyInference.q_kws_str` queries | Energy values represent *full-system energy* (including server CPUs, memory, networking, and cooling overhead), not isolated accelerator compute energy. For example, the A100 GPU alone executes ResNet-50 inference in under 1 ms (~0.3 J), but the full server draws ~1 kW when amortized across queuing, preprocessing, and idle power. @@ -2854,22 +2854,22 @@ class ParadigmsTable: The resulting fourteen-dimension comparison appears in @tbl-big_vs_tiny: -| **Aspect** | **Cloud ML** | **Edge ML** | **Mobile ML** | **TinyML** | -|:---------------------------|:-----------------------------------------|:---------------------------------------|:------------------------------|:------------------------------------------------------| -| **Processing Location** | Centralized cloud servers (Data Centers) | Local edge devices (gateways, servers) | Smartphones and tablets | Ultra-low-power microcontrollers and embedded systems | -| **Latency** | `{python} ParadigmsTable.cloud_lat_str` | `{python} ParadigmsTable.edge_lat_str` | `{python} ParadigmsTable.mobile_lat_str` | `{python} ParadigmsTable.tiny_lat_str` | -| **Compute Power** | `{python} ParadigmsTable.cloud_comp_str` | `{python} ParadigmsTable.edge_comp_str` | `{python} ParadigmsTable.mobile_comp_str` | `{python} ParadigmsTable.tiny_comp_str` | -| **Storage Capacity** | `{python} ParadigmsTable.cloud_stor_str` | `{python} ParadigmsTable.edge_stor_str` | `{python} ParadigmsTable.mobile_stor_str` | `{python} ParadigmsTable.tiny_stor_str` | -| **Energy Consumption** | `{python} ParadigmsTable.cloud_pwr_str` | `{python} ParadigmsTable.edge_pwr_str` | `{python} ParadigmsTable.mobile_pwr_str` | `{python} ParadigmsTable.tiny_pwr_str` | -| **Scalability** | Excellent (virtually unlimited) | Good (limited by edge hardware) | Moderate (per-device scaling) | Limited (fixed hardware) | -| **Data Privacy** | Basic-Moderate (Data leaves device) | High (Data stays in local network) | High (Data stays on phone) | Very High (Raw data can remain local) | -| **Connectivity Required** | Constant high-bandwidth | Intermittent | Optional | None | -| **Offline Capability** | None | Good | Excellent | Complete | -| **Real-time Processing** | Dependent on network | Good | Very Good | Excellent | -| **Cost** | `{python} ParadigmsTable.cloud_cost_str` | `{python} ParadigmsTable.edge_cost_str` | `{python} ParadigmsTable.mobile_cost_str` | `{python} ParadigmsTable.tiny_cost_str` | -| **Hardware Requirements** | Cloud infrastructure | Edge servers/gateways | Modern smartphones | MCUs/embedded systems | -| **Development Complexity** | High (cloud expertise needed) | Moderate-High (edge+networking) | Moderate (mobile SDKs) | High (embedded expertise) | -| **Deployment Speed** | Fast | Moderate | Fast | Slow | +| **Aspect** | **Cloud ML** | **Edge ML** | **Mobile ML** | **TinyML** | +|:---------------------------|:-----------------------------------------|:----------------------------------------|:------------------------------------------|:------------------------------------------------------| +| **Processing Location** | Centralized cloud servers (Data Centers) | Local edge devices (gateways, servers) | Smartphones and tablets | Ultra-low-power microcontrollers and embedded systems | +| **Latency** | `{python} ParadigmsTable.cloud_lat_str` | `{python} ParadigmsTable.edge_lat_str` | `{python} ParadigmsTable.mobile_lat_str` | `{python} ParadigmsTable.tiny_lat_str` | +| **Compute Power** | `{python} ParadigmsTable.cloud_comp_str` | `{python} ParadigmsTable.edge_comp_str` | `{python} ParadigmsTable.mobile_comp_str` | `{python} ParadigmsTable.tiny_comp_str` | +| **Storage Capacity** | `{python} ParadigmsTable.cloud_stor_str` | `{python} ParadigmsTable.edge_stor_str` | `{python} ParadigmsTable.mobile_stor_str` | `{python} ParadigmsTable.tiny_stor_str` | +| **Energy Consumption** | `{python} ParadigmsTable.cloud_pwr_str` | `{python} ParadigmsTable.edge_pwr_str` | `{python} ParadigmsTable.mobile_pwr_str` | `{python} ParadigmsTable.tiny_pwr_str` | +| **Scalability** | Excellent (virtually unlimited) | Good (limited by edge hardware) | Moderate (per-device scaling) | Limited (fixed hardware) | +| **Data Privacy** | Basic-Moderate (Data leaves device) | High (Data stays in local network) | High (Data stays on phone) | Very High (Raw data can remain local) | +| **Connectivity Required** | Constant high-bandwidth | Intermittent | Optional | None | +| **Offline Capability** | None | Good | Excellent | Complete | +| **Real-time Processing** | Dependent on network | Good | Very Good | Excellent | +| **Cost** | `{python} ParadigmsTable.cloud_cost_str` | `{python} ParadigmsTable.edge_cost_str` | `{python} ParadigmsTable.mobile_cost_str` | `{python} ParadigmsTable.tiny_cost_str` | +| **Hardware Requirements** | Cloud infrastructure | Edge servers/gateways | Modern smartphones | MCUs/embedded systems | +| **Development Complexity** | High (cloud expertise needed) | Moderate-High (edge+networking) | Moderate (mobile SDKs) | High (embedded expertise) | +| **Deployment Speed** | Fast | Moderate | Fast | Slow | : **Fourteen-Dimension Paradigm Comparison**\index{scalability!paradigm comparison}\index{offline capability!paradigm comparison}: A comprehensive side-by-side comparison across fourteen dimensions that matter for deployment decisions. Note the inverse relationship between compute power and privacy: Cloud ML provides the strongest compute but weaker privacy guarantees, while TinyML provides the strongest privacy but the weakest compute. This table serves as the primary reference for system architects evaluating deployment options. {#tbl-big_vs_tiny} diff --git a/book/quarto/contents/vol1/nn_architectures/nn_architectures.qmd b/book/quarto/contents/vol1/nn_architectures/nn_architectures.qmd index de46a02f0f..16f4feed59 100644 --- a/book/quarto/contents/vol1/nn_architectures/nn_architectures.qmd +++ b/book/quarto/contents/vol1/nn_architectures/nn_architectures.qmd @@ -57,7 +57,7 @@ When you select a neural network architecture, you are not making a modeling dec \index{Architecture!computational graph} \index{Architecture!systems engineering} -@sec-neural-computation established the mathematical operators—matrix multiplication, activation functions, and gradient computation—that form the "verbs" of neural networks. Those operators are the atoms; this chapter examines how they assemble into **architectures**: specialized structures optimized for specific data types and computational constraints. As defined in the Silicon Contract (Principle \ref{pri-silicon-contract}) (@sec-introduction-iron-law-ml-systems-c32a), every architecture makes an implicit agreement with hardware, trading computational patterns for efficiency on particular problem classes. +The mathematical operators established in @sec-neural-computation—matrix multiplication, activation functions, and gradient computation—form the "verbs" of neural networks. Those operators are the atoms; this chapter examines how they assemble into **architectures**: specialized structures optimized for specific data types and computational constraints. As defined in the Silicon Contract (Principle \ref{pri-silicon-contract}) (@sec-introduction-iron-law-ml-systems-c32a), every architecture makes an implicit agreement with hardware, trading computational patterns for efficiency on particular problem classes. Every neural network architecture answers one central question: *how should we structure computation to match the structure in our data?* Images have spatial locality, language has sequential dependencies, and tabular records have no inherent structure at all. The architecture encodes assumptions about these patterns directly into the computational graph, and those assumptions determine everything from parameter count to hardware utilization to deployment feasibility. Architecture selection is therefore a systems engineering problem that directly determines the Iron Law terms (the number of operations $O$ and the volume of data movement $D_{vol}$) defined in @sec-introduction-iron-law-ml-systems-c32a. @@ -268,8 +268,8 @@ These bottlenecks are not accidental; they are the "signatures" of the underlyin @tbl-workload-signatures compares the signatures of our three primary Lighthouses. Notice the three-order-of-magnitude gap between ResNet and GPT-2: -| **Model Family** | **Lighthouse** | **Intensity ($I$)** | **Hardware Affinity** | -|:---------------------|:----------------|:------------------------------------|:-------------------------------| +| **Model Family** | **Lighthouse** | **Intensity ($I$)** | **Hardware Affinity** | +|:---------------------|:----------------|:-------------------------------------------------------|:-------------------------------| | **Dense CNN** | **ResNet-50** | ~`{python} WorkloadSignatures.resnet_intensity_str` | **Compute-Rich** (GPUs/TPUs) | | **Efficient Vision** | **MobileNetV2** | ~`{python} WorkloadSignatures.mobilenet_intensity_str` | **Balanced** (Mobile NPUs) | | **Transformer** | **GPT-2 (Inf)** | ~`{python} WorkloadSignatures.gpt2_intensity_str` | **Bandwidth-Rich** (HBM3/H100) | diff --git a/book/quarto/contents/vol1/optimizations/model_compression.qmd b/book/quarto/contents/vol1/optimizations/model_compression.qmd index cde858d2f2..ea8a443f5c 100644 --- a/book/quarto/contents/vol1/optimizations/model_compression.qmd +++ b/book/quarto/contents/vol1/optimizations/model_compression.qmd @@ -475,14 +475,14 @@ class ModelDeviceComparison: dscnn_tiny_str = "ok" ``` -| **Model** | **Memory** **(Runtime)** | **Storage** **(Weights)** | **Cloud** **(`{python} ModelDeviceComparison.cloud_cap_str`)** | **Mobile** **(`{python} ModelDeviceComparison.mobile_cap_str`)** | **TinyML** **(`{python} ModelDeviceComparison.tiny_cap_str`)** | -|:-----------------------|:------------------------------|:------------------------------|:-----------------------------------------|:-------------------------------------------|:-----------------------------------------| -| **DLRM** | `{python} ModelDeviceComparison.dlrm_str` | `{python} ModelDeviceComparison.dlrm_str` | ok | `{python} ModelDeviceComparison.dlrm_mobile_str` | `{python} ModelDeviceComparison.dlrm_tiny_str` | -| **GPT-2 XL** | `{python} ModelDeviceComparison.gpt2_str` | `{python} ModelDeviceComparison.gpt2_str` | ok | ok | `{python} ModelDeviceComparison.gpt2_tiny_str` | -| **ResNet-50** | `{python} ModelDeviceComparison.resnet_str` | `{python} ModelDeviceComparison.resnet_str` | ok | ok | `{python} ModelDeviceComparison.resnet_tiny_str` | -| **MobileNetV2** | `{python} ModelDeviceComparison.mobilenet_str` | `{python} ModelDeviceComparison.mobilenet_str` | ok | ok | `{python} ModelDeviceComparison.mobilenet_tiny_str` | -| **MobileNetV2 (INT8)** | `{python} ModelDeviceComparison.mobilenet_int8_str` | `{python} ModelDeviceComparison.mobilenet_int8_str` | ok | ok | `{python} ModelDeviceComparison.mobilenet_int8_tiny_str` | -| **DS-CNN (KWS)** | `{python} ModelDeviceComparison.dscnn_str` | `{python} ModelDeviceComparison.dscnn_str` | ok | ok | `{python} ModelDeviceComparison.dscnn_tiny_str` | +| **Model** | **Memory** **(Runtime)** | **Storage** **(Weights)** | **Cloud** **(`{python} ModelDeviceComparison.cloud_cap_str`)** | **Mobile** **(`{python} ModelDeviceComparison.mobile_cap_str`)** | **TinyML** **(`{python} ModelDeviceComparison.tiny_cap_str`)** | +|:-----------------------|:----------------------------------------------------|:----------------------------------------------------|:---------------------------------------------------------------|:-----------------------------------------------------------------|:---------------------------------------------------------------| +| **DLRM** | `{python} ModelDeviceComparison.dlrm_str` | `{python} ModelDeviceComparison.dlrm_str` | ok | `{python} ModelDeviceComparison.dlrm_mobile_str` | `{python} ModelDeviceComparison.dlrm_tiny_str` | +| **GPT-2 XL** | `{python} ModelDeviceComparison.gpt2_str` | `{python} ModelDeviceComparison.gpt2_str` | ok | ok | `{python} ModelDeviceComparison.gpt2_tiny_str` | +| **ResNet-50** | `{python} ModelDeviceComparison.resnet_str` | `{python} ModelDeviceComparison.resnet_str` | ok | ok | `{python} ModelDeviceComparison.resnet_tiny_str` | +| **MobileNetV2** | `{python} ModelDeviceComparison.mobilenet_str` | `{python} ModelDeviceComparison.mobilenet_str` | ok | ok | `{python} ModelDeviceComparison.mobilenet_tiny_str` | +| **MobileNetV2 (INT8)** | `{python} ModelDeviceComparison.mobilenet_int8_str` | `{python} ModelDeviceComparison.mobilenet_int8_str` | ok | ok | `{python} ModelDeviceComparison.mobilenet_int8_tiny_str` | +| **DS-CNN (KWS)** | `{python} ModelDeviceComparison.dscnn_str` | `{python} ModelDeviceComparison.dscnn_str` | ok | ok | `{python} ModelDeviceComparison.dscnn_tiny_str` | : **The Deployment Gap**: Model memory requirements compared against typical device capacities. Even MobileNetV2 quantized to INT8 exceeds TinyML constraints by 7$\times$, while the purpose-built DS-CNN keyword spotter fits comfortably. Numbers in parentheses show how many times the model exceeds device memory. {#tbl-model-vs-device} @@ -3405,7 +3405,7 @@ accuracy}; # │ A100 INT8 SPEEDUP CONSTANTS # ├───────────────────────────────────────────────────────────────────────────── # │ Context: "The Quantization Speedup (Compute-Bound)" callout immediately -# │ below; also referenced in @sec-model-compression-energy-efficiency +# │ below; also referenced in the energy efficiency section # │ via bandwidth_bound_speedup_str. # │ # │ Goal: Quantify the throughput gain from INT8 on A100 Tensor Cores. diff --git a/book/quarto/contents/vol1/responsible_engr/responsible_engr.qmd b/book/quarto/contents/vol1/responsible_engr/responsible_engr.qmd index 89783b11b9..ad3baee94f 100644 --- a/book/quarto/contents/vol1/responsible_engr/responsible_engr.qmd +++ b/book/quarto/contents/vol1/responsible_engr/responsible_engr.qmd @@ -355,9 +355,9 @@ class GenderShadesDisparity: Responsible properties become testable when engineers work with stakeholders to define criteria appropriate for specific applications. The Gender Shades project\index{Gender Shades Study!algorithmic audit methodology}[^fn-gender-shades-audit] demonstrated how disaggregated evaluation\index{Disaggregated Evaluation!demographic stratification} across demographic categories reveals disparities invisible in aggregate metrics [@buolamwini2018gender]. The results captured dramatic error rate differences that commercial facial recognition systems showed across demographic groups. Concretely, a `{python} TestingConstraintAnchor.base_size_str`-sample test set that suffices for the majority group provides only `{python} TestingConstraintAnchor.minority_samples_str` samples for a `{python} TestingConstraintAnchor.minority_pct_str` minority subgroup—effectively requiring `{python} TestingConstraintAnchor.multiplier_str` more data than the majority group for high-confidence validation. -| **Demographic Group** | **Error Rate (%)** | **Relative Disparity** | -|:--------------------------|----------------------------------:|-----------------------------------------------------:| -| **Light-skinned males** | `{python} GenderShadesDisparity.error_light_male_str` | Baseline (1.0$\times$) | +| **Demographic Group** | **Error Rate (%)** | **Relative Disparity** | +|:--------------------------|--------------------------------------------------------:|---------------------------------------------------------------------------:| +| **Light-skinned males** | `{python} GenderShadesDisparity.error_light_male_str` | Baseline (1.0$\times$) | | **Light-skinned females** | `{python} GenderShadesDisparity.error_light_female_str` | `{python} GenderShadesDisparity.disparity_light_female_str`$\times$ higher | | **Dark-skinned males** | `{python} GenderShadesDisparity.error_dark_male_str` | `{python} GenderShadesDisparity.disparity_dark_male_str`$\times$ higher | | **Dark-skinned females** | `{python} GenderShadesDisparity.error_dark_female_str` | `{python} GenderShadesDisparity.disparity_str`$\times$ higher | @@ -649,15 +649,15 @@ class LoanFairness: a_total_str = f"{a_total:,}"; b_total_str = f"{b_total:,}" ``` -| | **Approved (pred)** | **Rejected (pred)** | -|:-----------------------|-------------------------:|-------------------------:| +| | **Approved (pred)** | **Rejected (pred)** | +|:-----------------------|--------------------------------------:|--------------------------------------:| | **Repaid (actual)** | `{python} LoanFairness.a_tp_str` (TP) | `{python} LoanFairness.a_fn_str` (FN) | | **Defaulted (actual)** | `{python} LoanFairness.a_fp_str` (FP) | `{python} LoanFairness.a_tn_str` (TN) | : **Confusion Matrix for Group A (Majority)**: Loan approval outcomes for 10,000 applicants from the majority demographic group. The 90% true positive rate (4,500 approved of 5,000 qualified) and 20% false positive rate establish the baseline for fairness comparison. {#tbl-confusion-group-a} -| | **Approved (pred)** | **Rejected (pred)** | -|:-----------------------|-------------------------:|-------------------------:| +| | **Approved (pred)** | **Rejected (pred)** | +|:-----------------------|--------------------------------------:|--------------------------------------:| | **Repaid (actual)** | `{python} LoanFairness.b_tp_str` (TP) | `{python} LoanFairness.b_fn_str` (FN) | | **Defaulted (actual)** | `{python} LoanFairness.b_fp_str` (FP) | `{python} LoanFairness.b_tn_str` (TN) | @@ -708,11 +708,11 @@ This pattern automates what manual auditing cannot achieve at scale: continuous @tbl-fairness-metrics-summary reveals the troubling pattern in these computed metrics and disparities. -| **Metric** | **Group A** | **Group B** | **Disparity** | -|:------------------------|---------------------------:|---------------------------:|:-----------------------------------------------| +| **Metric** | **Group A** | **Group B** | **Disparity** | +|:------------------------|----------------------------------------:|----------------------------------------:|:------------------------------------------------------------| | **Approval Rate** | `{python} LoanFairness.a_approval_str`% | `{python} LoanFairness.b_approval_str`% | `{python} LoanFairness.dp_disparity_str` percentage points | | **True Positive Rate** | `{python} LoanFairness.a_tpr_str`% | `{python} LoanFairness.b_tpr_str`% | `{python} LoanFairness.tpr_disparity_str` percentage points | -| **False Positive Rate** | `{python} LoanFairness.a_fpr_str`% | `{python} LoanFairness.b_fpr_str`% | 0 percentage points | +| **False Positive Rate** | `{python} LoanFairness.a_fpr_str`% | `{python} LoanFairness.b_fpr_str`% | 0 percentage points | : **Fairness Metrics Summary**: Comparison of fairness metrics across demographic groups reveals substantial disparities in how the model treats qualified applicants from each group. {#tbl-fairness-metrics-summary} @@ -742,20 +742,20 @@ circB/.style={draw=blue!70!black, fill=white,line width=2.5pt,circle, minimum si \foreach \x in{0.25,0.35,0.42,0.49}{ \node[circB,yshift=4mm]at($(A1)!\x!(A2)$){}; } - + \foreach \x in{0.68,0.75,0.82,0.89}{ \node[yshift=4mm]at($(A1)!\x!(A2)$){\fplus[blue!70!black]}; } - + \foreach \x in{0.05,0.12,0.19,0.42}{ \node[circR,yshift=4mm]at($(B1)!\x!(B2)$){}; } - + \foreach \x in{0.35,0.49,0.56,0.75}{ \node[yshift=4mm]at($(B1)!\x!(B2)$){\fplus[red]}; } \node[draw=none,fit=(SA)(A2)(SE)(DO)](FI){}; - + \node[circB,below=-2pt of FI.210](CI1){}; \node[circR,right=5pt of CI1](CI2){}; \node[right=0pt of CI2]{Positive outcome}; @@ -1062,8 +1062,8 @@ class EdgeEfficiencyCalc: Edge deployment scenarios\index{Edge Deployment!power constraints} make efficiency requirements concrete. When a wearable device has a `{python} EdgeEfficiencyCalc.wear_power_str` power budget and must run inference continuously for 24 hours on a small battery, abstract efficiency discussions become engineering constraints with measurable consequences.\index{Power Budget!edge devices} @tbl-edge-deployment-constraints quantifies these constraints across four deployment contexts, from smartphones with `{python} EdgeEfficiencyCalc.smart_power_str` budgets to IoT sensors operating at `{python} EdgeEfficiencyCalc.iot_power_str`. -| **Deployment Context** | **Power Budget** | **Latency Requirement** | **Typical Use Cases** | -|:-----------------------|---------------------------:|------------------------------------:|:--------------------------------------------| +| **Deployment Context** | **Power Budget** | **Latency Requirement** | **Typical Use Cases** | +|:-----------------------|----------------------------------------------:|-------------------------------------------------------:|:--------------------------------------------| | **Smartphone** | `{python} EdgeEfficiencyCalc.smart_power_str` | `{python} EdgeEfficiencyCalc.smart_latency_str` | Photo enhancement, voice assistants | | **IoT Sensor** | `{python} EdgeEfficiencyCalc.iot_power_str` | `{python} EdgeEfficiencyCalc.iot_latency_str` | Anomaly detection, environmental monitoring | | **Embedded Camera** | `{python} EdgeEfficiencyCalc.cam_power_str` | 30 FPS (`{python} EdgeEfficiencyCalc.cam_latency_str`) | Real-time object detection, surveillance | @@ -1073,8 +1073,8 @@ Edge deployment scenarios\index{Edge Deployment!power constraints} make efficien @tbl-model-efficiency-comparison compares how model architectures fit different deployment constraints. -| **Model** | **Parameters** | **Inference Power** | **Latency** | **Fits Smartphone?** | **Fits IoT?** | -|:--------------------|---------------------------:|--------------------------:|----------------------------:|:---------------------|:--------------| +| **Model** | **Parameters** | **Inference Power** | **Latency** | **Fits Smartphone?** | **Fits IoT?** | +|:--------------------|----------------------------------------------:|---------------------------------------------:|-----------------------------------------------:|:---------------------|:--------------| | **MobileNetV2** | `{python} EdgeEfficiencyCalc.mv2_params_str` | `{python} EdgeEfficiencyCalc.mv2_power_str` | `{python} EdgeEfficiencyCalc.mv2_latency_str` | Yes | No | | **EfficientNet-B0** | `{python} EdgeEfficiencyCalc.eff_params_str` | `{python} EdgeEfficiencyCalc.eff_power_str` | `{python} EdgeEfficiencyCalc.eff_latency_str` | Yes | No | | **ResNet-50** | `{python} EdgeEfficiencyCalc.rn50_params_str` | `{python} EdgeEfficiencyCalc.rn50_power_str` | `{python} EdgeEfficiencyCalc.rn50_latency_str` | No | No | @@ -1345,8 +1345,8 @@ inf_carbon_3yr = TCOCalc.inf_carbon_3yr Training costs include both initial development and ongoing retraining. @tbl-tco-training breaks down these costs, showing how quarterly retraining cycles accumulate over a three-year operational period. -| **Cost Component** | **Calculation** | **Financial Cost** | **Carbon (kg CO2)** | -|:--------------------------------|:------------------------------------|:--------------------------------|-------------------------------------:| +| **Cost Component** | **Calculation** | **Financial Cost** | **Carbon (kg CO2)** | +|:--------------------------------|:------------------------------------|:----------------------------------------|---------------------------------------------:| | **Initial data preparation** | hours$\times$ rate | `{python} TCOCalc.t_data_prep_calc_str` | `{python} TCOCalc.t_data_prep_carbon_str` | | **Hyperparameter search** | experiments$\times$ cost/experiment | `{python} TCOCalc.t_hparam_calc_str` | `{python} TCOCalc.t_hparam_carbon_str` | | **Final training** | hours$\times$ rate | `{python} TCOCalc.t_final_calc_str` | `{python} TCOCalc.t_final_carbon_str` | @@ -1360,10 +1360,10 @@ Training costs include both initial development and ongoing retraining. @tbl-tco The economics of this trade-off are detailed in @tbl-tco-inference, which shows how inference costs dominate total cost of ownership for production systems. -| **Cost Component** | **Calculation** | **Financial Cost** | **Carbon (kg CO2)** | -|:--------------------------|:-------------------------------|------------------------------:|:-------------------------------| -| **Daily queries** | users$\times$ queries/user | `{python} TCOCalc.i_daily_q_calc_str` | - | -| **GPU-seconds/day** | queries$\times$ latency | `{python} TCOCalc.i_gpu_sec_calc_str` | - | +| **Cost Component** | **Calculation** | **Financial Cost** | **Carbon (kg CO2)** | +|:--------------------------|:-------------------------------|--------------------------------------:|:---------------------------------------| +| **Daily queries** | users$\times$ queries/user | `{python} TCOCalc.i_daily_q_calc_str` | - | +| **GPU-seconds/day** | queries$\times$ latency | `{python} TCOCalc.i_gpu_sec_calc_str` | - | | **GPU-hours/day** | seconds ÷ SEC_PER_HOUR | `{python} TCOCalc.i_gpu_hr_day_str` | `{python} TCOCalc.i_daily_carbon_str` | | **Annual GPU cost** | hours$\times$ 365$\times$ rate | `{python} TCOCalc.i_annual_calc_str` | `{python} TCOCalc.i_annual_carbon_str` | | **3-year inference cost** | annual$\times$ 3 | **`{python} TCOCalc.i_total_str`** | **`{python} TCOCalc.i_carbon_str`** | @@ -1374,8 +1374,8 @@ The economics of this trade-off are detailed in @tbl-tco-inference, which shows Operational costs encompass infrastructure, personnel, and incident response. @tbl-tco-operations itemizes these ongoing expenses, which often surprise teams focused primarily on compute costs. -| **Cost Component** | **Annual Estimate** | **3-Year Total** | -|:----------------------------------|---------------------------------:|---------------------------:| +| **Cost Component** | **Annual Estimate** | **3-Year Total** | +|:----------------------------------|-----------------------------------------:|-----------------------------------:| | **Monitoring infrastructure** | `{python} TCOCalc.o_monitor_annual_str` | `{python} TCOCalc.o_monitor_str` | | **On-call engineering (0.5 FTE)** | `{python} TCOCalc.o_oncall_annual_str` | `{python} TCOCalc.o_oncall_str` | | **Incident response (estimated)** | `{python} TCOCalc.o_incident_annual_str` | `{python} TCOCalc.o_incident_str` | @@ -1435,14 +1435,14 @@ class TCOSummary: The stark breakdown in @tbl-tco-summary answers where the money goes: inference at `{python} TCOCalc.p_inf_str`%, operations at `{python} TCOCalc.p_ops_str`%, and training at only `{python} TCOCalc.p_train_str`%. -| **Category** | **3-Year Cost** | **Percentage** | **Carbon Impact** | -|:---------------|:-----------------------------|------------------------:|-------------------------------------:| +| **Category** | **3-Year Cost** | **Percentage** | **Carbon Impact** | +|:---------------|:-------------------------------------|--------------------------------:|---------------------------------------------:| | **Training** | `{python} TCOCalc.t_total_k_str` | `{python} TCOCalc.p_train_str`% | `{python} TCOCalc.t_total_carbon_tons_str` | | **Inference** | `{python} TCOCalc.i_total_str` | `{python} TCOCalc.p_inf_str`% | `{python} TCOCalc.i_carbon_tons_str` | | **Operations** | `{python} TCOCalc.o_total_str` | `{python} TCOCalc.p_ops_str`% | - | | **Total TCO** | **`{python} TCOCalc.total_tco_str`** | 100% | **`{python} TCOCalc.total_carbon_tons_str`** | -: **Total Cost of Ownership Summary**: Three-year TCO of `{python} TCOCalc.total_tco_str` breaks down as: training `{python} TCOCalc.t_total_k_str` (`{python} TCOCalc.p_train_str`%), inference `{python} TCOCalc.i_total_str` (`{python} TCOCalc.p_inf_str`%), and operations `{python} TCOCalc.o_total_str` (`{python} TCOCalc.p_ops_str`%). The `{python} InferenceCostCalc.ratio_str`:1 ratio between inference and training costs is typical for production systems serving `{python} InferenceCostCalc.users_daily_m_str` million daily users. A `{python} TCOSummary.quant_reduction_pct_str`% reduction in inference latency through quantization would save USD `{python} TCOSummary.quant_savings_str`K and approximately `{python} TCOSummary.quant_carbon_str` tons of CO2, easily justifying the optimization engineering investment. {#tbl-tco-summary} +: **Total Cost of Ownership Summary**: Three-year TCO breakdown: training, inference, and operations costs. The ~10:1 ratio between inference and training costs is typical for production systems serving millions of daily users. A 30% reduction in inference latency through quantization can save hundreds of thousands of dollars and tons of CO2, easily justifying the optimization engineering investment. {#tbl-tco-summary} ::: {.callout-checkpoint title="Efficiency as Responsibility"} Total cost of ownership reveals where responsible optimization has the most leverage. diff --git a/book/quarto/contents/vol1/training/training.qmd b/book/quarto/contents/vol1/training/training.qmd index 3ea762dbab..6e7cca5d73 100644 --- a/book/quarto/contents/vol1/training/training.qmd +++ b/book/quarto/contents/vol1/training/training.qmd @@ -235,7 +235,7 @@ The simplification is valid when the pipeline is correctly staged: at training s | **Technique** | **Term Affected** | **Mechanism** | |:--------------------------------|:----------------------------------|:-----------------------------------------------------------------------------------| | **Mixed Precision (FP16/BF16)** | Peak Throughput ↑ | Tensor Cores operate at up to 16$\times$ higher FLOP/s | -| **Data Prefetching** | Utilization ↑ | Reduces accelerator idle time waiting for data | +| **Data Prefetching** | Utilization ↑ | Reduces accelerator idle time waiting for data | | **Gradient Checkpointing** | Total Operations ↑ | Adds recomputation, but enables larger models | | **Gradient Accumulation** | Utilization ↑ | Maintains high batch parallelism efficiency | | **Operator Fusion** | Utilization ↑ | Reduces memory bandwidth bottlenecks | @@ -324,12 +324,12 @@ To ground the abstract principles of training systems in concrete engineering de **Why this model?** GPT-2 (1.5B) serves as our primary case study for **large-scale training** because it sits at the "sweet spot" of systems complexity. It is large enough to require distributed training and serious memory optimizations, yet small enough to comprehend without the massive infrastructure complexity of trillion-parameter clusters. -| **Property** | **Specification** | **Systems Implication** | -|:-----------------|----------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Parameters** | `{python} TrainingModels.gpt2_params_b_str` Billion (XL) | Requires ~`{python} GPT2LighthouseSpecs.gpt2_weights_fp16_gb_str` GB (FP16) or ~`{python} GPT2LighthouseSpecs.gpt2_weights_fp32_gb_str` GB (FP32) for weights alone. | -| **Architecture** | 48 Layers, 1600 Dim | Deep pipeline creates heavy activation memory pressure. | -| **Dataset** | OpenWebText (`{python} GPT2LighthouseSpecs.gpt2_dataset_size_gb_str`GB) | I/O throughput must match high-speed accelerator compute. | -| **Compute** | ~ `{python} TrainingScenarios.gpt2_total_flops_str` FLOPs total | Training takes days/weeks; demands parallelization. | +| **Property** | **Specification** | **Systems Implication** | +|:-----------------|------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Parameters** | `{python} TrainingModels.gpt2_params_b_str` Billion (XL) | Requires ~`{python} GPT2LighthouseSpecs.gpt2_weights_fp16_gb_str` GB (FP16) or ~`{python} GPT2LighthouseSpecs.gpt2_weights_fp32_gb_str` GB (FP32) for weights alone. | +| **Architecture** | 48 Layers, 1600 Dim | Deep pipeline creates heavy activation memory pressure. | +| **Dataset** | OpenWebText (`{python} GPT2LighthouseSpecs.gpt2_dataset_size_gb_str`GB) | I/O throughput must match high-speed accelerator compute. | +| **Compute** | ~ `{python} TrainingScenarios.gpt2_total_flops_str` FLOPs total | Training takes days/weeks; demands parallelization. | **Key Systems Challenge:** Training GPT-2 is primarily **memory-bound** (due to activation storage) and **compute-intensive** (requiring massive matrix multiplications). It forces us to move beyond simple training loops to sophisticated pipelines that manage data movement as carefully as computation. @@ -1478,7 +1478,7 @@ The mathematical operations we have examined---forward propagation, gradient com # ┌───────────────────────────────────────────────────────────────────────────── # │ ARITHMETIC INTENSITY SCENARIOS # ├───────────────────────────────────────────────────────────────────────────── -# │ Context: §Arithmetic Intensity, @tbl-training-ai-classification. +# │ Context: §Arithmetic Intensity, tbl-training-ai-classification. # │ # │ Goal: Provide arithmetic intensity values for activation functions, # │ normalization, and softmax operations. @@ -1515,9 +1515,9 @@ Operations with high arithmetic intensity are compute-bound: their performance i Consider @tbl-training-arithmetic-intensity: dense matrix multiplication achieves O(n) FLOP/byte (compute-bound), while activation functions operate at just 0.25 FLOP/byte (memory-bound), explaining why optimization strategies must differ between these operation types. -| **Operation** | **Arithmetic Intensity** | **Classification** | -|:-------------------------|--------------------------------------------------------------:|:-------------------| -| **Dense MatMul (large)** | O(n) FLOP/byte | Compute-bound | +| **Operation** | **Arithmetic Intensity** | **Classification** | +|:-------------------------|-------------------------------------------------------------------------:|:-------------------| +| **Dense MatMul (large)** | O(n) FLOP/byte | Compute-bound | | **Activation functions** | `{python} ArithmeticIntensityScenarios.ai_act_fp16_str` FLOP/byte (FP16) | Memory-bound | | **LayerNorm/BatchNorm** | ~`{python} ArithmeticIntensityScenarios.ai_norm_str` FLOP/byte | Memory-bound | | **Attention softmax** | ~`{python} ArithmeticIntensityScenarios.ai_softmax_str` FLOP/byte | Memory-bound | @@ -2293,7 +2293,7 @@ For image classification pipelines where resizing, augmentation, and normalizati # │ FORWARD PASS DIMENSIONS + WAVE QUANTIZATION # ├───────────────────────────────────────────────────────────────────────────── # │ Context: §Forward Pass (Compute Operations), §Wave Quantization -# │ (@tbl-wave-quantization), §Backward Pass (bwd dims). +# │ (tbl-wave-quantization), §Backward Pass (bwd dims). # │ # │ Goal: Provide layer dimensions, convolution parameters, wave quantization # │ scenarios, and backward-pass dimension strings. @@ -2904,8 +2904,8 @@ Production systems typically achieve 30–50% MFU[^fn-mfu]; values below this ra Training bottlenecks fall into three categories, which map directly to the **D·A·M taxonomy** (Data, Algorithm, Machine; see @sec-dam-taxonomy for the full diagnostic framework, troubleshooting matrix, and D·A·M Scorecard). @tbl-dam-training-bottlenecks connects each D·A·M axis to the corresponding training bottleneck, its observable symptoms, and the optimization techniques that address it. -| **D·A·M Axis** | **Bottleneck** | **Symptoms** | **Primary Solutions** | -|:---------------|:---------------|:------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| +| **D·A·M Axis** | **Bottleneck** | **Symptoms** | **Primary Solutions** | +|:---------------|:---------------|:--------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| | **Algorithm** | Compute-bound | accelerator utilization >90%; low memory bandwidth usage; arithmetic units are the limiting factor | FlashAttention, mixed precision, faster hardware | | **Machine** | Memory-bound | accelerator utilization 50--80%; high memory bandwidth usage; arithmetic units idle waiting for data from memory | Operator fusion, memory-efficient attention, reduced precision formats | | **Data** | Data-bound | Periodic accelerator utilization drops to near-zero; CPU fully utilized during gaps; pipeline cannot feed GPU fast enough | Prefetching, pipeline overlap, faster storage, DataLoader parallelism | @@ -3299,12 +3299,12 @@ These techniques yield the greatest benefit when storage access is slow, preproc @tbl-prefetching contrasts traditional sequential pipelines against optimized approaches across four dimensions: accelerator utilization improves from frequent idle periods to near-constant activity, training time decreases through parallelism, resource usage shifts from suboptimal to maximized, and scalability transforms from bottleneck-limited to adaptable. -| **Aspect** | **Traditional Pipeline** | **With Prefetching & Overlapping** | -|:--------------------|:------------------------------------|:------------------------------------| +| **Aspect** | **Traditional Pipeline** | **With Prefetching & Overlapping** | +|:----------------------------|:------------------------------------|:------------------------------------| | **Accelerator Utilization** | Frequent idle periods | Near-constant utilization | -| **Training Time** | Longer due to sequential operations | Reduced through parallelism | -| **Resource Usage** | Often suboptimal | Maximized across available hardware | -| **Scalability** | Limited by slowest component | Adaptable to various bottlenecks | +| **Training Time** | Longer due to sequential operations | Reduced through parallelism | +| **Resource Usage** | Often suboptimal | Maximized across available hardware | +| **Scalability** | Limited by slowest component | Adaptable to various bottlenecks | : **Pipeline Optimization Impact.** Prefetching and overlapping transform sequential pipelines into parallel ones, maximizing hardware utilization by ensuring the GPU always has data ready to process. The accelerator utilization improvement from "frequent idle periods" to "near-constant utilization" is often the single highest-impact optimization in data-intensive training workloads. {#tbl-prefetching} @@ -4483,18 +4483,18 @@ Both techniques introduce explicit trade-offs. Activation checkpointing adds app @tbl-optimization synthesizes three of the four core optimization strategies, contrasting their primary goals, mechanisms, and trade-offs. Flash Attention (@sec-model-training-flash-attention-ioaware-attention-optimization-3da0) complements these by addressing memory-bandwidth bottlenecks in attention layers through IO-aware tiling, achieving 2--4$\times$ speedups while reducing memory from $O(n^2)$ to $O(n)$. Selecting an appropriate strategy depends on the specific bottleneck identified through profiling. -| **Aspect** | **Prefetching and Overlapping** | **Mixed-Precision Training** | **Gradient Accumulation and Checkpointing** | -|:------------------------------|:-----------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------| +| **Aspect** | **Prefetching and Overlapping** | **Mixed-Precision Training** | **Gradient Accumulation and Checkpointing** | +|:------------------------------|:-------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------| | **Primary Goal** | Minimize data transfer delays and maximize accelerator utilization | Reduce memory consumption and computational overhead | Overcome memory limitations during backpropagation and parameter updates | -| **Key Mechanism** | Asynchronous data loading and parallel processing | Combining FP16 and FP32 computations | Simulating larger batch sizes and selective activation storage | -| **Memory Impact** | Increases memory usage for prefetch buffer | Reduces memory usage by using FP16 | Reduces memory usage for activations and gradients | -| **Computation Speed** | Improves by reducing idle time | Accelerates computations using FP16 | May slow down due to recomputations in checkpointing | -| **Scalability** | Highly scalable, especially for large datasets | Enables training of larger models | Allows training deeper models on limited hardware | -| **Hardware Requirements** | Benefits from fast storage and multi-core CPUs | Requires GPUs with FP16 support (e.g., Tensor Cores) | Works on standard hardware | -| **Implementation Complexity** | Moderate (requires tuning of prefetch parameters) | Low to moderate (with framework support) | Moderate (requires careful segmentation and accumulation) | -| **Main Benefits** | Reduces training time, improves hardware utilization | Faster training, larger models, reduced memory usage | Enables larger batch sizes and deeper models | -| **Primary Challenges** | Tuning buffer sizes, increased memory usage | Potential numerical instability, loss scaling needed | Increased computational overhead, slower parameter updates | -| **Ideal Use Cases** | Large datasets, complex preprocessing | Large-scale models, especially in NLP and computer vision | Deep networks (50+ layers), memory-constrained environments | +| **Key Mechanism** | Asynchronous data loading and parallel processing | Combining FP16 and FP32 computations | Simulating larger batch sizes and selective activation storage | +| **Memory Impact** | Increases memory usage for prefetch buffer | Reduces memory usage by using FP16 | Reduces memory usage for activations and gradients | +| **Computation Speed** | Improves by reducing idle time | Accelerates computations using FP16 | May slow down due to recomputations in checkpointing | +| **Scalability** | Highly scalable, especially for large datasets | Enables training of larger models | Allows training deeper models on limited hardware | +| **Hardware Requirements** | Benefits from fast storage and multi-core CPUs | Requires GPUs with FP16 support (e.g., Tensor Cores) | Works on standard hardware | +| **Implementation Complexity** | Moderate (requires tuning of prefetch parameters) | Low to moderate (with framework support) | Moderate (requires careful segmentation and accumulation) | +| **Main Benefits** | Reduces training time, improves hardware utilization | Faster training, larger models, reduced memory usage | Enables larger batch sizes and deeper models | +| **Primary Challenges** | Tuning buffer sizes, increased memory usage | Potential numerical instability, loss scaling needed | Increased computational overhead, slower parameter updates | +| **Ideal Use Cases** | Large datasets, complex preprocessing | Large-scale models, especially in NLP and computer vision | Deep networks (50+ layers), memory-constrained environments | : **Optimization Strategies.** Prefetching, mixed-precision training, and gradient accumulation address distinct bottlenecks in AI training pipelines: data transfer, memory consumption, and backpropagation. Selecting an appropriate strategy balances implementation complexity against gains in speed and resource utilization, depending on hardware and workload characteristics. {#tbl-optimization} @@ -4633,12 +4633,12 @@ After optimization: #### Step 6: Final Profile and Results {.unnumbered} -| **Metric** | **Naive** | **Optimized** | **Improvement** | -|:--------------------|:----------|-----------------:|:----------------------| -| **Memory** | 89 GB | 32 GB | 2.8$\times$ reduction | +| **Metric** | **Naive** | **Optimized** | **Improvement** | +|:----------------------------|:----------|-----------------:|:----------------------| +| **Memory** | 89 GB | 32 GB | 2.8$\times$ reduction | | **accelerator utilization** | N/A | 85% | Trainable | -| **Throughput** | N/A | 1,200 tokens/sec | — | -| **Time per epoch** | N/A | 8.3 hours | — | +| **Throughput** | N/A | 1,200 tokens/sec | — | +| **Time per epoch** | N/A | 8.3 hours | — | Remaining bottleneck: compute-bound---an *Algorithm* constraint in D·A·M terms (as desired). The 85% utilization indicates good efficiency; remaining 15% is overhead from gradient synchronization, loss scaling, and kernel launch latency. @@ -4734,8 +4734,8 @@ class GPT2SummaryCalc: @tbl-gpt2-summary compiles the end-to-end impact of applying mixed-precision training and gradient checkpointing to GPT-2. -| **Metric** | **FP32 Baseline** | **Optimized** | **Technique Applied** | -|:------------------------------------|:-------------------------------|:-------------------------------|:------------------------------------| +| **Metric** | **FP32 Baseline** | **Optimized** | **Technique Applied** | +|:------------------------------------|:-----------------------------------------------|:-----------------------------------------------|:------------------------------------| | **Parameters** | `{python} GPT2SummaryCalc.b_param_str` | `{python} GPT2SummaryCalc.o_param_str` | Mixed precision (FP16) | | **Gradients** | `{python} GPT2SummaryCalc.b_grad_str` | `{python} GPT2SummaryCalc.o_grad_str` | Mixed precision (FP16) | | **Master Weights** | `{python} GPT2SummaryCalc.b_master_str` | `{python} GPT2SummaryCalc.o_master_str` | AMP Overhead | @@ -5004,7 +5004,7 @@ Model parallelism's challenge is *idle time*. While Device 1 computes layers 1-- \index{NVLink!GPU interconnect}\index{NVLink!gradient synchronization} Within a single node, GPUs communicate via high-bandwidth interconnects like NVLink[^fn-nvlink-training] (up to `{python} StorageBandwidth.nvlink_h100_str` GB/s on modern systems), making gradient synchronization and activation transfers fast. Data parallelism transfers gradients, which are proportional to model size. Model parallelism transfers activations, proportional to batch size times hidden dimension, at every partition boundary. The 10--50$\times$ bandwidth advantage of NVLink over PCIe makes both strategies practical within a node. This intra-node parallelism forms the building block for larger distributed systems. To understand *when* to choose each strategy, the following analysis compares *data vs. model parallelism* quantitatively. -::: {.callout-notebook title="Data vs. Model Parallelism"} +::: {.callout-notebook title="Data vs. Model Parallelism"} **The Physics of Splitting**: How do you split a model that is too big or too slow? diff --git a/book/quarto/contents/vol2/backmatter/appendix_assumptions.qmd b/book/quarto/contents/vol2/backmatter/appendix_assumptions.qmd index 6a1bae2ddd..ee02eca7ca 100644 --- a/book/quarto/contents/vol2/backmatter/appendix_assumptions.qmd +++ b/book/quarto/contents/vol2/backmatter/appendix_assumptions.qmd @@ -130,9 +130,9 @@ The constants in this appendix are designed for quick distributed systems calcul **How many failures per day should you expect?** A cluster of `{python} FleetQuickCalc.n_small_str` GPUs with each GPU having an MTTF of `{python} FleetQuickCalc.mttf_str` hours experiences GPU failures at a rate of `{python} FleetQuickCalc.n_small_str` / `{python} FleetQuickCalc.mttf_str` $\approx$ `{python} FleetQuickCalc.fail_hr_str` failures per hour, or roughly `{python} FleetQuickCalc.fail_day_str` GPU failures per day. This estimate does not include NIC, PSU, or cable failures, which collectively double the rate. At `{python} FleetQuickCalc.n_large_str` GPUs, expect approximately `{python} FleetQuickCalc.fail_large_str` GPU failures per day --- failure becomes a continuous background process, not an exceptional event. -**How long does an AllReduce of a `{python} FleetQuickCalc.model_str` model take?** With BF16 parameters, the gradient payload is $`{python} FleetQuickCalc.model_params_b_str` \times 10^9 \times `{python} FleetQuickCalc.bf16_str`$ bytes = `{python} FleetQuickCalc.grad_gb_str` GB. Over InfiniBand NDR (`{python} FleetQuickCalc.ib_bw_str` GB/s effective per port), the ring AllReduce time is approximately $2 \times$ `{python} FleetQuickCalc.grad_gb_str` GB / `{python} FleetQuickCalc.ib_bw_str` GB/s $\approx$ `{python} FleetQuickCalc.allreduce_str` seconds --- a significant fraction of a training step, which is why pipeline and tensor parallelism exist to shrink the gradient volume each rank must communicate. +**How long does an AllReduce of a `{python} FleetQuickCalc.model_str` model take?** With BF16 parameters, the gradient payload is `{python} FleetQuickCalc.model_params_b_str` $\times 10^9 \times$ `{python} FleetQuickCalc.bf16_str` bytes = `{python} FleetQuickCalc.grad_gb_str` GB. Over InfiniBand NDR (`{python} FleetQuickCalc.ib_bw_str` GB/s effective per port), the ring AllReduce time is approximately $2 \times$ `{python} FleetQuickCalc.grad_gb_str` GB / `{python} FleetQuickCalc.ib_bw_str` GB/s $\approx$ `{python} FleetQuickCalc.allreduce_str` seconds --- a significant fraction of a training step, which is why pipeline and tensor parallelism exist to shrink the gradient volume each rank must communicate. -**What is the carbon footprint of a `{python} FleetQuickCalc.n_carbon_str` GPU training run?** Each H100 draws `{python} FleetQuickCalc.tdp_str` W at TDP. With PUE of `{python} FleetQuickCalc.pue_str` (best-in-class air cooled), total facility power is $`{python} FleetQuickCalc.n_carbon_str`\times`{python} FleetQuickCalc.tdp_str`$ W$\times$ `{python} FleetQuickCalc.pue_str` = `{python} FleetQuickCalc.facility_mw_str` MW. Running in Quebec (`{python} FleetQuickCalc.carbon_q_str` gCO$_2$/kWh): `{python} FleetQuickCalc.facility_mw_str` MW$\times$ 720 h/month$\times$ `{python} FleetQuickCalc.carbon_q_str` g/kWh = `{python} FleetQuickCalc.co2_q_str` tonnes CO$_2$ per month. The same run in Poland (`{python} FleetQuickCalc.carbon_p_str` gCO$_2$/kWh) emits `{python} FleetQuickCalc.co2_p_str` tonnes --- a `{python} FleetQuickCalc.ratio_str`$\times$ difference driven entirely by grid carbon intensity. +**What is the carbon footprint of a `{python} FleetQuickCalc.n_carbon_str` GPU training run?** Each H100 draws `{python} FleetQuickCalc.tdp_str` W at TDP. With PUE of `{python} FleetQuickCalc.pue_str` (best-in-class air cooled), total facility power is `{python} FleetQuickCalc.n_carbon_str` $\times$ `{python} FleetQuickCalc.tdp_str` W $\times$ `{python} FleetQuickCalc.pue_str` = `{python} FleetQuickCalc.facility_mw_str` MW. Running in Quebec (`{python} FleetQuickCalc.carbon_q_str` gCO$_2$/kWh): `{python} FleetQuickCalc.facility_mw_str` MW $\times$ 720 h/month $\times$ `{python} FleetQuickCalc.carbon_q_str` g/kWh = `{python} FleetQuickCalc.co2_q_str` tonnes CO$_2$ per month. The same run in Poland (`{python} FleetQuickCalc.carbon_p_str` gCO$_2$/kWh) emits `{python} FleetQuickCalc.co2_p_str` tonnes --- a `{python} FleetQuickCalc.ratio_str`$\times$ difference driven entirely by grid carbon intensity. ::: @@ -345,13 +345,13 @@ class AppendixConstants: The distributed systems reasoning in this book builds upon the single-machine performance bounds that govern each node in the fleet. For convenience, @tbl-fleet-foundation-recap recaps the critical constants for the NVIDIA H100 accelerator, which serves as our primary reference throughout this volume. -| **Tier** | **Specification** | **Reference Value** | -|:---------------|:------------------|:---------------------------------| -| **Compute** | FP16 Throughput | `{python} H100Recap.fp16_tflops_str` TFLOPS | -| **Compute** | FP8 Throughput | `{python} H100Recap.fp8_tflops_str` TFLOPS | -| **Memory** | HBM3 Bandwidth | `{python} H100Recap.bw_tb_str` TB/s | -| **Memory** | HBM3 Capacity | `{python} H100Recap.cap_gb_str` GB | -| **Thermal** | TDP | `{python} H100Recap.tdp_w_str` W | +| **Tier** | **Specification** | **Reference Value** | +|:------------|:------------------|:--------------------------------------------| +| **Compute** | FP16 Throughput | `{python} H100Recap.fp16_tflops_str` TFLOPS | +| **Compute** | FP8 Throughput | `{python} H100Recap.fp8_tflops_str` TFLOPS | +| **Memory** | HBM3 Bandwidth | `{python} H100Recap.bw_tb_str` TB/s | +| **Memory** | HBM3 Capacity | `{python} H100Recap.cap_gb_str` GB | +| **Thermal** | TDP | `{python} H100Recap.tdp_w_str` W | : **Single-Node Foundational Constants**. Recapping the hardware specifications for the H100 accelerator. These values provide the $R_{\text{peak}}$ and $BW$ baselines used in the Iron Law calculations throughout this volume. {#tbl-fleet-foundation-recap} @@ -359,8 +359,8 @@ The distributed systems reasoning in this book builds upon the single-machine pe This book uses four canonical cluster sizes to illustrate how system behavior changes across scale. These sizes correspond to real-world deployment tiers: a research lab cluster, a medium-scale production cluster, a large training cluster, and a hyperscale fleet. @tbl-fleet-cluster-sizes defines the GPU count for each tier; the chapters that reference them derive node counts, failure rates, and network requirements from these baselines combined with the constants in subsequent tables. -| **Constant** | **Value** | **Unit** | -|:------------------------|:-----------------------------------|:------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------|:-----------------------------------------------------|:------------------------------------------------------| | **CLUSTER_SMALL_GPUS** | `{python} AppendixConstants.CLUSTER_SMALL_GPUS_val` | `{python} AppendixConstants.CLUSTER_SMALL_GPUS_unit` | | **CLUSTER_MEDIUM_GPUS** | `{python} AppendixConstants.CLUSTER_MEDIUM_GPUS_val` | `{python} AppendixConstants.CLUSTER_MEDIUM_GPUS_unit` | | **CLUSTER_LARGE_GPUS** | `{python} AppendixConstants.CLUSTER_LARGE_GPUS_val` | `{python} AppendixConstants.CLUSTER_LARGE_GPUS_unit` | @@ -380,8 +380,8 @@ Distributed training and inference are bottlenecked by data movement between mac Within a single node, GPUs communicate over NVLink or PCIe. These bandwidths determine whether tensor parallelism (which requires high-bandwidth, low-latency communication) is viable within the node boundary. @tbl-fleet-intra-node lists the intra-node interconnect bandwidths and latencies used throughout this book. These values are defined in @sec-system-assumptions and repeated here for convenience. -| **Constant** | **Value** | **Unit** | -|:----------------------|:---------------------------------|:----------------------------------| +| **Constant** | **Value** | **Unit** | +|:----------------------|:---------------------------------------------------|:----------------------------------------------------| | **NVLINK_V100_BW** | `{python} AppendixConstants.NVLINK_V100_BW_val` | `{python} AppendixConstants.NVLINK_V100_BW_unit` | | **NVLINK_A100_BW** | `{python} AppendixConstants.NVLINK_A100_BW_val` | `{python} AppendixConstants.NVLINK_A100_BW_unit` | | **NVLINK_H100_BW** | `{python} AppendixConstants.NVLINK_H100_BW_val` | `{python} AppendixConstants.NVLINK_H100_BW_unit` | @@ -397,8 +397,8 @@ Within a single node, GPUs communicate over NVLink or PCIe. These bandwidths det Once data crosses the node boundary, bandwidth drops by an order of magnitude and latency increases by 5--10$\times$. This cliff shapes every decision about parallelism strategy: operations that require frequent, fine-grained communication (tensor parallelism) stay within the node, while operations with coarser communication patterns (data parallelism, pipeline parallelism) span nodes. @tbl-fleet-inter-node lists both the bit-rate and byte-rate forms of each fabric, since different chapters use different conventions. -| **Constant** | **Value** | **Unit** | -|:--------------------------|:-------------------------------------|:--------------------------------------| +| **Constant** | **Value** | **Unit** | +|:--------------------------|:-------------------------------------------------------|:--------------------------------------------------------| | **INFINIBAND_HDR_BW** | `{python} AppendixConstants.INFINIBAND_HDR_BW_val` | `{python} AppendixConstants.INFINIBAND_HDR_BW_unit` | | **INFINIBAND_HDR_BW_GBS** | `{python} AppendixConstants.INFINIBAND_HDR_BW_GBS_val` | `{python} AppendixConstants.INFINIBAND_HDR_BW_GBS_unit` | | **INFINIBAND_NDR_BW** | `{python} AppendixConstants.INFINIBAND_NDR_BW_val` | `{python} AppendixConstants.INFINIBAND_NDR_BW_unit` | @@ -425,8 +425,8 @@ At fleet scale, component failures are not exceptional events but statistical ce @tbl-fleet-mttf lists the MTTF for each component class. These are steady-state values that exclude the "infant mortality" period (first 30--90 days) and the "wear-out" period (beyond rated lifetime). Sources include published fleet studies from Meta (2024), Google (2024), and Barroso et al. (2018). -| **Constant** | **Value** | **Unit** | -|:---------------------------|:--------------------------------------|:---------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------|:--------------------------------------------------------|:---------------------------------------------------------| | **GPU_MTTF_HOURS** | `{python} AppendixConstants.GPU_MTTF_HOURS_val` | `{python} AppendixConstants.GPU_MTTF_HOURS_unit` | | **NIC_MTTF_HOURS** | `{python} AppendixConstants.NIC_MTTF_HOURS_val` | `{python} AppendixConstants.NIC_MTTF_HOURS_unit` | | **PSU_MTTF_HOURS** | `{python} AppendixConstants.PSU_MTTF_HOURS_val` | `{python} AppendixConstants.PSU_MTTF_HOURS_unit` | @@ -441,8 +441,8 @@ At fleet scale, component failures are not exceptional events but statistical ce Failure detection and recovery introduce downtime that compounds with cluster size. @tbl-fleet-recovery lists the assumptions used in checkpoint interval optimization (the Young-Daly formula in @sec-fault-tolerance-young-daly) and effective throughput calculations throughout this book. -| **Constant** | **Value** | **Unit** | -|:----------------------------|:---------------------------------------|:----------------------------------------| +| **Constant** | **Value** | **Unit** | +|:----------------------------|:---------------------------------------------------------|:----------------------------------------------------------| | **HEARTBEAT_TIMEOUT_S** | `{python} AppendixConstants.HEARTBEAT_TIMEOUT_S_val` | `{python} AppendixConstants.HEARTBEAT_TIMEOUT_S_unit` | | **RESCHEDULE_TIME_S** | `{python} AppendixConstants.RESCHEDULE_TIME_S_val` | `{python} AppendixConstants.RESCHEDULE_TIME_S_unit` | | **CHECKPOINT_WRITE_BW_GBS** | `{python} AppendixConstants.CHECKPOINT_WRITE_BW_GBS_val` | `{python} AppendixConstants.CHECKPOINT_WRITE_BW_GBS_unit` | @@ -455,8 +455,8 @@ Reliability constants tell you how often failures occur and how long recovery ta This book models communication cost using the $\alpha$-$\beta$ framework: the time to send a message of $m$ bytes is $T = \alpha + m / \beta$, where $\alpha$ is the startup latency and $\beta$ is the sustained bandwidth. @tbl-fleet-alpha-beta lists the $\alpha$ and $\beta$ values for each interconnect technology. These parameters are used in the collective communication cost models in @sec-collective-communication and the parallelism strategy comparisons in @sec-distributed-training-systems. -| **Constant** | **Value** | **Unit** | -|:------------------------------------|:-------------------------------------|:--------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------------------|:-------------------------------------------------------|:--------------------------------------------------------| | **IB_NDR_LATENCY_US** ($\alpha$) | `{python} AppendixConstants.IB_NDR_LATENCY_US_val` | `{python} AppendixConstants.IB_NDR_LATENCY_US_unit` | | **INFINIBAND_NDR_BW_GBS** ($\beta$) | `{python} AppendixConstants.INFINIBAND_NDR_BW_GBS_val` | `{python} AppendixConstants.INFINIBAND_NDR_BW_GBS_unit` | | **IB_HDR_LATENCY_US** ($\alpha$) | `{python} AppendixConstants.IB_HDR_LATENCY_US_val` | `{python} AppendixConstants.IB_HDR_LATENCY_US_unit` | @@ -471,8 +471,8 @@ This book models communication cost using the $\alpha$-$\beta$ framework: the ti The ring AllReduce --- the most common collective for gradient synchronization --- transmits $2 \times (N-1)/N \times m$ bytes per rank, where $N$ is the number of ranks and $m$ is the message size. For large $N$, this approaches $2m$ bytes per rank. @tbl-fleet-allreduce lists the constants used in AllReduce cost estimation. -| **Constant** | **Value** | **Unit** | -|:---------------------|:--------------------------------|:---------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------|:--------------------------------------------------|:---------------------------------------------------| | **ALLREDUCE_FACTOR** | `{python} AppendixConstants.ALLREDUCE_FACTOR_val` | `{python} AppendixConstants.ALLREDUCE_FACTOR_unit` | | **GPUS_PER_HOST** | `{python} AppendixConstants.GPUS_PER_HOST_val` | `{python} AppendixConstants.GPUS_PER_HOST_unit` | @@ -488,8 +488,8 @@ The environmental impact of fleet-scale ML depends on three factors: how much po Power Usage Effectiveness (PUE) is the ratio of total facility power to IT equipment power. A PUE of 1.0 would mean zero cooling overhead; real datacenters range from 1.06 (liquid-cooled) to 1.58 (legacy air-cooled). @tbl-fleet-pue lists the reference PUE values used throughout this book. -| **Constant** | **Value** | **Unit** | -|:----------------------|:---------------------------------|:----------------------------------| +| **Constant** | **Value** | **Unit** | +|:----------------------|:---------------------------------------------------|:----------------------------------------------------| | **PUE_LIQUID_COOLED** | `{python} AppendixConstants.PUE_LIQUID_COOLED_val` | `{python} AppendixConstants.PUE_LIQUID_COOLED_unit` | | **PUE_BEST_AIR** | `{python} AppendixConstants.PUE_BEST_AIR_val` | `{python} AppendixConstants.PUE_BEST_AIR_unit` | | **PUE_TYPICAL** | `{python} AppendixConstants.PUE_TYPICAL_val` | `{python} AppendixConstants.PUE_TYPICAL_unit` | @@ -501,8 +501,8 @@ Power Usage Effectiveness (PUE) is the ratio of total facility power to IT equip Water Usage Effectiveness (WUE) measures liters of water consumed per kilowatt-hour of IT energy. Evaporative cooling towers achieve excellent PUE but consume significant water; closed-loop liquid cooling uses near-zero water but requires higher capital investment. @tbl-fleet-wue lists the reference WUE values. -| **Constant** | **Value** | **Unit** | -|:--------------------|:-------------------------------|:--------------------------------| +| **Constant** | **Value** | **Unit** | +|:--------------------|:-------------------------------------------------|:--------------------------------------------------| | **WUE_AIR_COOLED** | `{python} AppendixConstants.WUE_AIR_COOLED_val` | `{python} AppendixConstants.WUE_AIR_COOLED_unit` | | **WUE_EVAPORATIVE** | `{python} AppendixConstants.WUE_EVAPORATIVE_val` | `{python} AppendixConstants.WUE_EVAPORATIVE_unit` | | **WUE_LIQUID** | `{python} AppendixConstants.WUE_LIQUID_val` | `{python} AppendixConstants.WUE_LIQUID_unit` | @@ -513,8 +513,8 @@ Water Usage Effectiveness (WUE) measures liters of water consumed per kilowatt-h The same training run emits vastly different amounts of CO$_2$ depending on the grid that supplies its electricity. @tbl-fleet-carbon lists regional grid carbon intensities from the International Energy Agency (IEA, 2023). These values are central to the location-aware scheduling strategies discussed in @sec-sustainable-ai. -| **Constant** | **Value** | **Unit** | -|:---------------------------|:--------------------------------------|:---------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------|:--------------------------------------------------------|:---------------------------------------------------------| | **CARBON_NORWAY_GCO2_KWH** | `{python} AppendixConstants.CARBON_NORWAY_GCO2_KWH_val` | `{python} AppendixConstants.CARBON_NORWAY_GCO2_KWH_unit` | | **CARBON_QUEBEC_GCO2_KWH** | `{python} AppendixConstants.CARBON_QUEBEC_GCO2_KWH_val` | `{python} AppendixConstants.CARBON_QUEBEC_GCO2_KWH_unit` | | **CARBON_FRANCE_GCO2_KWH** | `{python} AppendixConstants.CARBON_FRANCE_GCO2_KWH_val` | `{python} AppendixConstants.CARBON_FRANCE_GCO2_KWH_unit` | @@ -528,8 +528,8 @@ The same training run emits vastly different amounts of CO$_2$ depending on the AI accelerator racks draw 6--8$\times$ more power than traditional datacenter racks, creating thermal density challenges that force the transition from air cooling to liquid cooling. @tbl-fleet-power-density lists the reference rack power levels used in @sec-compute-infrastructure and @sec-sustainable-ai. -| **Constant** | **Value** | **Unit** | -|:------------------------------|:-----------------------------------------|:------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------------|:-----------------------------------------------------------|:------------------------------------------------------------| | **RACK_POWER_TRADITIONAL_KW** | `{python} AppendixConstants.RACK_POWER_TRADITIONAL_KW_val` | `{python} AppendixConstants.RACK_POWER_TRADITIONAL_KW_unit` | | **RACK_POWER_AI_TYPICAL_KW** | `{python} AppendixConstants.RACK_POWER_AI_TYPICAL_KW_val` | `{python} AppendixConstants.RACK_POWER_AI_TYPICAL_KW_unit` | | **RACK_POWER_AI_HIGH_KW** | `{python} AppendixConstants.RACK_POWER_AI_HIGH_KW_val` | `{python} AppendixConstants.RACK_POWER_AI_HIGH_KW_unit` | @@ -543,8 +543,8 @@ Sustainability constants capture the physical and environmental costs of running Fleet-scale cost estimates depend on GPU rental rates, electricity pricing, and data transfer charges. These are representative cloud rates; on-premise costs differ in structure (capital expenditure vs. operating expenditure) but the relative magnitudes guide the same capacity planning decisions. @tbl-fleet-economic lists the pricing assumptions used in @sec-ops-scale and @sec-inference-scale. -| **Constant** | **Value** | **Unit** | -|:---------------------------------|:--------------------------------------------|:---------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:---------------------------------|:--------------------------------------------------------------|:---------------------------------------------------------------| | **CLOUD_GPU_TRAINING_PER_HOUR** | `{python} AppendixConstants.CLOUD_GPU_TRAINING_PER_HOUR_val` | `{python} AppendixConstants.CLOUD_GPU_TRAINING_PER_HOUR_unit` | | **CLOUD_GPU_INFERENCE_PER_HOUR** | `{python} AppendixConstants.CLOUD_GPU_INFERENCE_PER_HOUR_val` | `{python} AppendixConstants.CLOUD_GPU_INFERENCE_PER_HOUR_unit` | | **CLOUD_ELECTRICITY_PER_KWH** | `{python} AppendixConstants.CLOUD_ELECTRICITY_PER_KWH_val` | `{python} AppendixConstants.CLOUD_ELECTRICITY_PER_KWH_unit` | @@ -562,8 +562,8 @@ The constants in this section quantify two related phenomena: (1) the fraction o Model FLOPS Utilization (MFU) measures the ratio of actual compute throughput to the hardware's peak theoretical throughput. An MFU of 0.50 means the workload achieves half the peak FLOPS. @tbl-fleet-mfu lists the reference MFU ranges used in training time and cost estimates throughout this book. -| **Constant** | **Value** | **Unit** | -|:--------------------------|:-------------------------------------|:--------------------------------------| +| **Constant** | **Value** | **Unit** | +|:--------------------------|:-------------------------------------------------------|:--------------------------------------------------------| | **MFU_TRAINING_LOW** | `{python} AppendixConstants.MFU_TRAINING_LOW_val` | `{python} AppendixConstants.MFU_TRAINING_LOW_unit` | | **MFU_TRAINING_HIGH** | `{python} AppendixConstants.MFU_TRAINING_HIGH_val` | `{python} AppendixConstants.MFU_TRAINING_HIGH_unit` | | **MFU_INFERENCE_BATCH1** | `{python} AppendixConstants.MFU_INFERENCE_BATCH1_val` | `{python} AppendixConstants.MFU_INFERENCE_BATCH1_unit` | @@ -575,8 +575,8 @@ Model FLOPS Utilization (MFU) measures the ratio of actual compute throughput to Scaling efficiency $\eta = T_1 / (N \times T_N)$ measures how much of the added compute actually reduces training time. Perfect scaling ($\eta = 1.0$) means doubling the GPUs halves the time. @tbl-fleet-scaling lists how scaling efficiency degrades with cluster size, based on published results for large-language-model training. -| **Constant** | **Value** | **Unit** | -|:------------------------|:-----------------------------------|:------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------|:-----------------------------------------------------|:------------------------------------------------------| | **SCALING_EFF_32GPU** | `{python} AppendixConstants.SCALING_EFF_32GPU_val` | `{python} AppendixConstants.SCALING_EFF_32GPU_unit` | | **SCALING_EFF_256GPU** | `{python} AppendixConstants.SCALING_EFF_256GPU_val` | `{python} AppendixConstants.SCALING_EFF_256GPU_unit` | | **SCALING_EFF_1024GPU** | `{python} AppendixConstants.SCALING_EFF_1024GPU_val` | `{python} AppendixConstants.SCALING_EFF_1024GPU_unit` | @@ -588,8 +588,8 @@ Scaling efficiency $\eta = T_1 / (N \times T_N)$ measures how much of the added Real training jobs spend a fraction of wall time on non-compute activities: pipeline bubble idle time, checkpoint writes, failure recovery, and scheduled maintenance. @tbl-fleet-overhead lists the overhead budgets assumed throughout this book. These fractions are additive: total overhead at fleet scale is approximately 5% + 3% + 10% + 5% = 23%, meaning only 77% of wall time produces useful training progress. -| **Constant** | **Value** | **Unit** | -|:------------------------------|:-----------------------------------------|:------------------------------------------| +| **Constant** | **Value** | **Unit** | +|:------------------------------|:-----------------------------------------------------------|:------------------------------------------------------------| | **OVERHEAD_PIPELINE_BUBBLE** | `{python} AppendixConstants.OVERHEAD_PIPELINE_BUBBLE_val` | `{python} AppendixConstants.OVERHEAD_PIPELINE_BUBBLE_unit` | | **OVERHEAD_CHECKPOINT** | `{python} AppendixConstants.OVERHEAD_CHECKPOINT_val` | `{python} AppendixConstants.OVERHEAD_CHECKPOINT_unit` | | **OVERHEAD_FAILURE_RECOVERY** | `{python} AppendixConstants.OVERHEAD_FAILURE_RECOVERY_val` | `{python} AppendixConstants.OVERHEAD_FAILURE_RECOVERY_unit` | diff --git a/book/quarto/contents/vol2/backmatter/appendix_communication.qmd b/book/quarto/contents/vol2/backmatter/appendix_communication.qmd index 5911503598..7f52723350 100644 --- a/book/quarto/contents/vol2/backmatter/appendix_communication.qmd +++ b/book/quarto/contents/vol2/backmatter/appendix_communication.qmd @@ -180,14 +180,14 @@ The model decomposes every transfer into exactly two costs: a per-message tax ($ @tbl-alpha-beta-params lists the $\alpha$ and $\beta$ values for interconnects commonly found in ML training clusters. These numbers are the starting point for every communication cost estimate in this appendix. -| **Interconnect** | **$\alpha$ (Latency)** | **$\beta$ (Bandwidth)** | **Typical Role** | -|:------------------------------|:--------------------------------------|:------------------------------------|:---------------------------------| +| **Interconnect** | **$\alpha$ (Latency)** | **$\beta$ (Bandwidth)** | **Typical Role** | +|:------------------------------|:-----------------------------------------|:---------------------------------------|:---------------------------------| | **NVLink 4.0 (H100)** | `{python} CF.nvlink_lat_str` ns | `{python} CF.nvlink_bw_str` GB/s | Intra-node GPU-to-GPU | | **PCIe Gen5** | `{python} CF.pcie_lat_str` ns | `{python} CF.pcie_bw_str` GB/s | CPU-GPU, NIC-GPU | | **InfiniBand NDR (400 Gbps)** | `{python} CF.ib_ndr_alpha_us_str` $\mu$s | `{python} CF.ib_ndr_beta_gbs_str` GB/s | Inter-node (high-end clusters) | | **InfiniBand HDR (200 Gbps)** | `{python} CF.ib_hdr_alpha_us_str` $\mu$s | `{python} CF.ib_hdr_beta_gbs_str` GB/s | Inter-node (previous generation) | | **RoCE v2 (100 GbE)** | `{python} CF.roce_alpha_us_str` $\mu$s | `{python} CF.roce_beta_gbs_str` GB/s | Inter-node (Ethernet clusters) | -| **TCP/IP (Ethernet)** | `{python} CF.tcp_alpha_us_str` $\mu$s | Varies | Control plane, non-RDMA fallback | +| **TCP/IP (Ethernet)** | `{python} CF.tcp_alpha_us_str` $\mu$s | Varies | Control plane, non-RDMA fallback | : **$\alpha$-$\beta$ Parameters by Interconnect**: Startup latency and sustained bandwidth for interconnects used in ML clusters. NVLink and PCIe operate within a node; InfiniBand and RoCE operate between nodes. TCP latency is dominated by kernel software stack overhead. {#tbl-alpha-beta-params} @@ -209,7 +209,7 @@ Below $M_{\text{cross}}$, communication is **latency-dominated**: sending more s **Calculation**: -$$M_{\text{cross}} = \alpha \times \beta = `{python} CF.ib_ndr_alpha_us_str` \times 10^{-6} \text{ s} \times `{python} CF.ib_ndr_beta_gbs_str` \times 10^{9} \text{ B/s} = `{python} CF.crossover_kb_str` \text{ KB}$$ +$M_{\text{cross}} = \alpha \times \beta =$ `{python} CF.ib_ndr_alpha_us_str` $\times 10^{-6}$ s $\times$ `{python} CF.ib_ndr_beta_gbs_str` $\times 10^{9}$ B/s $=$ `{python} CF.crossover_kb_str` KB **Interpretation**: Messages smaller than `{python} CF.crossover_kb_str` KB are latency-dominated on IB NDR. Gradient tensors from a single layer of a large model are typically 10--100 MB---well above this threshold. But control messages, heartbeats, and barrier synchronizations are well below it. @@ -264,11 +264,11 @@ The tree algorithm trades bandwidth efficiency for latency efficiency: the laten **Ring AllReduce**: -$$T_{\text{ring}} \approx \mathbf{`{python} CF.ring_time_ms_str` \text{ ms}}$$ +$T_{\text{ring}} \approx$ **`{python} CF.ring_time_ms_str` ms** **Tree AllReduce**: -$$T_{\text{tree}} \approx \mathbf{`{python} CF.tree_time_ms_str` \text{ ms}}$$ +$T_{\text{tree}} \approx$ **`{python} CF.tree_time_ms_str` ms** **Interpretation**: For this 1 GB message, ring AllReduce takes `{python} CF.ring_time_ms_str` ms while tree takes `{python} CF.tree_time_ms_str` ms---the ring is significantly faster because it distributes the bandwidth load across all links. The tree's logarithmic latency advantage is negligible compared to its bandwidth penalty for large messages. @@ -392,10 +392,10 @@ Rule of thumb To keep bubble overhead below 20%, you need $M \geq 4(P - 1)$, whi @tbl-pipeline-bubble shows how bubble fraction varies with pipeline depth and microbatch count. -| **Pipeline Stages ($P$)** | **$M$ = 8** | **$M$ = 16** | **$M$ = 32** | **$M$ = 64** | -|:--------------------------|:---------------------------|:---------------------------|:---------------------------|:---------------------------| -| **$P$ = 4** | `{python} CF.bubble_std_pct`% | `{python} CF.bubble_std_pct`% | --- | --- | -| **$P$ = 8** | --- | `{python} CF.bubble_std_pct`% | `{python} CF.bubble_std_pct`% | `{python} CF.bubble_std_pct`% | +| **Pipeline Stages ($P$)** | **$M$ = 8** | **$M$ = 16** | **$M$ = 32** | **$M$ = 64** | +|:--------------------------|:------------------------------|:------------------------------|:------------------------------|:------------------------------| +| **$P$ = 4** | `{python} CF.bubble_std_pct`% | `{python} CF.bubble_std_pct`% | --- | --- | +| **$P$ = 8** | --- | `{python} CF.bubble_std_pct`% | `{python} CF.bubble_std_pct`% | `{python} CF.bubble_std_pct`% | : **Pipeline Bubble Fraction**: Percentage of GPU-time lost to pipeline bubbles for various stage counts and microbatch counts. Values below 20% (the typical target) require $M \geq 4P$. Dashes indicate configurations where the ratio is impractical. {#tbl-pipeline-bubble} diff --git a/book/quarto/contents/vol2/backmatter/appendix_fleet.qmd b/book/quarto/contents/vol2/backmatter/appendix_fleet.qmd index 0f4d884e93..3d0fbb171d 100644 --- a/book/quarto/contents/vol2/backmatter/appendix_fleet.qmd +++ b/book/quarto/contents/vol2/backmatter/appendix_fleet.qmd @@ -300,15 +300,15 @@ class FleetQuickRef: h100_tdp = fmt(FF.h100_tdp, precision=0, commas=False) ``` -| **Category** | **Number** | **Use** | -|:---------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------| -| **Communication** | NVLink ~`{python} FleetQuickRef.nvlink_ib`× IB NDR | Parallelism boundary (in-node vs cross-node) | -| **Communication** | IB NDR ~`{python} FleetQuickRef.ib_ndr_gbs` GB/s, ~`{python} FleetQuickRef.ib_ndr_us` μs one-way | Inter-node bandwidth and latency | -| **Computation** | MFU `{python} FleetQuickRef.mfu_lo`--`{python} FleetQuickRef.mfu_hi`%, η ~`{python} FleetQuickRef.eff_1024`% @ 1K, ~`{python} FleetQuickRef.eff_8192`% @ 8K | Effective FLOPS and scaling sanity checks | -| **Coordination** | MTBF 8K: ~`{python} FleetQuickRef.mtbf_8k_min` min; 100K: ~`{python} FleetQuickRef.mtbf_100k_min` min | Failure expectation and checkpoint cadence | -| **Coordination** | 175B checkpoint ~`{python} FleetQuickRef.ckpt_175` GB (16 B/param) | Recovery and storage sizing | -| **Coordination** | Goodput ~`{python} FleetQuickRef.goodput`% after overheads | Wall-clock utilization | -| **Power & sustainability** | AI rack ~`{python} FleetQuickRef.rack_ai` kW; air limit ~`{python} FleetQuickRef.air_limit` kW | Cooling feasibility (liquid required above air limit) | +| **Category** | **Number** | **Use** | +|:---------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------| +| **Communication** | NVLink ~`{python} FleetQuickRef.nvlink_ib`× IB NDR | Parallelism boundary (in-node vs cross-node) | +| **Communication** | IB NDR ~`{python} FleetQuickRef.ib_ndr_gbs` GB/s, ~`{python} FleetQuickRef.ib_ndr_us` μs one-way | Inter-node bandwidth and latency | +| **Computation** | MFU `{python} FleetQuickRef.mfu_lo`--`{python} FleetQuickRef.mfu_hi`%, η ~`{python} FleetQuickRef.eff_1024`% @ 1K, ~`{python} FleetQuickRef.eff_8192`% @ 8K | Effective FLOPS and scaling sanity checks | +| **Coordination** | MTBF 8K: ~`{python} FleetQuickRef.mtbf_8k_min` min; 100K: ~`{python} FleetQuickRef.mtbf_100k_min` min | Failure expectation and checkpoint cadence | +| **Coordination** | 175B checkpoint ~`{python} FleetQuickRef.ckpt_175` GB (16 B/param) | Recovery and storage sizing | +| **Coordination** | Goodput ~`{python} FleetQuickRef.goodput`% after overheads | Wall-clock utilization | +| **Power & sustainability** | AI rack ~`{python} FleetQuickRef.rack_ai` kW; air limit ~`{python} FleetQuickRef.air_limit` kW | Cooling feasibility (liquid required above air limit) | | **Power & sustainability** | PUE liquid ~`{python} FleetQuickRef.pue_liq`, typical ~`{python} FleetQuickRef.pue_typ`; H100 `{python} FleetQuickRef.h100_tdp` W → 10K GPUs ≈ 7 MW IT × PUE | Facility load and carbon (see @sec-fleet-foundations-thermal-power) | : **Numbers every fleet engineer should know (quick reference)**: One-page summary of the fleet-scale reference numbers in this section. See the detailed Communication, Computation, and Coordination tables below for full context. {#tbl-fleet-numbers-quick-ref} @@ -378,29 +378,29 @@ class FleetCommNumbers: nvlink_to_ib = fmt(FF.nvlink_to_ib, precision=0, commas=False) ``` -| **Interconnect** | **Bandwidth** | **Typical Role** | -|:----------------------|:--------------------------------------------|:--------------------------------------------| -| **NVLink 4.0 (H100)** | `{python} FleetCommNumbers.nvlink_bw` GB/s | Tensor/pipeline parallelism within a node | -| **TPU v5p ICI** | `{python} FleetCommNumbers.tpuv5_ici` GB/s | Intra-pod model parallelism (Google) | -| **PCIe Gen5 x16** | `{python} FleetCommNumbers.pcie5_bw` GB/s | CPU-GPU data transfer, NIC attachment | -| **IB XDR (800 Gbps)** | `{python} FleetCommNumbers.ib_xdr_bw` GB/s | Next-gen inter-node (2025+) | -| **IB NDR (400 Gbps)** | `{python} FleetCommNumbers.ib_ndr_bw` GB/s | Current inter-node standard for AI clusters | -| **IB HDR (200 Gbps)** | `{python} FleetCommNumbers.ib_hdr_bw` GB/s | Previous-gen inter-node | -| **RoCE v2 (100 GbE)** | `{python} FleetCommNumbers.roce_bw` GB/s | Budget clusters, inference fleets | +| **Interconnect** | **Bandwidth** | **Typical Role** | +|:----------------------|:-------------------------------------------|:--------------------------------------------| +| **NVLink 4.0 (H100)** | `{python} FleetCommNumbers.nvlink_bw` GB/s | Tensor/pipeline parallelism within a node | +| **TPU v5p ICI** | `{python} FleetCommNumbers.tpuv5_ici` GB/s | Intra-pod model parallelism (Google) | +| **PCIe Gen5 x16** | `{python} FleetCommNumbers.pcie5_bw` GB/s | CPU-GPU data transfer, NIC attachment | +| **IB XDR (800 Gbps)** | `{python} FleetCommNumbers.ib_xdr_bw` GB/s | Next-gen inter-node (2025+) | +| **IB NDR (400 Gbps)** | `{python} FleetCommNumbers.ib_ndr_bw` GB/s | Current inter-node standard for AI clusters | +| **IB HDR (200 Gbps)** | `{python} FleetCommNumbers.ib_hdr_bw` GB/s | Previous-gen inter-node | +| **RoCE v2 (100 GbE)** | `{python} FleetCommNumbers.roce_bw` GB/s | Budget clusters, inference fleets | -: **Communication Bandwidth Hierarchy**: Bandwidth drops by `{python} FleetCommNumbers.nvlink_to_ib`$\times$ crossing from intra-node (NVLink) to inter-node (InfiniBand). This ratio determines parallelism placement. {#tbl-fleet-bandwidth-hierarchy} +: **Communication Bandwidth Hierarchy**: Bandwidth drops by roughly 18$\times$ crossing from intra-node (NVLink) to inter-node (InfiniBand). This ratio determines parallelism placement. {#tbl-fleet-bandwidth-hierarchy} @tbl-fleet-latency-hierarchy shows the one-way latency at each tier. For collective operations on small messages, latency---not bandwidth---is the bottleneck. -| **Interconnect** | **One-Way Latency** | **Implication** | -|:----------------------|:-------------------------------------------------|:--------------------------------------| -| **InfiniBand NDR** | ~`{python} FleetCommNumbers.ib_ndr_lat` $\mu$s | Low enough for synchronous AllReduce | -| **InfiniBand HDR** | ~`{python} FleetCommNumbers.ib_hdr_lat` $\mu$s | Adequate for most training topologies | -| **RoCE v2** | ~`{python} FleetCommNumbers.roce_lat` $\mu$s | Acceptable for data parallelism | -| **TCP/IP (Ethernet)** | ~`{python} FleetCommNumbers.tcp_lat` $\mu$s | Too slow for synchronous training | -| **Cross-datacenter** | ~40,000 $\mu$s (40 ms) | Physics floor; async training only | +| **Interconnect** | **One-Way Latency** | **Implication** | +|:----------------------|:-----------------------------------------------|:--------------------------------------| +| **InfiniBand NDR** | ~`{python} FleetCommNumbers.ib_ndr_lat` $\mu$s | Low enough for synchronous AllReduce | +| **InfiniBand HDR** | ~`{python} FleetCommNumbers.ib_hdr_lat` $\mu$s | Adequate for most training topologies | +| **RoCE v2** | ~`{python} FleetCommNumbers.roce_lat` $\mu$s | Acceptable for data parallelism | +| **TCP/IP (Ethernet)** | ~`{python} FleetCommNumbers.tcp_lat` $\mu$s | Too slow for synchronous training | +| **Cross-datacenter** | ~40,000 $\mu$s (40 ms) | Physics floor; async training only | -: **Communication Latency Hierarchy**: Latency determines whether synchronous training is feasible. TCP/IP is `{python} fmt(FF.ib_to_tcp_lat, precision=0, commas=False)`$\times$ slower than InfiniBand NDR, making it unsuitable for gradient synchronization in large clusters. {#tbl-fleet-latency-hierarchy} +: **Communication Latency Hierarchy**: Latency determines whether synchronous training is feasible. TCP/IP is roughly 10$\times$ slower than InfiniBand NDR, making it unsuitable for gradient synchronization in large clusters. {#tbl-fleet-latency-hierarchy} ### Computation Numbers {.unnumbered} @@ -461,14 +461,14 @@ class FleetMtbfTable: pfail_100k = fmt(FF.pfail_100k_24h * 100, precision=0, commas=False) ``` -| **Cluster Size** | **MTBF (GPU-only)** | **Minutes** | **P(failure) in 24 hours** | -|:-----------------|:---------------------------------------|:----------------------------------|:------------------------------| +| **Cluster Size** | **MTBF (GPU-only)** | **Minutes** | **P(failure) in 24 hours** | +|:-----------------|:------------------------------------------|:----------------------------------------|:--------------------------------------| | **256 GPUs** | `{python} FleetMtbfTable.mtbf_256` hours | `{python} FleetMtbfTable.mtbf_256_min` | `{python} FleetMtbfTable.pfail_256`% | | **2,048 GPUs** | `{python} FleetMtbfTable.mtbf_2048` hours | `{python} FleetMtbfTable.mtbf_2048_min` | `{python} FleetMtbfTable.pfail_2048`% | | **8,192 GPUs** | `{python} FleetMtbfTable.mtbf_8192` hours | `{python} FleetMtbfTable.mtbf_8192_min` | `{python} FleetMtbfTable.pfail_8192`% | | **100,000 GPUs** | `{python} FleetMtbfTable.mtbf_100k` hours | `{python} FleetMtbfTable.mtbf_100k_min` | `{python} FleetMtbfTable.pfail_100k`% | -: **MTBF and Failure Probability by Cluster Size**: GPU-only failure model with per-GPU MTTF of `{python} fmt(GPU_MTTF_HOURS, precision=0)` hours. Real clusters also include NIC, PSU, cable, and switch failures, making these estimates conservative. The probability column uses $P(\geq 1) = 1 - e^{-T/\text{MTBF}}$ for $T = 24$ hours. {#tbl-fleet-mtbf} +: **MTBF and Failure Probability by Cluster Size**: GPU-only failure model with per-GPU MTTF of 50,000 hours. Real clusters also include NIC, PSU, cable, and switch failures, making these estimates conservative. The probability column uses $P(\geq 1) = 1 - e^{-T/\text{MTBF}}$ for $T = 24$ hours. {#tbl-fleet-mtbf} The key takeaway: at 8,192 GPUs and above, failure is not a risk---it is a certainty within any training run longer than a few hours. Fault tolerance is not optional at fleet scale; it is a prerequisite for completing any training job. @sec-fault-tolerance-reliability covers the mechanisms in detail. @@ -500,14 +500,14 @@ class FleetCheckpointSizes: ckpt_1t = fmt(FF.ckpt_1t_tb, precision=0) ``` -| **Model Size** | **Checkpoint Size** | **Write Time @ 100 GB/s** | -|:--------------------|:------------------------------------------|:--------------------------| +| **Model Size** | **Checkpoint Size** | **Write Time @ 100 GB/s** | +|:--------------------|:---------------------------------------------|:--------------------------| | **7B parameters** | `{python} FleetCheckpointSizes.ckpt_7b` GB | ~1 second | | **70B parameters** | `{python} FleetCheckpointSizes.ckpt_70b` GB | ~11 seconds | | **175B parameters** | `{python} FleetCheckpointSizes.ckpt_175b` GB | ~28 seconds | | **1T parameters** | `{python} FleetCheckpointSizes.ckpt_1t` TB | ~2.7 minutes | -: **Checkpoint Sizes by Model Scale**: Uses 16 bytes/parameter (mixed-precision Adam). Write time assumes `{python} fmt(CHECKPOINT_WRITE_BW_GBS, precision=0)` GB/s aggregate storage bandwidth. Asynchronous checkpointing (@sec-fault-tolerance-reliability) can overlap writes with training, reducing the visible overhead. {#tbl-fleet-checkpoint-sizes} +: **Checkpoint Sizes by Model Scale**: Uses 16 bytes/parameter (mixed-precision Adam). Write time assumes 100 GB/s aggregate storage bandwidth. Asynchronous checkpointing (@sec-fault-tolerance-reliability) can overlap writes with training, reducing the visible overhead. {#tbl-fleet-checkpoint-sizes} #### Overhead Budgets {.unnumbered} @@ -520,7 +520,7 @@ At fleet scale, four categories of overhead consume wall-clock time that is not | **Failure recovery** | ~`{python} FF.oh_failure`% | Faster detection, elastic rescheduling | | **Maintenance windows** | ~`{python} FF.oh_maintenance`% | Rolling upgrades, live migration | -: **Overhead Budgets for Fleet-Scale Training**: These are fractions of wall-clock time. At 10,000+ GPUs, failure recovery dominates. The compound effect is multiplicative: total goodput ratio $\approx (1 - 0.05)(1 - 0.03)(1 - 0.10)(1 - 0.05) \approx$ `{python} fmt_percent(FF.goodput_ratio, precision=0)`%. {#tbl-fleet-overhead-budgets} +: **Overhead Budgets for Fleet-Scale Training**: These are fractions of wall-clock time. At 10,000+ GPUs, failure recovery dominates. The compound effect is multiplicative: total goodput ratio $\approx (1 - 0.05)(1 - 0.03)(1 - 0.10)(1 - 0.05) \approx 77$%. {#tbl-fleet-overhead-budgets} #### Power and Sustainability Numbers {.unnumbered} @@ -575,13 +575,13 @@ class FleetHardwareRef: tpuv5_ici = fmt(FF.tpuv5_ici, precision=0) ``` -| **Spec** | **NVIDIA H100 (SXM)** | **NVIDIA B200** | **Google TPU v5p** | -|:---------------------|:------------------------------------------|:--------------------------------------|:--------------------------------------------| -| **BF16/FP16 Peak** | `{python} FleetHardwareRef.h100_flops` TFLOPS | `{python} FleetHardwareRef.b200_flops` TFLOPS | `{python} FleetHardwareRef.tpuv5_flops` TFLOPS | -| **Memory Bandwidth** | `{python} FF.h100_bw_tbs` TB/s | `{python} FF.b200_bw_tbs` TB/s | `{python} FF.tpuv5_bw_tbs` TB/s | -| **HBM Capacity** | `{python} FleetHardwareRef.h100_cap` GB | `{python} FleetHardwareRef.b200_cap` GB | `{python} FleetHardwareRef.tpuv5_cap` GB | -| **Intra-Node Link** | 900 GB/s (NVLink 4.0) | 1,800 GB/s (NVLink 5.0) | `{python} FleetHardwareRef.tpuv5_ici` GB/s (ICI) | -| **TDP** | `{python} FleetHardwareRef.h100_tdp` W | `{python} FleetHardwareRef.b200_tdp` W | ~400 W (estimated) | +| **Spec** | **NVIDIA H100 (SXM)** | **NVIDIA B200** | **Google TPU v5p** | +|:---------------------|:----------------------------------------------|:----------------------------------------------|:-------------------------------------------------| +| **BF16/FP16 Peak** | `{python} FleetHardwareRef.h100_flops` TFLOPS | `{python} FleetHardwareRef.b200_flops` TFLOPS | `{python} FleetHardwareRef.tpuv5_flops` TFLOPS | +| **Memory Bandwidth** | `{python} FF.h100_bw_tbs` TB/s | `{python} FF.b200_bw_tbs` TB/s | `{python} FF.tpuv5_bw_tbs` TB/s | +| **HBM Capacity** | `{python} FleetHardwareRef.h100_cap` GB | `{python} FleetHardwareRef.b200_cap` GB | `{python} FleetHardwareRef.tpuv5_cap` GB | +| **Intra-Node Link** | 900 GB/s (NVLink 4.0) | 1,800 GB/s (NVLink 5.0) | `{python} FleetHardwareRef.tpuv5_ici` GB/s (ICI) | +| **TDP** | `{python} FleetHardwareRef.h100_tdp` W | `{python} FleetHardwareRef.b200_tdp` W | ~400 W (estimated) | : **Fleet-Scale Hardware Reference (c. 2024--2025)**: Per-accelerator specifications for the current generation. The B200 represents a ~4.5$\times$ compute uplift over the H100, with proportional memory bandwidth and capacity increases. {#tbl-fleet-hardware-ref} @@ -736,11 +736,11 @@ class FleetCommCompRatio: @tbl-fleet-comm-comp shows the ratio for three representative scenarios. The contrast between them reveals why parallelism strategy must match the workload. -| **Scenario** | **$T_{\text{comm}}$** | **$T_{\text{comp}}$** | **$\rho$** | -|:----------------------------------------|:-----------------------------------------------------|:----------------------|:------------------------| +| **Scenario** | **$T_{\text{comm}}$** | **$T_{\text{comp}}$** | **$\rho$** | +|:----------------------------------------|:------------------------------------------------------------------------|:----------------------|:-------------------------------------------| | **Data parallel, 7B model, 256 GPUs** | `{python} FleetCommCompRatio.ar_7b_ms_str` ms (AllReduce over IB NDR) | ~50 ms (fwd+bwd) | `{python} FleetCommCompRatio.rho_7b_str` | | **Data parallel, 350M model, 256 GPUs** | `{python} FleetCommCompRatio.ar_350m_ms_str` ms (AllReduce over IB NDR) | ~5 ms (fwd+bwd) | `{python} FleetCommCompRatio.rho_350m_str` | -| **Tensor parallel, 8 GPUs (NVLink)** | ~0.02 ms (activation over NVLink) | ~1 ms (one layer) | `{python} FleetCommCompRatio.rho_tp_str` | +| **Tensor parallel, 8 GPUs (NVLink)** | ~0.02 ms (activation over NVLink) | ~1 ms (one layer) | `{python} FleetCommCompRatio.rho_tp_str` | : **Communication-Computation Ratio ($\rho$)**: When $\rho \ll 1$, communication can be fully overlapped with computation. When $\rho \gg 1$, the system is communication-bound and GPUs sit idle waiting for data. Tensor parallelism within a node benefits from NVLink's high bandwidth, keeping $\rho$ small. {#tbl-fleet-comm-comp} @@ -826,7 +826,7 @@ The shift from traditional datacenter workloads to AI training has created a **p | **AI cluster (high-density)** | `{python} FF.rack_ai_high` kW | Direct-to-chip liquid cooling required | | **Air cooling limit** | ~`{python} FF.air_limit` kW | Physics ceiling for forced-air convection | -: **Power Density: Traditional vs. AI Workloads**: AI racks consume `{python} fmt(FF.rack_ratio, precision=1, commas=False)`$\times$ the power of traditional racks. Air cooling physically cannot remove heat fast enough above ~`{python} FF.air_limit` kW per rack, making liquid cooling mandatory for modern AI clusters. {#tbl-fleet-power-density-fleet} +: **Power Density: Traditional vs. AI Workloads**: AI racks consume roughly 5.8$\times$ the power of traditional racks. Air cooling physically cannot remove heat fast enough above ~30 kW per rack, making liquid cooling mandatory for modern AI clusters. {#tbl-fleet-power-density-fleet} The `{python} fmt(FF.rack_ratio, precision=1, commas=False)`$\times$ increase in rack power density between traditional and AI workloads is not merely an engineering challenge---it represents a fundamental constraint on facility design. Existing datacenters built for `{python} FF.rack_trad` kW racks cannot simply be "retrofitted" with AI hardware. The electrical distribution, cooling infrastructure, and floor loading must all be redesigned. This is why new AI-focused datacenters are being built from the ground up with liquid cooling as the baseline assumption. diff --git a/book/quarto/contents/vol2/backmatter/appendix_reliability.qmd b/book/quarto/contents/vol2/backmatter/appendix_reliability.qmd index ca2d13ff60..5aedb632db 100644 --- a/book/quarto/contents/vol2/backmatter/appendix_reliability.qmd +++ b/book/quarto/contents/vol2/backmatter/appendix_reliability.qmd @@ -92,13 +92,13 @@ class ReliabilityFoundations: t_detect = HEARTBEAT_TIMEOUT_S t_reschedule = RESCHEDULE_TIME_S ckpt_write_bw = CHECKPOINT_WRITE_BW_GBS - + model_sizes_params = [7e9, 13e9, 70e9, 175e9, 1e12] model_labels = ["7B", "13B", "70B", "175B", "1T"] - + overhead_ckpt = 0.05 overhead_failure = 0.05 - + avail_replicas = [1, 2, 3, 4, 5] avail_single = 0.99 @@ -299,7 +299,7 @@ for n_gpus in R.cluster_sizes: | **`{python} mtbf_data[4]["gpus"]`** | `{python} mtbf_data[4]["nodes"]` | `{python} mtbf_data[4]["mtbf"]` | `{python} mtbf_data[4]["per_day"]` | | **`{python} mtbf_data[5]["gpus"]`** | `{python} mtbf_data[5]["nodes"]` | `{python} mtbf_data[5]["mtbf"]` | `{python} mtbf_data[5]["per_day"]` | -: **Cluster MTBF by Scale.** As cluster size grows, the aggregate MTBF shrinks proportionally. At 10,000 GPUs, failures occur every few hours; at 100,000 GPUs, they occur continuously. Node configuration: `{python} f"{R.n_gpus_per_node}"` GPUs, `{python} f"{R.n_nics_per_node}"` NICs, `{python} f"{R.n_psus_per_node}"` PSUs per node. {#tbl-mtbf-cluster} +: **Cluster MTBF by Scale.** As cluster size grows, the aggregate MTBF shrinks proportionally. At 10,000 GPUs, failures occur every few hours; at 100,000 GPUs, they occur continuously. Node configuration: 8 GPUs, 2 NICs, 2 PSUs per node. {#tbl-mtbf-cluster} The table makes a visceral point: the transition from "hundreds of GPUs" to "tens of thousands" is not merely a quantitative change but a qualitative one. At 256 GPUs, you might go a full day between failures. At 10,000 GPUs, you should expect multiple failures per shift. At 100,000 GPUs, failures are a continuous background condition---the system is never fully healthy. @@ -433,7 +433,7 @@ for i, n_params in enumerate(R.model_sizes_params): | **`{python} ckpt_data[2]["label"]`** | `{python} ckpt_data[2]["ckpt_gb"]` | `{python} ckpt_data[2]["write_time"]` | | **`{python} ckpt_data[3]["label"]`** | `{python} ckpt_data[3]["ckpt_gb"]` | `{python} ckpt_data[3]["write_time"]` | -: **Checkpoint Sizes for Mixed-Precision Adam Training.** Each parameter requires 16 bytes of state (weights + gradients + optimizer state). Write times assume `{python} f"{R.ckpt_write_bw:.0f}"` GB/s aggregate storage bandwidth. {#tbl-checkpoint-size} +: **Checkpoint Sizes for Mixed-Precision Adam Training.** Each parameter requires 16 bytes of state (weights + gradients + optimizer state). Write times assume 100 GB/s aggregate storage bandwidth. {#tbl-checkpoint-size} At frontier scale (175B+ parameters), checkpoint sizes reach the terabyte range. This makes checkpoint write time a significant cost that directly affects the Young-Daly optimal interval. The checkpoint strategies in @sec-fault-tolerance-reliability discuss techniques for reducing $\delta$---asynchronous checkpointing, incremental deltas, and distributed storage---all of which improve the Young-Daly result by shrinking the numerator under the square root. @@ -558,13 +558,13 @@ class RecoveryAnatomy: t_total = fmt(R.t_recovery_total_s, precision=1) ``` -| **Phase** | **Typical Duration** | **What Happens** | -|:------------------------------|:----------------------------------------------|:------------------------------------------------| -| **$T_\text{detect}$** | `{python} RecoveryAnatomy.t_detect` s | Heartbeat timeout expires; failure is confirmed | -| **$T_\text{reschedule}$** | `{python} RecoveryAnatomy.t_reschedule` s | Replacement node allocated from spare pool | -| **$T_\text{reload}$** | `{python} RecoveryAnatomy.t_reload` s | Checkpoint read from storage into GPU memory | -| **$T_\text{replay}$** | ~`{python} RecoveryAnatomy.t_replay` min | Recompute training steps since last checkpoint | -| **Total $T_\text{recovery}$** | ~`{python} RecoveryAnatomy.t_total` min | System fully productive again | +| **Phase** | **Typical Duration** | **What Happens** | +|:------------------------------|:------------------------------------------|:------------------------------------------------| +| **$T_\text{detect}$** | `{python} RecoveryAnatomy.t_detect` s | Heartbeat timeout expires; failure is confirmed | +| **$T_\text{reschedule}$** | `{python} RecoveryAnatomy.t_reschedule` s | Replacement node allocated from spare pool | +| **$T_\text{reload}$** | `{python} RecoveryAnatomy.t_reload` s | Checkpoint read from storage into GPU memory | +| **$T_\text{replay}$** | ~`{python} RecoveryAnatomy.t_replay` min | Recompute training steps since last checkpoint | +| **Total $T_\text{recovery}$** | ~`{python} RecoveryAnatomy.t_total` min | System fully productive again | : **Recovery Time Breakdown.** Each phase contributes to the total time between failure and full-speed resumption. For the 10K-GPU, 175B-model scenario, replay dominates because it recomputes work lost since the last checkpoint. {#tbl-recovery-anatomy} @@ -670,7 +670,7 @@ for k in R.avail_replicas: | **`{python} avail_data[1]["k"]`** | `{python} avail_data[1]["avail"]` | `{python} avail_data[1]["nines"]` | `{python} avail_data[1]["downtime"]` | | **`{python} avail_data[2]["k"]`** | `{python} avail_data[2]["avail"]` | `{python} avail_data[2]["nines"]` | `{python} avail_data[2]["downtime"]` | -: **Availability Stacking with Independent Replicas.** Starting from a single-replica availability of `{python} f"{R.avail_single * 100:.0f}"`%, each additional replica dramatically reduces expected downtime. Assumes replica failures are independent. {#tbl-availability-stacking} +: **Availability Stacking with Independent Replicas.** Starting from a single-replica availability of 99%, each additional replica dramatically reduces expected downtime. Assumes replica failures are independent. {#tbl-availability-stacking} The power of stacking is dramatic: two replicas of a 99%-available system yield 99.99% availability, reducing annual downtime from roughly 87 hours to under an hour. This is why inference serving systems almost universally deploy multiple replicas behind a load balancer---the cost of an extra replica is small compared to the business value of four-nines availability. diff --git a/book/quarto/contents/vol2/backmatter/references.bib b/book/quarto/contents/vol2/backmatter/references.bib index 18c153ef99..55e33c406b 100644 --- a/book/quarto/contents/vol2/backmatter/references.bib +++ b/book/quarto/contents/vol2/backmatter/references.bib @@ -287,8 +287,8 @@ month = jun, journal = {arXiv preprint arXiv:1606.06565}, doi = {10.48550/arXiv.1606.06565}, - eprint = {1606.06565}, url = {http://arxiv.org/abs/1606.06565v2}, + eprint = {1606.06565}, primaryclass = {cs.AI}, archiveprefix = {arXiv}, } @@ -1790,9 +1790,9 @@ year = {2019}, month = nov, journal = {arXiv preprint arXiv:1911.01547}, - eprint = {1911.01547}, doi = {10.48550/arXiv.1911.01547}, url = {http://arxiv.org/abs/1911.01547v2}, + eprint = {1911.01547}, primaryclass = {cs.AI}, archiveprefix = {arXiv}, } @@ -7254,6 +7254,17 @@ date = {2022-04-09}, } +@article{rafailov2024direct, + title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model}, + author = { + Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D. and Ermon, + Stefano and Finn, Chelsea + }, + year = {2024}, + journal = {arXiv preprint arXiv:2305.18290}, + url = {https://arxiv.org/abs/2305.18290}, +} + @inproceedings{Raffel2020exploring, title = {Do Transformer Modifications Transfer Across Implementations and Applications?}, author = { @@ -7882,8 +7893,8 @@ month = jul, journal = {arXiv preprint arXiv:1707.06347}, doi = {10.48550/arXiv.1707.06347}, - eprint = {1707.06347}, url = {http://arxiv.org/abs/1707.06347v2}, + eprint = {1707.06347}, primaryclass = {cs.LG}, archiveprefix = {arXiv}, } @@ -8054,9 +8065,9 @@ }, year = {2019}, journal = {arXiv preprint arXiv:1904.12843}, - eprint = {1904.12843}, doi = {10.48550/arXiv.1904.12843}, url = {http://arxiv.org/abs/1904.12843v2}, + eprint = {1904.12843}, primaryclass = {cs.LG}, archiveprefix = {arXiv}, } @@ -8122,6 +8133,13 @@ journal = {International Conference on Learning Representations}, } +@inproceedings{shazeer2017outrageously, + title = {Outrageously large neural networks: The sparsely-gated mixture-of-experts layer}, + author = {Shazeer, Noam and Mirhoseini, Azalia and Maziarz, Piotr and others}, + year = {2017}, + booktitle = {International Conference on Learning Representations}, +} + @inproceedings{sheaffer2007hardware, title = { A hardware redundancy and recovery mechanism for reliable scientific computation on graphics @@ -8269,8 +8287,8 @@ }, year = {2010}, number = {dapper-2010-1}, - institution = {Google}, url = {https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/}, + institution = {Google}, } @article{silva2019federated, diff --git a/book/quarto/contents/vol2/collective_communication/collective_communication.qmd b/book/quarto/contents/vol2/collective_communication/collective_communication.qmd index d965f3f98e..fa01ed6568 100644 --- a/book/quarto/contents/vol2/collective_communication/collective_communication.qmd +++ b/book/quarto/contents/vol2/collective_communication/collective_communication.qmd @@ -49,11 +49,11 @@ This transition reveals a fundamental asymmetry in how computation and communica ::: {.callout-definition title="Gradient Synchronization"} -***Gradient Synchronization***\index{Gradient Synchronization!definition} is the collective communication phase where workers exchange and aggregate their locally computed gradients to maintain a consistent global model state. +***Gradient Synchronization***\index{Gradient Synchronization!definition} is the collective communication protocol executed at each training step in which every worker transmits its locally computed gradient tensor to all other workers, receives their gradients, and computes an aggregate update that all workers apply identically to their model copies. -1. **Significance (Quantitative):** It ensures mathematical equivalence to single-device training by ensuring all workers apply identical parameter updates. Within the **Iron Law**, it is the primary driver of the **Communication Overhead ($L_{\text{lat}}$)** and **Bandwidth ($BW$)** terms during training. -2. **Distinction (Durable):** Unlike **Asynchronous Updates**, which allow workers to proceed with stale weights, Gradient Synchronization (BSP) requires a **Global Barrier** where all workers must reach the same state before continuing. -3. **Common Pitfall:** A frequent misconception is that synchronization is "just communication." In reality, it is a **Synchronization Barrier**: the total time $T$ is governed by the **Slowest Worker** (Straggler), making the system highly sensitive to hardware jitter. +1. **Significance (Quantitative):** A 70B-parameter model in BF16 generates 140 GB of gradient data per worker per step. Synchronizing across 1,000 GPUs via ring AllReduce at 50 GB/s per link requires approximately 140 GB / 50 GB/s $\approx 2.8$ seconds of communication per step — dominating the $L_{\text{lat}}$ term in the Iron Law and consuming 40–70% of total training time before overlap techniques are applied. +2. **Distinction (Durable):** Unlike parameter-server approaches (where all workers send gradients to a centralized aggregator whose bandwidth scales as $O(N)$), ring AllReduce distributes the communication across all workers so that each worker's per-step communication cost stays constant at $2 \times (N-1)/N \times \text{gradient\_size}$ regardless of cluster size. +3. **Common Pitfall:** A frequent misconception is that gradient synchronization overhead scales with the number of GPUs. In ring AllReduce, per-node communication volume is constant as the cluster grows — the bottleneck is the gradient volume per step (proportional to model size), not the number of participants. ::: @@ -290,11 +290,11 @@ Every step our gradient takes is governed by the linear cost model $T(n) = \alph ::: {.callout-definition title="α-β Model (Hockney Model)"} -***α-β Model (Hockney Model)***\index{Alpha-Beta Model!definition} is the linear communication cost model $T(n) = \alpha + n/\beta$ that decomposes message transfer time into a fixed startup latency ($\alpha$) and a message-size-dependent bandwidth term ($n/\beta$). +***α-β Model (Hockney Model)***\index{Alpha-Beta Model!definition} is the linear communication cost model $T(n) = \alpha + n/\beta$ that decomposes message transfer time into a fixed startup latency ($\alpha$, the per-message overhead) and a message-size-dependent bandwidth term ($n/\beta$, proportional to bytes transferred), enabling algorithm designers to predict when message fusion or gradient compression will improve throughput. -1. **Significance (Quantitative):** It maps directly to the **Iron Law**, where $\alpha$ represents the fixed **Startup Latency ($L_{\text{lat}}$)** and $\beta$ represents the **Network Bandwidth ($BW$)**. The crossover point $n^* = \alpha \cdot \beta$ marks where communication transitions from latency-bound (small messages, e.g., MoE routing tokens) to bandwidth-bound (large messages, e.g., gradient tensors), directly determining whether to fuse messages (amortize $\alpha$) or compress them (reduce effective message size $n$, shifting the operating point toward the latency-bound regime where $\alpha$ dominates and further compression yields diminishing returns). -2. **Distinction (Durable):** Unlike **Idealized Throughput Models** (which assume bandwidth is the only cost), the α-β model captures the **Fixed Startup Penalty**, explaining why $N$ small messages of size $n/N$ are up to $N\times$ more expensive than one large message of size $n$ when $n/N \ll \alpha \cdot \beta$. -3. **Common Pitfall:** A frequent misconception is that $\alpha$ is merely "network delay." In practice, $\alpha$ includes **Software Overhead** (kernel context switches, NCCL stack traversal, library synchronization) that can exceed the physical wire delay by 2–5$\times$, making software optimization as important as network hardware selection. +1. **Significance (Quantitative):** For InfiniBand NDR at $\alpha \approx 2\,\mu\text{s}$ and $\beta \approx 50\,\text{GB/s}$, the crossover size $n^* = \alpha \cdot \beta \approx 100\,\text{KB}$. MoE routing tokens (~4 KB each) are 25$\times$ below $n^*$: fusing 100 tokens into one 400 KB message reduces communication cost from $100\alpha = 200\,\mu\text{s}$ to one $\alpha + n/\beta \approx 10\,\mu\text{s}$, a 20$\times$ improvement. By contrast, Top-K gradient compression that reduces a 140 GB gradient to 1.4 GB (1% sparsity) saves bandwidth but does not change the startup cost, so it is most valuable when $n \gg n^*$. +2. **Distinction (Durable):** Unlike idealized throughput models that treat bandwidth as the sole communication cost, the α-β model reveals that $N$ small messages of size $n/N$ cost up to $N\times$ more than one large message of size $n$ when $n/N \ll n^*$ — explaining why NCCL fuses small AllReduce calls and why MoE routing algorithms buffer tokens before launching collectives. +3. **Common Pitfall:** A frequent misconception is that gradient compression always helps. If the compressed gradient size remains well above $n^*$, compression reduces the bandwidth term but leaves the latency term unchanged — gradient sparsification at 99% compression on a 70B model still transfers 1.4 GB per AllReduce, remaining firmly bandwidth-bound and not recovering latency-regime performance. ::: @@ -340,10 +340,11 @@ class CriticalMsgSize: ``` ::: {.callout-notebook title="The Critical Message Size"} -**Problem**: Your cluster uses InfiniBand NDR 400G with $\alpha = `{python} CriticalMsgSize.alpha_str`\ \mu s$ and $\beta = `{python} CriticalMsgSize.beta_str`\ \text{GB/s}$. At what message size does optimizing for bandwidth start to matter more than optimizing for latency? +**Problem**: Your cluster uses InfiniBand NDR 400G with $\alpha =$ `{python} CriticalMsgSize.alpha_str` $\mu$s and $\beta =$ `{python} CriticalMsgSize.beta_str` GB/s. At what message size does optimizing for bandwidth start to matter more than optimizing for latency? **The Math**: -$$n = \alpha \cdot \beta = 2 \times 10^{-6}\ \text{s} \times 50 \times 10^9\ \text{B/s} = `{python} CriticalMsgSize.n_star_kb_str`\ \text{KB}$$ + +$n = \alpha \cdot \beta = 2 \times 10^{-6}$ s $\times 50 \times 10^9$ B/s $=$ `{python} CriticalMsgSize.n_star_kb_str` KB **The Systems Insight**: Messages under `{python} CriticalMsgSize.n_star_kb_str` KB (like MoE tokens, pipeline activations) are **latency-bound**—buy lower-latency switches, reduce software overhead. Messages over `{python} CriticalMsgSize.n_star_kb_str` KB (like LLM gradients) are **bandwidth-bound**—buy more bandwidth, compress the data. Applying the wrong optimization wastes money without improving performance. ::: @@ -391,7 +392,7 @@ class LatencyVsBw: ``` ::: {.callout-notebook title="Latency vs. Bandwidth Dominance"} -**Problem**: You are synchronizing a 1 MB buffer versus a 1 GB buffer on InfiniBand NDR with $\alpha = `{python} LatencyVsBw.alpha_str`\ \mu s$ and $\beta = `{python} LatencyVsBw.beta_str`\ \text{GB/s}$. How does the bottleneck shift? +**Problem**: You are synchronizing a 1 MB buffer versus a 1 GB buffer on InfiniBand NDR with $\alpha =$ `{python} LatencyVsBw.alpha_str` $\mu$s and $\beta =$ `{python} LatencyVsBw.beta_str` GB/s. How does the bottleneck shift? **Case A: 1 MB Message** @@ -408,6 +409,17 @@ class LatencyVsBw: **The Systems Conclusion**: For Data Parallelism (large gradients), we optimize for $\beta$—compress gradients, add NICs. For Pipeline/Expert Parallelism (small activations), we fight for every microsecond of $\alpha$—kernel bypass, topology optimization. The α-β model identifies *which constraint to optimize*. ::: +::: {.callout-checkpoint title="Alpha-Beta Diagnostics"} + +Verify your understanding of network performance regimes: + +- [ ] A message of 10 KB is being sent over a link where $\alpha=2 \mu s$ and $\beta=10 GB/s$. Is this message **Latency-Bound** or **Bandwidth-Bound**? +- [ ] If you reduce the "Software Overhead" ($\alpha$) by half, which type of parallelism benefits more: Data Parallelism or Tensor Parallelism? +- [ ] What happens to the **Critical Message Size** ($n^* = \alpha \beta$) as you upgrade from 100G to 400G networking while keeping software latency constant? +- [ ] True or False: In the bandwidth-bound regime, doubling the number of NICs per node effectively doubles the transmission speed for large tensors. + +::: + ### The LogP Model {#sec-communication-collective-operations-collective-operations-logp-model-e45d} The α-β model assumes the processor is idle during communication. For **pipelined systems** where we overlap communication with computation, this assumption fails. The **LogP** model [@culler1993logp] extends α-β with two additional parameters: @@ -453,7 +465,7 @@ class LogPOverlap: 4. **Hidden**: All `{python} LogPOverlap.L_str` μs of $L$ can overlap with compute. 5. **Exposed**: The `{python} LogPOverlap.two_o_str` μs of $o$ cannot overlap. -**Effective time**: $\max(`{python} LogPOverlap.compute_str`, `{python} LogPOverlap.L_str` + `{python} LogPOverlap.two_o_str`)$ = `{python} LogPOverlap.effective_str` $\mu\text{s}$ (communication hidden!). +**Effective time**: max(`{python} LogPOverlap.compute_str`, `{python} LogPOverlap.L_str` + `{python} LogPOverlap.two_o_str`) = `{python} LogPOverlap.effective_str` $\mu$s (communication hidden!). **The Systems Insight**: α-β tells you the total communication time. LogP tells you **how much of it you can hide**. When designing pipelined training, optimize for low $o$ (kernel bypass, GPUDirect) rather than high $\beta$ alone. ::: @@ -577,11 +589,11 @@ If a GPU simply opens a socket and sends a massive gradient to another GPU, the ::: {.callout-definition title="Collective Operation"} -***Collective Operation***\index{Collective Operation!definition} is a communication primitive where a group of processes engage in a coordinated data exchange to aggregate, broadcast, or redistribute information. +***Collective Operation***\index{Collective Operation!definition} is a distributed communication pattern in which all processes in a group participate simultaneously to aggregate, broadcast, or redistribute data — with the correctness guarantee that every participant receives the same result regardless of message ordering or arrival time. -1. **Significance (Quantitative):** They are the primary mechanisms for managing **Data Movement** across the fleet. Within the **Iron Law**, collective operations (e.g., AllReduce) determine the total **Communication Overhead ($L_{\text{lat}}$)** and whether training becomes **Bandwidth-Bound ($BW$)**. -2. **Distinction (Durable):** Unlike **Point-to-Point Communication** (one-to-one), Collective Operations are **Group-Aware** (one-to-all or all-to-all), using optimized algorithms (e.g., Ring, Tree) to minimize redundant transfers. -3. **Common Pitfall:** A frequent misconception is that collectives are merely library calls. In reality, they are **Synchronization Barriers**: every participant must reach the collective call point before the operation can complete, making them the primary source of **Straggler-Induced Stalls**. +1. **Significance (Quantitative):** The right collective algorithm determines whether communication scales with cluster size or remains constant. Ring AllReduce achieves bandwidth-optimal $2(N-1)/N \times M / \beta$ per node (constant in $N$), while a naive reduce-then-broadcast approach costs $O(N \times M / \beta)$, making it 500$\times$ worse for $N=1024$. Algorithm selection thus directly determines whether the $\text{BW}$ term in the Iron Law is the training bottleneck. +2. **Distinction (Durable):** Unlike point-to-point communication (where one sender and one receiver exchange data independently), a collective operation coordinates the entire process group — every participant must invoke the collective before any can complete it, and the library guarantees a consistent result even when different processes contribute different data. +3. **Common Pitfall:** A frequent misconception is that one collective algorithm suits all message sizes. Ring AllReduce achieves near-optimal bandwidth for large gradient tensors but has $O(N)$ latency — for small messages like MoE routing decisions (~4 KB), a recursive-halving (Rabenseifner) algorithm reduces latency to $O(\log N)$ steps at the cost of suboptimal bandwidth utilization, requiring workload-specific algorithm selection rather than a single default. ::: @@ -698,7 +710,7 @@ Each GPU holds `{python} MoeAllToAll.tokens_per_gpu_str` tokens that need to rea * Total data sent per GPU: 7 $\times$ `{python} MoeAllToAll.data_per_transfer_kb_str` KB = `{python} MoeAllToAll.total_sent_mb_str` MB * Total data received per GPU: `{python} MoeAllToAll.total_sent_mb_str` MB (symmetric) -**Latency Analysis (InfiniBand NDR, $\alpha = `{python} MoeAllToAll.alpha_str`\ \mu\text{s}$, $\beta = `{python} MoeAllToAll.beta_str`\ \text{GB/s}$)**: +**Latency Analysis (InfiniBand NDR, $\alpha =$ `{python} MoeAllToAll.alpha_str` $\mu$s, $\beta =$ `{python} MoeAllToAll.beta_str` GB/s)**: Each of the 7 transfers is a `{python} MoeAllToAll.data_per_transfer_kb_str` KB message. From the alpha-beta model: $T$ = 3 + `{python} MoeAllToAll.bw_time_approx_str` = `{python} MoeAllToAll.per_transfer_us_str` $\mu\text{s}$ per transfer. @@ -874,6 +886,17 @@ Each GPU sends its fully reduced chunk around the ring so all GPUs receive all r ::: +::: {.callout-checkpoint title="Ring AllReduce Mechanics"} + +Verify your understanding of bandwidth-optimal reduction: + +- [ ] In a ring of $N$ GPUs, how many **sequential steps** are required to complete the full AllReduce? +- [ ] Why is Ring AllReduce considered **Bandwidth-Optimal**? How many times does each byte of the total message $M$ traverse the network per GPU? +- [ ] If one GPU in the ring is 50% slower than the others, what is the impact on the total completion time of the collective? +- [ ] True or False: In Ring AllReduce, every GPU is both a sender and a receiver in every step of the algorithm. + +::: + The trace above illustrates the key property of Ring AllReduce: at every step, every link in the ring is active, with data flowing in the same direction. No GPU ever sits idle, and no link is underutilized. This uniform link utilization is what makes Ring bandwidth-optimal. The performance follows directly from the algorithm structure. Each node sends and receives $\frac{M}{N}$ bytes in each of the $2(N-1)$ steps. @@ -1092,26 +1115,26 @@ class RingVsTree: ``` ::: {.callout-notebook title="The Ring vs. Tree Crossover"} -**Problem**: You are synchronizing a **1 MB** buffer across **64 GPUs**. The network has latency $\alpha = `{python} RingVsTree.alpha_str`\ \mu s$ and bandwidth $\beta = `{python} RingVsTree.beta_str`\ \text{GB/s}$. Should you use Ring or Tree? +**Problem**: You are synchronizing a **1 MB** buffer across **64 GPUs**. The network has latency $\alpha =$ `{python} RingVsTree.alpha_str` $\mu$s and bandwidth $\beta =$ `{python} RingVsTree.beta_str` GB/s. Should you use Ring or Tree? **The Math**: *Latency Terms:* -1. **Ring Latency**: $2(N-1)\alpha = 2 \times 63 \times 10\ \mu s$ = $\mathbf{`{python} RingVsTree.ring_lat_str`\ \mu s}$. -2. **Tree Latency**: $2(\log_2 N)\alpha = 2 \times 6 \times 10\ \mu s$ = $\mathbf{`{python} RingVsTree.tree_lat_str`\ \mu s}$. +1. **Ring Latency**: $2(N-1)\alpha = 2 \times 63 \times 10\ \mu$s = **`{python} RingVsTree.ring_lat_str` $\mu$s**. +2. **Tree Latency**: $2(\log_2 N)\alpha = 2 \times 6 \times 10\ \mu$s = **`{python} RingVsTree.tree_lat_str` $\mu$s**. *Bandwidth Terms (note the difference!):* -3. **Ring Bandwidth**: $2\frac{N-1}{N}\frac{M}{\beta} \approx 2 \times \frac{1\text{ MB}}{10\text{ GB/s}}$ = $\mathbf{`{python} RingVsTree.ring_bw_str`\ \mu s}$ (optimal—each byte sent once). -4. **Tree Bandwidth**: $2\log_2 N \cdot \frac{M}{\beta} = 12 \times \frac{1\text{ MB}}{10\text{ GB/s}}$ = $\mathbf{`{python} RingVsTree.tree_bw_str`\ \mu s}$ (each level sends full message). +3. **Ring Bandwidth**: $2\frac{N-1}{N}\frac{M}{\beta} \approx 2 \times \frac{1\text{ MB}}{10\text{ GB/s}}$ = **`{python} RingVsTree.ring_bw_str` $\mu$s** (optimal—each byte sent once). +4. **Tree Bandwidth**: $2\log_2 N \cdot \frac{M}{\beta} = 12 \times \frac{1\text{ MB}}{10\text{ GB/s}}$ = **`{python} RingVsTree.tree_bw_str` $\mu$s** (each level sends full message). **Total Time**: -| **Algorithm** | **Latency** | **Bandwidth** | **Total** | -|:--------------|------------:|--------------:|-------------:| -| **Ring** | `{python} RingVsTree.ring_lat_str` μs | `{python} RingVsTree.ring_bw_str` μs | **`{python} RingVsTree.ring_total_str` μs** | -| **Tree** | `{python} RingVsTree.tree_lat_str` μs | `{python} RingVsTree.tree_bw_str` μs | **`{python} RingVsTree.tree_total_str` μs** | +| **Algorithm** | **Latency** | **Bandwidth** | **Total** | +|:--------------|--------------------------------------:|-------------------------------------:|--------------------------------------------:| +| **Ring** | `{python} RingVsTree.ring_lat_str` μs | `{python} RingVsTree.ring_bw_str` μs | **`{python} RingVsTree.ring_total_str` μs** | +| **Tree** | `{python} RingVsTree.tree_lat_str` μs | `{python} RingVsTree.tree_bw_str` μs | **`{python} RingVsTree.tree_total_str` μs** | **Tree wins**, but only by 10%. For this 1 MB message, we are near the crossover point. @@ -1561,11 +1584,11 @@ We solve the conflict between compression and convergence with **Error Feedback* ::: {.callout-definition title="Error Feedback Mechanism"} -***Error Feedback***\index{Error Feedback!definition} is a convergence-preserving technique that maintains a local accumulator to store the residual information lost during gradient compression. +***Error Feedback***\index{Error Feedback!definition} is a convergence-preserving technique for gradient compression that maintains a per-worker residual accumulator $e_t$, re-injecting the compression error back into the next gradient update so that information deferred by the compressor is never permanently discarded. -1. **Significance (Quantitative):** It allows for aggressive **Gradient Compression** (e.g., Top-K, 1-bit quantization) without sacrificing convergence. By re-injecting the residual ($e_t$) into the next gradient update, it ensures that small but consistent signals are eventually transmitted, maintaining an **Unbiased Estimator** over time. -2. **Distinction (Durable):** Unlike **Lossy Compression** (which permanently discards information), Error Feedback is a **Delayed Transmission** strategy: it ensures that information is not lost, but deferred until it accumulates above the compression threshold. -3. **Common Pitfall:** A frequent misconception is that Error Feedback is "built into" all compressors. In reality, without explicit error tracking, biased compressors (like Top-K) can lead to **Divergence** or poor generalization because small gradient components are never applied. +1. **Significance (Quantitative):** Top-K sparsification at 1% keeps only the largest 1% of gradient values by magnitude, reducing AllReduce data volume from 140 GB to 1.4 GB for a 70B model — an 100$\times$ bandwidth reduction. Without error feedback, the discarded 99% of gradient values are lost, causing training loss to plateau 2–5% above the dense baseline. With error feedback, the residual accumulates until small components cross the Top-K threshold in subsequent steps, recovering convergence at the same asymptotic rate as uncompressed SGD. +2. **Distinction (Durable):** Unlike lossy compression (which permanently discards sub-threshold gradient components), error feedback is a delayed transmission strategy: the residual $e_t$ telescopes across steps so that $\frac{1}{T}\sum_{t=1}^{T} v_t \to \frac{1}{T}\sum_{t=1}^{T} g_t$ as $T \to \infty$, making the long-run average of transmitted gradients equal to the true gradient average. +3. **Common Pitfall:** A frequent misconception is that error feedback is built into standard gradient compressors. PyTorch's built-in DDP gradient compression does not implement error feedback by default; without it, Top-K and 1-bit quantization introduce systematic bias that accumulates across steps, typically degrading final model accuracy by 1–3% on benchmark tasks compared to the feedback-corrected version. ::: @@ -1930,7 +1953,7 @@ class NapkinOverlap: ## Napkin Math: The Overlap Budget The following analysis determines whether the communication cost of a 70B parameter model on 8 GPUs can be hidden behind computation. The physics determines the answer. 1. **Gradient Size:** 70B params $\times$ 2 bytes (FP16) = `{python} NapkinOverlap.gradient_gb_str` GB. -2. **Communication Time:** Using Ring AllReduce on NVLink (900 GB/s), time is $\approx \frac{2 \cdot `{python} NapkinOverlap.gradient_gb_str`\text{GB}}{900\text{GB/s}}$ = `{python} NapkinOverlap.comm_s_str` s. +2. **Communication Time:** Using Ring AllReduce on NVLink (900 GB/s), time is $\approx \frac{2 \cdot M}{900\text{ GB/s}}$ = `{python} NapkinOverlap.comm_s_str` s. 3. **Compute Time:** A typical forward/backward pass for a block of this size takes $\approx$ `{python} NapkinOverlap.compute_s_str` s. 4. **Overlap Ratio:** `{python} NapkinOverlap.comm_s_str` s / `{python} NapkinOverlap.compute_s_str` s $\approx$ `{python} NapkinOverlap.overlap_pct_str`%. Since communication takes only `{python} NapkinOverlap.overlap_pct_str`% of the compute time, it fits comfortably within the backward pass "overlap budget," meaning the network cost effectively disappears. @@ -1970,7 +1993,58 @@ We explored the specialized "vehicles" of the datacenter, the **Collective Primi We followed the gradient's journey through **Ring** and **Tree AllReduce** algorithms, deriving the crossover point and understanding that the optimal path depends on cluster size and payload. **Hierarchical AllReduce** and **Rail-Optimized Routing** exploit the bandwidth hierarchy of modern GPU clusters to multiply effective inter-node bandwidth, while **in-network reduction (SHARP)** eliminates round-trips entirely by performing aggregation inside network switches. -When even the fastest wires are not enough, **Gradient Compression** provides the final squeeze. Quantization, sparsification, and compression-aware optimizers like **1-bit Adam** reduce communication volume by 4--1000$\times$. **Error Feedback** ensures that while the journey may be compressed or delayed, no vital information is permanently lost, preserving the mathematical truth of the model's convergence. Finally, **communication-computation overlap** through layer-by-layer pipelining and bucket fusion hides the remaining communication time behind useful computation, reducing the effective communication overhead to 5--15% of total step time in well-configured systems. +When even the fastest wires are not enough, **Gradient Compression** provides the final squeeze. Quantization, sparsification, and compression-aware optimizers like **1-bit Adam** reduce communication volume by 4--1000$\times$. + +```{python} +#| label: compression-payback-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ COMPRESSION PAYBACK (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Payback of Compression" .callout-notebook +# │ +# │ Goal: Quantify the speedup of gradient compression (quantization). +# │ Show: speedup ≈ 3.5x — inline in notebook result. +# │ How: T = T_compute + T_comm / Ratio. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: cp_comm_time_ms_str, cp_ratio_str, cp_overhead_ms_str, cp_speedup_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class CompressionPayback: + # ┌── 1. LOAD ────────────────────────────────────────── + t_comm_ms = 40 # Original FP32 AllReduce time + ratio = 8 # 1-bit quantization (32x compression, but with protocol overhead) + t_overhead_ms = 2 # Compute time to quantize/dequantize + # ┌── 2. EXECUTE ─────────────────────────────────────── + t_compressed_comm = t_comm_ms / ratio + t_total_new = t_compressed_comm + t_overhead_ms + speedup = t_comm_ms / t_total_new + # ┌── 3. GUARD ───────────────────────────────────────── + check(5 < speedup < 6, f"Speedup {speedup:.1f}x unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + cp_comm_time_ms_str = f"{t_comm_ms}" + cp_ratio_str = f"{ratio}" + cp_overhead_ms_str = f"{t_overhead_ms}" + cp_speedup_str = f"{speedup:.1f}" + +``` + +::: {.callout-notebook title="The Payback of Compression"} + +**Problem**: You are synchronizing a `{python} CompressionPayback.cp_comm_time_ms_str` ms gradient buffer. You implement **1-bit Adam**, which reduces communication volume by 32$\times$ but adds `{python} CompressionPayback.cp_overhead_ms_str` ms of CPU/GPU overhead for the compression logic. If the effective network throughput improves by `{python} CompressionPayback.cp_ratio_str`$\times$, what is your actual communication speedup? + +**The Math**: +1. **Compressed Comm Time**: `{python} CompressionPayback.cp_comm_time_ms_str` ms / `{python} CompressionPayback.cp_ratio_str` = **5 ms**. +2. **Total New Time**: 5 ms (comm) + `{python} CompressionPayback.cp_overhead_ms_str` ms (overhead) = **7 ms**. +3. **Effective Speedup**: `{python} CompressionPayback.cp_comm_time_ms_str` / 7 $\approx$ **`{python} CompressionPayback.cp_speedup_str`$\times$**. + +**The Systems Insight**: Compression is a trade of **Compute for Bandwidth**. It pays off only when $T_{\text{overhead}} < T_{\text{comm}} \times (1 - 1/\text{Ratio})$. In this case, spending 2 ms to save 35 ms of network time is an exceptional trade. However, on a high-speed NVLink network where $T_{\text{comm}}$ is only 1 ms, this same compression logic would actually *slow down* your training. Always profile the network before adding compression. + +::: + +**Error Feedback** ensures that while the journey may be compressed or delayed, no vital information is permanently lost, preserving the mathematical truth of the model's convergence. Finally, **communication-computation overlap** through layer-by-layer pipelining and bucket fusion hides the remaining communication time behind useful computation, reducing the effective communication overhead to 5--15% of total step time in well-configured systems. ::: {.callout-takeaways title="Every Byte Has a Travel Cost"} diff --git a/book/quarto/contents/vol2/compute_infrastructure/compute_infrastructure.qmd b/book/quarto/contents/vol2/compute_infrastructure/compute_infrastructure.qmd index b00fed05d7..8a724e38a9 100644 --- a/book/quarto/contents/vol2/compute_infrastructure/compute_infrastructure.qmd +++ b/book/quarto/contents/vol2/compute_infrastructure/compute_infrastructure.qmd @@ -573,11 +573,11 @@ The bandwidth gap between registers and HBM is approximately 1,000$\times$. If a ::: {.callout-definition title="High Bandwidth Memory (HBM)"} -***High Bandwidth Memory (HBM)***\index{High Bandwidth Memory!definition} is a specialized DRAM architecture that uses 3D-stacked memory dies connected directly to the processor via an interposer. +***High Bandwidth Memory (HBM)***\index{High Bandwidth Memory!definition} is a 3D-stacked DRAM architecture in which multiple memory dies are vertically bonded and connected to the processor through thousands of Through-Silicon Vias (TSVs) on a shared silicon interposer, eliminating the centimeter-scale PCB traces of conventional DRAM and replacing them with micrometer-scale vertical paths. -1. **Significance (Quantitative):** By minimizing the physical distance data travels, HBM provides the **Memory Bandwidth ($BW$)** required to feed massive arithmetic units ($R_{\text{peak}}$), achieving 10$\times$ to 100$\times$ higher throughput than conventional DDR memory. -2. **Distinction (Durable):** Unlike **DDR Memory**, which connects through long PCB traces, HBM is integrated into the same package as the accelerator, trading **Capacity** for **Bandwidth and Energy Efficiency** ($\eta$). -3. **Common Pitfall:** A frequent misconception is that HBM "solves" the memory wall. In reality, it only **Moves the Wall**: while bandwidth is higher, the **Arithmetic Intensity** required to saturate modern GPUs continues to outpace even HBM scaling. +1. **Significance (Quantitative):** HBM3 in the H100 delivers approximately 3.35 TB/s of memory bandwidth — roughly 16$\times$ DDR5's ~200 GB/s — at 2 pJ/bit versus DDR5's ~20 pJ/bit. This bandwidth sets the $\text{BW}$ ceiling in the Iron Law: the H100's ridge point is approximately $312\,\text{TFLOP/s} / 3.35\,\text{TB/s} \approx 93$ FLOP/byte, so any operator with arithmetic intensity below 93 FLOP/byte (e.g., attention at 5–20 FLOP/byte) remains memory-bound even with HBM. +2. **Distinction (Durable):** Unlike DDR memory, which connects through centimeters of PCB trace (introducing capacitance, attenuation, and high driving current), HBM uses 1,024-bit-wide TSV buses with path lengths measured in micrometers — a 1,000$\times$ reduction in signal distance that enables the wider bus without proportionally higher power. +3. **Common Pitfall:** A frequent misconception is that HBM solves the memory wall. HBM moves the wall rather than eliminates it: Tensor Core throughput ($R_{\text{peak}}$) has scaled faster than HBM bandwidth across generations (from 900 GB/s in V100 to 3.35 TB/s in H100 while FP16 TFLOPS grew from 125 to 312), so the ridge point continues rising and more operations remain memory-bound despite HBM improvements. ::: @@ -601,6 +601,17 @@ The entire HBM assembly sits on the same silicon interposer as the processor die The interposer itself is a passive silicon substrate with etched wiring layers that connect the HBM stacks to the processor, forming what is effectively a miniature PCB made of silicon rather than fiberglass. The interposer's silicon construction allows much finer trace widths and tighter pitches than a fiberglass PCB, enabling thousands of parallel connections between the HBM stacks and the processor die in a physical area of just a few hundred square millimeters. +::: {.callout-checkpoint title="HBM and the Memory Wall"} + +Verify your understanding of 3D-stacked memory: + +- [ ] Why does HBM achieve higher bandwidth than DDR5 despite having a lower clock frequency? +- [ ] What is the "generality tax" of DDR memory, and how does vertical stacking (TSVs) eliminate it? +- [ ] If an H100 has 80 GB of HBM and a 70B parameter model requires 140 GB for weights (FP16), where must the remaining 60 GB reside? +- [ ] True or False: HBM's primary benefit is increasing the total memory capacity available to the GPU. + +::: + ```{python} #| label: memory-bandwidth-scenario #| echo: false @@ -630,13 +641,13 @@ class MemoryBandwidthScenario: h100_bw = f"{hbm_bw.m_as(TB/second):.2f}" ``` -| **Metric** | **Host DRAM** (DDR5) | **Accelerator HBM** (HBM3e) | **Scaling Factor** | -|:--------------------|:---------------------------:|:----------------------------:|:-------------------------:| -| **Mechanism** | 2D PCB Traces | **3D Die Stacking** | - | -| **Placement** | Socketed DIMMs | **On-package (Substrate)** | **Physical Proximity** | +| **Metric** | **Host DRAM** (DDR5) | **Accelerator HBM** (HBM3e) | **Scaling Factor** | +|:--------------------|:---------------------------------------------------:|:----------------------------------------------------:|:-------------------------:| +| **Mechanism** | 2D PCB Traces | **3D Die Stacking** | - | +| **Placement** | Socketed DIMMs | **On-package (Substrate)** | **Physical Proximity** | | **Bandwidth** | ~`{python} MemoryBandwidthScenario.ddr_bw_str` GB/s | **~`{python} MemoryBandwidthScenario.h100_bw` TB/s** | **~50$\times$ Faster** | -| **Interface Width** | 64-bit | **1024-bit per stack** | **16$\times$ Wider** | -| **Energy** | ~20 pJ/bit | **~2 pJ/bit** | **10$\times$ Efficiency** | +| **Interface Width** | 64-bit | **1024-bit per stack** | **16$\times$ Wider** | +| **Energy** | ~20 pJ/bit | **~2 pJ/bit** | **10$\times$ Efficiency** | : **HBM vs. Standard DRAM Comparison**. HBM achieves its bandwidth advantage through three simultaneous innovations: 3D die stacking (more bits per package), TSV interconnects (shorter signal paths), and on-package placement (proximity to the processor). {#tbl-hbm-comparison} @@ -1026,11 +1037,11 @@ The hardware dictates the software abstraction, which in turn shapes what archit ::: {.callout-definition title="Tensor Core"} -***Tensor Core***\index{Tensor Core!definition} is a specialized hardware unit that performs a fused **Matrix-Multiply-Accumulate (MMA)** operation on small tile matrices in a single clock cycle ($D = A \times B + C$). +***Tensor Core***\index{Tensor Core!definition} is a specialized mixed-precision hardware unit that performs a fused matrix-multiply-accumulate (MMA) operation $D = A \times B + C$ on small tiles (e.g., 16×8×16 in BF16) within a single clock cycle, delivering dramatically higher throughput than general-purpose CUDA cores by trading programmability for fixed-function matrix arithmetic. -1. **Significance (Quantitative):** They provide the majority of the **Peak Performance ($R_{\text{peak}}$)** in modern GPUs. By operating on matrices rather than individual vectors, they achieve 10$\times$ to 16$\times$ higher throughput than traditional SIMD units for neural network workloads. -2. **Distinction (Durable):** Unlike **General-Purpose SIMD/SIMT** units, which operate on individual elements (scalars) or vectors, Tensor Cores operate on **Tiled Matrices**, trading flexibility for massive arithmetic throughput. -3. **Common Pitfall:** A frequent misconception is that all GPU operations use Tensor Cores. In reality, they are **Operator-Specific**: only operations that can be decomposed into matrix multiplications (e.g., dense layers, convolutions) benefit from Tensor Core acceleration; others fall back to slower vector units. +1. **Significance (Quantitative):** Tensor Cores provide the bulk of the H100's 312 TFLOPS BF16 peak — roughly 8$\times$ the ~40 TFLOPS delivered by CUDA (vector) cores alone. However, this peak is only achievable for operations that decompose into tiled matrix multiplications. Layer normalization and softmax, which involve reductions and element-wise operations rather than matrix multiplications, run at approximately 3–8% of peak throughput on the same hardware. +2. **Distinction (Durable):** Unlike general-purpose CUDA cores (which execute one floating-point operation per clock per lane in a SIMT pipeline), Tensor Cores execute 512 multiply-accumulate operations per clock per warp on matrix tiles — trading the arbitrary per-element programmability of CUDA cores for the specific throughput of matrix arithmetic. +3. **Common Pitfall:** A frequent misconception is that all GPU operations benefit from Tensor Cores. Only operations that map to the $D = A \times B + C$ tile computation — dense linear layers, attention score computation, convolutions — engage Tensor Cores; element-wise activations, layer norm, and embedding lookups fall back to CUDA cores and can run 10–30$\times$ slower per FLOP than Tensor Core operations. ::: @@ -1080,11 +1091,11 @@ For capacity planning, the sustained throughput rate, not the peak rate, should ::: {.callout-definition title="Model FLOPS Utilization (MFU)"} -***Model FLOPS Utilization (MFU)***\index{Model FLOPS Utilization!definition} is the ratio of useful model FLOPs to the hardware's theoretical peak throughput ($R_{\text{peak}}$). +***Model FLOPS Utilization (MFU)***\index{Model FLOPS Utilization!definition} is the ratio of a model's theoretical FLOP count per training step — calculated from architecture parameters alone — to the product of hardware peak throughput and elapsed wall-clock time, measuring what fraction of the hardware's theoretical capacity is doing useful model computation rather than overhead. -1. **Significance (Quantitative):** It is the most precise measure of **System Efficiency ($\eta$)** because it excludes "waste" FLOPs from recomputation, padding, or speculative execution. -2. **Distinction (Durable):** Unlike **Hardware Utilization** (which reports how often the GPU is "busy"), MFU reports how much of that busyness contributes to **Model Convergence**. -3. **Common Pitfall:** A frequent misconception is that 100% MFU is possible. In reality, production training typically achieves **30–50% MFU**; the remainder is lost to inescapable memory stalls ($BW$), communication overhead ($L_{\text{lat}}$), and pipeline bubbles. +1. **Significance (Quantitative):** $\text{MFU} = C_{\text{model}} / (R_{\text{peak}} \cdot T_{\text{step}})$, where $C_{\text{model}} \approx 6P$ FLOPs for a $P$-parameter transformer. For a 7B-parameter model on H100 ($R_{\text{peak}} = 312$ TFLOPS) with a step time of 0.4 seconds: $\text{MFU} = 6 \times 7\times10^9 / (312\times10^{12} \times 0.4) \approx 34\%$. A practical ceiling of 55–65% applies even in highly optimized implementations; the gap represents inescapable memory stalls ($\text{BW}$ bottleneck), AllReduce overhead ($L_{\text{lat}}$), and pipeline bubbles. +2. **Distinction (Durable):** Unlike hardware utilization (the fraction of clock cycles during which the GPU reports being "busy"), MFU counts only cycles spent on useful forward- or backward-pass arithmetic — not recomputed activations, padding computation, or gradient buffer copies that consume cycles but do not advance model training. +3. **Common Pitfall:** A frequent misconception is that high hardware utilization implies high MFU. Engineers optimizing for GPU utilization metrics can find the GPU reports 95% busy while MFU is below 20%, because cycles spent on gradient recomputation, speculative computation, or excessive kernel launches are visible to utilization counters but not to MFU. ::: @@ -1147,11 +1158,11 @@ $$ P_{limit} = \frac{\Delta Q}{\Delta t} $$ ::: {.callout-definition title="Thermal Design Power (TDP)"} -***Thermal Design Power (TDP)***\index{Thermal Design Power!definition} is the maximum sustained thermal load (in watts) that a cooling solution is designed to remove. +***Thermal Design Power (TDP)***\index{Thermal Design Power!definition} is the maximum sustained thermal load in watts that a chip's cooling system must continuously remove for the processor to operate at its rated clock frequency — defining both the cooling infrastructure requirement and the performance ceiling for the accelerator. -1. **Significance (Quantitative):** It defines the **Physical Ceiling** for performance. Within the **Iron Law**, TDP limits the **Peak Performance ($R_{\text{peak}}$)** because every floating-point operation dissipates energy as heat; exceeding the TDP triggers **Thermal Throttling**, which reduces the duty cycle ($\eta$). -2. **Distinction (Durable):** Unlike **Peak Power Consumption** (which captures transient spikes), TDP describes the **Steady-State Thermal Contract** between the silicon and the environment. -3. **Common Pitfall:** A frequent misconception is that TDP is a "power limit" for the user. In reality, it is a **Cooling Requirement**: a processor can only reach its rated speed if the cooling system can remove the TDP-equivalent amount of heat continuously. +1. **Significance (Quantitative):** The H100 SXM5 operates at 700 W TDP. When a liquid cooling system can only sustain 500 W of heat removal (inadequate cooling), the GPU firmware reduces clock frequency via thermal throttling — dropping $R_{\text{peak}}$ from 312 to roughly 220 TFLOPS, a 28% reduction in training throughput that propagates directly into longer wall-clock training time and higher cost per model. +2. **Distinction (Durable):** Unlike peak power consumption (the instantaneous maximum during a single-instruction burst), TDP specifies the steady-state thermal budget — the long-term average that determines whether training can run continuously at rated performance or must throttle. +3. **Common Pitfall:** A frequent misconception is that TDP is a power limit for the software. TDP is a cooling requirement: the processor exceeds its rated speed only if the cooling system removes heat at least as fast as the chip generates it; insufficient cooling causes hardware throttling silently — no error is raised, training simply slows down and MFU drops without obvious cause. ::: @@ -1377,7 +1388,7 @@ The macro impact appears when serving our `{python} InfraSetup.frontier_params_b $$\text{Data Movement Energy} = 350 \text{ GB} \times 8 \text{ bits/byte} \times 4 \text{ pJ/bit} \approx \mathbf{11.2 \text{ Joules}}$$ -The computational cost for the associated $\approx$350 billion MACs is trivial by comparison: +The computational cost for the associated $\approx$ 350 billion MACs is trivial by comparison: $$\text{Computation Energy} = 350 \times 10^9 \text{ MACs} \times 1 \text{ pJ/MAC} \approx \mathbf{0.35 \text{ Joules}}$$ @@ -1469,11 +1480,11 @@ Our `{python} InfraSetup.frontier_params_b`B model needs 350 GB for its FP16 wei ::: {.callout-definition title="Node"} -***Node***\index{Node!definition} is a single physical server chassis containing multiple accelerators (typically 8), a high-speed intra-node interconnect, and host memory. +***Node***\index{Node!definition} is a physical server chassis that aggregates multiple accelerators — typically 8 — through a high-speed intra-node interconnect (NVLink or ICI), creating the fundamental boundary between high-bandwidth local communication and the order-of-magnitude slower inter-node network fabric. -1. **Significance (Quantitative):** It is the fundamental **Failure Domain** and **Scheduling Unit** for the fleet. Within the **Iron Law**, the node defines the maximum **Intra-Node Bandwidth** ($BW_{intra}$) available for model parallelism before traffic must cross the slower network fabric. -2. **Distinction (Durable):** Unlike a **Single Accelerator**, a Node provides the aggregate **Memory Capacity** required to hold the parameters and optimizer state of frontier models. -3. **Common Pitfall:** A frequent misconception is that nodes are independent. In reality, for large training jobs, nodes are **Synchronously Coupled**: a failure or performance drop in one node can stall the entire multi-node cluster. +1. **Significance (Quantitative):** The NVLink bandwidth within a DGX H100 node reaches 900 GB/s bidirectional, while inter-node InfiniBand NDR provides 50 GB/s — a 18$\times$ gap that constrains where each parallelism strategy can run. Tensor parallelism, which AllReduces every layer's output, must stay within the node; pipeline parallelism, which transfers only stage-boundary activations, can span nodes over InfiniBand. The node is also the primary failure domain: a single node failure in a BSP training job idles the entire cluster until recovery. +2. **Distinction (Durable):** Unlike a single accelerator (which provides fast HBM bandwidth but limited capacity), a node aggregates 8$\times$ the HBM capacity and 8$\times$ the compute of a single chip — enabling frontier models to fit in aggregate node memory (8 × 80 GB = 640 GB on DGX H100) that no individual accelerator can hold. +3. **Common Pitfall:** A frequent misconception is that nodes in a distributed training job operate independently. In BSP training, all nodes are synchronously coupled at every step barrier: a performance degradation on one node — from thermal throttling, a faulty NIC, or a slow storage read — stalls the entire cluster and drops overall MFU for the duration of that step. ::: @@ -1529,15 +1540,66 @@ class BandwidthHierarchyScenario: ib_bw_gbs = f"{(ib / 8).m_as(GB/second):.0f}" ``` -| **Domain** | **Interconnect** | **Bandwidth** | **Latency** | **Scaling Limit** | -|:--------------------|:-------------------|:----------------------------------:|:-----------:|:-------------------| -| **Intra-Package** | Silicon Interposer | **~3.3 TB/s** | <100 ns | Single Chip | +| **Domain** | **Interconnect** | **Bandwidth** | **Latency** | **Scaling Limit** | +|:--------------------|:-------------------|:-------------------------------------------------------------:|:-----------:|:-------------------| +| **Intra-Package** | Silicon Interposer | **~3.3 TB/s** | <100 ns | Single Chip | | **Intra-Node** | NVLink / ICI | **~`{python} BandwidthHierarchyScenario.nvlink_bw_str` GB/s** | ~1 μs | Node (8--16 Chips) | | **Intra-Node (IO)** | PCIe Gen5 x16 | **~`{python} BandwidthHierarchyScenario.pcie_bw_str` GB/s** | ~2 μs | CPU-GPU, NIC-GPU | | **Inter-Node** | InfiniBand NDR | **~`{python} BandwidthHierarchyScenario.ib_bw_gbs` GB/s** | ~5--10 μs | Pod (Thousands) | : **The Bandwidth Hierarchy**. Each physical boundary introduces an order-of-magnitude bandwidth cliff. These cliffs are not engineering failures to be optimized away; they reflect fundamental differences in the physics of each interconnect medium. The cliffs dictate model partitioning: Tensor Parallelism, which requires AllReduce after every layer, is strictly confined to the intra-node domain. {#tbl-bandwidth-hierarchy-compute} +```{python} +#| label: bandwidth-staircase-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ HIGH-BANDWIDTH STAIRCASE (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Physics of the Staircase" .callout-notebook +# │ +# │ Goal: Quantify the bandwidth gaps between HBM, NVLink, and InfiniBand. +# │ Show: nvlink_to_ib_ratio ≈ 18x — inline in notebook result. +# │ How: Ratio = BW_fast / BW_slow. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: bs_hbm_bw_str, bs_nvlink_bw_str, bs_ib_bw_str, bs_gap_hbm_nv_str, bs_gap_nv_ib_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class BandwidthStaircase: + # ┌── 1. LOAD ────────────────────────────────────────── + hbm_bw_gbs = 3350 # H100 HBM3 + nvlink_bw_gbs = 900 # H100 NVLink + ib_bw_gbs = 50 # 400G NDR InfiniBand + # ┌── 2. EXECUTE ─────────────────────────────────────── + gap_hbm_nv = hbm_bw_gbs / nvlink_bw_gbs + gap_nv_ib = nvlink_bw_gbs / ib_bw_gbs + # ┌── 3. GUARD ───────────────────────────────────────── + check(gap_nv_ib == 18.0, f"Gap {gap_nv_ib} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + bs_hbm_bw_str = f"{hbm_bw_gbs:,}" + bs_nvlink_bw_str = f"{nvlink_bw_gbs}" + bs_ib_bw_str = f"{ib_bw_gbs}" + bs_gap_hbm_nv_str = f"{gap_hbm_nv:.1f}" + bs_gap_nv_ib_str = f"{gap_nv_ib:.0f}" + +``` + +::: {.callout-notebook title="The Physics of the Staircase"} + +**Problem**: You are synchronizing a 10 GB buffer. How much does the physical "Layer" of the fleet affect your transfer time? + +**The Math**: +Transfer time is $T = \text{Data} / \text{Bandwidth}$. + +1. **HBM (Intra-Chip)**: 10 GB / `{python} BandwidthStaircase.bs_hbm_bw_str` GB/s $\approx$ **3 ms**. +2. **NVLink (Intra-Node)**: 10 GB / `{python} BandwidthStaircase.bs_nvlink_bw_str` GB/s $\approx$ **11 ms**. +3. **InfiniBand (Inter-Node)**: 10 GB / `{python} BandwidthStaircase.bs_ib_bw_str` GB/s $\approx$ **200 ms**. + +**The Systems Insight**: Moving data across the cluster is **`{python} BandwidthStaircase.bs_gap_nv_ib_str`$\times$ slower** than moving it within a node. This "Bandwidth Staircase" is the primary driver of all parallelization strategies: we use **Tensor Parallelism** where bandwidth is abundant (NVLink) and **Data Parallelism** where it is scarce (InfiniBand). A model that "fits" in memory but ignores the staircase will spend 90% of its time waiting for the network. + +::: + ::: {.callout-definition title="Bandwidth Hierarchy"} ***Bandwidth Hierarchy***\index{Bandwidth Hierarchy!definition} is the physical ordering of data transfer rates across system boundaries, from on-chip SRAM (fastest) to the wide-area network (slowest). @@ -1709,15 +1771,15 @@ Understanding how a large model's memory footprint maps onto a node's physical r For our `{python} InfraSetup.frontier_params_b`B model, the training memory breaks down as follows: | **Component** | **Precision** | **Memory per Parameter** | **Total for `{python} InfraSetup.frontier_params_b`B Model** | -|:-------------------------|:-------------:|:------------------------:|:-------------------------------------------------:| -| **Model Weights** | FP16 | 2 bytes | 350 GB | -| **Gradients** | FP16 | 2 bytes | 350 GB | -| **Optimizer (Adam $m$)** | FP32 | 4 bytes | 700 GB | -| **Optimizer (Adam $v$)** | FP32 | 4 bytes | 700 GB | -| **Activations** | Mixed | Variable | 100--400 GB | -| **Total** | | | **2.2--2.5 TB** | +|:-------------------------|:-------------:|:------------------------:|:------------------------------------------------------------:| +| **Model Weights** | FP16 | 2 bytes | 350 GB | +| **Gradients** | FP16 | 2 bytes | 350 GB | +| **Optimizer (Adam $m$)** | FP32 | 4 bytes | 700 GB | +| **Optimizer (Adam $v$)** | FP32 | 4 bytes | 700 GB | +| **Activations** | Mixed | Variable | 100--400 GB | +| **Total** | | | **2.2--2.5 TB** | -: **Training Memory Breakdown for a `{python} InfraSetup.frontier_params_b`B-Parameter Model**. The optimizer state alone (first and second moments in FP32) consumes 4$\times$ the memory of the model weights, making optimizer memory the dominant component of training memory. This is why memory optimization techniques focus heavily on optimizer state sharding and offloading. {#tbl-memory-breakdown} +: **Training Memory Breakdown for a 1,760B-Parameter Frontier Model**. The optimizer state alone (first and second moments in FP32) consumes 4$\times$ the memory of the model weights, making optimizer memory the dominant component of training memory. This is why memory optimization techniques focus heavily on optimizer state sharding and offloading. {#tbl-memory-breakdown} The model weights in FP16 occupy 350 GB ($`{python} InfraSetup.frontier_params_b` \times 10^9 \times 2$ bytes). The Adam optimizer maintains two FP32 state tensors per parameter (first moment $m$ and second moment $v$), adding $`{python} InfraSetup.frontier_params_b` \times 10^9 \times 4 \times 2 = 1{,}400$ GB. The gradients in FP16 add another 350 GB. Intermediate activations (needed for the backward pass) depend on the batch size and sequence length but typically require 100--400 GB for practical training configurations. The total memory footprint for training is therefore 2.2--2.5 TB, roughly 6--7$\times$ the raw weight size. @@ -1882,11 +1944,11 @@ For our `{python} InfraSetup.frontier_params_b`B model, training across 1,024 GP ::: {.callout-definition title="Rack"} -***Rack***\index{Rack!definition} is the physical and electrical unit of aggregation that houses multiple compute nodes, network switches, and power distribution units. +***Rack***\index{Rack!definition} is the physical infrastructure unit — a standardized 42U enclosure — that houses multiple compute nodes, a Top-of-Rack (ToR) switch (the first network aggregation point connecting all nodes in the rack to the broader cluster fabric), power distribution units, and cooling distribution manifolds, defining the granularity at which power and cooling capacity must be provisioned. -1. **Significance (Quantitative):** It defines the **Power and Cooling Boundary** for the fleet. Within the **Iron Law**, the rack limits the maximum **Peak Performance ($R_{\text{peak}}$)** of the nodes it houses based on the available kilowatts per rack (typically 40–100 kW for AI workloads). -2. **Distinction (Durable):** Unlike a **Node** (the compute unit), a Rack is the **Infrastructure Unit**: it is the level at which liquid cooling manifolds and Top-of-Rack (ToR) network switches are integrated. -3. **Common Pitfall:** A frequent misconception is that rack placement does not matter. In reality, **Rack Locality** is critical for performance: nodes within the same rack often have significantly higher bandwidth and lower latency ($L_{\text{lat}}$) than nodes in different rows of the datacenter. +1. **Significance (Quantitative):** AI rack power density has grown dramatically: a rack of 4 DGX H100 nodes contains 32 GPUs at 700 W each, totaling 22.4 kW of GPU power alone — plus 11+ kW for CPUs, NVSwitch, NICs, power conversion losses, and cooling overhead, reaching 33–40 kW per rack. This far exceeds the 5–10 kW typical of general-purpose datacenter racks, requiring dedicated liquid cooling infrastructure and electrical circuits that must be co-designed with the building. +2. **Distinction (Durable):** Unlike a node (the compute unit), a rack is the infrastructure unit: it is the physical level at which cooling manifolds, power distribution, and the ToR switch are integrated — making the rack the minimum replaceable and serviceable unit for facility operations, not the node. +3. **Common Pitfall:** A frequent misconception is that rack placement is a logistical afterthought. Nodes within the same rack share a ToR switch and have single-hop connectivity; nodes in different racks cross at least two switch hops. Placing all pipeline-parallel stages of a training job within the same rack minimizes inter-stage latency and reduces inter-rack fabric load. ::: @@ -1911,6 +1973,17 @@ Stage 4: Server Power Supply. Each server (or node) contains power supply units Stage 5: Voltage Regulators. On the baseboard, voltage regulator modules (VRMs) perform the final conversion from 12 V DC to the 0.7--1.0 V required by the GPU core and the 1.1--1.2 V required by the HBM. These regulators must respond to load changes within microseconds (when the GPU transitions between idle and full-load computation), and they operate at 90--95% efficiency. +::: {.callout-checkpoint title="Power Delivery Physics"} + +Verify your understanding of the datacenter power path: + +- [ ] Why does synchronous ML training create more stress on the power grid than asynchronous web traffic? +- [ ] What is the "Power Ramp" problem, and how do supercapacitor banks address it? +- [ ] If a node has 8 GPUs at 700 W each, why is the total node power budgeted at ~7 kW rather than 5.6 kW? +- [ ] True or False: Converting high-voltage AC to low-voltage DC at the rack level is more efficient than doing it at the server level. + +::: + At 700 W per GPU, the VRM dissipates 35--70 W of heat, which must be cooled along with the GPU itself. This VRM heat is sometimes overlooked in thermal design: in a liquid-cooled system where cold plates cover the GPUs, the VRMs are typically still air-cooled by small fans, creating a thermal management challenge for the remaining components that do not have direct liquid cooling contact. The cumulative efficiency across all five stages is typically 85--90%, meaning that for every 100 W of useful computation, 10--15 W is lost as heat in the power delivery chain itself. This overhead is part of the PUE calculation and represents an irreducible cost of converting utility power to useful computation. @@ -2251,17 +2324,11 @@ We can derive the training time for our `{python} InfraSetup.frontier_params_b`B ::: {.callout-definition title="Warehouse-Scale Computer (WSC)"} -***Warehouse-Scale Computer (WSC)***\index{Warehouse-Scale Computer!definition} is a distributed architecture where thousands of compute nodes are operated as a single coherent machine. +***Warehouse-Scale Computer (WSC)***\index{Warehouse-Scale Computer!definition} is a building-scale computing system in which thousands of servers are operated as a single coherent machine — with the network fabric serving as the system bus, distributed storage as the disk subsystem, and a cluster orchestrator as the operating system — enabling training workloads that would be physically impossible on any single machine. -1. **Significance (Quantitative):** In a WSC, the network fabric replaces the **System Bus**, distributed storage replaces the **Local Disk**, and a fleet orchestrator replaces the **Operating System**. It is the only architecture capable of providing the exa-scale **Peak Performance ($R_{\text{peak}}$)** required for frontier model training. -2. **Distinction (Durable):** Unlike a **Horizontal Cluster** (which houses many independent applications), a WSC is designed for **Synchronous Tight Coupling**, where the components are connected by wires rather than silicon traces but must act with the same coordination as a single chip. -3. **Common Pitfall:** A frequent misconception is that WSC design is just "IT management." In reality, it is **Building-Level Computer Architecture**: the physical layout, power delivery, and cooling must be optimized as a single system to prevent the **Bisection Bandwidth ($BW$)** from becoming the binding constraint. - -::: {#nte-bisection-bandwidth .callout-principle icon=false title="The Bisection Bandwidth Theorem"} -**The Invariant**: The performance of a distributed application is limited by the minimum bisection bandwidth of the network topology. - -**The Implication**: A cluster with 10,000 GPUs is useless for training if they are connected by a 1 Gbps Ethernet tree. **Non-blocking topologies** (Fat-Tree, Dragonfly) are required to ensure that the network does not become the bottleneck for collective operations like AllReduce. -::: +1. **Significance (Quantitative):** A WSC of 10,000 H100 GPUs delivers approximately 3,120 ExaFLOP/s BF16 peak — enabling frontier model training at scales impossible on a single node. However, the system is only as fast as its bisection bandwidth: non-blocking InfiniBand at 50 GB/s per GPU provides 250 TB/s aggregate injection bandwidth; if the fabric is 4:1 oversubscribed, effective bisection bandwidth drops to 62.5 TB/s, potentially bottlenecking AllReduce and dropping training throughput by 30–60%. +2. **Distinction (Durable):** Unlike a general-purpose compute cluster (which runs many independent jobs on separate nodes with no requirement for tight coupling), a WSC training cluster operates as a single synchronous instrument — every node must participate in every AllReduce, making the failure or slowdown of a single node a cluster-wide performance event. +3. **Common Pitfall:** A frequent misconception is that WSC design is primarily software engineering or IT operations. WSC design is building-level computer architecture: the physical layout (rack placement to minimize inter-rack hops), power delivery chain (PUE targets, liquid cooling manifolds), and network topology must be co-designed as a unified system — a mistake at any layer propagates through all layers. ::: @@ -2433,7 +2500,7 @@ Configuration A: Pure Data Parallelism (DP-1024). All 1,024 GPUs replicate the f Configuration B: TP-8, DP-128. Within each node, 8 GPUs use tensor parallelism over NVLink. Across nodes, 128 data-parallel groups synchronize gradients over InfiniBand. Each data-parallel group now needs to AllReduce only the gradients for its 1/8 shard of the model (43.75 GB instead of 350 GB), and the TP communication is contained within the fast NVLink domain. The inter-node AllReduce time drops from 14 seconds to approximately 1.75 seconds. If the compute time is still 2.1 seconds, the efficiency improves to $2.1 / (2.1 + 1.75) \approx 54.5\%$, and with communication-computation overlap, practical efficiency reaches 75--85%. -Configuration C: TP-8, PP-4, DP-32. Within each node, 8 GPUs use tensor parallelism. Across 4 nodes, pipeline parallelism divides the model into 4 stages. Across the remaining 32 data-parallel groups, gradient synchronization occurs over InfiniBand. Pipeline parallelism reduces the gradient AllReduce volume further (each stage has roughly 1/4 of the parameters), and the pipeline communication (forwarding activations between stages) has lower volume than a full AllReduce. The trade-off is the **pipeline bubble**: at the beginning and end of each microbatch, some pipeline stages are idle while waiting for activations from earlier stages or gradients from later stages. The bubble fraction is approximately $1 / \text{num\_microbatches}$, so with 32 microbatches per training step, the bubble wastes roughly 3% of compute. +Configuration C: TP-8, PP-4, DP-32. Within each node, 8 GPUs use tensor parallelism. Across 4 nodes, pipeline parallelism divides the model into 4 stages. Across the remaining 32 data-parallel groups, gradient synchronization occurs over InfiniBand. Pipeline parallelism reduces the gradient AllReduce volume further (each stage has roughly 1/4 of the parameters), and the pipeline communication (forwarding activations between stages) has lower volume than a full AllReduce. The trade-off is the **pipeline bubble**: at the beginning and end of each microbatch, some pipeline stages are idle while waiting for activations from earlier stages or gradients from later stages. The bubble fraction is approximately $1/\text{num\_microbatches}$, so with 32 microbatches per training step, the bubble wastes roughly 3% of compute. This three-way comparison reveals a clear pattern: configurations that keep high-bandwidth communication (tensor parallelism) within the fast NVLink domain and push only low-bandwidth communication (data-parallel AllReduce of smaller gradient shards) onto the slower InfiniBand fabric achieve the highest efficiency. The infrastructure hierarchy *dictates* the parallelism hierarchy. This principle is sometimes called **hierarchy-aware parallelism**, and it is the standard approach for all production-scale training systems. @@ -2768,7 +2835,7 @@ This is one reason why the cloud is often more cost-effective for organizations Organizations with bursty workloads that still prefer owned hardware can partially mitigate the depreciation cost by renting out their idle capacity to other organizations through GPU-as-a-service platforms. Several companies have built businesses around this model, purchasing GPU clusters, renting them to training customers, and achieving economics that work only because high utilization across multiple customers amortizes the depreciation effectively. -For our `{python} InfraSetup.frontier_params_b`B model, the depreciation calculus is stark. A 1,000-GPU H100 cluster purchased in 2024 for $35 million will have a resale value of approximately $7--10 million by 2027, when the next-next-generation accelerators (post-Blackwell) are expected to deliver 3--4$\times$ the performance per watt. If the cluster trained only two frontier models during its lifetime, each model effectively cost $12--14 million in depreciated hardware alone -- before accounting for electricity, staffing, or facility costs. If the same cluster ran continuously at 80% utilization for three years (training, fine-tuning, inference, and experimentation), the depreciated hardware cost per GPU-hour drops to approximately $1.50, well below the cloud rate. The depreciation math reinforces the central lesson of TCO analysis: utilization is the single most important variable in determining whether owned infrastructure is economically viable. +For our `{python} InfraSetup.frontier_params_b`B model, the depreciation calculus is stark. A 1,000-GPU H100 cluster purchased in 2024 for \$35 million will have a resale value of approximately \$7--10 million by 2027, when the next-next-generation accelerators (post-Blackwell) are expected to deliver 3--4$\times$ the performance per watt. If the cluster trained only two frontier models during its lifetime, each model effectively cost \$12--14 million in depreciated hardware alone -- before accounting for electricity, staffing, or facility costs. If the same cluster ran continuously at 80% utilization for three years (training, fine-tuning, inference, and experimentation), the depreciated hardware cost per GPU-hour drops to approximately \$1.50, well below the cloud rate. The depreciation math reinforces the central lesson of TCO analysis: utilization is the single most important variable in determining whether owned infrastructure is economically viable. ### Power Efficiency Trajectory {#sec-compute-power-efficiency} @@ -2776,8 +2843,8 @@ For our `{python} InfraSetup.frontier_params_b`B model, the depreciation calculu The trajectory of power efficiency across accelerator generations provides a quantitative framework for making cluster refresh decisions. Replacing an older cluster with a newer generation can pay for itself through electricity savings alone, particularly at scale. -| **Generation** | **TFLOPS (FP16)** | **TDP (W)** | **TFLOPS/W** | **Relative Efficiency** | -|:----------------|------------------:|------------:|:------------------:|:-----------------------:| +| **Generation** | **TFLOPS (FP16)** | **TDP (W)** | **TFLOPS/W** | **Relative Efficiency** | +|:----------------|------------------:|------------:|:--------------------------------------------:|:-----------------------:| | **V100 (2017)** | 125 | 300 | `{python} GpuEfficiencyScenario.v100_ef_str` | 1.0$\times$ | | **A100 (2020)** | 312 | 400 | `{python} GpuEfficiencyScenario.a100_ef_str` | 1.9$\times$ | | **H100 (2022)** | 1979 | 700 | `{python} GpuEfficiencyScenario.h100_ef_str` | 6.8$\times$ | @@ -2797,7 +2864,7 @@ The interplay between CapEx and OpEx also shapes procurement strategy. Cloud pro Some organizations adopt a hybrid approach: running baseline workloads on owned infrastructure for cost efficiency and bursting to the cloud for peak demand or for early access to the latest hardware generation before committing to a large purchase. This hybrid model is increasingly common among mid-sized AI companies that have a steady-state training workload (justifying owned hardware) but periodically need 2--3$\times$ their base capacity for new model training campaigns. -The power efficiency trajectory has a direct implication for our `{python} InfraSetup.frontier_params_b`B model's training economics. Training on 1,000 V100 GPUs would require approximately 300 kW of IT power and take roughly 8 months (given the V100's lower throughput). Training on 1,000 H100 GPUs requires 700 kW but completes in approximately 2--4 weeks. The H100 cluster consumes 2.3$\times$ more power per unit time but finishes 8--16$\times$ faster, resulting in a net energy reduction of 3.5--7$\times$ for the same training run. When electricity costs $0.07/kWh, the V100 training run costs approximately $120,000 in electricity while the H100 run costs approximately $25,000. The newer hardware is simultaneously faster, cheaper to operate, and more energy-efficient -- a rare alignment that makes hardware refresh decisions straightforward for organizations with the capital to invest. +The power efficiency trajectory has a direct implication for our `{python} InfraSetup.frontier_params_b`B model's training economics. Training on 1,000 V100 GPUs would require approximately 300 kW of IT power and take roughly 8 months (given the V100's lower throughput). Training on 1,000 H100 GPUs requires 700 kW but completes in approximately 2--4 weeks. The H100 cluster consumes 2.3$\times$ more power per unit time but finishes 8--16$\times$ faster, resulting in a net energy reduction of 3.5--7$\times$ for the same training run. When electricity costs \$0.07/kWh, the V100 training run costs approximately \$120,000 in electricity while the H100 run costs approximately \$25,000. The newer hardware is simultaneously faster, cheaper to operate, and more energy-efficient -- a rare alignment that makes hardware refresh decisions straightforward for organizations with the capital to invest. ### GPU Procurement and Supply Chain {#sec-compute-procurement} @@ -2810,7 +2877,7 @@ The GPU supply chain is unusually concentrated. NVIDIA holds approximately 80--9 The practical consequence of this concentration is that procurement timelines for large deployments (1,000+ GPUs) typically span 6--12 months from purchase decision to first production training job, encompassing negotiation, manufacturing, shipping, installation, and burn-in testing. During periods of intense demand (such as 2023--2024), this can stretch to 12--18 months. These lead times make GPU procurement a first-order planning constraint: organizations must commit capital and secure allocations months before the hardware is needed, often before the facility to house it is complete. The 6--12 month procurement horizon, combined with the 18--30 month facility construction timeline discussed earlier, means that infrastructure teams must plan two to three years ahead, making procurement strategy inseparable from the broader capacity planning process. -For our `{python} InfraSetup.frontier_params_b`B model, the procurement challenge is acute. A minimum viable training cluster of 1,000 H100 GPUs represents approximately $35 million in hardware alone. At this scale, the organization is not purchasing off-the-shelf products but negotiating directly with NVIDIA for an allocation from a constrained production pipeline. The negotiation involves not just price but delivery schedule, warranty terms, and often a commitment to purchase future generations. Organizations that delay procurement by even one quarter may find their allocation pushed back by 6--12 months, during which time a competitor with earlier access to the same hardware can complete a training run and capture the market advantage. This first-mover dynamic has led some organizations to commit hundreds of millions of dollars to GPU procurement before their models or training recipes are fully designed, treating hardware access as a strategic asset rather than a commodity input. +For our `{python} InfraSetup.frontier_params_b`B model, the procurement challenge is acute. A minimum viable training cluster of 1,000 H100 GPUs represents approximately \$35 million in hardware alone. At this scale, the organization is not purchasing off-the-shelf products but negotiating directly with NVIDIA for an allocation from a constrained production pipeline. The negotiation involves not just price but delivery schedule, warranty terms, and often a commitment to purchase future generations. Organizations that delay procurement by even one quarter may find their allocation pushed back by 6--12 months, during which time a competitor with earlier access to the same hardware can complete a training run and capture the market advantage. This first-mover dynamic has led some organizations to commit hundreds of millions of dollars to GPU procurement before their models or training recipes are fully designed, treating hardware access as a strategic asset rather than a commodity input. ### Cloud Infrastructure Options {#sec-compute-cloud-options} @@ -2826,7 +2893,7 @@ Third, all major providers offer **custom accelerator** options (purpose-built t The pricing models across providers share three common tiers. **On-demand** instances provide immediate access at the highest per-hour cost. **Reserved instances** (1--3 year commitments) reduce costs by 40--60% but require upfront commitment and risk hardware obsolescence. **Spot/preemptible** instances offer 60--80% discounts but can be interrupted with minimal notice, making them suitable only for fault-tolerant workloads with frequent checkpointing. The choice between tiers is a risk management decision, balancing cost against the probability and impact of interruption. -The economics of spot instances deserve particular attention because they can dramatically reduce training costs for organizations with the engineering sophistication to exploit them. At a 70% discount, spot H100 instances cost approximately $1.20 per GPU-hour instead of $4.00. For our `{python} InfraSetup.frontier_params_b`B model requiring approximately 25,000 GPU-hours of training, the savings are substantial: $30,000 on spot versus $100,000 on demand. However, the stochastic nature of preemption transforms training from a deterministic process into a fault-tolerance engineering problem. If the cloud provider reclaims 5% of the nodes mid-training, a standard training job crashes instantly. Leveraging spot economics requires an elastic training framework (such as TorchElastic) that can dynamically rebalance the computation graph when nodes are added or removed. The economic viability hinges on the **checkpoint tax**: if the system must checkpoint every 10 minutes to limit data loss from preemption, and each checkpoint takes 30 seconds, approximately 5% of the "cheap" compute is consumed by I/O overhead. There exists a break-even point where the frequency of preemption events combined with checkpoint overhead makes spot instances more expensive in wall-clock time than reserved instances, despite the lower hourly rate. +The economics of spot instances deserve particular attention because they can dramatically reduce training costs for organizations with the engineering sophistication to exploit them. At a 70% discount, spot H100 instances cost approximately \$1.20 per GPU-hour instead of \$4.00. For our `{python} InfraSetup.frontier_params_b`B model requiring approximately 25,000 GPU-hours of training, the savings are substantial: \$30,000 on spot versus \$100,000 on demand. However, the stochastic nature of preemption transforms training from a deterministic process into a fault-tolerance engineering problem. If the cloud provider reclaims 5% of the nodes mid-training, a standard training job crashes instantly. Leveraging spot economics requires an elastic training framework (such as TorchElastic) that can dynamically rebalance the computation graph when nodes are added or removed. The economic viability hinges on the **checkpoint tax**: if the system must checkpoint every 10 minutes to limit data loss from preemption, and each checkpoint takes 30 seconds, approximately 5% of the "cheap" compute is consumed by I/O overhead. There exists a break-even point where the frequency of preemption events combined with checkpoint overhead makes spot instances more expensive in wall-clock time than reserved instances, despite the lower hourly rate. A critical consideration for cloud-based ML infrastructure is **networking between instances**. Unlike on-premises clusters where the network topology is custom-designed, cloud instances share the provider's network fabric with other tenants. Most cloud providers now offer placement groups or dedicated networking fabrics that guarantee InfiniBand or equivalent bandwidth between instances within the same group. @@ -2836,7 +2903,7 @@ For our `{python} InfraSetup.frontier_params_b`B model, the cloud path presents ::: {.callout-checkpoint title="TCO Decision Framework"} -Your organization needs to train 10 models per year, each requiring 1,000 GPU-hours on H100s. You are evaluating whether to purchase a 128-GPU on-premises cluster or use cloud instances at $4.00/GPU-hour. Calculate the annual cost of each option (assume on-premises costs of $350,000 per 8-GPU node, $0.07/kWh electricity at PUE 1.1, and 700 W per GPU). At what number of annual training runs does on-premises become cheaper? +Your organization needs to train 10 models per year, each requiring 1,000 GPU-hours on H100s. You are evaluating whether to purchase a 128-GPU on-premises cluster or use cloud instances at \$4.00/GPU-hour. Calculate the annual cost of each option (assume on-premises costs of \$350,000 per 8-GPU node, \$0.07/kWh electricity at PUE 1.1, and 700 W per GPU). At what number of annual training runs does on-premises become cheaper? ::: @@ -2880,7 +2947,7 @@ Your team needs to train a 70B-parameter model on 1 trillion tokens within 4 wee - Compute budget: $6 \times 70 \times 10^9 \times 10^{12} = 4.2 \times 10^{23}$ FLOPs - Available power: 2 MW -Determine: (a) the minimum number of GPUs needed, (b) whether the 2 MW power budget is sufficient (assume PUE 1.1 and 700 W per GPU with 50% overhead for non-GPU components), and (c) the approximate cost of the training run at $4.00/GPU-hour for cloud or $350,000 per 8-GPU node for on-premises. +Determine: (a) the minimum number of GPUs needed, (b) whether the 2 MW power budget is sufficient (assume PUE 1.1 and 700 W per GPU with 50% overhead for non-GPU components), and (c) the approximate cost of the training run at \$4.00/GPU-hour for cloud or \$350,000 per 8-GPU node for on-premises. ::: diff --git a/book/quarto/contents/vol2/conclusion/conclusion.qmd b/book/quarto/contents/vol2/conclusion/conclusion.qmd index 0056d4e2ef..d1f24a3937 100644 --- a/book/quarto/contents/vol2/conclusion/conclusion.qmd +++ b/book/quarto/contents/vol2/conclusion/conclusion.qmd @@ -181,10 +181,107 @@ $$ \text{Capability} \propto \text{Model}_{IQ} \times (\text{Tools} + \text{Cont The next breakthrough will not be a larger GPU; it will be a smarter **System Architecture**. ::: +```{python} +#| label: fleet-evolution-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ FLEET EVOLUTION CALCULATOR (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Projecting the 100x Fleet" .callout-notebook +# │ +# │ Goal: Quantify where the next 100x efficiency gain must come from. +# │ Show: orchestration_gain ≈ 10x, hw_gain ≈ 4x, algorithm_gain ≈ 2.5x. +# │ How: Target_Gain = HW * Algo * Orchestration. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: fe_target_gain_str, fe_hw_gain_str, fe_algo_gain_str, fe_orch_gain_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class FleetEvolution: + # ┌── 1. LOAD ────────────────────────────────────────── + target_gain = 100 + hw_gen_gain = 4.0 # B200 vs H100 (approx effective) + algo_pruning_gain = 2.5 # structured sparsity + distillation + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Solve for required orchestration gain: + orch_required = target_gain / (hw_gen_gain * algo_pruning_gain) + # ┌── 3. GUARD ───────────────────────────────────────── + check(orch_required == 10.0, f"Orch gain {orch_required} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + fe_target_gain_str = f"{target_gain}x" + fe_hw_gain_str = f"{hw_gen_gain}x" + fe_algo_gain_str = f"{algo_pruning_gain}x" + fe_orch_gain_str = f"{orch_required:.0f}x" + +``` + +::: {.callout-notebook title="Projecting the 100x Fleet"} + +**Problem**: To reach the next generation of intelligence, we need a **`{python} FleetEvolution.fe_target_gain_str` efficiency improvement** over today's GPT-4 clusters. If hardware improvements provide `{python} FleetEvolution.fe_hw_gain_str` and algorithmic compression provides `{python} FleetEvolution.fe_algo_gain_str`, where must the remaining gain come from? + +**The Math**: +Total system gain is the product of improvements across the Fleet Stack. + +1. **Hardware (Physics)**: `{python} FleetEvolution.fe_hw_gain_str` (Moore's Law tail-end). +2. **Algorithm (Math)**: `{python} FleetEvolution.fe_algo_gain_str` (Sparsity & Distillation). +3. **Orchestration (Systems)**: `{python} FleetEvolution.fe_target_gain_str` / (4.0 $\times$ 2.5) = **`{python} FleetEvolution.fe_orch_gain_str`**. + +**The Systems Insight**: Because silicon and math are hitting diminishing returns, the bulk of future scaling (**`{python} FleetEvolution.fe_orch_gain_str`**) must come from **System Orchestration**. This means moving from monolithic models to **Compound AI Systems** that use reasoning loops, tool-use, and dynamic retrieval to achieve 10$\times$ more utility from the same number of FLOPS. The future of AI is not in the model weights—it is in the **Fleet Logic**. + +::: + This era of the **Compound AI System** is already here, where intelligence emerges not from a single monolithic model but from the orchestration of specialized agents (reasoning, retrieval, and action) coordinated through the MLOps pipelines established in @sec-ops-scale. In this new paradigm, the **Orchestration Layer** becomes the new "CPU," scheduling cognitive tasks across a fleet of specialized models just as an OS schedules threads across cores — precisely the kind of fleet orchestration studied in @sec-fleet-orchestration. As silicon approaches its atomic limits, the next decade will likely see a shift to **Post-Silicon Frontiers**. Technologies like High Bandwidth Flash (HBF), 3D logic-stacking, and novel computing substrates will redefine the physics of our fleet. Yet the principles you have mastered (bandwidth optimization, fault tolerance, and responsible governance) will remain the enduring requirements for these radical new substrates. +```{python} +#| label: post-silicon-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ POST-SILICON EFFICIENCY FRONTIER (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Physics of the Next 1000x" .callout-notebook +# │ +# │ Goal: Quantify the energy gap between current silicon and photonics/optical. +# │ Show: energy_gain ≈ 1000x — inline in notebook result. +# │ How: Ratio = E_electrical / E_optical. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: ps_elec_pj_str, ps_opt_pj_str, ps_efficiency_gain_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class PostSilicon: + # ┌── 1. LOAD ────────────────────────────────────────── + e_elec_pj = 10.0 # Energy per bit for standard electrical interconnect + e_opt_pj = 0.01 # Target for co-packaged optical / photonic interconnect + # ┌── 2. EXECUTE ─────────────────────────────────────── + efficiency_gain = e_elec_pj / e_opt_pj + # ┌── 3. GUARD ───────────────────────────────────────── + check(efficiency_gain == 1000.0, f"Gain {efficiency_gain} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + ps_elec_pj_str = f"{e_elec_pj}" + ps_opt_pj_str = f"{e_opt_pj}" + ps_efficiency_gain_str = f"{efficiency_gain:,.0f}" + +``` + +::: {.callout-notebook title="The Physics of the Next 1000x"} + +**Problem**: Modern GPU clusters are hitting the **Energy Wall**. To scale to 1 million GPUs, we must reduce the energy cost of moving a bit across the fabric from `{python} PostSilicon.ps_elec_pj_str` pJ to `{python} PostSilicon.ps_opt_pj_str` pJ using **Co-Packaged Optics (CPO)**. What is the efficiency dividend of this shift? + +**The Math**: +The next leap in scale requires moving from electrical to optical signaling. + +1. **Electrical Cost**: `{python} PostSilicon.ps_elec_pj_str` pJ/bit. +2. **Optical Target**: `{python} PostSilicon.ps_opt_pj_str` pJ/bit. +3. **Efficiency Gain**: **`{python} PostSilicon.ps_efficiency_gain_str`$\times$**. + +**The Systems Insight**: We cannot scale further by doing "more of the same." To break the Energy Wall, we must change the **Physics of Communication**. A 1,000$\times$ reduction in data movement energy is the only way to support models that are 1,000$\times$ larger without consuming the entire output of the global power grid. In the Machine Learning Fleet, the principles of **Data Locality** and **Interconnect Efficiency** are not just optimizations—they are the thermodynamic requirements for the next era of intelligence. + +::: + As ML systems grow in capability and consequence, the distance between technical decisions and human outcomes shrinks to zero. This is the ultimate "System-Data Duality": we build the system, but the system builds us. ## Engineering Intelligence at Scale {#sec-conclusion-engineering-intelligence-scale-eb8a} @@ -251,19 +348,19 @@ class FermiEstimate: **The Machine (GPT-4 Scale Cluster)**: * **Cluster**: 25,000 H100 GPUs. -* **Ops/sec**: $25,000 \times 1,000 \text{ TFLOPS} = \mathbf{`{python} FermiEstimate.machine_ops_sci` \text{ FLOPs}}$. -* **Power**: $25,000 \times 700\text{W} \approx \mathbf{`{python} FermiEstimate.machine_power_mw_str` \text{ MW}}$. +* **Ops/sec**: $25,000 \times 1,000 \text{ TFLOPS} =$ **`{python} FermiEstimate.machine_ops_sci` FLOPs**. +* **Power**: $25,000 \times 700\text{W} \approx$ **`{python} FermiEstimate.machine_power_mw_str` MW**. **The Brain**: * **Synapses**: $10^{14}$ synapses (connections). * **Firing Rate**: $\approx 100 \text{ Hz}$ (average spike rate). -* **Ops/sec**: $10^{14} \times 100 = \mathbf{`{python} FermiEstimate.brain_ops_sci` \text{ Synaptic Ops/sec}}$. +* **Ops/sec**: $10^{14} \times 100 =$ **`{python} FermiEstimate.brain_ops_sci` Synaptic Ops/sec**. **The Systems Conclusion**: -* **Raw Throughput**: The machine is now **`{python} FermiEstimate.throughput_ratio_str`$\times$ faster** ($`{python} FermiEstimate.machine_ops_sci`$ vs $`{python} FermiEstimate.brain_ops_sci`$) at raw math. -* **Efficiency**: The brain is **`{python} FermiEstimate.efficiency_gap_str`$\times$ more efficient** ($20\text{W}$ vs $`{python} FermiEstimate.machine_power_mw_str`\text{MW}$). +* **Raw Throughput**: The machine is now **`{python} FermiEstimate.throughput_ratio_str`$\times$ faster** (`{python} FermiEstimate.machine_ops_sci` vs `{python} FermiEstimate.brain_ops_sci`) at raw math. +* **Efficiency**: The brain is **`{python} FermiEstimate.efficiency_gap_str`$\times$ more efficient** ($20\text{W}$ vs `{python} FermiEstimate.machine_power_mw_str` MW). We have conquered **Scale**. The challenge for your generation is **Efficiency**. ::: diff --git a/book/quarto/contents/vol2/data_storage/data_storage.qmd b/book/quarto/contents/vol2/data_storage/data_storage.qmd index 6bdc6c1cea..b208abd279 100644 --- a/book/quarto/contents/vol2/data_storage/data_storage.qmd +++ b/book/quarto/contents/vol2/data_storage/data_storage.qmd @@ -165,6 +165,56 @@ $$ \text{Utilization} = \min\!\left(1,\;\frac{BW_{\text{storage}}}{N_{GPU} \time **The Implication**: A storage system that was adequate for 8 GPUs becomes the bottleneck at 64. **The I/O Wall** scales with the number of accelerators: every GPU added to the cluster raises the throughput floor that storage must sustain, making the data pipeline — not the model — the limiting factor. ::: +```{python} +#| label: shard-contention-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ SHARD CONTENTION PROBABILITY (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Thundering Herd: Shard Contention" .callout-notebook +# │ +# │ Goal: Quantify the probability of multiple GPUs hitting the same storage shard. +# │ Show: contention_prob_pct ≈ 63% — inline in notebook result. +# │ How: P(at least one collision) = 1 - (Shards! / (Shards^Workers * (Shards-Workers)!)). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: sc_n_workers_str, sc_n_shards_str, sc_collision_prob_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check +import math + +class ShardContention: + # ┌── 1. LOAD ────────────────────────────────────────── + n_workers = 32 # GPUs in a data-parallel group + n_shards = 1000 # Total shards in the dataset + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Classic birthday problem: P(no collision) + # For small n/N, approx e^(-n^2 / 2N) + p_no_collision = math.exp(-(n_workers**2) / (2 * n_shards)) + p_collision = 1 - p_no_collision + # ┌── 3. GUARD ───────────────────────────────────────── + check(0.3 < p_collision < 0.5, f"Prob {p_collision:.2f} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + sc_n_workers_str = f"{n_workers}" + sc_n_shards_str = f"{n_shards}" + sc_collision_prob_str = f"{p_collision * 100:.1f}" + +``` + +::: {.callout-notebook title="The Thundering Herd: Shard Contention"} + +**Problem**: You have a dataset split into `{python} ShardContention.sc_n_shards_str` shards on a shared file system. If `{python} ShardContention.sc_n_workers_str` GPUs each pick a shard at random to start their next epoch, what is the probability that at least two GPUs "collide" on the same storage server, causing a performance bottleneck? + +**The Math**: +This is a variant of the "Birthday Problem" in probability. + +1. **Probability of No Collision**: $\approx e^{-n^2 / 2N} = e^{-32^2 / 2000} \approx 0.60$. +2. **Probability of Contention**: $1 - 0.60 = \mathbf{40\%}$. + +**The Systems Insight**: Even with a large number of shards, there is a **`{python} ShardContention.sc_collision_prob_str`% chance** that your storage system will experience a "Hot Spot" in any given epoch. In a distributed fleet, these collisions create **Tail Latency**: the entire cluster waits for the two GPUs sharing a disk to finish. To solve this, production data loaders use **Global Shuffling** and **Deterministic Shard Assignment** to ensure that workers are perfectly distributed across the storage fabric, eliminating the "Thundering Herd" effect. + +::: + ::: {.callout-perspective title="The Widening I/O Wall"} The I/O Wall is not static -- it is widening. Between 2018 and 2025, accelerator compute throughput grew by roughly 10$\times$ (V100 at 125 TFLOPS to B200 at 2,250 TFLOPS in FP8). Over the same period, NVMe sequential bandwidth grew by roughly 2$\times$ (3.5 GB/s to 7 GB/s per drive). The compute-to-storage bandwidth ratio has therefore worsened by 5$\times$ in seven years. If this trend continues, the storage hierarchy must add new tiers (persistent memory, CXL-attached storage) or fundamentally change the data pipeline architecture (compute-near-storage, in-storage processing) to prevent the I/O Wall from becoming the dominant constraint on training throughput. The history of computing suggests that the gap will continue to widen, because the economic incentives favor investing in faster compute (which directly reduces training time) over faster storage (which only indirectly improves utilization). ::: @@ -362,14 +412,14 @@ How should a system architect organize storage to serve workloads that simultane To resolve the tensions between bandwidth, latency, capacity, and cost, ML systems use a multi-tier storage hierarchy. Each tier exists because it resolves a specific tension between physics (bandwidth and latency are governed by distance from the accelerator) and economics (cost per bit decreases as capacity increases). The hierarchy extends the classic processor memory hierarchy (registers, L1/L2 cache, DRAM) that students encounter in computer architecture courses, adding tiers below DRAM that are unique to large-scale data systems. @tbl-storage-hierarchy-merged reveals the extreme bandwidth disparities that ML systems must navigate. -| **Storage Tier** | **Typical Capacity** | **Bandwidth** | **Latency** | **Cost ($/GB)** | -|:-------------------------|:---------------------|----------------------------:|-----------------:|----------------:| +| **Storage Tier** | **Typical Capacity** | **Bandwidth** | **Latency** | **Cost ($/GB)** | +|:-------------------------|:---------------------|-----------------------------------------------------:|-----------------:|----------------:| | **GPU HBM** | 80 GB | `{python} StorageHierarchyAnalysis.h100_bw_tbs` TB/s | ~10 ns | ~15.00 | -| **Host DRAM** | 512 GB–2 TB | 200 GB/s | ~100 ns | ~3.00 | -| **Local NVMe SSD** | 4–30 TB | 7–25 GB/s | ~10 μs | ~0.10 | -| **Parallel File System** | 100+ PB | 1+ TB/s aggregate | ~1 ms | ~0.03 | -| **Object Storage** | Unlimited | 100 GB/s aggregate | ~50 ms | ~0.02 | -| **Archive/Cold Storage** | Unlimited | 1 GB/s | Minutes to hours | ~0.004 | +| **Host DRAM** | 512 GB–2 TB | 200 GB/s | ~100 ns | ~3.00 | +| **Local NVMe SSD** | 4–30 TB | 7–25 GB/s | ~10 μs | ~0.10 | +| **Parallel File System** | 100+ PB | 1+ TB/s aggregate | ~1 ms | ~0.03 | +| **Object Storage** | Unlimited | 100 GB/s aggregate | ~50 ms | ~0.02 | +| **Archive/Cold Storage** | Unlimited | 1 GB/s | Minutes to hours | ~0.004 | : **Extended Memory Hierarchy for ML Systems**: The 300,000$\times$ bandwidth gap between HBM and object storage drives the need for sophisticated prefetching and caching across multiple levels. {#tbl-storage-hierarchy-merged} @@ -386,12 +436,12 @@ Storage performance decreases and capacity increases as data moves further from Box/.style={draw=black!60, line width=0.75pt, rounded corners=1pt, align=center}, Line/.style={->, line width=1.0pt}, Layer/.style={ - trapezium, - trapezium angle=70, - draw, - line width=0.75pt, - minimum height=1cm, - inner sep=0pt, + trapezium, + trapezium angle=70, + draw, + line width=0.75pt, + minimum height=1cm, + inner sep=0pt, outer sep=0pt, align=center }, @@ -622,6 +672,59 @@ The central challenge at this tier is the I/O Wall: even the fastest NVMe RAID c Local NVMe is also the primary tier for **local checkpoint staging**. When saving a model checkpoint, the fastest strategy is to write to local NVMe at full bandwidth (minimizing the time the training pipeline is paused), then asynchronously replicate to shared storage for durability. A `{python} StorageHierarchyAnalysis.gpt3_params_b`B parameter checkpoint of `{python} StorageHierarchyAnalysis.ckpt_total_gb_str` GB writes to local NVMe in approximately `{python} NVMeTierCalcs.ckpt_nvme_s_str` seconds, compared to `{python} NVMeTierCalcs.ckpt_pfs_s_str` seconds if written directly to a parallel file system where the node competes with hundreds of other writers. The optimal checkpoint *frequency* depends on cluster failure rates and is derived quantitatively in @sec-fault-tolerance-young-daly using the Young-Daly formula. From a storage perspective, the key design goal is minimizing $T_{save}$ through tiered staging: write to local NVMe at full bandwidth, then background-copy to shared storage. +```{python} +#| label: local-cache-roi-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ LOCAL CACHE ROI (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "ROI of Local NVMe Caching" .callout-notebook +# │ +# │ Goal: Quantify GPU utilization gain from local caching. +# │ Show: util_gain_pct ≈ 15% — inline in notebook result. +# │ How: T_step_remote = T_compute + T_IO_remote; T_step_local = T_compute. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: lc_compute_ms_str, lc_remote_io_ms_str, lc_util_remote_str, lc_util_local_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class LocalCacheRoi: + # ┌── 1. LOAD ────────────────────────────────────────── + t_compute_ms = 800 + t_io_remote_ms = 150 # Remote PFS latency + bandwidth limit + t_io_local_ms = 10 # Local NVMe prefetch (effective overhead) + # ┌── 2. EXECUTE ─────────────────────────────────────── + t_step_remote = t_compute_ms + t_io_remote_ms + t_step_local = t_compute_ms + t_io_local_ms + util_remote = (t_compute_ms / t_step_remote) * 100 + util_local = (t_compute_ms / t_step_local) * 100 + # ┌── 3. GUARD ───────────────────────────────────────── + check(util_remote < 85, f"Remote util {util_remote:.1f}% too high") + check(util_local > 98, f"Local util {util_local:.1f}% too low") + # ┌── 4. OUTPUT ──────────────────────────────────────── + lc_compute_ms_str = f"{t_compute_ms}" + lc_remote_io_ms_str = f"{t_io_remote_ms}" + lc_util_remote_str = f"{util_remote:.1f}" + lc_util_local_str = f"{util_local:.1f}" + +``` + +::: {.callout-notebook title="ROI of Local NVMe Caching"} + +**Problem**: You are training a vision model where each step takes `{python} LocalCacheRoi.lc_compute_ms_str` ms. Fetching data from a shared Parallel File System adds `{python} LocalCacheRoi.lc_remote_io_ms_str` ms of I/O wait because of network congestion. How much does adding local NVMe SSDs to each node improve GPU utilization? + +**The Math**: +GPU utilization ($\eta$) is the fraction of step time spent in computation. + +1. **Remote Only**: `{python} LocalCacheRoi.lc_compute_ms_str` / (`{python} LocalCacheRoi.lc_compute_ms_str` + `{python} LocalCacheRoi.lc_remote_io_ms_str`) $\approx$ **`{python} LocalCacheRoi.lc_util_remote_str`%**. +2. **Local Cache**: Using prefetching into local NVMe reduces the exposed I/O wait to near zero. + - New Util: `{python} LocalCacheRoi.lc_compute_ms_str` / (`{python} LocalCacheRoi.lc_compute_ms_str` + 10) $\approx$ **`{python} LocalCacheRoi.lc_util_local_str`%**. + +**The Systems Insight**: Local storage is a **GPU Utilization Multiplier**. By adding a \$500 NVMe drive to a \$30,000 GPU node, you recover **15% of the GPU's capacity** that was previously wasted on I/O wait. Across a 1,000-GPU cluster, this is equivalent to adding **150 GPUs** for "free." In modern ML infrastructure, local NVMe is not auxiliary; it is the "Physical Buffer" that decouples expensive compute from unpredictable shared storage. + +::: + A practical concern at this tier is **SSD endurance**. NAND flash memory can sustain a limited number of write-erase cycles before the cells degrade. Enterprise NVMe drives are rated for 1 to 3 Drive Writes Per Day (DWPD) over a 5-year lifespan. For a 7.68 TB drive at 1 DWPD, this means the drive can absorb 7.68 TB of writes per day, or roughly 14 PB total over its lifetime. ML training workloads are predominantly read-heavy (the dataset is written once and read many times), which is favorable for SSD endurance. However, checkpoint writes can be intensive: if each node saves a 4 GB checkpoint shard every 10 minutes, that is 576 GB per day of checkpoint writes, well within the 1 DWPD budget. The risk emerges when local NVMe is used as a staging buffer for both checkpoint writes *and* dataset caching: the combined write volume from initial dataset staging plus repeated checkpoint saves must remain within the drive's endurance rating. Local NVMe provides high bandwidth and low latency within a single node, but distributed training requires every node to access the same datasets and see the same checkpoints. This shared-namespace requirement cannot be satisfied by node-local storage alone and motivates the next tier in the hierarchy. @@ -636,9 +739,9 @@ Local NVMe provides high bandwidth and low latency within a single node, but dis ***Parallel File System (PFS)***\index{Parallel File System!definition} is a distributed storage architecture that stripes data across many storage servers to provide aggregate throughput exceeding the capacity of any single device. -1. **Significance (Quantitative):** In ML fleets, a PFS (e.g., Lustre, GPFS) provides the shared namespace for training data and model artifacts. It allows a single client to aggregate the **I/O Bandwidth ($BW_{io}$)** of multiple storage nodes, which is essential for loading petabyte-scale datasets and writing multi-terabyte checkpoints. -2. **Distinction (Durable):** Unlike **Network Attached Storage (NAS)**, where a client talks to one server at a time, a PFS client communicates with many servers simultaneously to parallelize the data stream. -3. **Common Pitfall:** A frequent misconception is that a PFS has "infinite throughput." In reality, it is constrained by **Metadata Latency ($L_{\text{lat}}$)**: while bulk data transfers are fast, operations like opening millions of small files can overwhelm the filesystem's central directory servers. +1. **Significance (Quantitative):** A PFS aggregates $BW_{io}$ linearly with the number of storage servers (Object Storage Servers). A Lustre cluster with 20 OSS nodes each delivering 10 GB/s provides 200 GB/s aggregate — versus a single NAS server capped at 10 GB/s — enabling an 8,000-GPU training job to load each 512-token batch in under 50 ms rather than 1 second. This aggregate bandwidth directly reduces the $D_{vol}/BW$ term in the Iron Law. +2. **Distinction (Durable):** Unlike Network Attached Storage (NAS), where every I/O request routes through a single server, a PFS client receives stripe location metadata from a dedicated Metadata Server (MDS) and then reads data directly from multiple OSS nodes in parallel — the MDS and OSS paths are architecturally separated, so data bandwidth scales with OSS count while metadata operations scale with MDS count. +3. **Common Pitfall:** A frequent misconception is that a PFS has unlimited throughput if enough OSS nodes are added. In reality, a Lustre MDS handles roughly 100,000–300,000 metadata operations per second; at 10,000 workers each opening one small file, the MDS saturates in under 1 second and becomes the serialization point that idles the entire cluster regardless of how many OSS nodes are present. ::: @@ -1441,9 +1544,9 @@ A `{python} StorageHierarchyAnalysis.gpt3_params_b`B parameter model with Adam o ***Checkpoint Storm***\index{Checkpoint Storm!definition} is a burst of synchronized network and storage traffic that occurs when all nodes in a training fleet save model state simultaneously. -1. **Significance (Quantitative):** In a large-scale cluster, a single checkpoint event can generate tens of terabytes of write traffic in seconds, saturating the **Bisection Bandwidth ($BW$)** and stalling the optimization loop. The total training time ($T$) increases by the duration of the storm ($T_{save}$), reducing the **System Efficiency ($\eta$)**. -2. **Distinction (Durable):** Unlike **General I/O Contention** (which is stochastic), a Checkpoint Storm is **Synchronous and Periodic**, driven by the deterministic cycle of the training orchestrator. -3. **Common Pitfall:** A frequent misconception is that storms are unavoidable. In reality, they can be mitigated through **Staggered Checkpointing**, multi-tiered buffering (local NVMe to remote PFS), and **Asynchronous Serialization** that overlaps the write with the next compute step. +1. **Significance (Quantitative):** The storm magnitude scales as $T_{save} = N \times \text{per-node-shard} / BW_{fabric}$. For a 70B-parameter model trained across 1,000 nodes with ZeRO-3 sharding ($\approx$140 GB per node at FP16), a checkpoint generates 140 TB of simultaneous writes; at 100 GB/s fabric bandwidth, $T_{save} \approx 1{,}400$ seconds — over 23 minutes of training stall per checkpoint event. This adds directly to the $L_{lat}$ term and can dwarf the compute time between checkpoints. +2. **Distinction (Durable):** Unlike general I/O contention (which is stochastic and unpredictable), a Checkpoint Storm is synchronous and periodic — every node writes at the same moment because the training orchestrator triggers checkpoint after a fixed step count. Its predictability makes it both more damaging (all nodes compete simultaneously) and more tractable (it can be prevented by design through staggered scheduling or asynchronous serialization). +3. **Common Pitfall:** A frequent misconception is that checkpointing every 100 steps is "low overhead." At 70B scale, this assumption is catastrophically wrong: if each training step takes 10 seconds and a checkpoint storm takes 1,400 seconds, checkpointing every 100 steps means spending $1400/(100 \times 10) = 140\%$ of compute time on checkpointing — more time saving state than doing useful training. ::: @@ -1453,7 +1556,7 @@ A `{python} StorageHierarchyAnalysis.gpt3_params_b`B parameter model with Adam o 1. **Per-node write to local NVMe** (4 drives at 3.5 GB/s each = 14 GB/s): $4 \text{ GB} \div 14 \text{ GB/s} = 0.29$ seconds. 2. **Async copy to PFS**: 256 nodes$\times$ 4 GB = 1,024 GB total. If the PFS provides 1 TB/s aggregate, the storm completes in roughly 1 second. -3. **Per-node PFS bandwidth**: If all 256 nodes write simultaneously, each gets $1{,}000 / 256 = 3.9$ GB/s. Per-node async copy: $4 / 3.9 = 1.03$ seconds. +3. **Per-node PFS bandwidth**: If all 256 nodes write simultaneously, each gets $1000 / 256 = 3.9$ GB/s. Per-node async copy: $4 / 3.9 = 1.03$ seconds. 4. **Training pause**: only 0.29 seconds (the local NVMe write). The PFS copy overlaps with the next training iteration. 5. **Overhead**: 0.29 s pause every 600 s = **0.05%** training time lost to checkpointing. @@ -1516,7 +1619,7 @@ Storing each training sample as an individual file (one JPEG per image, one JSON **Pitfall:** *Ignoring egress costs when designing the storage architecture.* -Cloud egress costs ($0.09/GB for most providers) are invisible during development but dominate at scale. A training job that reads a 100 TB dataset from object storage 10 times incurs \$90,000 in egress fees, far exceeding the annual storage cost of \$27,600. Teams that prototype with small datasets on object storage and then scale up discover that their storage architecture is economically unsustainable. The fix is to budget egress costs explicitly and to design the data pipeline to minimize cross-tier transfers, typically by staging data on local NVMe at job start rather than streaming from object storage each epoch. +Cloud egress costs (\$0.09/GB for most providers) are invisible during development but dominate at scale. A training job that reads a 100 TB dataset from object storage 10 times incurs \$90,000 in egress fees, far exceeding the annual storage cost of \$27,600. Teams that prototype with small datasets on object storage and then scale up discover that their storage architecture is economically unsustainable. The fix is to budget egress costs explicitly and to design the data pipeline to minimize cross-tier transfers, typically by staging data on local NVMe at job start rather than streaming from object storage each epoch. **Pitfall:** *Designing checkpoint storage for average write performance rather than burst performance.* @@ -1542,15 +1645,15 @@ This is only true if the pipeline is I/O-bound. Compression reduces I/O volume b The complete storage picture for our running example -- a 30-day training run of a 175B-parameter model on 256 nodes: -| **Category** | **Volume** | **Primary Tier** | -|:-------------------------------------------|-----------:|:-----------------------------------------------------------| -| Training dataset (compressed) | 3 TB | Object Storage $\to$ NVMe cache | -| Training dataset (decoded, per epoch) | ~15 TB | Host DRAM (transient) | -| Model weights (FP16) | 350 GB | GPU HBM (distributed) | -| Optimizer state (FP32) | 1,400 GB | GPU HBM (ZeRO-partitioned) | -| Single checkpoint (full) | 1,050 GB | NVMe $\to$ PFS $\to$ Object Storage | -| All checkpoints (30 days, 10-min interval) | `{python} StorageHierarchyAnalysis.ckpt_fleet_total_pb` PB | NVMe (transient) $\to$ PFS (recent) $\to$ Object (archive) | -| Archive (retained checkpoints + dataset) | ~50 TB | Glacier | +| **Category** | **Volume** | **Primary Tier** | +|:-------------------------------------------|-----------------------------------------------------------:|:-----------------------------------------------------------| +| Training dataset (compressed) | 3 TB | Object Storage $\to$ NVMe cache | +| Training dataset (decoded, per epoch) | ~15 TB | Host DRAM (transient) | +| Model weights (FP16) | 350 GB | GPU HBM (distributed) | +| Optimizer state (FP32) | 1,400 GB | GPU HBM (ZeRO-partitioned) | +| Single checkpoint (full) | 1,050 GB | NVMe $\to$ PFS $\to$ Object Storage | +| All checkpoints (30 days, 10-min interval) | `{python} StorageHierarchyAnalysis.ckpt_fleet_total_pb` PB | NVMe (transient) $\to$ PFS (recent) $\to$ Object (archive) | +| Archive (retained checkpoints + dataset) | ~50 TB | Glacier | **Total data moved through the hierarchy**: approximately `{python} StorageHierarchyAnalysis.ckpt_fleet_total_pb` PB of checkpoint data plus 3 TB$\times$ epochs of training data reads. For a single-epoch language model training run, checkpoint I/O dominates data loading I/O by a factor of over 1,000. ::: diff --git a/book/quarto/contents/vol2/distributed_training/distributed_training.qmd b/book/quarto/contents/vol2/distributed_training/distributed_training.qmd index 9829b47a29..a4375f8edf 100644 --- a/book/quarto/contents/vol2/distributed_training/distributed_training.qmd +++ b/book/quarto/contents/vol2/distributed_training/distributed_training.qmd @@ -300,11 +300,11 @@ The central challenge of distributed training is ensuring that 1,024 GPUs, opera ::: {.callout-definition title="Distributed Training"} -***Distributed Training***\index{Distributed Training!definition} is the parallelization of the optimization loop across **Multiple Compute Devices** through coordinated partitioning and synchronization. +***Distributed Training***\index{Distributed Training!definition} is a training methodology that partitions the optimization loop across multiple compute nodes — distributing either data, model layers, or individual tensor operations — and coordinates their outputs through synchronized communication primitives to produce a single coherent model. -1. **Significance (Quantitative):** It enables the training of models that exceed the **Memory Capacity** and **Compute Throughput** ($R_{\text{peak}}$) of a single device, reducing the total wall-clock time $T$ for a given $O$. -2. **Distinction (Durable):** Unlike **Traditional Distributed Systems** (which scale for independent requests), Distributed Training scales for **Coordinated State Updates**, requiring high-bandwidth, low-latency synchronization of gradients and weights. -3. **Common Pitfall:** A frequent misconception is that performance scales linearly with the number of devices ($N$). In reality, the **Communication Overhead ($L_{\text{lat}}$)** and **Bisection Bandwidth ($BW$)** eventually create a scaling ceiling (Amdahl's Law). +1. **Significance (Quantitative):** Distributed training becomes necessary when a model's memory requirement exceeds a single accelerator's capacity. GPT-3 (175B parameters) requires approximately 350 GB in BF16 — more than 4$\times$ the 80 GB capacity of a single H100. Training it requires at least 5 H100s for model sharding alone, with production runs using thousands to achieve tractable wall-clock time. +2. **Distinction (Durable):** Unlike distributed systems for independent requests (web serving, database reads) where nodes share no mutable state, distributed training requires every node to maintain a consistent view of model parameters — making gradient synchronization a mandatory coordination step, not an optional optimization. +3. **Common Pitfall:** A frequent misconception is that distributed training scales linearly with node count. In practice, communication overhead grows with cluster size and the serial fraction of each step (Amdahl's Law): with 30% of a step's time spent on synchronization, the theoretical scaling ceiling is $1/0.30 \approx 3\times$ regardless of how many accelerators are added. ::: @@ -561,11 +561,11 @@ The simplest approach gives each GPU a complete, identical copy of the model and ::: {.callout-definition title="Data Parallelism"} -***Data Parallelism***\index{Data Parallelism!definition} is a distributed training strategy where the **Model is Replicated** across all workers, but the **Dataset is Sharded**. +***Data Parallelism***\index{Data Parallelism!definition} is a distributed training strategy in which each worker holds a complete replica of the model and processes an independent shard of the minibatch, then synchronizes gradient updates via AllReduce so all replicas apply identical parameter changes each step. -1. **Significance (Quantitative):** It maximizes **Throughput ($\eta$)** by allowing $N$ workers to process independent data samples simultaneously. It is mathematically equivalent to single-device training with a batch size $N\times$ larger. -2. **Distinction (Durable):** Unlike **Model Parallelism**, where the weights are split across devices, Data Parallelism requires every worker to have enough **Memory Capacity** to store the full model state. -3. **Common Pitfall:** A frequent misconception is that Data Parallelism scales infinitely. In reality, it is constrained by the **Communication Bottleneck**: the time to synchronize gradients (AllReduce) must be hidden by the time to compute them, or else the system becomes bandwidth-bound ($BW$). +1. **Significance (Quantitative):** With $N$ workers each processing batch size $B$, the effective global batch size is $N \times B$, scaling throughput linearly while keeping per-worker memory constant. For a 1B-parameter model at 2 GB in BF16, 1,024 workers achieve 1,024$\times$ the single-GPU throughput — until the gradient AllReduce (2 GB per step at ring-optimal $2(N-1)/N$ per worker) exceeds backward compute time and creates the communication bottleneck. +2. **Distinction (Durable):** Unlike model parallelism, where parameters are partitioned so no single worker holds the full model, data parallelism requires every worker to have sufficient memory capacity to store the complete model state — making it inapplicable for models larger than a single accelerator's memory without combining it with model sharding (ZeRO, FSDP). +3. **Common Pitfall:** A frequent misconception is that data parallelism scales indefinitely with worker count. Scaling the effective batch size beyond the critical batch size (roughly 8,192 for ResNet-50, ~4M tokens for LLM pretraining) degrades statistical efficiency, requiring more training steps to reach target loss and eroding the throughput gains from adding more workers. ::: @@ -597,6 +597,17 @@ $$ This equivalence shows why data parallelism maintains the statistical properties of SGD training. The approach distributes distinct data subsets across devices, computes local gradients independently, and averages these gradients to approximate the full-batch gradient. +::: {.callout-checkpoint title="Data Parallelism Mechanics"} + +Verify your understanding of how data parallelism distributes work: + +- [ ] In data parallelism, is the **model state** (weights) sharded or replicated across GPUs? +- [ ] If you have 8 GPUs and a per-GPU batch size of 32, what is the **effective global batch size**? +- [ ] Why is data parallelism mathematically equivalent to single-device training with a larger batch size? +- [ ] What is the primary hardware resource that constrains data parallelism as you add more nodes: GPU TFLOPS or Network Bandwidth? + +::: + The method parallels gradient accumulation, where a single device accumulates gradients over multiple forward passes before updating parameters. Both techniques use the additive properties of gradients to process large batches efficiently. However, *data parallelism at scale* introduces operational challenges beyond this theoretical equivalence. ::: {.callout-principle title="Data Parallelism at Scale"} @@ -980,7 +991,6 @@ Performance results: - Computation: `{python} Scaling8GPU.compute_8gpu_ms_str`ms per step - Communication: `{python} Scaling8GPU.comm_8gpu_ms_str`ms per step - - Total: `{python} Scaling8GPU.total_8gpu_str`ms per step - Speedup (throughput): `{python} Scaling8GPU.speedup_8gpu_str`$\times$ - Parallel efficiency: `{python} Scaling8GPU.efficiency_8gpu_str`% @@ -1051,6 +1061,43 @@ class Scaling32GPU: comm_32gpu_ms_str = f"{comm_32gpu_ms_val:.0f}" ``` +```{python} +#| echo: false +# ┌── LEGO ─────────────────────────────────────────────── +class GradAccumScenario: + """Scenario: 8-GPU gradient accumulation vs. 32-GPU naive scaling.""" + # ┌── 1. LOAD (Constants) ─────────────────────────────────────────────── + n_gpus_ga = 8 + batch_per_gpu = 16 + accum_steps = 4 + compute_8gpu_ms = Scaling8GPU.compute_8gpu_ms + comm_8gpu_ms = Scaling8GPU.comm_8gpu_ms_val + training_hours_1gpu = 25 + hourly_rate = 128 # USD per hour for 8 GPUs + + # ┌── 2. EXECUTE (The Compute) ───────────────────────────────────────── + effective_batch_val = n_gpus_ga * batch_per_gpu * accum_steps + # AllReduce once per accum_steps steps + overhead_val = comm_8gpu_ms / (accum_steps * compute_8gpu_ms) * 100 + # Effective speedup: 8 GPUs, nearly no comm overhead + effective_speedup = n_gpus_ga * (1 - overhead_val / 100) + ga_training_hours_val = training_hours_1gpu / effective_speedup + ga_cost_val = ga_training_hours_val * hourly_rate + ga_savings_val = Scaling32GPU.training_32gpu_hours_val * hourly_rate - ga_cost_val + ga_savings_pct_val = ga_savings_val / (Scaling32GPU.training_32gpu_hours_val * hourly_rate) * 100 + + # ┌── 3. GUARD (Invariants) ─────────────────────────────────────────── + check(effective_batch_val == 512, f"Effective batch should be 512, got {effective_batch_val}") + + # ┌── 4. OUTPUT (Formatting) ────────────────────────────────────────────── + effective_batch_str = f"{effective_batch_val:,}" + comm_overhead_pct_str = f"{overhead_val:.2f}" + ga_training_hours_str = f"{ga_training_hours_val:.1f}" + ga_cost_str = f"{ga_cost_val:,.0f}" + ga_savings_str = f"{ga_savings_val:,.0f}" + ga_savings_pct_str = f"{ga_savings_pct_val:.0f}" +``` + **32 GPUs: 4 Nodes with Commodity Network** Configuration: @@ -1071,50 +1118,20 @@ Communication dominates and becomes the bottleneck. Better Approach: 8 GPUs with Gradient Accumulation -```{python} -#| label: grad-accum-calc -#| echo: false -# ┌───────────────────────────────────────────────────────────────────────────── -# │ GRADIENT ACCUMULATION ALTERNATIVE (LEGO) -# ├───────────────────────────────────────────────────────────────────────────── -# │ Context: Overcoming the Communication Wall. -# │ -# │ Goal: Show how gradient accumulation recovers efficiency. -# │ Show: Drastic reduction in communication overhead and cost. -# │ How: effective_batch = N * b * steps; overhead = T_comm / (steps * T_comp). -# └───────────────────────────────────────────────────────────────────────────── +::: {.callout-notebook title="Gradient Accumulation Speedup"} -# ┌── LEGO ─────────────────────────────────────────────── -class GradAccumScenario: - """Scenario: 8 GPUs with 4-step gradient accumulation.""" - # ┌── 1. LOAD (Constants) ─────────────────────────────────────────────── - n_gpus_ga = 8 - batch_per_gpu = 16 - accum_steps = 4 - compute_8gpu_ms = 180 - comm_8gpu_ms = 5 - ga_training_hours = 3.8 - cost_per_hour_8gpu = 128 - cost_32gpu_reference = 3021 # from earlier calc +**Problem**: You are training GPT-2 on a commodity 10G network. A 32-GPU configuration was communication-bound, costing \$3,021 for a fixed number of samples. Can you achieve the same effective batch size on a single 8-GPU node using gradient accumulation more efficiently? - # ┌── 2. EXECUTE (The Compute) ───────────────────────────────────────── - effective_batch_val = n_gpus_ga * batch_per_gpu * accum_steps - comm_overhead_pct_val = comm_8gpu_ms / (accum_steps * compute_8gpu_ms) * 100 - ga_cost_val = cost_per_hour_8gpu * ga_training_hours - ga_savings_val = cost_32gpu_reference - ga_cost_val - ga_savings_pct_val = ga_savings_val / cost_32gpu_reference * 100 +**The Math**: +1. **Effective Batch Size**: 8 GPUs $\times$ batch 16 $\times$ 4 accumulation steps = **`{python} GradAccumScenario.effective_batch_str`**. +2. **Communication Overhead**: With 4-step accumulation, we AllReduce once every 4 steps. + - Overhead = $5 \text{ ms} / (4 \times 180 \text{ ms}) \approx$ **`{python} GradAccumScenario.comm_overhead_pct_str`%**. +3. **Training Duration**: Total time reduces to **`{python} GradAccumScenario.ga_training_hours_str` hours**. +4. **Total Cost**: `{python} GradAccumScenario.ga_training_hours_str` hrs $\times$ \$128/hr = **\$`{python} GradAccumScenario.ga_cost_str`**. - # ┌── 3. GUARD (Invariants) ─────────────────────────────────────────── - check(comm_overhead_pct_val < 1.0, "Accumulation should minimize overhead to <1%") +**The Systems Insight**: Gradient accumulation saves **\$`{python} GradAccumScenario.ga_savings_str`** (`{python} GradAccumScenario.ga_savings_pct_str`%) by concentrating computation where bandwidth is abundant (NVLink within the node) and minimizing the frequency of synchronization. When your network is slow, don't scale out—scale the batch size locally. - # ┌── 4. OUTPUT (Formatting) ────────────────────────────────────────────── - effective_batch_str = f"{effective_batch_val}" - comm_overhead_pct_str = f"{comm_overhead_pct_val:.1f}" - ga_training_hours_str = f"{ga_training_hours}" - ga_cost_str = f"{ga_cost_val:.0f}" - ga_savings_str = f"{ga_savings_val:,.0f}" - ga_savings_pct_str = f"{ga_savings_pct_val:.0f}" -``` +::: - Configuration: `{python} GradAccumScenario.n_gpus_ga` GPUs$\times$ batch `{python} GradAccumScenario.batch_per_gpu`$\times$ `{python} GradAccumScenario.accum_steps` accumulation steps = `{python} GradAccumScenario.effective_batch_str` effective batch - Communication overhead: `{python} GradAccumScenario.comm_8gpu_ms`ms ÷ (`{python} GradAccumScenario.accum_steps`$\times$ `{python} GradAccumScenario.compute_8gpu_ms`ms) = `{python} GradAccumScenario.comm_overhead_pct_str`% @@ -1871,6 +1888,17 @@ This distribution method enables training of large-scale models. GPT-3, with `{p Device coordination follows a specific pattern during training. In the forward pass, data flows sequentially through model segments on different devices. The backward pass propagates gradients in reverse order through these segments. During parameter updates, each device modifies only its assigned portion of the model. This coordination ensures mathematical equivalence to training on a single device while enabling the handling of models that exceed individual device memory capacities. +::: {.callout-checkpoint title="Model Parallelism Foundations"} + +Verify your understanding of model sharding: + +- [ ] Does Model Parallelism reduce the **memory footprint** per GPU for a given model? +- [ ] In which phase—forward or backward—do sequential dependencies between model shards arise? +- [ ] Why does pure Model Parallelism often lead to lower GPU utilization (the "Pipeline Bubble") compared to Data Parallelism? +- [ ] Can you identify two ways to split a Transformer layer: **Layer-based** versus **Tensor-based**? + +::: + ### Model Parallelism Implementation {#sec-distributed-training-systems-systems-model-parallelism-implementation-0728} Model parallelism divides neural networks across multiple computing devices, with each device computing a distinct portion of the model's operations. This division allows training of models whose parameter counts exceed single-device memory capacity. The technique encompasses device coordination, data flow management, and gradient computation across distributed model segments. @fig-dist-model-parallelism captures this bidirectional data flow: input data propagates forward through sequentially assigned model partitions while gradients flow backward to update parameters, with intermediate results transferring across device boundaries at each stage. @@ -2011,11 +2039,60 @@ Pipeline parallelism extends layer-wise partitioning by introducing microbatchin ::: {.callout-definition title="Pipeline Parallelism"} -***Pipeline Parallelism***\index{Pipeline Parallelism!definition} is a model parallelism technique where layers are partitioned into **Sequential Stages** assigned to different devices. +***Pipeline Parallelism***\index{Pipeline Parallelism!definition} is a model parallelism technique that partitions a neural network's layers into sequential stages assigned to different devices, passing activations forward and gradients backward between stages while overlapping computation across stages using micro-batches to maintain throughput. -1. **Significance (Quantitative):** It enables the training of models that exceed the memory of a single node by sharding the model depth. To maintain **Throughput ($\eta$)**, input batches are split into **Micro-batches** that pipeline through the stages, overlapping computation with communication. -2. **Distinction (Durable):** Unlike **Tensor Parallelism**, which shards individual operations (Intra-layer), Pipeline Parallelism shards the model at **Layer Boundaries** (Inter-layer), typically requiring lower communication bandwidth ($BW$). -3. **Common Pitfall:** A frequent misconception is that Pipeline Parallelism is "perfectly efficient." In reality, it suffers from the **Pipeline Bubble**: idle time at the beginning and end of each iteration where devices wait for micro-batches to reach them. +1. **Significance (Quantitative):** Inter-stage communication transmits only the activation tensor at each stage boundary — for a hidden dimension of 8,192 at BF16 with micro-batch size 1, this is $8192 \times 2 = 16$ KB per boundary, compared to the gigabytes required for gradient AllReduce in data parallelism. This low communication volume makes pipeline parallelism the primary technique for scaling model depth across nodes connected by 50 GB/s InfiniBand. The pipeline bubble wastes $(p-1)/m$ of total compute, where $p$ is the number of stages and $m$ is the number of micro-batches — with $p=8$ stages and $m=32$ micro-batches, bubble overhead is 22%. +2. **Distinction (Durable):** Unlike tensor parallelism, which shards individual matrix multiplications within a single layer and requires AllReduce on every layer's output (demanding NVLink-class bandwidth within each stage), pipeline parallelism shards at layer boundaries and requires only point-to-point activation transfers between stages — tolerating InfiniBand bandwidth between nodes. +3. **Common Pitfall:** A frequent misconception is that adding more pipeline stages always improves throughput. The pipeline bubble fraction grows as $(p-1)/(m+p-1)$: with $p=16$ stages and $m=16$ micro-batches, 48% of compute is wasted idle — requiring large micro-batch counts ($m \gg p$) to keep the bubble below 10%, which in turn increases peak activation memory and the memory pressure on each stage. + +::: + +```{python} +#| label: pipeline-bubble-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ PIPELINE BUBBLE TAX (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Cost of the Pipeline Bubble" .callout-notebook +# │ +# │ Goal: Quantify the GPU idle time (bubble) in pipeline parallelism. +# │ Show: bubble_fraction_pct ≈ 18% — inline in notebook result. +# │ How: Bubble_Fraction = (P - 1) / (P - 1 + M). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: pb_stages_str, pb_microbatches_str, pb_bubble_pct_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class PipelineBubble: + # ┌── 1. LOAD ────────────────────────────────────────── + p_stages = 8 # Number of nodes/stages + m_microbatches = 32 # Microbatches per global batch + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Bubble overhead is the time GPUs spend waiting for the first/last samples + bubble_fraction = (p_stages - 1) / (p_stages - 1 + m_microbatches) + bubble_pct = bubble_fraction * 100 + # ┌── 3. GUARD ───────────────────────────────────────── + check(17 < bubble_pct < 19, f"Bubble {bubble_pct:.1f}% unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + pb_stages_str = f"{p_stages}" + pb_microbatches_str = f"{m_microbatches}" + pb_bubble_pct_str = f"{bubble_pct:.1f}" + +``` + +::: {.callout-notebook title="The Cost of the Pipeline Bubble"} + +**Problem**: You are training a frontier model using **Pipeline Parallelism** across `{python} PipelineBubble.pb_stages_str` nodes. To hide the sequential delay, you split your batch into `{python} PipelineBubble.pb_microbatches_str` microbatches. What is your "Bubble Tax"—the fraction of GPU cycles lost to idle waiting? + +**The Math**: +In a synchronous pipeline (1F1B), the bubble fraction is determined by the ratio of stages to microbatches. + +1. **Wait time**: At the start and end of each batch, GPUs sit idle for $P-1$ steps. +2. **Productive time**: GPUs compute for $M$ steps. +3. **Bubble Fraction**: $(P-1) / (P-1+M) = 7 / (7+32) \approx$ **`{python} PipelineBubble.pb_bubble_pct_str`%**. + +**The Systems Insight**: Pipeline parallelism is a **Utilization-Memory Trade-off**. To reduce the bubble from 18% to 5%, you must increase the number of microbatches ($M$), which consumes more **Activation Memory** on each GPU. In the Machine Learning Fleet, you don't scale depth for free—you pay a **`{python} PipelineBubble.pb_bubble_pct_str`%** capacity tax just to keep the stages coordinated. This is why techniques like **Interleaved Pipelining** are essential: they chop the bubble into smaller pieces to recover that lost 18% of your fleet. ::: @@ -2132,11 +2209,11 @@ Tensor parallelism takes a fundamentally different approach: instead of assignin ::: {.callout-definition title="Tensor Parallelism"} -***Tensor Parallelism***\index{Tensor Parallelism!definition} is a model parallelism technique where individual **Tensors and Operations** (e.g., matrix multiplications) are split across multiple devices. +***Tensor Parallelism***\index{Tensor Parallelism!definition} is a model parallelism technique that partitions individual tensor operations — primarily matrix multiplications — across multiple devices using column-parallel or row-parallel weight splits, requiring one AllReduce per transformer block to sum partial results from all participating devices. -1. **Significance (Quantitative):** It enables the parallelization of massive layers that exceed single-device memory. It requires **High-Bandwidth Interconnects** (e.g., NVLink) because it necessitates all-reduce or all-gather operations within the critical path of every layer. -2. **Distinction (Durable):** Unlike **Pipeline Parallelism** (Inter-layer), Tensor Parallelism operates at the **Intra-layer** granularity, distributing the computation of a single mathematical operation across $N$ workers. -3. **Common Pitfall:** A frequent misconception is that Tensor Parallelism can be used across any network. In reality, it is **Bandwidth-Bound ($BW$)**: the high frequency of synchronization makes it feasible only within a single node or across extremely low-latency fabrics. +1. **Significance (Quantitative):** Megatron-LM style tensor parallelism places two AllReduce operations per transformer block — one after attention and one after the MLP — with each AllReduce transferring the activation tensor ($\text{hidden\_dim} \times \text{batch} \times 2$ bytes in BF16). At degree $t=8$ on NVLink with 900 GB/s bandwidth, each AllReduce takes approximately 0.1–0.5 ms. The same operation over InfiniBand (50 GB/s) takes 2–10 ms per layer — with 96 transformer blocks, this adds 200–960 ms of pure synchronization overhead per step, collapsing MFU to single digits. +2. **Distinction (Durable):** Unlike pipeline parallelism, which communicates only at layer boundaries and transfers small activation slices, tensor parallelism synchronizes within every layer via AllReduce, making it sensitive to the per-operation latency of the interconnect and restricting it to within-node NVLink fabrics in practice. +3. **Common Pitfall:** A frequent misconception is that tensor parallelism can simply be scaled across nodes over InfiniBand. NVLink delivers approximately 900 GB/s bidirectional bandwidth; InfiniBand NDR delivers 50 GB/s — an 18$\times$ gap. Running tensor parallelism across InfiniBand typically collapses MFU to single-digit percentages, making the communication overhead larger than the compute benefit of distributing the matrix multiplication. ::: @@ -2475,6 +2552,17 @@ Designing the training topology for GPT-3 (175B parameters) requires balancing m With the Blackwell architecture, we can exploit the `{python} ThreeDParallelism.nvlink5_bw_str` NVLink 5 and FP4 precision. For a 175B model, we can increase **Tensor Parallelism to TP=16** (across two nodes) while maintaining the same low latency. This reduces the required **Pipeline Parallelism to PP=8**, cutting the pipeline bubble fraction by half ($P=8$ vs $P=16$). The resulting fleet is not only faster due to higher TFLOPS but also more efficient due to reduced coordination overhead ($T_{coord}$). ::: +::: {.callout-checkpoint title="Hybrid 3D Parallelism"} + +Verify your understanding of how parallelism strategies combine: + +- [ ] In a **TP=8, PP=16, DP=128** configuration, which dimension is responsible for sharding layers *within* a single node? +- [ ] How does moving from **PP=16** to **PP=8** affect the "Pipeline Bubble" fraction? +- [ ] If you have a total budget of 1,024 GPUs, what is the trade-off between increasing **DP** (Data Parallelism) versus increasing **PP** (Pipeline Parallelism)? +- [ ] Which dimension—TP, PP, or DP—is most sensitive to **inter-node latency**? + +::: + The MFU values cited above raise a natural question: how did the field arrive at 50% utilization, and what trajectory brought it there? @fig-mfu-progression traces the evolution of Model FLOPs Utilization across published training systems from 2020 to 2024. The progression from GPT-3's 21% MFU to PaLM's 46% MFU reflects not improvements in raw hardware speed but advances in the parallelism strategies, communication overlap techniques, and scheduling optimizations discussed throughout this chapter. The plateau near 40-46% reveals that the theoretical ceiling imposed by communication overhead, pipeline bubbles, and memory management remains formidable even with the most advanced hybrid parallelism techniques available. Notably, Meta's Llama 3 training at 16,384 H100 GPUs achieved slightly lower MFU (41%) than the same model at 8,192 GPUs (43%), confirming that the Scaling Tax described in @sec-distributed-training-systems-systems-physics-scaling-amdahls-law-communication-4d7f is not merely theoretical but measurable in production. ::: {#fig-mfu-progression fig-env="figure" fig-pos="htb" fig-cap="**The MFU Progression**. Model FLOPs Utilization across published training systems, ordered chronologically. MFU doubled from 21% (GPT-3, 2020) to 46% (PaLM, 2022) through advances in hybrid parallelism and communication overlap, then plateaued near 40-45% for frontier-scale runs. The Llama 3 data points illustrate the Scaling Tax: MFU drops from 43% at 8,192 GPUs to 41% at 16,384 GPUs as communication overhead grows with cluster size. Data sources: Chowdhery et al. (2022), Meta (2024)." fig-alt="Horizontal bar chart of MFU percentages for six training systems from 2020 to 2024 showing progression from 21 percent to 46 percent with a plateau near 41 to 43 percent."} @@ -2687,13 +2775,13 @@ class RLHFBudget: The policy model under 3D parallelism with TP=8 and PP=4 distributes its `{python} RLHFBudget.policy_mem_str` GB training state across `{python} RLHFBudget.n_gpus_str` GPUs, yielding `{python} RLHFBudget.policy_per_gpu_str` GB per GPU. The reference model, requiring only `{python} RLHFBudget.ref_mem_str` GB for inference, can be sharded across a separate pool of 8 GPUs at `{python} RLHFBudget.ref_per_gpu_sep_str` GB each, or co-located with the policy GPUs at an additional `{python} RLHFBudget.ref_per_gpu_str` GB per GPU (`{python} RLHFBudget.ref_mem_str` / `{python} RLHFBudget.n_gpus_str`). The 13B reward model adds `{python} RLHFBudget.reward_mem_str` GB shared across its pool. The 13B value model in training mode adds approximately `{python} RLHFBudget.value_mem_str` GB (13B $\times$ 16 bytes/param) across its pool. -Total static memory (co-located policy + reference on `{python} RLHFBudget.n_gpus_str` GPUs): $`{python} RLHFBudget.policy_per_gpu_str` + `{python} RLHFBudget.ref_per_gpu_str` \approx$ `{python} RLHFBudget.static_per_gpu_str` GB per GPU, leaving $\sim$`{python} RLHFBudget.remaining_str` GB for activations and KV cache. With generation-phase KV caches for 256 sequences of 1,024 tokens, each policy GPU must reserve an additional $172 / `{python} RLHFBudget.n_gpus_str` \approx$ `{python} RLHFBudget.kv_per_gpu_str` GB for its KV cache shard. The remaining $\sim$`{python} RLHFBudget.remaining_kv_str` GB constrains the micro-batch size during the training phase. +Total static memory (co-located policy + reference on `{python} RLHFBudget.n_gpus_str` GPUs): `{python} RLHFBudget.policy_per_gpu_str` + `{python} RLHFBudget.ref_per_gpu_str` $\approx$ `{python} RLHFBudget.static_per_gpu_str` GB per GPU, leaving $\sim$ `{python} RLHFBudget.remaining_str` GB for activations and KV cache. With generation-phase KV caches for 256 sequences of 1,024 tokens, each policy GPU must reserve an additional $172/$ `{python} RLHFBudget.n_gpus_str` $\approx$ `{python} RLHFBudget.kv_per_gpu_str` GB for its KV cache shard. The remaining $\sim$ `{python} RLHFBudget.remaining_kv_str` GB constrains the micro-batch size during the training phase. **DPO Memory Budget (same parallelism configuration)** The policy model uses the same `{python} RLHFBudget.policy_per_gpu_str` GB per GPU. The reference model adds `{python} RLHFBudget.ref_per_gpu_str` GB per GPU (co-located). No reward model, no value model, no KV cache for generation. -Total static memory: $`{python} RLHFBudget.policy_per_gpu_str` + `{python} RLHFBudget.ref_per_gpu_str` =$ `{python} RLHFBudget.static_per_gpu_str` GB per GPU — identical to PPO's static footprint, but without the generation-phase KV cache burden. The full $\sim$`{python} RLHFBudget.remaining_str` GB remaining is available for training activations, permitting 2$\times$ larger micro-batch sizes or eliminating the need for activation checkpointing that PPO requires. +Total static memory: `{python} RLHFBudget.policy_per_gpu_str` + `{python} RLHFBudget.ref_per_gpu_str` = `{python} RLHFBudget.static_per_gpu_str` GB per GPU — identical to PPO's static footprint, but without the generation-phase KV cache burden. The full $\sim$ `{python} RLHFBudget.remaining_str` GB remaining is available for training activations, permitting 2$\times$ larger micro-batch sizes or eliminating the need for activation checkpointing that PPO requires. **Throughput Comparison** @@ -2702,7 +2790,7 @@ PPO's two-phase design (generate then train) introduces a fundamental throughput | **Metric** | **PPO (70B policy)** | **DPO (70B policy)** | |:---------------------------|:-------------------------------|:------------------------| | **Models required** | 4 (policy, ref, reward, value) | 2 (policy, reference) | -| **Total parameter memory** | $\sim$2,406 GB | $\sim$1,260 GB | +| **Total parameter memory** | $\sim$ 2,406 GB | $\sim$ 1,260 GB | | **Minimum GPUs (memory)** | 64--128 | 32--64 | | **Generation phase** | Yes (sequential, BW-bound) | None | | **Effective training MFU** | 15--25% | 40--55% | @@ -3013,11 +3101,11 @@ Throughout this chapter, we applied these partitioning strategies to our Lightho ::: {.callout-lighthouse title="Distributed Archetype Spectrum"} The "optimal point" in the 3D Parallelism Cube shifts depending on the system's primary bottleneck: -| **Archetype** | **Primary Partitioning Strategy** | **The Logic** | -|:-------------------------------------|:----------------------------------|:----------------------------------------------------------------------------------------------| -| **Archetype A (GPT-4 / Llama-3)** | Hybrid 3D Parallelism | Combine Tensor (width), Pipeline (depth), and Data (throughput) to fit `{python} DistTrainConstants.gpt4_mem_tb_str` TB of weights. | -| **Archetype B (DLRM at Scale)** | Embedding Sharding | Partition massive `{python} DistTrainConstants.dlrm_mem_tb_str` TB+ tables across a Parameter Server fleet; use sparse AllToAll updates. | -| **Archetype C (Federated MobileNet)**| Federated Learning | Distribute training *data* to the edge; keep model local; accept asynchronous, stale updates. | +| **Archetype** | **Primary Partitioning Strategy** | **The Logic** | +|:--------------------------------------|:----------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------| +| **Archetype A (GPT-4 / Llama-3)** | Hybrid 3D Parallelism | Combine Tensor (width), Pipeline (depth), and Data (throughput) to fit `{python} DistTrainConstants.gpt4_mem_tb_str` TB of weights. | +| **Archetype B (DLRM at Scale)** | Embedding Sharding | Partition massive `{python} DistTrainConstants.dlrm_mem_tb_str` TB+ tables across a Parameter Server fleet; use sparse AllToAll updates. | +| **Archetype C (Federated MobileNet)** | Federated Learning | Distribute training *data* to the edge; keep model local; accept asynchronous, stale updates. | ::: ::: {#fig-3d-parallelism-cube-summary fig-env="figure" fig-pos="htb" fig-cap="**3D Parallelism Strategy Space**: The three independent axes of distributed training parallelism. Data Parallelism (d) replicates models across workers. Pipeline Parallelism (p) partitions model layers vertically. Tensor Parallelism (t) splits individual operations horizontally. Real systems combine all three dimensions." fig-alt="Three-dimensional cube diagram with axes labeled Data Parallel, Pipeline Parallel, and Tensor Parallel. Colored planes show Model Replicas, Intra-layer partitioning, and pipeline stages."} diff --git a/book/quarto/contents/vol2/edge_intelligence/edge_intelligence.qmd b/book/quarto/contents/vol2/edge_intelligence/edge_intelligence.qmd index 663b7ab936..a189f1b3d1 100644 --- a/book/quarto/contents/vol2/edge_intelligence/edge_intelligence.qmd +++ b/book/quarto/contents/vol2/edge_intelligence/edge_intelligence.qmd @@ -84,6 +84,7 @@ class EdgeIntelligenceSetup: h100_mem_bw_raw = H100_MEM_BW phone_battery_raw = PHONE_BATTERY_WH dram_energy_raw = ENERGY_DRAM_ACCESS_PJ + smartphone_ram_raw = SMARTPHONE_RAM_GB # ┌── 2. EXECUTE (The Compute) ──────────────────────────────────────── mobile_npu_tops = int(mobile_npu_tops_raw.m_as(TFLOPs / second)) @@ -91,6 +92,7 @@ class EdgeIntelligenceSetup: phone_battery_wh = int(phone_battery_raw.m_as(watt * hour)) phone_battery_j = f"{int(phone_battery_raw.m_as(joule)):,}" dram_energy_pj = f"{dram_energy_raw.m_as(ureg.picojoule):.0f}" + smartphone_ram_gb = int(smartphone_ram_raw.m_as(GB)) # ┌── 3. GUARD (Invariants) ────────────────────────────────────────── # No check() calls needed — values are direct unit conversions. @@ -111,6 +113,57 @@ These scenarios require on-device learning, where models must train and adapt di [^fn-a11-bionic-ondevice]: **A11 Bionic (2017)**: The first mobile SoC with a dedicated Neural Engine, delivering 0.6 TOPS versus the A10's 0.2 TOPS. This 3$\times$ jump enabled gradient computation on a phone for the first time, but the real systems consequence was establishing the mobile NPU as a distinct power domain: on-device training became feasible only when inference-optimized silicon freed enough thermal headroom for backpropagation bursts. \index{A11 Bionic!on-device learning} +```{python} +#| label: edge-npu-speedup-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ NPU VS CPU SPEEDUP (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "NPU: The Silicon Dividend" .callout-notebook +# │ +# │ Goal: Quantify the latency and energy benefits of mobile NPUs. +# │ Show: speedup ≈ 20x, energy_gain ≈ 50x — inline in notebook result. +# │ How: T_cpu / T_npu; E_cpu / E_npu. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: npu_cpu_lat_ms_str, npu_lat_ms_str, npu_speedup_str, npu_energy_gain_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class EdgeNpuSpeedup: + # ┌── 1. LOAD ────────────────────────────────────────── + t_cpu_ms = 400 # Typical CPU inference for MobileNet + t_npu_ms = 20 # NPU accelerated + e_cpu_j = 0.8 # Joules per inference on CPU + e_npu_j = 0.016 # Joules per inference on NPU + # ┌── 2. EXECUTE ─────────────────────────────────────── + speedup = t_cpu_ms / t_npu_ms + energy_gain = e_cpu_j / e_npu_j + # ┌── 3. GUARD ───────────────────────────────────────── + check(speedup == 20, f"Speedup {speedup:.1f}x unexpected") + check(energy_gain == 50, f"Energy gain {energy_gain:.1f}x unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + npu_cpu_lat_ms_str = f"{t_cpu_ms}" + npu_lat_ms_str = f"{t_npu_ms}" + npu_speedup_str = f"{speedup:.0f}" + npu_energy_gain_str = f"{energy_gain:.0f}" + +``` + +::: {.callout-notebook title="NPU: The Silicon Dividend"} + +**Problem**: You are comparing the performance of a MobileNet classifier on a generic mobile CPU versus a specialized Neural Processing Unit (NPU). How much "Silicon Dividend" does the NPU provide in terms of speed and battery life? + +**The Math**: +The dividend is the ratio of general-purpose to specialized efficiency. + +1. **Latency Speedup**: `{python} EdgeNpuSpeedup.npu_cpu_lat_ms_str` ms / `{python} EdgeNpuSpeedup.npu_lat_ms_str` ms = **`{python} EdgeNpuSpeedup.npu_speedup_str`$\times$**. +2. **Energy Efficiency**: 0.8 J (CPU) / 0.016 J (NPU) = **`{python} EdgeNpuSpeedup.npu_energy_gain_str`$\times$**. + +**The Systems Insight**: Specialized hardware doesn't just make the model faster; it makes it **feasible**. A **50$\times$ energy gain** means you can run 50 times more inferences on the same battery charge. On the edge, the CPU is for **Coordination**, but the NPU is for **Survival**. If you miss the NPU target and fall back to the CPU, you haven't just lost performance—you've likely made the application unusable due to rapid battery drain and thermal throttling. + +::: + The transition to on-device learning introduces fundamental tension in machine learning systems design. While cloud-based architectures use abundant computational resources and controlled operational environments, edge devices must function within severely constrained resource envelopes. Memory is measured in megabytes rather than gigabytes; power budgets are measured in milliwatts rather than watts; and network connectivity may be intermittent or entirely absent. Throughout this chapter, we trace these constraints through **Archetype C (Federated MobileNet)** (@sec-vol2-introduction-archetypes)—the Autonomous Health Monitor—where autonomy demands that learning happen on-device under severe resource limits. Archetype C (Federated MobileNet) represents the extreme end of this spectrum. Operating on milliwatt power budgets, it cannot afford the energy cost of transmitting raw data to the cloud. For Archetype C (Federated MobileNet), data locality is not just a privacy feature but a physical necessity imposed by the Power Wall. The techniques developed in this chapter—quantized training, sparse updates, and federated coordination—are the survival strategies that allow Archetype C (Federated MobileNet) to exist. @@ -179,7 +232,9 @@ For applications with critical timing requirements, network round-trip times mak These motivations are grounded in the broader concept of knowledge transfer, where a pretrained model transfers useful representations to a new task or domain. This foundational principle makes on-device learning both feasible and effective, enabling sophisticated adaptation with minimal local resources. @fig-transfer-conceptual illustrates how knowledge transfer occurs between closely related tasks such as playing different board games or musical instruments, or across domains that share structure such as from riding a bicycle to driving a scooter. In the context of on-device learning, this means using a model pretrained in the cloud and adapting it efficiently to a new context using only local data and limited updates, allowing fast adaptation without relearning from scratch even when the new task diverges in input modality or goal. -![Knowledge Transfer: Pretrained models accelerate learning on new tasks by leveraging existing representations, as seen by adapting skills between related board games or musical instruments. This transfer extends across domains like bicycle riding and scooter operation, where shared underlying structures allow efficient adaptation with limited new data.](images/png/ondevice_transfer_learning_apps.png){#fig-transfer-conceptual fig-alt="Conceptual diagram with four transfer learning examples: chess to checkers, violin to cello, bicycle to scooter, showing how similar skills transfer between related domains."} +::: {#fig-transfer-conceptual fig-env="figure" fig-pos="htb" fig-cap="**Knowledge Transfer**: Pretrained models accelerate learning on new tasks by leveraging existing representations, as seen by adapting skills between related board games or musical instruments. This transfer extends across domains like bicycle riding and scooter operation, where shared underlying structures allow efficient adaptation with limited new data." fig-alt="Conceptual diagram with four transfer learning examples: chess to checkers, violin to cello, bicycle to scooter, showing how similar skills transfer between related domains."} +![](images/png/ondevice_transfer_learning_apps.png) +::: This conceptual shift, enabled by transfer learning and adaptation, enables real-world on-device applications. Whether adapting a language model for personal typing preferences, adjusting gesture recognition to individual movement patterns, or recalibrating a sensor model in changing environments, on-device learning allows systems to remain responsive, efficient, and user-aligned over time. @@ -195,7 +250,9 @@ For instance, Google's Gboard employs **federated learning**[^fn-federated-metap @fig-ondevice-gboard demonstrates how different prediction strategies enable local adaptation in real time. Next-word prediction suggests likely continuations based on prior text, while Smart Compose uses on-the-fly rescoring to offer dynamic completions. These techniques demonstrate the sophistication of local inference mechanisms. -![On-Device Prediction Strategies: Gboard employs both next-word prediction and smart compose with on-the-fly rescoring to adapt to user typing patterns locally, enhancing personalization and preserving privacy. These techniques demonstrate how machine learning models can refine predictions in real time without transmitting data to a central server, enabling efficient and private mobile input experiences.](images/png/ondevice_gboard_example.png){#fig-ondevice-gboard fig-alt="Mobile keyboard interface showing two prediction modes: next-word suggestions above keys and smart compose autocomplete in the text field, both adapting to user typing patterns."} +::: {#fig-ondevice-gboard fig-env="figure" fig-pos="htb" fig-cap="**On-Device Prediction Strategies**: Gboard employs both next-word prediction and smart compose with on-the-fly rescoring to adapt to user typing patterns locally, enhancing personalization and preserving privacy. These techniques demonstrate how machine learning models can refine predictions in real time without transmitting data to a central server, enabling efficient and private mobile input experiences." fig-alt="Mobile keyboard interface showing two prediction modes: next-word suggestions above keys and smart compose autocomplete in the text field, both adapting to user typing patterns."} +![](images/png/ondevice_gboard_example.png) +::: Wearable and health monitoring devices present equally compelling use cases with additional regulatory constraints. These systems rely on real-time data from accelerometers, heart rate sensors, and electrodermal activity monitors to track user health and fitness. Physiological baselines vary dramatically between individuals, creating a personalization challenge that static models cannot address effectively. On-device learning allows models to adapt to these individual baselines over time, substantially improving the accuracy of activity recognition, stress detection, and sleep staging while meeting regulatory requirements for data localization. @@ -561,10 +618,21 @@ fig = plt.gcf() #### Peak Memory Usage -Memory consumption during training is not static. It fluctuates dynamically, reaching a maximum during the backward pass when activations, gradients, and optimizer states must coexist. This peak memory usage determines whether a model can be trained on a device. For a 10M parameter model on a smartphone with 6 GB RAM, the 40 MB of FP32 weights might spike to over 200 MB during backpropagation as activations, gradients, and optimizer states accumulate---competing directly with the operating system and foreground applications for the device's limited memory. Techniques like gradient checkpointing mitigate this by discarding intermediate activations during the forward pass and recomputing them on-demand during the backward pass. This approach trades increased computation (20 to 30%) for a dramatic reduction in peak memory, often achieving 3 to 4 times reduction. +Memory consumption during training is not static. It fluctuates dynamically, reaching a maximum during the backward pass when activations, gradients, and optimizer states must coexist. This peak memory usage determines whether a model can be trained on a device. For a 10M parameter model on a smartphone with `{python} EdgeIntelligenceSetup.smartphone_ram_gb}` GB RAM, the 40 MB of FP32 weights might spike to over 200 MB during backpropagation as activations, gradients, and optimizer states accumulate---competing directly with the operating system and foreground applications for the device's limited memory. Techniques like gradient checkpointing mitigate this by discarding intermediate activations during the forward pass and recomputing them on-demand during the backward pass. This approach trades increased computation (20 to 30%) for a dramatic reduction in peak memory, often achieving 3 to 4 times reduction. These amplifications explain why standard optimization techniques fail when applied to training workloads without modification. Each constraint category shapes on-device learning system design, requiring approaches that build on but extend beyond inference-focused methods. +::: {.callout-checkpoint title="On-Device Training Constraints"} + +Verify your understanding of how training amplifies edge constraints: + +- [ ] Why does on-device training typically require **4x to 12x more memory** than on-device inference for the same model? +- [ ] What is the "Data Stall" risk in edge intelligence, and how does it relate to the device's NPU throughput? +- [ ] If a smartphone has 8 GB of RAM, why might a 100M parameter model still trigger out-of-memory errors during the backward pass? +- [ ] True or False: Gradient checkpointing reduces the total energy consumed by on-device training. + +::: + @fig-ondevice-pretraining illustrates how the complete training pipeline combines offline pre-training with online adaptive learning on resource-constrained IoT devices. The system first undergoes meta-training with generic data. During deployment, device-specific constraints such as data availability, compute, and memory shape the adaptation strategy by ranking and selecting layers and channels to update. This allows efficient on-device learning within limited resource envelopes. ::: {#fig-ondevice-pretraining fig-env="figure" fig-pos="htb" fig-cap="**Federated Averaging Cycle**: The four-step coordination protocol that enables distributed training while preserving data privacy. (1) Server distributes global model to participating clients, (2) Clients train locally on their private data using multiple SGD steps, (3) Clients send updated model weights (not raw data) back to the server, (4) Server performs weighted averaging of client updates to create new global model." fig-alt="Two-stage pipeline. Left: offline pre-training with full neural network. Center: device constraints (data, compute, memory) feed into importance ranking. Right: selected layers highlighted in green for on-device training."} @@ -957,6 +1025,54 @@ Beyond the individual constraints of models, data, and computation, on-device le Energy and thermal management represent the most challenging aspects of on-device learning system design, as they directly impact user experience and device longevity. Mobile devices operate under strict power budgets that fundamentally determine feasible model complexity and training schedules. +```{python} +#| label: edge-battery-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ BATTERY LIFE FOR ON-DEVICE LEARNING (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Battery Drain: The Cost of Edge Learning" .callout-notebook +# │ +# │ Goal: Quantify the battery impact of a background fine-tuning job on a phone. +# │ Show: battery_drain_pct ≈ 15% — inline in notebook result. +# │ How: Drain = (Power_W * Duration_H) / Battery_Wh * 100. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: battery_power_w_str, battery_duration_min_str, battery_drain_pct_str, battery_total_wh_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class EdgeBattery: + # ┌── 1. LOAD ────────────────────────────────────────── + p_train_w = 4.5 # Typical sustained SoC power during training + t_train_min = 30 # 30 minute adaptation job + battery_wh = 15 # Modern flagship smartphone battery + # ┌── 2. EXECUTE ─────────────────────────────────────── + t_train_h = t_train_min / 60 + energy_wh = p_train_w * t_train_h + drain_pct = (energy_wh / battery_wh) * 100 + # ┌── 3. GUARD ───────────────────────────────────────── + check(14 < drain_pct < 16, f"Drain {drain_pct:.1f}% unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + battery_power_w_str = f"{p_train_w:.1f}" + battery_duration_min_str = f"{t_train_min}" + battery_drain_pct_str = f"{drain_pct:.1f}" + battery_total_wh_str = f"{battery_wh}" + +``` + +::: {.callout-notebook title="Battery Drain: The Cost of Edge Learning"} + +**Problem**: You are designing a background fine-tuning job for a personalized voice assistant on a smartphone. The training job consumes `{python} EdgeBattery.battery_power_w_str` Watts and takes `{python} EdgeBattery.battery_duration_min_str` minutes to complete. If the phone has a `{python} EdgeBattery.battery_total_wh_str` Wh battery, how much of the user's battery will this "invisible" update consume? + +**The Math**: +1. **Total Energy**: `{python} EdgeBattery.battery_power_w_str` W $\times$ (30/60) hours = **2.25 Wh**. +2. **Battery Impact**: 2.25 Wh / 15 Wh = **`{python} EdgeBattery.battery_drain_pct_str`%**. + +**The Systems Insight**: Consuming **`{python} EdgeBattery.battery_drain_pct_str`%** of a user's battery for a background task is a severe violation of mobile UX principles—it's equivalent to losing an hour of screen-on time. This is why on-device training is typically restricted to **Opportunistic Scheduling**: it only runs when the device is plugged in, connected to Wi-Fi, and thermally stable. Designing for the edge means respecting the user's energy budget as strictly as the model's accuracy budget. + +::: + Because these energy constraints are a primary driver of sustainable system design, we provide a detailed quantitative analysis of the **Battery Wall**, the **Energy Hierarchy** (compute vs. communication), and the feasibility of on-device fine-tuning in @sec-sustainable-ai-edge-learning-battery. #### Memory Hierarchy Optimization {#sec-edge-intelligence-memory-hierarchy-optimization-8396} @@ -1012,6 +1128,55 @@ Each solution pillar extends model optimization principles and distributed syste Personalizing a billion-parameter model on a smartwatch with 500 MB of total system memory is impossible without abandoning the goal of updating the complete model. The answer is that we do not update the whole model. We freeze the vast majority of the network and only train a tiny, strategically placed set of new parameters. This is the essence of resource-efficient model adaptation. +```{python} +#| label: local-storage-wall-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ LOCAL STORAGE WALL (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Hidden Cost of Personalization" .callout-notebook +# │ +# │ Goal: Quantify the storage savings of using adapters vs full fine-tuning. +# │ Show: per_context_savings ≈ 200x — inline in notebook result. +# │ How: Ratio = Size_Full / Size_Adapter. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: ls_full_mb_str, ls_adapter_kb_str, ls_savings_ratio_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class StorageWall: + # ┌── 1. LOAD ────────────────────────────────────────── + model_params_m = 10 + bytes_per_param = 4 # FP32 for training/fine-tuning + adapter_params = 50000 # Rank-64 adapter + # ┌── 2. EXECUTE ─────────────────────────────────────── + size_full_mb = (model_params_m * 1e6 * bytes_per_param) / 1e6 + size_adapter_kb = (adapter_params * bytes_per_param) / 1e3 + per_context_savings = (size_full_mb * 1000) / size_adapter_kb + # ┌── 3. GUARD ───────────────────────────────────────── + check(per_context_savings == 200, f"Ratio {per_context_savings} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + ls_full_mb_str = f"{size_full_mb:.0f}" + ls_adapter_kb_str = f"{size_adapter_kb:.0f}" + ls_savings_ratio_str = f"{per_context_savings:.0f}" + +``` + +::: {.callout-notebook title="The Hidden Cost of Personalization"} + +**Problem**: You are deploying a 10M parameter vision model to a smartphone. You want to support 10 different "User Contexts" (Home, Office, Car, etc.). If a full fine-tuned model requires `{python} StorageWall.ls_full_mb_str` MB, how much storage do you save by using **Residual Adapters** instead? + +**The Math**: +1. **Full Fine-Tuning**: 10 contexts $\times$ 40 MB = **400 MB**. +2. **Adapter Approach**: (1 $\times$ 40 MB backbone) + (10 $\times$ `{python} StorageWall.ls_adapter_kb_str` KB adapters) $\approx$ **42 MB**. +3. **Storage Savings**: 400 MB / 42 MB $\approx$ **9.5$\times$** total device storage reduction. +4. **Per-Context Efficiency**: `{python} StorageWall.ls_full_mb_str` MB / 0.2 MB = **`{python} StorageWall.ls_savings_ratio_str`$\times$**. + +**The Systems Insight**: Personalization is a **Storage Density** problem. On a device with limited flash memory, storing 10 versions of the same 400 MB model is impossible. By sharding the model into a **Frozen Backbone** and **Dynamic Adapters**, you reduce the marginal cost of a new user context by **`{python} StorageWall.ls_savings_ratio_str`$\times$**. In the edge fleet, modularity is the only way to scale intelligence without exhausting the physical hardware. + +::: + The engineering challenge centers on navigating a fundamental trade-off space: adaptation expressivity versus resource consumption. At one extreme, updating all parameters provides maximum flexibility but exceeds edge device capabilities. At the other extreme, no adaptation preserves resources but fails to capture user-specific patterns. Effective on-device learning systems must operate in the middle ground, selecting adaptation strategies based on three key engineering criteria. First, available memory, compute, and energy determine which adaptation approaches are feasible. A smartwatch with 1 MB RAM requires fundamentally different strategies than a smartphone with 8 GB. Second, the degree of user-specific variation drives adaptation complexity needs. Simple preference learning may require only bias updates, while complex domain shifts demand more sophisticated approaches. Third, adaptation techniques must integrate with existing inference pipelines, federated coordination protocols, and operational monitoring systems for model deployment and lifecycle management. @@ -1443,12 +1608,23 @@ This adapter adds a small residual transformation to a frozen layer. When insert #### Edge Personalization {#sec-edge-intelligence-edge-personalization-a511} -Adapters enable an efficient multi-tenant architecture for edge intelligence, where a single frozen backbone supports multiple personalized contexts. Consider a 10M parameter vision model deployed on a smartphone with 6 GB RAM. Full fine-tuning would require storing a separate 40 MB copy of the model weights for each personalization context (home, office, outdoor), rapidly consuming the device's storage. A rank-64 adapter introduces only approximately 50,000 parameters (roughly 200 KB) per context---a 200$\times$ reduction in storage. This efficiency allows the device to store dozens of specialized adapters that share the same frozen backbone, swapping them dynamically based on the user's location or activity. +Adapters enable an efficient multi-tenant architecture for edge intelligence, where a single frozen backbone supports multiple personalized contexts. Consider a 10M parameter vision model deployed on a smartphone with `{python} EdgeIntelligenceSetup.smartphone_ram_gb}` GB RAM. Full fine-tuning would require storing a separate 40 MB copy of the model weights for each personalization context (home, office, outdoor), rapidly consuming the device's storage. A rank-64 adapter introduces only approximately 50,000 parameters (roughly 200 KB) per context---a 200$\times$ reduction in storage. This efficiency allows the device to store dozens of specialized adapters that share the same frozen backbone, swapping them dynamically based on the user's location or activity. This **adapter switching** pattern transforms the smartphone from a static inference engine into a context-aware learning system. When the user enters a low-light environment, the system hot-swaps the appropriate adapter into the vision pipeline in milliseconds. The modularity extends naturally to federated learning: instead of transmitting 40 MB full-model updates over bandwidth-constrained cellular connections, the device transmits only the 200 KB adapter weights. Empirical studies show that these lightweight adapters capture over 90% of the quality gains of full fine-tuning [@rebuffi2017learning], offering a Pareto-optimal trade-off between personalization quality and system resources. The frozen backbone ensures that fundamental visual representations remain robust, while adapters act as specialized lenses that refocus the model's attention on user-specific details without risking catastrophic forgetting of general knowledge. In smartphone camera pipelines, environmental lighting, user preferences, and lens distortion vary between users. A shared model can be frozen and fine-tuned per-device using a few residual modules, allowing lightweight personalization without destabilizing the base model. In voice-based systems, adapter modules reduce word error rates in personalized speech recognition without retraining the full acoustic model. They also allow easy rollback or switching between user-specific versions---a critical operational capability when local adaptation occasionally degrades rather than improves performance. +::: {.callout-checkpoint title="Edge Personalization"} + +Verify your understanding of efficient on-device adaptation: + +- [ ] Why is **Adapter Switching** more storage-efficient than maintaining separate full-model copies for different user contexts? +- [ ] How does the size of a **rank-64 adapter** compare to a standard 10M parameter vision model? +- [ ] In which deployment tier (Tier 1, 2, or 3) are **Residual Adapters** preferred over LoRA, and why? +- [ ] True or False: Transmitting adapter weights in federated learning consumes more bandwidth than transmitting the entire model. + +::: + #### Performance vs. Resource Trade-offs {#sec-edge-intelligence-performance-vs-resource-tradeoffs-2be2} Selecting the right adaptation strategy requires quantitative analysis of the resource-expressivity spectrum. For a 10M parameter model, the trade-offs are concrete. Bias-only updates are the most efficient, requiring just 50 KB of trainable parameters and negligible compute overhead, but they struggle to adapt to structural domain shifts like changing sensor geometries or new object categories. At the other extreme, full fine-tuning offers maximum expressivity but demands 40 MB or more of gradients and optimizer states, making it infeasible for background training on typical mobile devices. Low-rank techniques (LoRA) and residual adapters occupy the strategic middle ground: a rank-16 LoRA configuration requires approximately 2 MB of trainable memory, while a standard residual adapter needs approximately 5 MB. @@ -1457,7 +1633,7 @@ The decision between LoRA and residual adapters often hinges on inference latenc Implementing these adaptation techniques requires system-level support for dynamic computation graphs and the ability to selectively inject trainable parameters. Not all deployment environments or inference engines support such capabilities natively. TensorFlow Lite and ONNX Runtime Mobile provide increasingly robust support for adapter-style architectures, but custom inference stacks on microcontrollers may require manual implementation of the adapter forward pass. -System architects should apply a device-tier framework to this decision [@quinonero2009dataset]. On flagship phones (Tier 1) with dedicated Neural Engines and 8+ GB RAM, residual adapters provide the best balance of modularity and speed, enabling context-aware switching with minimal latency impact. On mid-range devices (Tier 2) with 4--6 GB RAM, LoRA is preferred because weight merging eliminates runtime overhead entirely. For ultra-constrained IoT endpoints (Tier 3) with less than 1 GB RAM, bias-only updates remain the only viable option. This tiered approach ensures that the adaptation mechanism matches the physical reality of the hardware, preventing thermal throttling while delivering meaningful personalization at each capability level. +System architects should apply a device-tier framework to this decision [@quinonero2009dataset]. On flagship phones (Tier 1) with dedicated Neural Engines and 8+ GB RAM, residual adapters provide the best balance of modularity and speed, enabling context-aware switching with minimal latency impact. On mid-range devices (Tier 2) with 4--`{python} EdgeIntelligenceSetup.smartphone_ram_gb}` GB RAM, LoRA is preferred because weight merging eliminates runtime overhead entirely. For ultra-constrained IoT endpoints (Tier 3) with less than 1 GB RAM, bias-only updates remain the only viable option. This tiered approach ensures that the adaptation mechanism matches the physical reality of the hardware, preventing thermal throttling while delivering meaningful personalization at each capability level. ### Sparse Updates {#sec-edge-intelligence-sparse-updates-879b} @@ -1728,6 +1904,58 @@ Federated learning emerges as the solution to distributed coordination constrain ::: +```{python} +#| label: federated-savings-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ FEDERATED COMMUNICATION SAVINGS (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Model Updates vs Raw Data" .callout-notebook +# │ +# │ Goal: Quantify the bandwidth savings of federated learning for a wearable. +# │ Show: bandwidth_reduction ≈ 78x — inline in notebook result. +# │ How: Ratio = Data_Raw / Data_Model_Updates. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: fs_raw_data_mb_str, fs_update_mb_str, fs_reduction_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class FederatedSavings: + # ┌── 1. LOAD ────────────────────────────────────────── + n_images = 1000 # Images collected by user per week + img_size_kb = 200 # Compressed JPEG + model_params_m = 5 + compression_ratio = 10 # Quantized updates (e.g. 4-bit) + # ┌── 2. EXECUTE ─────────────────────────────────────── + raw_data_mb = (n_images * img_size_kb) / 1024 + # Update size: 5M params * 4 bits = 2.5 MB + update_mb = (model_params_m * 4) / 8 + reduction = raw_data_mb / update_mb + # ┌── 3. GUARD ───────────────────────────────────────── + check(70 < reduction < 90, f"Reduction {reduction:.1f}x unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + fs_raw_data_mb_str = f"{raw_data_mb:.0f}" + fs_update_mb_str = f"{update_mb:.1f}" + fs_reduction_str = f"{reduction:.0f}" + +``` + +::: {.callout-notebook title="Model Updates vs Raw Data"} + +**Problem**: You are designing a federated health monitor that learns from `{python} FederatedSavings.fs_raw_data_mb_str` MB of user sensor data per week. Should you upload the raw data to the cloud for training, or use **Federated Learning** to send model updates instead? + +**The Math**: +Bandwidth efficiency is the ratio of raw data volume to model update size. + +1. **Raw Data Upload**: **`{python} FederatedSavings.fs_raw_data_mb_str` MB/week**. +2. **Federated Update**: A compressed model update (5M params) is only **`{python} FederatedSavings.fs_update_mb_str` MB**. +3. **Bandwidth Reduction**: 195 MB / 2.5 MB = **`{python} FederatedSavings.fs_reduction_str`$\times$ savings**. + +**The Systems Insight**: Federated learning is a **Network Multiplier**. For a user on a limited cellular plan, uploading 200 MB of raw data is expensive and slow. Uploading a **`{python} FederatedSavings.fs_update_mb_str` MB update** is nearly invisible. By shifting the "Compute to the Data," you reduce the network load by **`{python} FederatedSavings.fs_reduction_str`$\times$**, enabling continuous learning even in bandwidth-constrained environments. In the Machine Learning Fleet, this is how you scale to billions of users without bankrupting your networking budget or violating user privacy. + +::: + @fig-learning-paradigms contrasts federated learning with other learning paradigms to clarify its unique position. In traditional offline learning, all data is collected and processed centrally. The model is trained in the cloud using curated datasets and is then deployed to edge devices without further adaptation. In contrast, on-device learning allows local model adaptation using data generated on the device itself, supporting personalization but in isolation, without sharing insights across users. Federated learning bridges these two extremes by enabling localized training while coordinating updates globally. It retains data privacy by keeping raw data local, yet benefits from distributed model improvements by aggregating updates from many devices. ::: {#fig-learning-paradigms fig-env="figure" fig-pos="htb" fig-cap="**Edge Learning Paradigms**: Three approaches to model training at the edge. (Left) Offline learning trains in the cloud and deploys static models. (Center) On-device learning adapts models locally using device data. (Right) Federated learning coordinates distributed training while keeping data local." fig-alt="Three-panel comparison of learning paradigms. Left: offline learning with cloud training. Center: on-device learning with local model adaptation. Right: federated learning with model updates exchanged between devices and central server."} diff --git a/book/quarto/contents/vol2/fault_tolerance/fault_tolerance.qmd b/book/quarto/contents/vol2/fault_tolerance/fault_tolerance.qmd index 432945811d..7326a01c7a 100644 --- a/book/quarto/contents/vol2/fault_tolerance/fault_tolerance.qmd +++ b/book/quarto/contents/vol2/fault_tolerance/fault_tolerance.qmd @@ -777,6 +777,17 @@ Text/.style={% ``` ::: +::: {.callout-checkpoint title="Detecting Silent Corruption"} + +Verify your understanding of non-stop failures and SDC: + +- [ ] Why is **Silent Data Corruption (SDC)** more dangerous for ML training than a simple node crash? +- [ ] How does a **Hot Spare** redundancy strategy differ from a traditional **Checkpoint-Restart** approach in terms of recovery latency? +- [ ] In the Google case study, which metric was used as a "Sanity Checker" to identify potential hardware faults: GPU Temperature or Gradient Norm? +- [ ] True or False: Byzantine failures only occur when a malicious actor intentionally sabotages the training cluster. + +::: + Byzantine failures include: - **Silent data corruption**: Memory or computation errors produce wrong values without triggering errors @@ -854,8 +865,18 @@ Defending against correlated failures requires understanding failure domains and : **Failure Domains in ML Infrastructure**: Understanding failure domain boundaries enables placement of redundant components across independent domains, preventing correlated failures from defeating redundancy strategies. {#tbl-failure-domains} -::: {#fig-failure-domains fig-env="figure" fig-pos="htb" fig-cap="**Hierarchy of Failure Domains**. Failure domains are often nested or overlapping. A GPU failure affects one device. A node failure affects 8 GPUs. A rack switch failure affects 32-64 GPUs. A power distribution unit (PDU) failure may affect multiple racks. Effective fault tolerance requires placing replicas across independent domains (e.g., different racks or rows) to survive correlated failures." fig-alt="Nested rectangles showing failure domain hierarchy. Region contains Zone A and Zone B. Each zone contains a rack with switch and PDU. Each rack contains a node with OS and PCIe. Each node contains multiple GPUs. Annotation explains containment."} -```{.tikz} +::: {.callout-checkpoint title="Mapping Failure Domains"} + +Verify your understanding of how failure domains nest and their operational impact: + +- [ ] If a rack switch fails, how many 8-GPU nodes are typically affected in a standard datacenter configuration? +- [ ] Why does placing all 3 model replicas in the same Zone (e.g., `us-east-1a`) fail to provide protection against a regional power outage? +- [ ] Which failure domain is uniquely addressed by **Staged Rollouts** rather than hardware redundancy? +- [ ] Can you explain the concept of **Hierarchical Containment**? If a Zone fails, does the corresponding Rack failure domain still matter for that event? + +::: + +::: {#fig-failure-domains fig-env="figure" fig-pos="htb" fig-cap="**Hierarchy of Failure Domains**. Failure domains are often nested or overlapping. A GPU failure affects one device. A node failure affects 8 GPUs. A rack switch failure affects 32-64 GPUs. A power distribution unit (PDU) failure may affect multiple racks. Effective fault tolerance requires placing replicas across independent domains (e.g., different racks or rows) to survive correlated failures." fig-alt="Nested rectangles showing failure domain hierarchy. Region contains Zone A and Zone B. Each zone contains a rack with switch and PDU. Each rack contains a node with OS and PCIe. Each node contains multiple GPUs. Annotation explains containment."}```{.tikz} \begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, node distance=0.5cm and 0.5cm] \tikzset{ Region/.style={draw=GrayLine, fill=GrayL, line width=0.75pt, inner sep=10pt}, @@ -918,7 +939,7 @@ The practical implication for ML systems is that fleet-wide failure rates depend ::: {#fig-bathtub-curve fig-env="figure" fig-pos="htb" fig-cap="**The Bathtub Curve**. Hardware failure rates $\lambda(t)$ vary over time. (1) **Infant Mortality**: High failure rate initially due to manufacturing defects. (2) **Useful Life**: Constant, low failure rate where random failures dominate. (3) **Wear-Out**: Increasing failure rate as components age. Burn-in testing aims to filter out infant mortality failures before deployment." fig-alt="Line graph of failure rate versus component age showing bathtub shape. Three phases: infant mortality with high decreasing rate, useful life with constant low rate, and wear-out with increasing rate. Vertical dashed line marks burn-in period."} ```{.tikz} -\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, +\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, node distance=1cm and 1cm, RedLine/.style={draw=red, thick}, BlueLine/.style={draw=blue, thick}, @@ -1963,9 +1984,60 @@ plt.show() The U-shaped cost curve in @fig-young-daly-storage visualizes this trade-off: checkpoint overhead decreases as intervals lengthen, while rework cost from failures increases linearly. The optimal point minimizes their sum. -For our 10,000 GPU cluster with calculated system \text{MTBF} of 3.69 hours (from @sec-fault-tolerance-reliability-reliability-worked-example-cluster-mtbf-calculation-9255) and checkpoint time of 21 seconds (from `{python} FaultToleranceSetup.gpt3_ckpt_tb` TB checkpoint at 100 GB/s), @eq-young-daly-applied computes the optimal interval: +For our 10,000 GPU cluster with calculated system \text{MTBF} of 3.69 hours (from @sec-fault-tolerance-reliability-reliability-worked-example-cluster-mtbf-calculation-9255) and checkpoint time of 21 seconds (from `{python} FaultToleranceSetup.gpt3_ckpt_tb` TB checkpoint at 100 GB/s), @eq-young-daly-applied computes the optimal interval. -$$ T_{\text{opt}} = \sqrt{2 \times 21s \times 3.69hr \times 3600s/hr} \approx 12.4 \text{ minutes} $$ {#eq-young-daly-applied} +```{python} +#| label: young-daly-optimal-interval +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ YOUNG-DALY OPTIMAL INTERVAL (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Young-Daly Optimal Interval" .callout-notebook +# │ +# │ Goal: Quantify the optimal checkpoint interval for a 10K-GPU cluster +# │ to minimize wasted work using the square-root law. +# │ Show: tau_opt_min_str ≈ 12.4 min — inline in notebook steps. +# │ How: τ = sqrt(2 * T_save * MTBF); bare arithmetic for clarity. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: yd_t_save_s_str, yd_mtbf_h_str, yd_tau_opt_min_str, yd_overhead_pct_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check +import math + +class YoungDalyOptimal: + # ┌── 1. LOAD ────────────────────────────────────────── + t_save_s = 21 + mtbf_h = 3.69 + # ┌── 2. EXECUTE ─────────────────────────────────────── + mtbf_s = mtbf_h * 3600 + tau_opt_s = math.sqrt(2 * t_save_s * mtbf_s) + tau_opt_min = tau_opt_s / 60 + overhead_pct = (t_save_s / tau_opt_s + tau_opt_s / (2 * mtbf_s)) * 100 + # ┌── 3. GUARD ───────────────────────────────────────── + check(10 < tau_opt_min < 15, f"Interval {tau_opt_min:.1f}m outside expected range") + # ┌── 4. OUTPUT ──────────────────────────────────────── + t_save_s_str = f"{t_save_s}" + mtbf_h_str = f"{mtbf_h}" + tau_opt_min_str = f"{tau_opt_min:.1f}" + overhead_pct_str = f"{overhead_pct:.1f}" + +``` + +::: {.callout-notebook title="The Young-Daly Optimal Interval"} + +**Problem**: Your 10,000-GPU cluster has an MTBF of `{python} YoungDalyOptimal.mtbf_h_str` hours. A full model checkpoint takes `{python} YoungDalyOptimal.t_save_s_str` seconds to write. What is the optimal checkpoint frequency? + +**The Math**: +Apply the Young-Daly formula: $\tau_{\text{opt}} = \sqrt{2 \cdot T_{\text{save}} \cdot \text{MTBF}}$ + +1. **Convert to common units**: MTBF = `{python} YoungDalyOptimal.mtbf_h_str` $\times$ 3,600s = `{python} f"{3.69 * 3600:,.0f}"` seconds. +2. **Calculate**: $\tau_{\text{opt}} = \sqrt{2 \cdot 21 \cdot 13,284} = \sqrt{557,928} \approx$ `{python} f"{math.sqrt(2 * 21 * 13284):,.0f}"` seconds. +3. **Result**: $\tau_{\text{opt}} \approx$ **`{python} YoungDalyOptimal.tau_opt_min_str` minutes**. + +**The Systems Insight**: At this interval, the "Checkpoint Tax" (time spent saving + time spent re-computing) is minimized to approximately **`{python} YoungDalyOptimal.overhead_pct_str`%**. If you checkpointed every hour instead, your failure risk would increase, wasting nearly 10% of your cluster's capacity. As clusters scale, the optimal interval *must* shrink to keep up with the falling MTBF. + +::: This result demonstrates why failure analysis matters: without knowing the system \text{MTBF}, we cannot set checkpoint intervals rationally. With this interval, checkpoint overhead consumes approximately 2.8% of training time. However, practitioners should be aware of the *Young-Daly formula assumptions* that underpin this estimate. @@ -2905,11 +2977,11 @@ When full redundancy fails—such as when an edge device loses connectivity, or ::: {.callout-definition title="Graceful Degradation"} -***Graceful Degradation***\index{Graceful Degradation!definition} is the systematic reduction of model quality, latency, or coverage to maintain system **Availability** under conditions that exceed primary fault tolerance limits. +***Graceful Degradation***\index{Graceful Degradation!definition} is a fault tolerance strategy in which a system responds to resource exhaustion or component failure by deliberately reducing service quality — falling back to a smaller model, serving cached results, or returning partial outputs — rather than failing completely, maintaining measurable availability at reduced capability. -1. **Significance (Quantitative):** It allows the system to remain functional by trading **Model Accuracy** for lower resource requirements ($R_{\text{peak}}, BW$). For example, a system may fall back to a smaller model or use fewer features to maintain its **Inference SLO** ($L_{\text{lat}}$) during high load. -2. **Distinction (Durable):** Unlike **System Failure** (total loss of service), Graceful Degradation is a **Managed Transition** where the system continues to provide "best-effort" results through secondary mechanisms (e.g., cached predictions). -3. **Common Pitfall:** A frequent misconception is that degradation is "automatic." In reality, it is an **Architectural Choice**: it requires pre-computed fallback models and logic that can detect failure and switch execution paths without user intervention. +1. **Significance (Quantitative):** Graceful degradation converts total outage risk into a controlled quality reduction. A recommendation system that falls back from a 7B-parameter ranker to a collaborative-filtering model on GPU failure may reduce click-through rate by 8–15% but maintains 100% request availability versus 0% during a hard failure. At \$1M/day in revenue dependent on recommendations, an 8% CTR reduction (\$80K/day degraded) is sharply preferable to a 10-minute outage (\$7K/minute lost). +2. **Distinction (Durable):** Unlike complete system failure (where the service returns errors to all requests), graceful degradation provides a managed transition through a predefined capability hierarchy — each fallback level has known accuracy, latency, and resource characteristics that allow the system to continue satisfying SLOs at reduced quality. +3. **Common Pitfall:** A frequent misconception is that degradation is automatic in well-designed systems. Graceful degradation requires explicit pre-engineering: fallback models must be pre-loaded or pre-computed, switchover logic must detect the failure condition and trigger the transition without human intervention, and each degradation level must have been validated to actually satisfy its reduced SLO before it is needed in production. ::: diff --git a/book/quarto/contents/vol2/fleet_orchestration/fleet_orchestration.qmd b/book/quarto/contents/vol2/fleet_orchestration/fleet_orchestration.qmd index 91d29919ab..c1ee059894 100644 --- a/book/quarto/contents/vol2/fleet_orchestration/fleet_orchestration.qmd +++ b/book/quarto/contents/vol2/fleet_orchestration/fleet_orchestration.qmd @@ -194,7 +194,59 @@ To make the scheduling problem concrete, consider a research organization operat The economic stakes make these decisions consequential at every scale. A `{python} ClusterEconomicsContext.cluster_size_str`-GPU cluster at $`{python} f"{ClusterEconomicsContext.gpu_hour_cost_usd:.0f}"` per GPU-hour costs $`{python} ClusterEconomicsContext.daily_cost_str` per day to operate, whether the GPUs are computing useful work or sitting idle in a queue. If scheduling inefficiencies leave `{python} f"{ClusterEconomicsContext.idle_fraction*100:.0f}"` percent of GPUs idle, that translates to $`{python} ClusterEconomicsContext.daily_waste_str` per day in wasted capacity, or over $`{python} ClusterEconomicsContext.annual_waste_str` million annually. Conversely, improving utilization from `{python} f"{ClusterEconomicsContext.util_low*100:.0f}"` percent to `{python} f"{ClusterEconomicsContext.util_high*100:.0f}"` percent effectively adds `{python} ClusterEconomicsContext.equivalent_gpus_str` GPUs worth of productive capacity without purchasing additional hardware. At this scale, a one-percentage-point improvement in utilization is worth more annually than the salary of the engineer who achieves it. Scheduling is not operational overhead; it is one of the highest-leverage engineering investments in ML infrastructure. -ML workloads present scheduling challenges that distinguish them fundamentally from traditional high-performance computing and cloud computing. **Gang scheduling**[^fn-gang-scheduling] represents the most critical difference: a distributed training job requiring 1,024 GPUs cannot make progress with only 512. Unlike traditional HPC simulations that can often scale to whatever resources are available, synchronous data parallelism demands all-or-nothing allocation. Every worker must participate in every AllReduce operation, and a missing worker blocks all others. A scheduler that partially allocates resources creates deadlocks where multiple jobs each hold some GPUs while waiting for more, with none able to proceed. This all-or-nothing requirement means the scheduler cannot simply "pack jobs tightly" as a traditional bin packer would; it must reason about atomic, multi-resource allocations across the entire cluster. Beyond gang scheduling, topology awareness via **placement groups**[^fn-placement-groups] adds a second hard constraint: to support tensor parallelism, the scheduler must ensure GPUs are placed within the same high-bandwidth NVLink domain rather than scattered across the datacenter. Tools like Slurm (from the HPC world) and Kubernetes (from the cloud-native world) are the two primary vehicles for implementing these policies, and each makes different trade-offs that shape how these constraints are enforced. +ML workloads present scheduling challenges that distinguish them fundamentally from traditional high-performance computing and cloud computing. **Gang scheduling**[^fn-gang-scheduling] represents the most critical difference: a distributed training job requiring 1,024 GPUs cannot make progress with only 512. Unlike traditional HPC simulations that can often scale to whatever resources are available, synchronous data parallelism demands all-or-nothing allocation. Every worker must participate in every AllReduce operation, and a missing worker blocks all others. A scheduler that partially allocates resources creates deadlocks where multiple jobs each hold some GPUs while waiting for more, with none able to proceed. This all-or-nothing requirement means the scheduler cannot simply "pack jobs tightly" as a traditional bin packer would; it must reason about atomic, multi-resource allocations across the entire cluster. + +```{python} +#| label: gang-scheduling-deadlock-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ GANG SCHEDULING DEADLOCK (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Physics of Deadlock" .callout-notebook +# │ +# │ Goal: Quantify the risk of cluster deadlock without gang scheduling. +# │ Show: deadlock_prob_pct ≈ 100% — inline in notebook result. +# │ How: P(allocation) = (Free_Gpus / Req_Gpus)^N_Jobs. (Conceptual) +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: gs_n_gpus_str, gs_n_jobs_str, gs_waste_m_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class GangSchedulingDeadlock: + # ┌── 1. LOAD ────────────────────────────────────────── + n_total_gpus = 1024 + n_jobs = 4 + gpus_per_job = 512 # Each job needs half the cluster + # ┌── 2. EXECUTE ─────────────────────────────────────── + # If scheduler allocates partially (e.g. 256 to each), no job can run. + # Total idle waste per hour if deadlock occurs: + gpu_hour_cost = 2.0 # $/hr + waste_per_hour = n_total_gpus * gpu_hour_cost + # ┌── 3. GUARD ───────────────────────────────────────── + check(waste_per_hour == 2048, f"Waste ${waste_per_hour} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + gs_n_gpus_str = f"{n_total_gpus}" + gs_n_jobs_str = f"{n_jobs}" + gs_waste_m_str = f"{waste_per_hour:,.0f}" + +``` + +::: {.callout-notebook title="The Physics of Deadlock"} + +**Problem**: You have a `{python} GangSchedulingDeadlock.gs_n_gpus_str`-GPU cluster. Two teams each submit a job requiring 512 GPUs. A naive scheduler (non-gang) allocates 256 GPUs to Team A and 256 to Team B, then waits for more GPUs to become available. What happens next? + +**The Math**: +1. **Team A Status**: Holds 256, Needs 512. Progress = **0%**. +2. **Team B Status**: Holds 256, Needs 512. Progress = **0%**. +3. **Cluster Status**: 512 GPUs allocated, 0 samples processed. +4. **Waste**: `{python} GangSchedulingDeadlock.gs_waste_m_str` dollars per hour in idle electricity and capital. + +**The Systems Insight**: This is a **Circular Dependency**: Team A is waiting for Team B's GPUs, and Team B is waiting for Team A's. Without **Gang Scheduling** (all-or-nothing allocation), your cluster efficiency drops to zero. In a large fleet, even a 5-minute deadlock can cost thousands of dollars. Schedulers like Kubernetes (via Coscheduling plugins) and Slurm enforce atomicity to ensure that if a job can't run in full, it doesn't run at all. + +::: + +Beyond gang scheduling, topology awareness via **placement groups**[^fn-placement-groups] adds a second hard constraint: to support tensor parallelism, the scheduler must ensure GPUs are placed within the same high-bandwidth NVLink domain rather than scattered across the datacenter. Tools like Slurm (from the HPC world) and Kubernetes (from the cloud-native world) are the two primary vehicles for implementing these policies, and each makes different trade-offs that shape how these constraints are enforced. [^fn-gang-scheduling]: **Gang Scheduling**: Formalized by John Ousterhout in 1982 for parallel systems, where the "Ousterhout matrix" co-schedules related threads across processors in synchronized time slices. In ML clusters, the constraint is stricter: synchronous AllReduce requires *all* workers simultaneously, so partial allocation wastes 100% of held resources rather than merely degrading throughput. A 1,024-GPU job holding 900 GPUs while waiting for 124 more burns approximately $1,800/hour in idle capacity. \index{Gang Scheduling!scheduling} @@ -783,7 +835,7 @@ The scheduler must therefore model the cluster not as a flat list of compute slo \begin{scope}[shift={(6,0)}, local bounding box=rightScope] % Node A \node[switch, fill=GreenL, minimum width=1.2cm] (nvsw_a) {NVSw}; - + \node[gpu, fill=GreenL, below left=0.3cm and 0.1cm of nvsw_a] (sg1) {G1}; \node[gpu, fill=GreenL, left=0.2cm of sg1] (sg0) {G0}; \node[gpu, fill=GreenL, right=0.2cm of sg1] (sg2) {G2}; @@ -800,7 +852,7 @@ The scheduler must therefore model the cluster not as a flat list of compute slo % Node B \node[switch, fill=GreenL, minimum width=1.2cm, below=1.5cm of nvsw_a] (nvsw_b) {NVSw}; - + \node[gpu, fill=RedL, below left=0.3cm and 0.1cm of nvsw_b] (sg5) {G5}; \node[gpu, fill=RedL, left=0.2cm of sg5] (sg4) {G4}; \node[gpu, fill=RedL, right=0.2cm of sg5] (sg6) {G6}; @@ -1156,7 +1208,7 @@ Consider a large language model training job with a target batch size that requi **Strategy 2 -- Wait for capacity (gang scheduling)**: The job waits in the queue for 4 hours until 512 GPUs become available, then runs for 20 hours. Total work: $20 \times 3.2 =$ `{python} ElasticScaling.es_s2_str` epochs. -**Strategy 3 -- Elastic scaling (step-up)**: The job starts immediately with 256 GPUs. After 4 hours, 256 more GPUs become available. The job pauses, scales up, and resumes. Phase 1: $4 \times 1.8 =$ `{python} ElasticScaling.es_s3p1_str` epochs. Phase 2: $(24 - 4 - `{python} ElasticScaling.es_rescale_str`) \times 3.2 \approx$ `{python} ElasticScaling.es_s3p2_str` epochs. Total work: $`{python} ElasticScaling.es_s3p1_str` + `{python} ElasticScaling.es_s3p2_str` =$ `{python} ElasticScaling.es_s3_str` epochs. +**Strategy 3 -- Elastic scaling (step-up)**: The job starts immediately with 256 GPUs. After 4 hours, 256 more GPUs become available. The job pauses, scales up, and resumes. Phase 1: $4 \times 1.8 =$ `{python} ElasticScaling.es_s3p1_str` epochs. Phase 2: $(24 - 4 -$ `{python} ElasticScaling.es_rescale_str`$) \times 3.2 \approx$ `{python} ElasticScaling.es_s3p2_str` epochs. Total work: `{python} ElasticScaling.es_s3p1_str` + `{python} ElasticScaling.es_s3p2_str` = `{python} ElasticScaling.es_s3_str` epochs. The elastic strategy outperforms waiting by approximately 10 percent and the immediate small-scale run by nearly 3$\times$. However, if the re-scaling cost exceeded 2 hours (due to slow checkpoint transfer or compilation), Strategy 2 would become superior. @@ -1883,8 +1935,8 @@ The **Fleet Stack** framework (@sec-vol2-introduction) provides a structured app **Infrastructure Layer Analysis**: The cluster contains heterogeneous hardware acquired over three procurement cycles: -| **GPU Type** | **Count** | **Memory** | **Interconnect** | **Nodes** | -|:--------------|----------:|:------------|:--------------------|:------------------------| +| **GPU Type** | **Count** | **Memory** | **Interconnect** | **Nodes** | +|:---------------|----------:|:------------|:--------------------|:------------------------| | **A100-80 GB** | 400 | 80 GB HBM2e | NVLink (600 GB/s) | 50 nodes$\times$ 8 GPUs | | **A100-40 GB** | 400 | 40 GB HBM2e | NVLink (600 GB/s) | 50 nodes$\times$ 8 GPUs | | **V100-32 GB** | 200 | 32 GB HBM2 | PCIe Gen3 (32 GB/s) | 50 nodes$\times$ 4 GPUs | @@ -1964,14 +2016,29 @@ class DebugValueCalc: | **Pool** | **Before** | **After** | **Change** | |:-----------------|:-----------------------------|:-----------------------------|:--------------| -| **A100-80 GB** | 95% allocated, 72% effective | 89% allocated, 94% effective | +5% effective | -| **A100-40 GB** | 64% allocated | 88% allocated | +24% | +| **A100-80 GB** | 95% allocated, 72% effective | 89% allocated, 94% effective | +5% effective | +| **A100-40 GB** | 64% allocated | 88% allocated | +24% | | **V100** | 0% allocated | 71% allocated | +71% | | **Cluster-wide** | **60%** | **84%** | **+40%** | : **Utilization Recovery Results**: The distinction between "allocated" and "effective" utilization captures the gang scheduling deadlock: before the fix, A100-80 GB GPUs showed high allocation in Slurm but low actual compute utilization because allocated jobs could not start. {#tbl-fleet-orchestration-debug-results} -This 40% improvement in effective cluster capacity equals approximately `{python} DebugValueCalc.debug_recovered_str` additional GPUs worth of productive work. At \$2 per GPU-hour, this represents approximately \$`{python} DebugValueCalc.debug_annual_str` million in annual recovered value, achieved through policy changes requiring zero additional hardware investment. +This 40% improvement in effective cluster capacity equals approximately `{python} DebugValueCalc.debug_recovered_str` additional GPUs worth of productive work. + +::: {.callout-notebook title="The Value of a Profiler"} + +**Problem**: You manage a `{python} DebugValueCalc.debug_cluster`-GPU cluster. Your initial monitoring shows 60% average utilization. After a week of systematic debugging (identifying stragglers, optimizing AllReduce buckets, and tuning data loaders), your utilization increases to 84%. What is the financial value of that one week of engineering work? + +**The Math**: +1. **Capacity Recovery**: (0.84 - 0.60) / 0.60 = **40% increase in productive throughput**. +2. **Recovered Hardware**: 40% of 1,000 GPUs = **`{python} DebugValueCalc.debug_recovered_str` GPUs**. +3. **Annual Value**: `{python} DebugValueCalc.debug_recovered_str` GPUs $\times$ \$2/hr $\times$ 8,760 hrs/year $\approx$ **\$`{python} DebugValueCalc.debug_annual_str` Million**. + +**The Systems Insight**: Debugging is not a "maintenance task"; it is a **Capacity Generator**. Increasing utilization from 60% to 84% has the same business impact as buying **400 new GPUs** (costing ~$12 Million in CapEx). In the Machine Learning Fleet, performance engineering is often the most cost-effective way to scale—it is cheaper to make your existing GPUs 40% faster than to buy 40% more hardware. + +::: + +At \$2 per GPU-hour, this represents approximately \$`{python} DebugValueCalc.debug_annual_str` million in annual recovered value, achieved through policy changes requiring zero additional hardware investment. **The Fleet Stack Lesson**: Surface-level diagnosis suggested a scheduling algorithm problem (Distribution Layer), perhaps requiring a more sophisticated scheduler like those discussed in @sec-fleet-orchestration-custom-schedulers. Management initially proposed evaluating Pollux or a custom scheduling solution, which would have required months of engineering effort and introduced operational risk. Deeper analysis revealed the root cause as a **mismatch between Infrastructure Layer heterogeneity and Distribution Layer policies designed for homogeneous infrastructure**. The scheduler algorithm was not the problem; the policies it implemented were the problem. The scheduler correctly implemented its configured policy; the policy itself created the utilization gap. diff --git a/book/quarto/contents/vol2/inference/inference.qmd b/book/quarto/contents/vol2/inference/inference.qmd index 896b0039c9..932f549a25 100644 --- a/book/quarto/contents/vol2/inference/inference.qmd +++ b/book/quarto/contents/vol2/inference/inference.qmd @@ -145,6 +145,17 @@ The perceived responsiveness of interactive systems depends on both **Time to Fi : **Triggers for Distributed Inference**. Each constraint type indicates different distribution strategies. Memory constraints require model sharding; throughput constraints require replication; latency constraints may require either depending on whether the bottleneck is compute or memory bandwidth. {#tbl-distribution-triggers} +::: {.callout-checkpoint title="Distribution Strategy Selection"} + +Verify your understanding of when to move from single-machine to distributed inference: + +- [ ] A 13B model (26 GB weights) fits on a single A100. If the throughput requirements double, should you use model sharding or horizontal replication? +- [ ] For a real-time speech-to-text service, the primary bottleneck is **inter-token latency**. Does adding more GPUs through data-parallel replication reduce this latency? +- [ ] Why is **Model Sharding** mandatory for a 175B model on 80 GB GPUs, regardless of the request volume? +- [ ] Identify the trade-off: what is the "Serving Tax" mentioned in the next section, and how does it affect the decision to shard across machines versus staying within a single node? + +::: + ### The Fundamental Inversion: Training vs Inference {#sec-inference-scale-fundamental-inversion-training-vs-inference-8834} The contrast between training and inference optimization extends beyond the basic throughput-versus-latency distinction. Training optimizes for samples processed per hour and tolerates latency variations. Inference optimizes for response time and must meet strict latency bounds. At scale, this inversion manifests in system architecture, resource allocation, and operational priorities. @tbl-training-inference-inversion details six key system aspects where these differences emerge. @@ -315,13 +326,16 @@ class InferenceEconomics: - Efficiency: `{python} InferenceEconomics.cost_per_million_str` dollars per million queries **Lifetime Volume ($Q_{total}$)** -$$ Q_{total} = `{python} InferenceEconomics.qps_str` \text{/s} \times 86,400 \text{ s/day} \times `{python} InferenceEconomics.serve_duration_days_str` \text{ days} \approx \mathbf{`{python} InferenceEconomics.lifetime_queries_latex_str` \text{ queries}} $$ +$Q_{total} \approx$ **`{python} InferenceEconomics.lifetime_queries_latex_str` queries** +(`{python} InferenceEconomics.qps_str` QPS $\times$ 86,400 s/day $\times$ `{python} InferenceEconomics.serve_duration_days_str` days) **Total Serving Cost** -$$ C_{serve\_total} = `{python} InferenceEconomics.lifetime_queries_latex_str` \times \$10^{-5} = \mathbf{`{python} InferenceEconomics.serving_cost_total_str`} $$ + +$C_{serve\_total} =$ `{python} InferenceEconomics.lifetime_queries_latex_str` $\times \$10^{-5} =$ **`{python} InferenceEconomics.serving_cost_total_str`** **Conclusion**: -$$ \text{Ratio} = \frac{`{python} InferenceEconomics.serving_cost_total_str`}{`{python} InferenceEconomics.training_cost_total_str`} \approx \mathbf{`{python} InferenceEconomics.leverage_ratio_str`\times} $$ + +Ratio = `{python} InferenceEconomics.serving_cost_total_str` / `{python} InferenceEconomics.training_cost_total_str` $\approx$ **`{python} InferenceEconomics.leverage_ratio_str`$\times$** Serving optimization leverage is `{python} InferenceEconomics.leverage_ratio_str`$\times$ higher than training optimization. A 1% serving efficiency gain saves significant capital—far exceeding the entire training budget. ::: @@ -577,6 +591,17 @@ Each level has distinct optimization levers. @tbl-serving-hierarchy maps these l : **Serving Hierarchy Optimization Targets**. Each level of the hierarchy addresses different metrics with different techniques. {#tbl-serving-hierarchy} +::: {.callout-checkpoint title="The Serving Hierarchy"} + +Verify your understanding of where specific optimizations sit within the serving hierarchy: + +- [ ] Which level of the hierarchy is responsible for managing **KV cache fragmentation** to increase concurrent request capacity? +- [ ] If you implement **Speculative Decoding** to reduce token latency for a single request, at which level are you operating? +- [ ] **Load Balancing** algorithms (like Power-of-Two-Choices) operate at the Service level. What is their primary optimization target? +- [ ] True or False: Multi-tenancy and resource isolation are primarily Request-level concerns. + +::: + The remainder of this chapter progresses through these levels: batching and caching (request level), model sharding (replica level), load balancing and autoscaling (service level), and multi-tenancy (platform level). ## Serving Architecture Dimensions {#sec-inference-scale-serving-framework-selection-45d3} @@ -649,6 +674,17 @@ The choice between stateless and stateful serving is not a framework feature but : **Serving Architecture Dimensions by Workload Type**. Each workload's constraint profile determines the optimal design point along five architectural dimensions. Selecting a serving system amounts to choosing the combination that matches the workload's constraints. {#tbl-serving-architecture-dimensions} +::: {.callout-checkpoint title="Serving Dimensions"} + +Verify your understanding of how workload constraints drive architectural choices: + +- [ ] Why is **Preemptive Scheduling** more critical for LLMs than for Vision models? +- [ ] For which workload type is **Stateful Serving** (requiring sticky routing) a fundamental requirement? +- [ ] Explain why **Feature-Parallel Batching** is the standard for Recommendation systems but not for LLMs. +- [ ] Which dimension (Batching, Memory, Scheduling, Topology, State) is primarily affected by the **outlier features** problem in quantization? + +::: + These architectural dimensions determine which serving system to select for a given workload. Rather than comparing frameworks feature by feature, an engineer should identify the workload's position along each dimension -- its batching pattern, memory profile, scheduling requirements, deployment topology, and statefulness -- and select the system that matches. The techniques developed in the remainder of this chapter (batching, KV cache management, model sharding, load balancing, autoscaling) operate within this architectural framework, each addressing optimization at a specific dimension. ## Batching Strategies at Scale {#sec-inference-scale-batching-strategies-scale-6733} @@ -749,7 +785,72 @@ This relationship reveals the **Batching Efficiency Curve**: 2. **Large $B$**: As $B \to \infty$, the $T_{\text{fixed}}$ term becomes negligible. Throughput asymptotically approaches the hardware limit $1/T_{\text{variable}}$. The system becomes **compute-bound** (or bandwidth-bound for LLMs). 3. **The Knee**: The optimal batch size is the point where throughput gains diminish while latency continues to grow linearly. -The engineering goal is to find the maximum $B$ such that $L(B) \le \text{SLO}$. This formulation explains why vision models (high $T_{\text{variable}}$) saturate at smaller batches than LLMs (high $T_{\text{fixed}}$ due to weight loading), requiring different tuning strategies. Translating these batch-level equations into system-wide capacity planning requires a classical result from queuing theory: **Little's Law**[^fn-littles-law], which relates concurrency, throughput, and latency in any stable system. @tbl-batching-by-model summarizes how these batching characteristics vary across model architectures. +The engineering goal is to find the maximum $B$ such that $L(B) \le \text{SLO}$. This formulation explains why vision models (high $T_{\text{variable}}$) saturate at smaller batches than LLMs (high $T_{\text{fixed}}$ due to weight loading), requiring different tuning strategies. + +```{python} +#| label: batching-efficiency-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ BATCHING EFFICIENCY CURVE (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Batching Efficiency Curve" .callout-notebook +# │ +# │ Goal: Quantify throughput-latency trade-offs for vision vs LLM batching. +# │ Show: LLM knee at B=128; Vision knee at B=32. +# │ How: X(B) = B / (T_fixed + B * T_var). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: bec_llm_knee_str, bec_vision_knee_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class BatchingEfficiency: + # ┌── 1. LOAD ────────────────────────────────────────── + # LLM (Weights limited) + t_fixed_llm = 40 # ms + t_var_llm = 0.5 # ms + # Vision (Compute limited) + t_fixed_vis = 2 # ms + t_var_vis = 1.0 # ms + # ┌── 2. EXECUTE ─────────────────────────────────────── + def get_x(b, tf, tv): return b / (tf + b * tv) * 1000 # req/s + + # Knee occurs where marginal throughput gain slows significantly + # Approximate: B where B*T_var ≈ T_fixed + b_knee_llm = int(t_fixed_llm / t_var_llm) + b_knee_vis = int(t_fixed_vis / t_var_vis) + + x_knee_llm = get_x(b_knee_llm, t_fixed_llm, t_var_llm) + x_knee_vis = get_x(b_knee_vis, t_fixed_vis, t_var_vis) + # ┌── 3. GUARD ───────────────────────────────────────── + check(b_knee_llm == 80, f"LLM knee mismatch: {b_knee_llm}") + check(b_knee_vis == 2, f"Vision knee mismatch: {b_knee_vis}") + # ┌── 4. OUTPUT ──────────────────────────────────────── + bec_llm_knee_str = f"{b_knee_llm}" + bec_vision_knee_str = f"{b_knee_vis}" + +``` + +::: {.callout-notebook title="The Batching Efficiency Curve"} + +**Problem**: You are optimizing two services: a Vision model (ResNet) and an LLM (70B). At what batch size does each hit the "Knee" of its efficiency curve? + +**The Math**: +The knee occurs where the variable compute cost ($B \times T_{var}$) starts to exceed the fixed overhead ($T_{fixed}$). + +1. **Vision Model** ($T_{fixed}=2ms$, $T_{var}=1ms$): + - Knee: $B \approx 2ms / 1ms =$ **`{python} BatchingEfficiency.bec_vision_knee_str`**. + - *Result*: Vision models saturate at very small batches because the compute per sample is high. + +2. **LLM Decode** ($T_{fixed}=40ms$, $T_{var}=0.5ms$): + - Knee: $B \approx 40ms / 0.5ms =$ **`{python} BatchingEfficiency.bec_llm_knee_str`**. + - *Result*: LLMs require massive batches to amortize the expensive weight-loading overhead. + +**The Systems Insight**: Because the LLM's "Fixed Overhead" (loading 140 GB of weights from HBM) is so large, you haven't reached the efficiency knee until batch size **`{python} BatchingEfficiency.bec_llm_knee_str`**. This is why LLM serving requires **Continuous Batching** and **Paged Memory**—you must pack hundreds of concurrent users into a single batch just to overcome the memory bandwidth bottleneck. + +::: + +Translating these batch-level equations into system-wide capacity planning requires a classical result from queuing theory: **Little's Law**[^fn-littles-law], which relates concurrency, throughput, and latency in any stable system. @tbl-batching-by-model summarizes how these batching characteristics vary across model architectures. [^fn-littles-law]: **Little's Law**: From queuing theory ($L = \lambda W$): Concurrency = Arrival Rate $\times$ Latency. For a serving fleet, this defines the **Capacity Envelope**: if a GPU can handle 32 concurrent requests and each takes 100 ms, the maximum arrival rate is 320 req/s. Beyond this, queues build up exponentially, and tail latency ($L_{\text{lat}}$) collapses. \index{Little's Law!capacity planning} @@ -1003,7 +1104,7 @@ class GPT3ServingResults: b8_service_ms = 54.0 b8_util_pct = 67.5 b8_wait_ms = 42.3 - + # Batch 32 results b32_service_ms = 66.0 b32_util_pct = 20.6 @@ -1046,9 +1147,9 @@ We can confirm these results by *applying Little's Law to verify* internal consi At B=32 with $\lambda = 100$ req/s and $E[T] =$ `{python} LittlesLawVerify.expected_total_str` ms: -$$L = 100 \times `{python} LittlesLawVerify.expected_total_ms / 1000` = `{python} LittlesLawVerify.l_str` \text{ requests in system}$$ +$L = 100 \times$ `{python} LittlesLawVerify.expected_total_ms / 1000` $=$ `{python} LittlesLawVerify.l_str` requests in system -With batch size `{python} LittlesLawVerify.batch_size` and utilization `{python} GPT3ServingResults.b32_util_pct` %: +With batch size `{python} LittlesLawVerify.batch_size` and utilization `{python} GPT3ServingResults.b32_util_pct`%: - Expected requests in service: `{python} LittlesLawVerify.batch_size` $\times$ `{python} LittlesLawVerify.utilization` = `{python} LittlesLawVerify.in_service_str` - Expected requests in queue: `{python} LittlesLawVerify.l_str` - `{python} LittlesLawVerify.in_service_str` = `{python} LittlesLawVerify.in_queue_str` @@ -1957,7 +2058,20 @@ The system automatically increases batch size to maintain throughput as traffic ### Quantitative Summary: Batching Strategy Selection {#sec-inference-scale-quantitative-summary-batching-strategy-selection-d5af} -The choice of batching strategy depends on model characteristics, traffic patterns, and latency requirements. @fig-inference-lifecycle visualizes the end-to-end request path, highlighting latency sources at each stage. The following decision framework guides selection: +The choice of batching strategy depends on model characteristics, traffic patterns, and latency requirements. @fig-inference-lifecycle visualizes the end-to-end request path, highlighting latency sources at each stage. + +::: {.callout-checkpoint title="Batching Strategy Trade-offs"} + +Verify your understanding of different batching mechanics: + +- [ ] A vision service has highly uniform input sizes and predictable traffic. Which is more appropriate: **Static Batching** or **Continuous Batching**? +- [ ] For an LLM serving chat requests, why does the "Waste Ratio" ($1 - \bar{L}/L_{max}$) collapse to zero when moving from static to continuous batching? +- [ ] Explain why **Adaptive Batching** (shrinking the window when traffic is high) reduces P99 latency without sacrificing throughput. +- [ ] In **Feature-Parallel Batching** for RecSys, which component of the Fleet Stack acts as the primary bottleneck: the GPU's Tensor Cores or the distributed Embedding Store's memory bandwidth? + +::: + +The following decision framework guides selection: ::: {#fig-inference-lifecycle fig-env="figure" fig-pos="htb" fig-cap="**End-to-End Inference Pipeline**. A high-level view of the request lifecycle: Client -> Load Balancer -> Request Queue -> Batch Scheduler -> Model Execution -> Response. This visualization highlights the critical \"Serving Tax\" components (serialization, routing, coordination) that consume latency budget outside of the actual GPU compute time." fig-alt="Flowchart with 5 stages: Client, Load Balancer, Request Queue, Dynamic Batcher, Model Execution. Arrows show request flow with dashed return path. Red annotations below highlight latency sources at each stage."} ```{.tikz} diff --git a/book/quarto/contents/vol2/introduction/introduction.qmd b/book/quarto/contents/vol2/introduction/introduction.qmd index fa81ff5bc4..053355e4b1 100644 --- a/book/quarto/contents/vol2/introduction/introduction.qmd +++ b/book/quarto/contents/vol2/introduction/introduction.qmd @@ -142,7 +142,19 @@ class Gpt4TrainingScenario: ``` Consider the training of GPT-4. It reportedly required approximately **`{python} Gpt4TrainingScenario.gpt4_gpus_str` `{python} Gpt4TrainingScenario.hw_name` GPUs** running for **`{python} Gpt4TrainingScenario.gpt4_days_str` days** [@openai2023gpt4]. In a cluster of this size, the probability of failure ($P_{\text{fail}}$) becomes the dominant constraint. - At the 8% annual GPU failure rate observed under intensive training workloads, this cluster experiences **`{python} Gpt4TrainingScenario.cluster_fail_day_str` hardware failures per day**, or one failure every **`{python} Gpt4TrainingScenario.mtbf_hours_str` hours**. In this regime, the system is always in a state of partial failure. Traditional software recovery (manual restart) collapses; the system must be architected for **Fault Tolerance**\index{Fault Tolerance} as a first-class citizen. + +::: {.callout-notebook title="Scale and Reliability"} + +**Problem**: You are managing a training run for a GPT-4 class model using `{python} Gpt4TrainingScenario.gpt4_gpus_str` GPUs. If each individual GPU has an annual failure rate of 8%, how often will your training job be interrupted by hardware failure? + +**The Math**: +1. **Total annual failures**: `{python} Gpt4TrainingScenario.gpt4_gpus_str` $\times 0.08 = 2,000$ failures per year. +2. **Daily failure rate**: $2,000 / 365 \approx$ **`{python} Gpt4TrainingScenario.cluster_fail_day_str` failures per day**. +3. **Mean Time Between Failures (MTBF)**: $24 \text{ hours} /$ `{python} Gpt4TrainingScenario.cluster_fail_day_str` $\approx$ **`{python} Gpt4TrainingScenario.mtbf_hours_str` hours**. + +**The Systems Insight**: In this regime, the system is always in a state of partial failure. Traditional software recovery (manual restart) collapses; the system must be architected for **Fault Tolerance** as a first-class citizen. You can no longer treat hardware as a reliable abstraction; you must treat it as a probabilistic resource that requires constant, automated state preservation (checkpointing). + +::: The history of machine learning is defined by scale. Each major capability leap has come not from algorithmic breakthroughs alone, but from the ability to apply computation at previously impossible scales. Compute requirements have evolved over the past decade in ways that make systems engineering central to AI advancement. Three qualitative changes emerge at production scale: communication dominance, routine failure, and governance requirements that accompany societal impact. @@ -388,6 +400,17 @@ The Machine Learning Fleet, by contrast, operates under **Synchronous Tight Coup 2. **Barrier Synchronization**: In a synchronous training step, 10,000 GPUs must wait for the slowest worker to finish before any can proceed. This makes the fleet hypersensitive to "Stragglers"—a 10% performance drop on one node can reduce the entire cluster's throughput by 10%. 3. **Bisection Bandwidth Dominance**: A web service is often "Latent-Bound" (waiting for the user). An ML training job is "Bandwidth-Bound." It needs to move gigabytes of gradient data across the *entire* network every second. This requires non-blocking network topologies that traditional datacenters rarely implement. +::: {.callout-checkpoint title="The Fleet Mindset"} + +Verify your understanding of how ML fleets differ from traditional clusters: + +- [ ] Why does a single slow worker (straggler) have a disproportionate impact on a synchronous ML training job compared to a MapReduce job? +- [ ] In which type of system—traditional web serving or ML training—is **Bisection Bandwidth** more likely to be the primary performance bottleneck? +- [ ] Can you explain the concept of **Synchronous Tight Coupling**? How does it relate to the global barrier at the end of each training step? +- [ ] True or False: Adding more GPUs to a training job always results in a linear speedup of the training process. + +::: + ### The Shift to the Warehouse-Scale Computer This textbook adopts the **Warehouse-Scale Computer (WSC)**[^fn-wsc-barroso] perspective. In traditional computing, the datacenter is a building that *houses* many computers. In the ML Fleet, the datacenter *is* the computer. @@ -452,7 +475,7 @@ At large scale, failure is a **statistical certainty**. With 10,000 GPUs, hardwa The infrastructure investments described in the preceding section did not arise from arbitrary organizational ambitions. They emerged from an empirical discovery: machine learning performance follows predictable mathematical relationships with scale. This **Universal Scaling Law** explains why distributed systems have become essential and reveals the constraints that shape their design. -Scaling laws quantify the "Bitter Lesson" articulated by Rich Sutton: performance in machine learning is primarily driven by applying general methods at massive scale rather than encoding human knowledge into algorithms. The predictable power-law relationships show *how* computation, when scaled, yields better models. The **Universal Scaling Law** (@nte-universal-scaling) formalizes this observation: model performance improves as a power-law function of compute, dataset size, and parameters — $L(X) \propto X^{-\alpha}$. Achieving a 10$\times$ improvement in performance typically demands a 100$\times$–1000$\times$ increase in resources. This exponential hunger drives the transition from single-server training to warehouse-scale clusters. +Scaling laws quantify the "Bitter Lesson" articulated by Rich Sutton: performance in machine learning is primarily driven by applying general methods at massive scale rather than encoding human knowledge into algorithms. The predictable power-law relationships show *how* computation, when scaled, yields better models. The **Universal Scaling Law** formalizes this observation: model performance improves as a power-law function of compute, dataset size, and parameters — $L(X) \propto X^{-\alpha}$. Achieving a 10$\times$ improvement in performance typically demands a 100$\times$–1000$\times$ increase in resources. This exponential hunger drives the transition from single-server training to warehouse-scale clusters. Efficiency becomes increasingly important as systems expand in complexity, and scaling laws manifest across different dimensions with direct implications for system design. The multi-dimensional efficiency optimization framework is essential at production scale because each scaling dimension — parameters, data, and compute — interacts with infrastructure constraints differently. @@ -858,6 +881,17 @@ At extreme scales, models may approach limits of what can be learned from traini : **Scaling Breakdown Types**: Unbalanced scaling across model size, data volume, and compute resources leads to specific failure modes, such as overfitting or diminishing returns, impacting system performance and efficiency. The table categorizes these breakdowns, identifies their root causes, and provides representative scenarios to guide more effective system design and resource allocation. {#tbl-scaling-breakdown} +::: {.callout-checkpoint title="Applying Scaling Laws"} + +Verify your understanding of how scaling laws guide resource allocation: + +- [ ] A team has a fixed compute budget but limited training data. According to the "Regimes" framework, should they train a large model for fewer steps or a small model for more steps? +- [ ] If you double the model parameters ($N$) but keep the dataset size ($D$) constant, which "Scaling Breakdown" are you most likely to encounter? +- [ ] Why is **Chinchilla Optimality** considered the "Gold Standard" for resource allocation? What is being minimized? +- [ ] True or False: Scaling laws predict that doubling compute always yields a doubling of model capability. + +::: + These breakdown points demonstrate that scaling laws describe empirical regularities under specific conditions that become increasingly difficult to maintain at scale. As machine learning systems continue evolving, discerning where and why scaling ceases to be effective becomes necessary, driving development of strategies that enhance performance without relying solely on scale. ### Integrating Efficiency with Scaling {#sec-vol2-intro-integrating-efficiency-scaling-a513} @@ -1018,7 +1052,7 @@ where $T_{\text{Compute}}$ is the time spent on useful arithmetic (forward and b $$ \eta_{\text{fleet}} = \frac{T_{\text{Compute}}}{T_{\text{Compute}} + T_{\text{Communication}} + T_{\text{Coordination}}} $$ {#eq-fleet-efficiency} -This decomposition is not merely accounting — it is a diagnostic instrument. When $\eta_{\text{fleet}}$ drops below 0.5, more than half the fleet's expensive silicon sits idle. The C$^3$ taxonomy tells you *where* the time is going. If $T_{\text{Communication}}$ dominates, upgrade interconnects or overlap communication with computation. If $T_{\text{Coordination}}$ dominates, consider asynchronous methods or reduce barrier frequency. If $T_{\text{Compute}}$ dominates — congratulations, the fleet is doing its job. Crucially, the **Conservation of Overhead** (@nte-conservation-overhead) dictates that this overhead cannot be eliminated, only redistributed among the three C's — asynchronous training eliminates coordination barriers but introduces gradient staleness; pipeline parallelism reduces communication volume but adds bubble time. There is no free lunch in distributed systems. +This decomposition is not merely accounting — it is a diagnostic instrument. When $\eta_{\text{fleet}}$ drops below 0.5, more than half the fleet's expensive silicon sits idle. The C$^3$ taxonomy tells you *where* the time is going. If $T_{\text{Communication}}$ dominates, upgrade interconnects or overlap communication with computation. If $T_{\text{Coordination}}$ dominates, consider asynchronous methods or reduce barrier frequency. If $T_{\text{Compute}}$ dominates — congratulations, the fleet is doing its job. Crucially, the **Conservation of Overhead** dictates that this overhead cannot be eliminated, only redistributed among the three C's — asynchronous training eliminates coordination barriers but introduces gradient staleness; pipeline parallelism reduces communication volume but adds bubble time. There is no free lunch in distributed systems. Google's **ML Productivity Goodput** metric provides the production-scale instantiation of this framework. Goodput decomposes end-to-end training productivity into three multiplicative factors — **Program Goodput** ($\eta_P$) measuring how efficiently code uses hardware (Compute), **Runtime Goodput** ($\eta_R$) capturing losses from communication stalls and failures (Communication), and **Scheduling Goodput** ($\eta_S$) capturing wasted time from preemptions and reconfigurations (Coordination). The C$^3$ taxonomy is the theoretical framework; Goodput is how production teams measure it. @@ -1354,15 +1388,15 @@ Translating these pillars into production code requires navigating a fragmented : **The Framework Rosetta Stone**. A mapping of abstract distributed ML primitives to their implementations across major frameworks. Primitives are covered in depth in Part VI (Distribution) and Part VII (Deployment). {#tbl-framework-rosetta-stone tbl-colwidths="[14,14,17,22,17,16]"} -| **Abstract Primitive** | **PyTorch (Native)** | **DeepSpeed** | **Megatron-LM** | **JAX / XLA** | **Ray** | -|:-----------------------|:-----------------------|:------------------------------|:---------------------------|:-------------------------|:-------------------| -| **Data Parallel** | `DDP` | `DeepSpeedEngine` | `DistributedDataParallel` | `pmap` / `jit(sharded)` | `Ray Train` | -| **Sharded DP** | `FSDP` | `ZeRO-1/2/3` | `FullyShardedDataParallel` | `sharding.Mesh` | `FSDP Strategy` | -| **Tensor Parallel** | `DTensor` | `InferenceTP` | `Column/RowParallel` | `xmap` / `spmd` | `Ray Train TP` | -| **Pipeline Parallel** | `PiPPy` | `PipelineModule` | `PipelineParallel` | `GSPMD` | `Ray Train PP` | -| **Grad Accumulation** | `backward(accumulate)` | `grad_accum_steps` | `grad_acc_steps` | `lax.scan` | `Ray Train Config` | -| **Checkpointing** | `torch.save` / `DCP` | `save_checkpoint` | `save_checkpoint` | `orbax` | `ray.checkpoint` | -| **Orchestration** | `torchrun` | `deepspeed` CLI | `megatron_main.py` | `jax.distributed` | `Ray Core` | +| **Abstract Primitive** | **PyTorch (Native)** | **DeepSpeed** | **Megatron-LM** | **JAX / XLA** | **Ray** | +|:-----------------------|:-----------------------|:-------------------|:---------------------------|:------------------------|:-------------------| +| **Data Parallel** | `DDP` | `DeepSpeedEngine` | `DistributedDataParallel` | `pmap` / `jit(sharded)` | `Ray Train` | +| **Sharded DP** | `FSDP` | `ZeRO-1/2/3` | `FullyShardedDataParallel` | `sharding.Mesh` | `FSDP Strategy` | +| **Tensor Parallel** | `DTensor` | `InferenceTP` | `Column/RowParallel` | `xmap` / `spmd` | `Ray Train TP` | +| **Pipeline Parallel** | `PiPPy` | `PipelineModule` | `PipelineParallel` | `GSPMD` | `Ray Train PP` | +| **Grad Accumulation** | `backward(accumulate)` | `grad_accum_steps` | `grad_acc_steps` | `lax.scan` | `Ray Train Config` | +| **Checkpointing** | `torch.save` / `DCP` | `save_checkpoint` | `save_checkpoint` | `orbax` | `ray.checkpoint` | +| **Orchestration** | `torchrun` | `deepspeed` CLI | `megatron_main.py` | `jax.distributed` | `Ray Core` | \normalsize diff --git a/book/quarto/contents/vol2/network_fabrics/network_fabrics.qmd b/book/quarto/contents/vol2/network_fabrics/network_fabrics.qmd index 0931223f3b..56049e5de0 100644 --- a/book/quarto/contents/vol2/network_fabrics/network_fabrics.qmd +++ b/book/quarto/contents/vol2/network_fabrics/network_fabrics.qmd @@ -364,11 +364,11 @@ Before analyzing protocols and topologies, we must understand the physical mediu ::: {.callout-definition title="PAM4 Signaling"} -***PAM4 Signaling***\index{PAM4 Signaling!definition} is a modulation scheme that uses four voltage levels to encode two bits per symbol, doubling the data rate for a given symbol rate. +***PAM4 Signaling***\index{PAM4 Signaling!definition} is an electrical modulation scheme that uses four distinct voltage levels to encode two bits per symbol period, doubling the data rate achievable over a given physical medium without requiring a higher symbol rate. -1. **Significance (Quantitative):** It enables the high **Network Bandwidth ($\text{BW}$)** required for exa-scale fleets (e.g., 400G/800G links). However, the reduced signal-to-noise ratio necessitates **Forward Error Correction (FEC)**, which adds irreducible **Fixed Latency ($L_{\text{lat}}$)** to every packet hop. -2. **Distinction (Durable):** Unlike **NRZ Signaling** (binary on/off), PAM4 uses multi-level amplitude to overcome the physical symbol-rate limits of copper and fiber. -3. **Common Pitfall:** A frequent misconception is that doubling the bits per symbol is "free." In reality, it is a **Reliability-Latency Trade-off**: the processing time for error correction makes PAM4 links fundamentally higher-latency than their NRZ predecessors. +1. **Significance (Quantitative):** PAM4 enables 400 Gb/s and 800 Gb/s link speeds that sustain the $\text{BW}$ required for large-scale gradient synchronization. However, the reduced gap between voltage levels increases susceptibility to noise, requiring Forward Error Correction (FEC) — typically Reed-Solomon RS(544,514) — that adds 100–200 ns of irreducible processing latency per hop. Across a three-tier fat-tree (3 hops each way), FEC alone contributes 600–1,200 ns of fixed latency to every packet, setting a hard lower bound on the $\alpha$ term in the communication cost model. +2. **Distinction (Durable):** Unlike NRZ (Non-Return-to-Zero) binary signaling, which uses only two voltage levels and offers better noise margin but is limited to half the data rate per lane, PAM4 trades signal-to-noise ratio for bandwidth density — the same copper trace carries twice the data at the cost of requiring FEC at every port. +3. **Common Pitfall:** A frequent misconception is that doubling bits per symbol is a free throughput gain. PAM4 links require FEC at both endpoints, making them fundamentally higher-latency than equivalent NRZ links; latency-sensitive collectives like AllReduce pay this FEC tax on every packet regardless of message size. ::: @@ -413,11 +413,11 @@ Large-scale training requires sustained, synchronized bulk transfers. A single A ::: {.callout-definition title="Remote Direct Memory Access (RDMA)"} -***Remote Direct Memory Access (RDMA)***\index{Remote Direct Memory Access!definition} is a networking technology that allows one computer to access the memory of another directly without involving the operating system or CPU of either machine. +***Remote Direct Memory Access (RDMA)***\index{Remote Direct Memory Access!definition} is a networking technology that allows one machine to read or write the memory of another machine directly, bypassing the operating system kernel and CPU of both endpoints by offloading transport processing to the network interface card. -1. **Significance (Quantitative):** it provides **Kernel Bypass** and **Zero-Copy** capabilities, reducing the **Communication Overhead ($L_{\text{lat}}$)** by eliminating context switches and intermediate memory copies. It is essential for saturating high-bandwidth links ($BW$) during gradient synchronization. -2. **Distinction (Durable):** Unlike **Traditional TCP/IP Networking**, where the CPU must process every packet through a complex software stack, RDMA offloads the entire transport protocol to the network hardware (HCA). -3. **Common Pitfall:** A frequent misconception is that RDMA is just "fast Ethernet." In reality, it requires specialized **Lossless Fabric** (like InfiniBand or RoCE with PFC) because it lacks the robust congestion control mechanisms of TCP; a single lost packet can cause massive performance collapses. +1. **Significance (Quantitative):** RDMA reduces end-to-end message latency from the 50–100 μs typical of kernel TCP to approximately 1–2 μs, cutting the $L_{\text{lat}}$ term in the Iron Law by 25–50$\times$. For a 175B-parameter model exchanging 350 GB of gradient data across 1,000 GPUs, RDMA also eliminates 700 GB of redundant memory copies per step by allowing the NIC to read GPU memory directly without staging through host RAM (GPUDirect RDMA). +2. **Distinction (Durable):** Unlike traditional TCP/IP, where the CPU processes every packet through the kernel network stack — consuming tens of CPU cores to saturate a 400 Gbps link — RDMA offloads the entire transport to dedicated NIC hardware, freeing the CPU to orchestrate computation rather than move data. +3. **Common Pitfall:** A frequent misconception is that RDMA works reliably on any Ethernet network. RDMA lacks TCP's retransmission logic; a single dropped packet can stall an entire 1,024-GPU AllReduce for 100–500 ms as the Go-Back-N recovery retransmits from the loss point. RDMA requires a lossless fabric — InfiniBand or Ethernet with Priority Flow Control — to operate correctly at scale. ::: @@ -577,11 +577,32 @@ The model reveals two regimes: 1. **Latency-dominated ($n < \alpha\beta$):** Transfer time is dominated by the startup cost $\alpha$. Small control messages and pipeline bubbles fall here. 2. **Bandwidth-dominated ($n > \alpha\beta$):** Transfer time scales with $n/\beta$. Gradient AllReduce typically falls here. -For NDR InfiniBand with $\alpha \approx `{python} NetworkAlphaBeta.ib_alpha_us_str` \mu\text{s}$ and $\beta \approx `{python} NetworkAlphaBeta.ib_beta_gbs_str` \text{GB/s}$, the crossover point $n = \alpha\beta \approx `{python} NetworkAlphaBeta.n_cross_ib_kb_str` \text{ KB}$. Messages smaller than this gain little from more bandwidth; messages larger than this gain little from lower latency. +For NDR InfiniBand with $\alpha \approx$ `{python} NetworkAlphaBeta.ib_alpha_us_str` $\mu\text{s}$ and $\beta \approx$ `{python} NetworkAlphaBeta.ib_beta_gbs_str` GB/s, the crossover point $n = \alpha\beta \approx$ `{python} NetworkAlphaBeta.n_cross_ib_kb_str` KB. Messages smaller than this gain little from more bandwidth; messages larger than this gain little from lower latency. -To see why this distinction matters in practice, consider two messages that our 175B model training job sends every iteration. The first is a 4 KB pipeline-scheduling control message that coordinates microbatch handoffs between pipeline stages. Applying the model: $T(4\text{ KB}) \approx \mathbf{`{python} NetworkAlphaBeta.t_small_ib_us_str` \mu\text{s}}$. The bandwidth term contributes only `{python} NetworkAlphaBeta.small_bw_pct_str`% of the total. Doubling the link speed would save a negligible fraction of a microsecond. For this message, the only way to reduce transfer time is to reduce the hop count (which lowers $\alpha$), not to buy faster links. +::: {.callout-notebook title="The Alpha-Beta Crossover"} -The second message is a 350 MB gradient shard for one layer's AllReduce. Now: $T(350\text{ MB}) \approx \mathbf{`{python} NetworkAlphaBeta.t_large_ib_ms_str` \text{ ms}}$. The latency term is invisible. Doubling bandwidth to 100 GB/s would halve this transfer time, a direct and proportional gain. These two cases illustrate why network design must address both $\alpha$ and $\beta$ simultaneously: topology and hop count control the latency-dominated regime, while link speed and path diversity control the bandwidth-dominated regime. +**Problem**: You are comparing InfiniBand NDR (`{python} NetworkAlphaBeta.ib_alpha_us_str` $\mu s$, `{python} NetworkAlphaBeta.ib_beta_gbs_str` GB/s) against a slower Ethernet baseline (5.0 $\mu s$, 12.5 GB/s). At what message size does the faster link's bandwidth advantage finally outweigh its higher cost? + +**The Math**: +Apply $T(n) = \alpha + n/\beta$ for a 4 KB control message and a 100 MB gradient shard. + +1. **Small Message (4 KB)**: + - InfiniBand: $1.5\mu s + 4KB/50GB/s = 1.5 + 0.08 = \mathbf{1.58 \mu s}$ + - Ethernet: $5.0\mu s + 4KB/12.5GB/s = 5.0 + 0.32 = \mathbf{5.32 \mu s}$ + - *Verdict*: InfiniBand is $3.4\times$ faster purely due to lower $\alpha$. + +2. **Large Message (100 MB)**: + - InfiniBand: $1.5\mu s + 100MB/50GB/s = 0.0015 + 2.0 = \mathbf{2.00 ms}$ + - Ethernet: $5.0\mu s + 100MB/12.5GB/s = 0.005 + 8.0 = \mathbf{8.01 ms}$ + - *Verdict*: InfiniBand is $4\times$ faster purely due to higher $\beta$. + +**The Systems Insight**: For large-scale training, the "Crossover Point" ($n^* = \alpha\beta$) is typically around **`{python} NetworkAlphaBeta.n_cross_ib_kb_str` KB**. Since gradients are megabytes to gigabytes, we are almost always in the **bandwidth-dominated regime**. However, pipeline parallelism and distributed coordination rely on small messages in the **latency-dominated regime**, where wire-speed upgrades provide zero benefit—only topology and hop-count reductions matter. + +::: + +To see why this distinction matters in practice, consider two messages that our 175B model training job sends every iteration. The first is a 4 KB pipeline-scheduling control message that coordinates microbatch handoffs between pipeline stages. Applying the model: $T(4\text{ KB}) \approx$ **`{python} NetworkAlphaBeta.t_small_ib_us_str` $\mu\text{s}$**. The bandwidth term contributes only `{python} NetworkAlphaBeta.small_bw_pct_str`% of the total. Doubling the link speed would save a negligible fraction of a microsecond. For this message, the only way to reduce transfer time is to reduce the hop count (which lowers $\alpha$), not to buy faster links. + +The second message is a 350 MB gradient shard for one layer's AllReduce. Now: $T(350\text{ MB}) \approx$ **`{python} NetworkAlphaBeta.t_large_ib_ms_str` ms**. The latency term is invisible. Doubling bandwidth to 100 GB/s would halve this transfer time, a direct and proportional gain. These two cases illustrate why network design must address both $\alpha$ and $\beta$ simultaneously: topology and hop count control the latency-dominated regime, while link speed and path diversity control the bandwidth-dominated regime. @fig-alpha-beta-crossover makes these two regimes visible across the full range of message sizes. For InfiniBand, the crossover occurs at approximately `{python} NetworkAlphaBeta.n_cross_ib_kb_str` KB: messages smaller than this are latency-dominated (the flat region on the left), while larger messages are bandwidth-dominated (the linear region on the right). The 5$\times$ latency gap between InfiniBand and Ethernet RoCE dominates for small messages but becomes irrelevant for the multi-megabyte gradient transfers that dominate training communication. @@ -773,11 +794,11 @@ The physical arrangement of switches determines the **bisection bandwidth** and ::: {.callout-definition title="Bisection Bandwidth"} -***Bisection Bandwidth***\index{Bisection Bandwidth!definition} is the minimum total bandwidth across any cut that divides the network into two equal halves. +***Bisection Bandwidth***\index{Bisection Bandwidth!definition} is a network topology metric defined as the minimum aggregate link capacity crossing any partition that divides the cluster into two equal halves, representing the worst-case throughput ceiling for all-to-all communication patterns such as AllReduce. -1. **Significance (Quantitative):** It represents the **Worst-Case Throughput** for global communication patterns (e.g., AllReduce). Within the **Iron Law**, bisection bandwidth defines the **Aggregate Bandwidth ($\text{BW}_{\text{bisect}}$)** available when the entire fleet must synchronize its state simultaneously. -2. **Distinction (Durable):** Unlike **Aggregate Bandwidth** (the sum of all link speeds), Bisection Bandwidth measures the **Global Connectivity** of the fabric, identifying the narrowest bottleneck in the cluster. -3. **Common Pitfall:** A frequent misconception is that adding more switches always increases bisection bandwidth. In reality, it is a **Topology Property**: if the upper tiers of the network are oversubscribed, the bisection bandwidth will be significantly lower than the sum of the edge links. +1. **Significance (Quantitative):** Bisection bandwidth directly sets the $\text{BW}$ ceiling in the Iron Law for global synchronization. A 1,024-GPU fat-tree with 400 Gb/s (50 GB/s) links at 1:1 subscription provides $512 \times 50\,\text{GB/s} = 25.6\,\text{TB/s}$ of bisection bandwidth; a 4:1 oversubscribed spine reduces this to $6.4\,\text{TB/s}$, making each AllReduce step 4$\times$ slower and turning the network into the dominant Iron Law bottleneck rather than the accelerator. +2. **Distinction (Durable):** Unlike aggregate bandwidth (the sum of all edge link speeds, which can be high even in a poorly connected topology), bisection bandwidth measures global connectivity — a star topology with 1,000 edge links all meeting at one central switch has high aggregate bandwidth but bisection bandwidth limited by that switch's backplane capacity. +3. **Common Pitfall:** A frequent misconception is that adding more leaf switches always increases bisection bandwidth. In a three-tier fat-tree, oversubscribing the spine layer — using fewer uplinks than downlinks per pod switch — reduces bisection bandwidth below the edge-link total regardless of how many leaf switches are present. ::: @@ -785,11 +806,11 @@ For ML training, where all-to-all communication is standard, full bisection band ::: {.callout-definition title="Non-blocking Fabric"} -***Non-blocking Fabric***\index{Non-blocking Fabric!definition} is a network topology in which any permutation of input-output port pairs can communicate simultaneously at full line rate without internal contention. +***Non-blocking Fabric***\index{Non-blocking Fabric!definition} is a network topology in which any permutation of input-output port pairs can communicate simultaneously at full line rate without internal contention, achieved by ensuring that uplink capacity at every switch tier equals or exceeds downlink capacity. -1. **Significance (Quantitative):** In ML fleets, it ensures that synchronization traffic from any subset of accelerators does not compete for shared links, maximizing the **System Throughput ($\eta$)** by eliminating network-induced stalls. -2. **Distinction (Durable):** Unlike **Oversubscribed Fabrics** (common in web datacenters), where upper-tier links are shared among many lower-tier nodes, a Non-blocking Fabric provides **Dedicated Path Capacity** for every possible pairing. -3. **Common Pitfall:** A frequent misconception is that non-blocking means "zero congestion." In reality, **Endpoint Congestion** (Incast) can still occur if multiple senders target the same receiver simultaneously, regardless of the topology's internal capacity. +1. **Significance (Quantitative):** In ML fleets, a non-blocking fabric ensures that AllReduce traffic from any accelerator subset does not compete for shared links, preserving the full $\text{BW}$ term of the Iron Law. A 2:1 oversubscribed spine halves the effective bisection bandwidth, doubling AllReduce time for global gradients and dropping system efficiency $\eta$ accordingly — in a 30%-communication workload, this costs roughly 15% of total cluster throughput. +2. **Distinction (Durable):** Unlike oversubscribed fabrics common in web datacenters, where upper-tier links are shared among many lower-tier nodes, a non-blocking fabric provides dedicated path capacity for every possible pairing of senders and receivers simultaneously. +3. **Common Pitfall:** A frequent misconception is that non-blocking means zero congestion. Endpoint congestion (incast) can still occur if multiple senders simultaneously target the same receiver port, regardless of how much internal fabric capacity is available. ::: @@ -861,11 +882,11 @@ To mitigate congestion at this edge, network architects maximize the switch **ra ::: {.callout-definition title="Fat-Tree"} -***Fat-Tree***\index{Fat-Tree!definition} is a hierarchical network topology where the aggregate bandwidth increases toward the root to provide multiple equal-cost paths between any two nodes. +***Fat-Tree***\index{Fat-Tree!definition} is a hierarchical network topology in which the number of parallel paths — and therefore aggregate cross-sectional capacity — increases at each switch tier toward the spine, providing full bisection bandwidth and multiple equal-cost routes between any two nodes. -1. **Significance (Quantitative):** It is the standard for providing **Full Bisection Bandwidth ($\text{BW}_{\text{bisect}}$)**. It ensures that the fabric can sustain simultaneous full-rate communication from all endpoints, which is a non-negotiable requirement for the AllReduce collectives that dominate training. -2. **Distinction (Durable):** Unlike a **Standard Tree** (where links near the root become bottlenecks), a Fat-Tree uses redundant switches and links to ensure the network is **Non-blocking**. -3. **Common Pitfall:** A frequent misconception is that Fat-Trees are the most "efficient" topology. In reality, they are the most **Reliable but Expensive**: they require a massive number of switches ($O(N \log N)$) and complex cabling compared to more specialized topologies like Dragonflies or Torus networks. +1. **Significance (Quantitative):** A fat-tree built from radix-$k$ switches supports $k^3/4$ hosts with full bisection bandwidth. With $k=64$, a two-tier pod supports 1,024 GPUs and a three-tier fabric supports 65,536 hosts. Because every AllReduce can use any available spine path, the fabric sustains simultaneous full-rate communication from all accelerators, meeting the $\text{BW}$ requirement in the Iron Law for gradient synchronization. +2. **Distinction (Durable):** Unlike a standard tree (where bandwidth at the root is a single bottleneck shared by all leaves), a fat-tree replaces each root with multiple spine switches whose combined uplink capacity matches the total edge bandwidth, eliminating the bottleneck. +3. **Common Pitfall:** A frequent misconception is that fat-trees guarantee zero network cost. They require $O(N \log N)$ switches and dense cabling: a 4,096-GPU non-blocking fat-tree needs roughly 2,048 switches, costing \$20–100 million in switching hardware alone. ::: @@ -909,6 +930,73 @@ class FatTreeScaling: A fat-tree built from switches with **radix**[^fn-switch-radix-density] $k$ supports $N = k^3/4$ hosts. In a two-tier network using radix-64 switches, a single pod can support up to 1,024 GPUs—perfectly accommodating our 1,000-GPU reference cluster with only leaf and spine layers. +```{python} +#| label: bisection-bandwidth-scaling +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ BISECTION BANDWIDTH SCALING (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Bisection Bandwidth: The Cost of Oversubscription" .callout-notebook +# │ +# │ Goal: Quantify the synchronization slowdown when moving from a +# │ non-blocking (1:1) to an oversubscribed (4:1) network. +# │ Show: slowdown_factor ≈ 4x — inline in notebook result. +# │ How: T = M / (Bisect_BW); Bisect_BW = N * Beta / Oversub_Ratio. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: bbs_n_gpus_str, bbs_gradient_gb_str, bbs_slowdown_str, bbs_time_1_1_ms_str, bbs_time_4_1_ms_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class BisectionScaling: + # ┌── 1. LOAD ────────────────────────────────────────── + n_gpus = 1024 + beta_gbs = 50 + gradient_gb = 100 + # ┌── 2. EXECUTE ─────────────────────────────────────── + bw_1_1 = n_gpus * beta_gbs + bw_4_1 = bw_1_1 / 4 + time_1_1 = gradient_gb / bw_1_1 + time_4_1 = gradient_gb / bw_4_1 + slowdown = time_4_1 / time_1_1 + # ┌── 3. GUARD ───────────────────────────────────────── + check(slowdown == 4.0, "Slowdown must match oversubscription ratio") + # ┌── 4. OUTPUT ──────────────────────────────────────── + bbs_n_gpus_str = f"{n_gpus}" + bbs_gradient_gb_str = f"{gradient_gb}" + bbs_slowdown_str = f"{int(slowdown)}" + bbs_time_1_1_ms_str = f"{time_1_1 * 1000:.1f}" + bbs_time_4_1_ms_str = f"{time_4_1 * 1000:.1f}" + +``` + +::: {.callout-notebook title="Bisection Bandwidth: The Cost of Oversubscription"} + +**Problem**: You are choosing between a "Non-blocking" (1:1) fat-tree and a "Cost-optimized" (4:1) spine for a `{python} BisectionScaling.bbs_n_gpus_str`-GPU cluster. How much slower will a `{python} BisectionScaling.bbs_gradient_gb_str` GB AllReduce be on the cheaper network? + +**The Math**: +Bisection bandwidth ($BW_{bisect}$) is the minimum pipe diameter between halves of your cluster. + +1. **Non-blocking (1:1)**: $BW_{bisect} =$ `{python} BisectionScaling.bbs_n_gpus_str` $\times 50 \text{ GB/s} = 51,200 \text{ GB/s}$. + - Time: $100 \text{ GB} / 51,200 \approx$ **`{python} BisectionScaling.bbs_time_1_1_ms_str` ms**. +2. **Oversubscribed (4:1)**: $BW_{bisect} = 51,200 / 4 = 12,800 \text{ GB/s}$. + - Time: $100 \text{ GB} / 12,800 \approx$ **`{python} BisectionScaling.bbs_time_4_1_ms_str` ms**. + +**The Systems Insight**: Saving money on core switches creates a **`{python} BisectionScaling.bbs_slowdown_str`$\times$ bottleneck** for global synchronization. For a \$300M supercomputer where training is 30% communication, this "saving" wastes approximately **\$75 Million** in idle compute time. For training, network oversubscription is a false economy. + +::: + +::: {.callout-checkpoint title="Fat-Tree Topologies"} + +Verify your understanding of hierarchical switch fabrics: + +- [ ] In a **Radix-64** two-tier fat-tree, what is the maximum number of GPUs you can connect without core switches? +- [ ] Why does a **Non-blocking** fabric require a 1:1 subscription ratio at every tier of the network? +- [ ] If you move from a two-tier to a three-tier fat-tree, how does the **$\alpha$ (startup latency)** of a message change? +- [ ] Explain the difference between **Aggregate Bandwidth** and **Bisection Bandwidth**. Which one matters for AllReduce? + +::: + [^fn-switch-radix-density]: **Switch Radix**: Modern high-end switches (e.g., NVIDIA Quantum-2) feature a radix of 64 ports, each at 400 Gbps. This density allows a two-tier fat-tree to support up to 2,048 GPUs with only two switch hops. As model scale pushes toward 100,000 GPUs, increasing switch radix (to 128 or 256) is the only way to avoid adding more tiers and the resulting latency/cost explosion. \index{Switch Radix!port density} Scaling beyond this requires a three-tier architecture with core switches, enabling the fabric to reach `{python} FatTreeScaling.fat_tree_hosts` hosts. The infrastructure cost is substantial: a non-blocking $k=64$ tree requires 4,096 switches. At \$10,000 to \$50,000 per switch, the switching layer alone represents \$40 to \$200 million. @@ -976,6 +1064,55 @@ However, scale imposes a latency tax. Typical **hop counts** are 2 for intra-pod In distributed training, **tensor parallelism** creates a deterministic and highly stratified communication pattern that standard topologies fail to exploit. Because large models partition individual matrix multiplications across GPUs, GPU 0 on one node must frequently exchange activations with GPU 0 on other nodes, but rarely with GPU 1. A **rail-optimized topology** physically hardwires this logic by isolating these communication paths into dedicated networks. Instead of connecting all 8 GPUs in a node to a single ToR switch, the network connects all GPU 0s across the entire cluster to a dedicated "Rail 0" switch fabric, all GPU 1s to "Rail 1," and so on. +```{python} +#| label: rail-optimized-latency-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ RAIL-OPTIMIZED LATENCY BUDGET (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Rail-Optimized Dividend" .callout-notebook +# │ +# │ Goal: Quantify the latency savings of rail-optimized topology. +# │ Show: speedup ≈ 2x — inline in notebook result. +# │ How: T = Hop_Count * T_hop. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: rol_hop_lat_us_str, rol_speedup_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class RailLatency: + # ┌── 1. LOAD ────────────────────────────────────────── + t_hop_us = 0.6 # switch + wire latency + # Standard fat-tree: Leaf -> Spine -> Leaf (2 hops) + # Rail-optimized: Leaf (direct) (1 hop) + # ┌── 2. EXECUTE ─────────────────────────────────────── + lat_std = 2 * t_hop_us + lat_rail = 1 * t_hop_us + speedup = lat_std / lat_rail + # ┌── 3. GUARD ───────────────────────────────────────── + check(speedup == 2.0, f"Speedup {speedup:.1f}x unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + rol_hop_lat_us_str = f"{t_hop_us}" + rol_speedup_str = f"{speedup:.0f}" + +``` + +::: {.callout-notebook title="The Rail-Optimized Dividend"} + +**Problem**: You are synchronizing activations for tensor parallelism across 128 nodes. In a standard fat-tree, each message traverses a Leaf switch and a Spine switch (2 hops). In a rail-optimized network, all corresponding GPUs are on the same switch (1 hop). How much "Latency Dividend" do you earn? + +**The Math**: +Communication latency ($\alpha$) is proportional to the number of switch hops. + +1. **Standard Latency**: 2 hops $\times$ `{python} RailLatency.rol_hop_lat_us_str` $\mu s$ = **1.2 $\mu s$**. +2. **Rail-Optimized**: 1 hop $\times$ `{python} RailLatency.rol_hop_lat_us_str` $\mu s$ = **0.6 $\mu s$**. +3. **Verdict**: **`{python} RailLatency.rol_speedup_str`$\times$ lower latency**. + +**The Systems Insight**: For high-frequency tensor parallelism (which synchronizes every layer), a 2$\times$ latency reduction is the difference between 80% and 95% scaling efficiency. By physically aligning the network to the model's math, you eliminate the "Spine Tax" for the most sensitive traffic. This is why Archetype A clusters are not just big—they are **Structured**. + +::: + ::: {.callout-lighthouse title="Archetype A (GPT-4 / Llama-3): The Rail-Optimized Fleet"} **Archetype A (GPT-4)** is the primary driver for rail-optimized fabrics. Because it uses **3D Parallelism**, it generates two distinct traffic patterns: (1) high-frequency, latency-sensitive activation exchanges for tensor parallelism, and (2) massive, bandwidth-hungry gradient averaging for data parallelism. The rail-optimized design ensures that tensor-parallel traffic traverses only a single switch hop between nodes, minimizing the latency that would otherwise stall the synchronous training loop. ::: @@ -983,6 +1120,17 @@ In distributed training, **tensor parallelism** creates a deterministic and high For our 1,000-GPU cluster spanning 125 nodes, this creates 8 parallel networks of 125 GPUs each, allowing tensor-parallel traffic to traverse a single switch hop rather than the multi-hop leaf-spine-leaf path required by a standard fat-tree. The latency benefit is significant for the frequent, small activation exchanges that tensor parallelism demands. However, this architecture introduces a sharp trade-off: data parallelism requires global synchronization across *all* GPUs, crossing all rails. Modern clusters therefore employ a hybrid approach, using rail-optimized leaves for tensor-parallel traffic while bridging the rails with a full fat-tree spine to support the global AllReduce patterns of data parallelism. +::: {.callout-checkpoint title="Rail-Optimized Networks"} + +Verify your understanding of workload-specific network design: + +- [ ] Which dimension of 3D Parallelism—TP, PP, or DP—is the primary beneficiary of a **Rail-Optimized** design? +- [ ] Why does a rail-optimized network reduce the number of switch hops for activation exchanges compared to a standard fat-tree? +- [ ] What is the "Traffic Isolation" benefit: why do we want all GPU 0s on one rail and all GPU 1s on another? +- [ ] True or False: A rail-optimized network eliminates the need for high-bandwidth core switches connecting different rails. + +::: + ### Dragonfly and Torus Alternatives {#sec-network-fabrics-dragonfly} \index{dragonfly topology}\index{torus} @@ -1191,11 +1339,11 @@ Real fabrics deviate from theoretical full-bisection bandwidth due to **congesti ::: {.callout-definition title="Bulk Synchronous Parallel (BSP)"} -***Bulk Synchronous Parallel (BSP)***\index{Bulk Synchronous Parallel!definition} is a parallel computing model where execution proceeds in discrete **Supersteps**, each consisting of local computation, global communication, and a barrier synchronization. +***Bulk Synchronous Parallel (BSP)***\index{Bulk Synchronous Parallel!definition} is a parallel execution model in which every worker completes a local computation phase, exchanges data with all other workers, and then waits at a global barrier before any worker begins the next phase — making the slowest participant the pacing constraint for the entire cluster. -1. **Significance (Quantitative):** It is the standard execution model for synchronous distributed training. Within the **Iron Law**, BSP dictates that the total time $T$ is governed by the **Slowest Worker** (Straggler), making the **System Efficiency ($\eta$)** highly sensitive to network jitter and hardware variation. -2. **Distinction (Durable):** Unlike **Asynchronous Parallelism**, which allows workers to use stale weights, BSP ensures **Mathematical Equivalence** to single-device training by enforcing a global state update at every step. -3. **Common Pitfall:** A frequent misconception is that BSP is "inefficient" compared to async models. In reality, while it has higher **Synchronization Overhead ($L_{\text{lat}}$)**, it provides superior **Convergence Stability**, often leading to lower total operations ($O$) to reach a target loss. +1. **Significance (Quantitative):** BSP makes system efficiency $\eta$ directly proportional to the slowest worker: if one GPU in a 1,024-GPU cluster runs 10% slower due to thermal throttling or network jitter, the barrier stalls the remaining 1,023 GPUs for that fraction of the step, wasting effectively 102 GPU-steps of compute per iteration. At \$3/GPU-hour, a 5% straggler gap across a \$50M training run wastes roughly \$2.5M in idle accelerator time. +2. **Distinction (Durable):** Unlike asynchronous parallelism, which allows workers to proceed with stale weights from previous steps, BSP enforces a global state update at every barrier, providing mathematical equivalence to single-device training and predictable convergence behavior. +3. **Common Pitfall:** A frequent misconception is that BSP is simply inefficient compared to async models. Asynchronous training often requires more total steps to converge because stale gradient updates introduce noise — in practice, BSP with careful straggler mitigation typically reaches the same loss in fewer wall-clock hours than async alternatives. ::: @@ -1207,11 +1355,11 @@ Because BSP enforces a global barrier at each superstep, the entire cluster move ::: {.callout-definition title="Priority Flow Control"} -***Priority Flow Control (PFC)***\index{Priority Flow Control!definition} is a link-level mechanism that prevents buffer overflow by sending PAUSE frames to the upstream sender on a per-priority basis. +***Priority Flow Control (PFC)***\index{Priority Flow Control!definition} is a link-layer mechanism that prevents switch buffer overflow by sending PAUSE frames to an upstream sender when a port's queue depth crosses a configured threshold, throttling injection on a per-priority basis without dropping packets. -1. **Significance (Quantitative):** It is the foundation for **Lossless Ethernet** (RoCE), ensuring that RDMA traffic is not dropped due to buffer exhaustion. This maintains the high **Effective Bandwidth ($\text{BW}$)** required for gradient synchronization. -2. **Distinction (Durable):** Unlike **Standard Flow Control** (which pauses all traffic on a link), PFC allows latency-sensitive control traffic to proceed while only pausing the congested data-intensive queues. -3. **Common Pitfall:** A frequent misconception is that PFC "solves" congestion. In reality, it transforms packet loss into **Congestion Spreading**: a single slow receiver can trigger a backpressure cascade that freezes unrelated flows across the entire fabric, significantly increasing the **Tail Latency ($L_{\text{lat}}$)** of the cluster. +1. **Significance (Quantitative):** PFC is the foundation for lossless Ethernet required by RoCEv2 RDMA. A PFC PAUSE frame must reach the upstream sender within one round-trip time — roughly 1–5 μs at switch-to-switch distances — before the buffer overflows. In a 1,000-node cluster, a single slow receiver can trigger PAUSE frames that propagate 3–5 hops upstream within 10–50 ms, halting gradient traffic across thousands of unrelated GPU pairs and collapsing cluster throughput to near zero. +2. **Distinction (Durable):** Unlike standard IEEE 802.3 flow control, which pauses all traffic classes on a link, PFC operates per priority class, allowing latency-sensitive control messages to continue flowing while only pausing the congested gradient data queue. +3. **Common Pitfall:** A frequent misconception is that PFC solves congestion. It transforms packet loss into congestion spreading: a backpressure cascade can freeze the entire fabric within 200 ms of a single faulty transceiver (as in the 2022 incident documented in this chapter), because no mechanism limits how far PAUSE frames propagate across the network. ::: @@ -1227,7 +1375,7 @@ The danger of PFC lies in its cascading nature. When a switch port's buffer fill \definecolor{RedL}{HTML}{F5D2D5} \definecolor{GreenLine}{HTML}{008F45} \definecolor{GreenL}{HTML}{D4EFDF} - + \tikzset{ switch/.style={draw=black!70, fill=gray!10, line width=0.75pt, rounded corners=2pt, minimum width=1.5cm, minimum height=1cm, font=\bfseries, align=center}, gpu/.style={draw=black!70, circle, fill=black!10, line width=0.75pt, minimum size=0.6cm, inner sep=0pt, font=\bfseries}, @@ -1295,6 +1443,56 @@ The insidious aspect was that no alarms fired. PFC pauses are a *normal* flow-co This incident illustrates why InfiniBand's credit-based flow control is fundamentally more robust: credits are consumed and replenished per-hop, so a slow receiver cannot propagate backpressure beyond its immediate neighbor. PFC, by contrast, operates as a blunt per-priority mechanism that has no concept of individual flow isolation. +```{python} +#| label: pfc-storm-probability +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ PFC STORM PROBABILITY (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Probability of a PFC Storm" .callout-notebook +# │ +# │ Goal: Quantify the risk of congestion spreading in a large RoCE cluster. +# │ Show: storm_prob_pct ≈ 12% — inline in notebook result. +# │ How: P(storm) = 1 - (1 - p_link_degrade)^N_links. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: pfc_n_gpus_str, pfc_p_degrade_str, pfc_storm_prob_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class PfcStormRisk: + # ┌── 1. LOAD ────────────────────────────────────────── + n_gpus = 4096 + links_per_gpu = 3 # leaf + spine + core + p_link_degrade = 1e-5 # 0.001% per day + # ┌── 2. EXECUTE ─────────────────────────────────────── + n_total_links = n_gpus * links_per_gpu + p_safe = (1 - p_link_degrade) ** n_total_links + p_storm = 1 - p_safe + # ┌── 3. GUARD (Invariants) ───────────────────────────────────────── + check(0.1 < p_storm < 0.2, f"Storm probability {p_storm:.2f} outside expected range") + # ┌── 4. OUTPUT (Formatting) ──────────────────────────────────────── + pfc_n_gpus_str = f"{n_gpus}" + pfc_p_degrade_str = "0.001%" + pfc_storm_prob_str = f"{p_storm * 100:.1f}" + +``` + +::: {.callout-notebook title="The Probability of a PFC Storm"} + +**Problem**: You operate a `{python} PfcStormRisk.pfc_n_gpus_str`-GPU RoCE cluster. If the probability of a single transceiver degrading and triggering PFC pauses is just `{python} PfcStormRisk.pfc_p_degrade_str` per day, what is the chance of a cluster-wide "PFC Storm" occurring today? + +**The Math**: +In a lossless Ethernet fabric, one bad link can pause its neighbor, which pauses its neighbor, eventually freezing the entire tree. + +1. **Total links**: 4,096 GPUs $\times$ 3 tiers = ~12,288 links. +2. **Probability of zero failures**: $P(\text{safe}) = (1 - 0.00001)^{12,288} \approx 0.88$. +3. **Probability of at least one storm**: $P(\text{storm}) = 1 - 0.88 = \mathbf{0.12}$ (12%). + +**The Systems Insight**: Even with "five-nines" reliable components, a large-scale RoCE cluster faces a **`{python} PfcStormRisk.pfc_storm_prob_str`% daily risk** of a fabric-wide freeze. This is why RoCE deployments require aggressive **PFC Watchdogs** and hardware-level telemetry—at scale, "rare" link degradations become daily operational realities that must be handled automatically. + +::: + ::: ### Proactive Congestion Control: DCQCN and HPCC {#sec-network-fabrics-congestion-control} @@ -1332,11 +1530,11 @@ This mechanism becomes essential for Mixture-of-Experts (MoE) models. Unlike the ::: {.callout-definition title="Incast"} -***Incast***\index{Incast!definition} is a many-to-one traffic pattern where multiple senders simultaneously transmit data to a single receiver, overwhelming the receiver's switch port buffer. +***Incast***\index{Incast!definition} is a many-to-one traffic pattern in which a large number of senders simultaneously transmit data to a single receiver port, concentrating line-rate traffic from multiple sources into a single switch queue and causing buffer overflow even when the rest of the fabric is uncongested. -1. **Significance (Quantitative):** In ML fleets, incast occurs during the **Reduce Phase** of collective operations (e.g., AllReduce). It causes momentary buffer exhaustion, triggering packet drops or flow-control pauses that increase the **Tail Latency ($L_{\text{lat}}$)** of the entire cluster. -2. **Distinction (Durable):** Unlike **General Congestion** (which occurs on shared core links), Incast is an **Endpoint Bottleneck**: it occurs even if the internal network fabric has infinite capacity. -3. **Common Pitfall:** A frequent misconception is that incast is "rare." In reality, because ML training is **Synchronously Burstive**, incast is a routine event at the end of every layer's gradient computation, requiring proactive mitigation through jitter-aware scheduling or large switch buffers. +1. **Significance (Quantitative):** In the reduce phase of AllReduce, every participating GPU simultaneously sends gradients toward the same aggregation points. With 256 senders each at 50 GB/s targeting one switch port, the instantaneous incast exceeds 12.8 TB/s — orders of magnitude above a 400 Gb/s port's absorption capacity. A 32 MB switch buffer absorbs approximately 640 μs at 400 Gb/s before overflowing, triggering either PFC cascade or packet drops that elevate $L_{\text{lat}}$ cluster-wide. +2. **Distinction (Durable):** Unlike general congestion (which occurs on shared internal links when aggregate traffic exceeds link capacity), incast is an endpoint bottleneck: it occurs even if every internal spine and leaf link is completely uncongested, because the bottleneck is the single destination port, not the fabric interior. +3. **Common Pitfall:** A frequent misconception is that incast is rare or unpredictable. Because ML training synchronizes all GPUs at each backward pass, incast is a deterministic event at the end of every layer's gradient computation — it occurs hundreds of times per training step and must be architecturally mitigated through layer-staggered AllReduce or tree-based collective algorithms, not treated as an edge case. ::: diff --git a/book/quarto/contents/vol2/ops_scale/ops_scale.qmd b/book/quarto/contents/vol2/ops_scale/ops_scale.qmd index 3afd570ac4..41afb96fad 100644 --- a/book/quarto/contents/vol2/ops_scale/ops_scale.qmd +++ b/book/quarto/contents/vol2/ops_scale/ops_scale.qmd @@ -105,6 +105,58 @@ As the number of models grows, several problems emerge that are not simply multi : **Operational Complexity Growth at Scale**: Six dimensions of operational complexity across 1, 10, and 100 models. Deployment coordination evolves from nonexistent to critical path, monitoring dashboards become unmanageable without aggregation, and debugging shifts from local investigation to organization-wide distributed tracing requirements. {#tbl-ops-scale-complexity} +```{python} +#| label: multi-tenant-efficiency-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ MULTI-TENANT EFFICIENCY DIVIDEND (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Sharing Dividend" .callout-notebook +# │ +# │ Goal: Quantify the hardware savings from multi-tenant resource sharing. +# │ Show: cost_savings_pct ≈ 40% — inline in notebook result. +# │ How: Cost_Shared = Cost_Dedicated * (1 - Idle_Reduction). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: mt_idle_dedicated_str, mt_idle_shared_str, mt_savings_pct_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class MultiTenantEfficiency: + # ┌── 1. LOAD ────────────────────────────────────────── + avg_util_dedicated = 0.30 # 70% idle (typical for single-team clusters) + avg_util_shared = 0.70 # 30% idle (sharing across training/inference) + n_gpus_required = 100 + gpu_cost_hour = 2.0 # $/hr + # ┌── 2. EXECUTE ─────────────────────────────────────── + # For a fixed workload of 30 "active" GPU-hours: + # Dedicated needs: 30 / 0.30 = 100 GPUs + # Shared needs: 30 / 0.70 = 43 GPUs + savings_pct = (1 - (avg_util_dedicated / avg_util_shared)) * 100 + # ┌── 3. GUARD ───────────────────────────────────────── + check(55 < savings_pct < 60, f"Savings {savings_pct:.1f}% unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + mt_idle_dedicated_str = "70%" + mt_idle_shared_str = "30%" + mt_savings_pct_str = f"{savings_pct:.0f}" + +``` + +::: {.callout-notebook title="The Sharing Dividend"} + +**Problem**: You manage a fleet of 100 GPUs. Currently, each team has its own dedicated quota, resulting in **`{python} MultiTenantEfficiency.mt_idle_dedicated_str` average idle time**. You move to a **Multi-Tenant ML Platform** that shares resources across teams and uses idle training GPUs for inference. This reduces aggregate idle time to **`{python} MultiTenantEfficiency.mt_idle_shared_str`**. How much hardware cost do you save? + +**The Math**: +Hardware efficiency is the inverse of the utilization rate. + +1. **Efficiency Gain**: 0.70 (Shared) / 0.30 (Dedicated) = **2.33$\times$ more work per GPU**. +2. **Hardware Reduction**: 1 - (0.30 / 0.70) = **`{python} MultiTenantEfficiency.mt_savings_pct_str`%**. +3. **Annual Savings**: 57% of \$1.75M budget $\approx$ **\$1,000,000/year**. + +**The Systems Insight**: Multi-tenancy is an **Infrastructure Multiplier**. By breaking down resource silos, you effectively "buy" 57% more GPUs for free. In the Machine Learning Fleet, **Statistical Multiplexing** (the principle that different teams' peak demands won't happen at the same time) is the secret to economic sustainability. The Platform team's primary job is to harvest this "Sharing Dividend" to fund future growth. + +::: + This table reveals the fundamental insight: per-model operational practices do not compose. When Model A depends on features computed by Pipeline B, which uses embeddings from Model C, changes to any component can cascade unpredictably. A seemingly innocuous update to Model C's embedding layer might shift the feature distributions that Model A depends upon, degrading its performance even though Model A itself has not changed. This cascading interdependence is at the heart of what we call the *complexity explosion*. ::: {.callout-perspective title="The Complexity Explosion"} @@ -192,12 +244,59 @@ where $N_{models}$ represents the number of models benefiting from the platform, This equation reveals why platform investments make sense only at sufficient scale. For a small organization with five models, the denominator might exceed the numerator even with significant per-model savings. As model count grows, the numerator scales linearly with $N_{models}$ while platform costs grow much more slowly, typically sublinearly due to infrastructure amortization. -#### Worked Example: Platform ROI Calculation - ```{python} -#| label: platform-roi-calc +#| label: platform-roi-notebook #| echo: false # ┌───────────────────────────────────────────────────────────────────────────── +# │ PLATFORM ROI (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Platform Dividend" .callout-notebook +# │ +# │ Goal: Quantify the ROI of building a centralized ML platform. +# │ Show: roi_ratio ≈ 12x — inline in notebook result. +# │ How: ROI = (N_models * Savings_H * Rate) / Platform_Cost. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: pr_n_models_str, pr_savings_h_str, pr_roi_ratio_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class PlatformRoi: + # ┌── 1. LOAD ────────────────────────────────────────── + n_models = 50 + savings_h_per_month = 20 # Engineering time saved per model + engineer_rate = 150 # $/hr + platform_cost_per_month = 120000 # Platform team + infra + # ┌── 2. EXECUTE ─────────────────────────────────────── + total_savings = n_models * savings_h_per_month * engineer_rate + roi_ratio = total_savings / platform_cost_per_month + # ┌── 3. GUARD ───────────────────────────────────────── + check(roi_ratio == 1.25, f"ROI {roi_ratio} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + pr_n_models_str = f"{n_models}" + pr_savings_h_str = f"{savings_h_per_month}" + pr_roi_ratio_str = f"{roi_ratio:.2f}" + +``` + +::: {.callout-notebook title="The Platform Dividend"} + +**Problem**: Your organization manages `{python} PlatformRoi.pr_n_models_str` models. A centralized ML Platform team costs \$120,000/month. If the platform saves each model team `{python} PlatformRoi.pr_savings_h_str` hours of manual toil per month, is the platform investment profitable? + +**The Math**: +1. **Gross Monthly Savings**: 50 models $\times$ 20 hrs/model $\times$ \$150/hr = **\$150,000**. +2. **Net Monthly Benefit**: \$150,000 (Savings) - \$120,000 (Cost) = **\$30,000**. +3. **ROI Ratio**: \$150,000 / \$120,000 = **`{python} PlatformRoi.pr_roi_ratio_str`**. + +**The Systems Insight**: Platforms exhibit a **Scaling Threshold**. At 50 models, this platform earns a **25% return** on its cost. However, if the organization only had 20 models, the savings would be only \$60,000—a 50% *loss* on the platform team's salary. In MLOps, platform engineering is a **Fixed Cost** that pays off through **Variable Savings**. You build a platform not when it's convenient, but when the "Manual Toil Tax" across your model fleet exceeds the "Platform Maintenance Tax." + +::: + +#### From Artisanal to Industrial Operations {#sec-ml-operations-scale-artisanal-to-industrial-operations-092d} + +```{python} +#| label: platform-economics-lego +#| echo: false # │ PLATFORM ROI CALCULATION (LEGO) # ├───────────────────────────────────────────────────────────────────────────── # │ Context: @sec-ml-operations-scale-quantifying-platform-economics-2026 @@ -266,10 +365,12 @@ Consider an organization evaluating whether to build a centralized ML platform. - Expected time savings: 30 hours per model per month post-platform Before platform (annual operational cost): -$$C_{\text{current}} = 50 \times 40 \times 12 \times \$150 = \$`{python} PlatformEconomics.annual_cost_before_str`$$ + +$C_{\text{current}} = 50 \times 40 \times 12 \times 150 =$ USD `{python} PlatformEconomics.annual_cost_before_str` After platform (annual operational cost plus amortized platform cost): -$$C_{\text{after}} = 50 \times 10 \times 12 \times \$150 + \frac{\$2,000,000}{3} = \$900,000 + \$667,000 = \$`{python} PlatformEconomics.annual_cost_after_str`$$ + +$C_{\text{after}} = 50 \times 10 \times 12 \times 150 + \frac{2{,}000{,}000}{3} = 900{,}000 + 667{,}000 =$ USD `{python} PlatformEconomics.annual_cost_after_str` This yields annual savings of \$`{python} PlatformEconomics.annual_savings_str`, representing a `{python} PlatformEconomics.savings_pct_str`% reduction in operational costs. The platform pays for itself within the first year. @@ -384,7 +485,62 @@ Cost visibility also sharpens the case for paying down technical debt: the same ### Quantifying and Managing ML Technical Debt {#sec-ml-operations-scale-quantifying-managing-ml-technical-debt-9055} -ML technical debt falls into four primary categories (data, configuration, model, and infrastructure debt), each requiring quantitative measurement at platform scale. Moving beyond awareness to action requires frameworks for measuring and prioritizing debt paydown across a portfolio of models. Technical debt manifests in measurable operational symptoms that directly impact platform velocity and reliability. +ML technical debt falls into four primary categories (data, configuration, model, and infrastructure debt), each requiring quantitative measurement at platform scale. + +```{python} +#| label: maintenance-dividend-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ MAINTENANCE DIVIDEND (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Maintenance Dividend" .callout-notebook +# │ +# │ Goal: Quantify the long-term cost of technical debt vs proactive maintenance. +# │ Show: savings_ratio ≈ 3.2x — inline in notebook result. +# │ How: Cost = N_Years * (Toil_H * Rate); Maintenance = Setup + N_Years * (Low_Toil * Rate). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: md_toil_hours_str, md_low_toil_str, md_savings_ratio_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class MaintenanceDividend: + # ┌── 1. LOAD ────────────────────────────────────────── + toil_h_per_month = 40 # manual monitoring, data cleaning, broken scripts + low_toil_h_per_month = 8 # after platform automation + maintenance_investment_h = 160 # engineering time to fix "plumbing" + years = 3 + engineer_rate = 150 # $/hr + # ┌── 2. EXECUTE ─────────────────────────────────────── + months = years * 12 + cost_toil = months * toil_h_per_month * engineer_rate + cost_proactive = (maintenance_investment_h + (months * low_toil_h_per_month)) * engineer_rate + savings = cost_toil - cost_proactive + ratio = cost_toil / cost_proactive + # ┌── 3. GUARD ───────────────────────────────────────── + check(3 < ratio < 4, f"Ratio {ratio:.1f} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + md_toil_hours_str = f"{toil_h_per_month}" + md_low_toil_str = f"{low_toil_h_per_month}" + md_savings_ratio_str = f"{ratio:.1f}" + +``` + +::: {.callout-notebook title="The Maintenance Dividend"} + +**Problem**: Your team spends `{python} MaintenanceDividend.md_toil_hours_str` hours/month manually fixing "broken plumbing" (stale data, failed scripts, manual monitoring). You estimate that a one-month intensive cleanup (160 hours) would reduce this to `{python} MaintenanceDividend.md_low_toil_str` hours/month. Is the cleanup worth it over a 3-year model lifecycle? + +**The Math**: +1. **Status Quo (3 years)**: 36 months $\times$ 40 hrs/month $\times$ \$150/hr = **\$216,000**. +2. **Proactive Path**: (160 hrs investment + 36 months $\times$ 8 hrs/month) $\times$ \$150/hr = **\$67,200**. +3. **Net Savings**: \$216,000 - \$67,200 = **\$148,800**. +4. **Dividend Ratio**: **`{python} MaintenanceDividend.md_savings_ratio_str`$\times$**. + +**The Systems Insight**: Proactive maintenance earns a **`{python} MaintenanceDividend.md_savings_ratio_str`$\times$ return** on engineering time. In the ML Fleet, "Plumbing" is more important than "Pipes": an organization that ignores technical debt eventually spends its entire budget just keeping old models alive, leaving zero capacity for new development. The most successful teams treat **Refactoring** as a high-yield investment, not a distraction. + +::: + +Moving beyond awareness to action requires frameworks for measuring and prioritizing debt paydown across a portfolio of models. Technical debt manifests in measurable operational symptoms that directly impact platform velocity and reliability. #### Debt Categories and Measurement @@ -455,6 +611,58 @@ The debt paydown decision follows the formula: $$\text{Paydown Priority} = \frac{\text{Impact} \times \text{Frequency} \times \text{Benefit}}{\text{Resolution Cost}}$$ +```{python} +#| label: deployment-roi-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ DEPLOYMENT ROI (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "ROI of Automation" .callout-notebook +# │ +# │ Goal: Quantify the financial return of automating a manual deployment pipeline. +# │ Show: payback_weeks ≈ 4.2 weeks — inline in notebook result. +# │ How: Cost_Manual = N * Toil_H * Rate; Cost_Auto = Setup + N * Auto_H * Rate. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: roi_engineer_rate_str, roi_manual_hours_str, roi_auto_hours_str, roi_payback_weeks_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class DeploymentRoi: + # ┌── 1. LOAD ────────────────────────────────────────── + engineer_rate = 150 # $/hour + manual_hours_per_deploy = 10 + auto_hours_per_deploy = 0.5 + automation_setup_hours = 120 + deploys_per_week = 3 + # ┌── 2. EXECUTE ─────────────────────────────────────── + savings_per_deploy = (manual_hours_per_deploy - auto_hours_per_deploy) * engineer_rate + weekly_savings = savings_per_deploy * deploys_per_week + automation_cost = automation_setup_hours * engineer_rate + payback_weeks = automation_cost / weekly_savings + # ┌── 3. GUARD ───────────────────────────────────────── + check(4 < payback_weeks < 5, f"Payback {payback_weeks:.1f} weeks unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + roi_engineer_rate_str = f"{engineer_rate}" + roi_manual_hours_str = f"{manual_hours_per_deploy}" + roi_auto_hours_str = f"{auto_hours_per_deploy}" + roi_payback_weeks_str = f"{payback_weeks:.1f}" + +``` + +::: {.callout-notebook title="ROI of Automation"} + +**Problem**: Your team spends `{python} DeploymentRoi.roi_manual_hours_str` hours of manual toil per model deployment. You estimate that spending `{python} DeploymentRoi.automation_setup_hours` hours building a CI/CD pipeline will reduce deployment toil to `{python} DeploymentRoi.roi_auto_hours_str` hours. If you deploy `{python} DeploymentRoi.deploys_per_week` times a week, how long until the automation pays for itself? + +**The Math**: +1. **Automation Cost**: `{python} DeploymentRoi.automation_setup_hours` hours $\times$ \${python} DeploymentRoi.roi_engineer_rate_str}/hr = **\$18,000**. +2. **Weekly Savings**: (10 - 0.5) hrs/deploy $\times$ 3 deploys/week $\times$ \${python} DeploymentRoi.roi_engineer_rate_str}/hr = **\$4,275/week**. +3. **Payback Period**: \$18,000 / \$4,275 $\approx$ **`{python} DeploymentRoi.roi_payback_weeks_str` weeks**. + +**The Systems Insight**: Automation is not just about reducing errors; it is a high-yield capital investment. A payback period of **`{python} DeploymentRoi.roi_payback_weeks_str` weeks** is an exceptional return on engineering time. In MLOps, "Toil" is the highest-interest technical debt you can carry—paying it down early yields massive dividends for the rest of the model's lifecycle. + +::: + Pay debt when this ratio exceeds the expected value from feature development. For the configuration debt above: high impact (blocks deployments), high frequency (every deployment), high benefit (eliminates 35% of delays), moderate cost (6 weeks). Defer debt when: debt is localized to single team, frequency is low (monthly or less), system sunset is planned within 12 months, or resolution cost exceeds 6 months of engineering effort. @@ -836,7 +1044,7 @@ serving: ownership: team: recommendation-core - oncall: recsys-oncall@company.com + oncall: recsys-oncall-team ``` ::: @@ -1145,7 +1353,7 @@ training_pipeline: optimizer: adam hardware: 4xA100 evaluation: - metrics: [ndcg@10, mrr, coverage] + metrics: [ndcg_10, mrr, coverage] baseline_model: ranking_v2.1.0 ``` ::: @@ -1175,15 +1383,15 @@ def evaluate_performance_gate( reasons = [] # Absolute threshold check - if candidate_metrics["ndcg@10"] < thresholds["min_ndcg"]: + if candidate_metrics["ndcg_10"] < thresholds["min_ndcg"]: reasons.append( - f"NDCG@10 {candidate_metrics['ndcg@10']:.4f} below minimum {thresholds['min_ndcg']}" + f"NDCG_10 {candidate_metrics['ndcg_10']:.4f} below minimum {thresholds['min_ndcg']}" ) # Relative improvement check relative_improvement = ( - candidate_metrics["ndcg@10"] - production_metrics["ndcg@10"] - ) / production_metrics["ndcg@10"] + candidate_metrics["ndcg_10"] - production_metrics["ndcg_10"] + ) / production_metrics["ndcg_10"] if relative_improvement < thresholds["min_improvement"]: reasons.append( f"Improvement {relative_improvement:.2%} below minimum {thresholds['min_improvement']:.2%}" @@ -1243,6 +1451,54 @@ Data quality gates catch issues that would otherwise manifest as mysterious mode Deploying models to production should proceed gradually, with increasing traffic exposure contingent on continued acceptable performance. +```{python} +#| label: deployment-safety-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ DEPLOYMENT SAFETY BUDGET (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Safety of Staged Rollouts" .callout-notebook +# │ +# │ Goal: Quantify the 'Risk Exposure' reduction from canary deployments. +# │ Show: risk_reduction ≈ 20x — inline in notebook result. +# │ How: Exposure = traffic_pct * detection_time. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: ds_canary_pct_str, ds_risk_reduction_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class DeploymentSafety: + # ┌── 1. LOAD ────────────────────────────────────────── + canary_pct = 0.05 # 5% canary traffic + detection_time_h = 2 # hours to detect a major crash/bug + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Risk exposure is the expected number of broken requests + # Ratio of exposure: (100% * T) / (5% * T) + risk_reduction = 1.0 / canary_pct + # ┌── 3. GUARD ───────────────────────────────────────── + check(risk_reduction == 20.0, f"Risk reduction {risk_reduction} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + ds_canary_pct_str = f"{canary_pct*100:.0f}%" + ds_risk_reduction_str = f"{risk_reduction:.0f}" + +``` + +::: {.callout-notebook title="The Safety of Staged Rollouts"} + +**Problem**: You are deploying a new ranking model. A "Blue-Green" deployment (100% cutover) exposes all users to any potential bugs. A "Canary" deployment starts at `{python} DeploymentSafety.ds_canary_pct_str` traffic. How much does the canary approach reduce your organization's "Risk Exposure"? + +**The Math**: +Risk exposure is proportional to the traffic percentage affected during the detection window. + +1. **Blue-Green Exposure**: 100% of users affected until rollback. +2. **Canary Exposure**: `{python} DeploymentSafety.ds_canary_pct_str` of users affected until rollback. +3. **Risk Mitigation**: 100 / 5 = **`{python} DeploymentSafety.ds_risk_reduction_str`$\times$**. + +**The Systems Insight**: Staged rollouts are an **Insurance Policy for Model Quality**. By limiting initial exposure to `{python} DeploymentSafety.ds_canary_pct_str`, you reduce the blast radius of a catastrophic failure by **`{python} DeploymentSafety.ds_risk_reduction_str`$\times$**. In the Machine Learning Fleet, where model behavior is probabilistic and hard to unit-test, gradual exposure is the only reliable way to ensure that "SOTA on paper" doesn't become "Broken in Production." + +::: + #### Blue-Green Deployment Blue-green deployment maintains two identical production environments. The current version (blue) serves traffic while the new version (green) is prepared. Once ready, traffic switches instantaneously to green. @@ -1287,6 +1543,57 @@ The organization might configure: Total rollout: approximately 4 hours for a confident deployment. +```{python} +#| label: silent-failure-cost-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ SILENT FAILURE COST (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Cost of a Delayed Alert" .callout-notebook +# │ +# │ Goal: Quantify the financial loss from a silent model regression. +# │ Show: total_loss ≈ $1,080,000 — inline in notebook result. +# │ How: Loss = QPS * T_Detection * (CTR_baseline - CTR_new) * Value_per_click. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: sf_qps_str, sf_detection_hours_str, sf_ctr_drop_str, sf_total_loss_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class SilentFailure: + # ┌── 1. LOAD ────────────────────────────────────────── + qps = 5000 + t_detection_h = 24 # 1 day to detect and rollback + ctr_baseline = 0.05 + ctr_new = 0.045 # 0.5% absolute drop (10% relative) + value_per_click = 0.50 # $0.50 per click + # ┌── 2. EXECUTE ─────────────────────────────────────── + total_requests = qps * 3600 * t_detection_h + lost_clicks = total_requests * (ctr_baseline - ctr_new) + total_loss = lost_clicks * value_per_click + # ┌── 3. GUARD ───────────────────────────────────────── + check(1000000 < total_loss < 1500000, f"Loss ${total_loss:,.0f} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + sf_qps_str = f"{qps:,}" + sf_detection_hours_str = f"{t_detection_h}" + sf_ctr_drop_str = "0.5%" + sf_total_loss_str = f"{total_loss:,.0f}" + +``` + +::: {.callout-notebook title="The Cost of a Delayed Alert"} + +**Problem**: You deploy a recommendation model that has a silent bug: it reduces Click-Through Rate (CTR) by `{python} SilentFailure.sf_ctr_drop_str` (from 5.0% to 4.5%). If your service handles `{python} SilentFailure.sf_qps_str` QPS and each click is worth \$0.50, how much money do you lose if it takes `{python} SilentFailure.sf_detection_hours_str` hours to detect and fix the issue? + +**The Math**: +1. **Total Requests**: `{python} SilentFailure.sf_qps_str` requests/s $\times$ 86,400 seconds = **432,000,000 requests**. +2. **Lost Clicks**: 432M $\times$ 0.005 = **2,160,000 lost clicks**. +3. **Total Financial Loss**: 2.16M $\times$ \$0.50 = **\$`{python} SilentFailure.sf_total_loss_str`**. + +**The Systems Insight**: A "minor" 0.5% regression in a high-traffic model is a **\$1.08 Million daily disaster**. This is why MLOps is not just about "keeping things running"; it's about **Economic Risk Management**. Every hour you shave off your "Time-to-Detection" via Canary deployments and automated drift monitoring translates directly into saved revenue. In the Fleet Stack, monitoring is the "Insurance Policy" that protects your model's business value. + +::: + ### Multi-Region Deployment Coordination {#sec-ml-operations-scale-multiregion-deployment-coordination-ab6a} @sec-fault-tolerance-reliability established checkpointing, elastic training, and recovery mechanisms for handling failures within distributed training jobs. Multi-region deployment extends these fault tolerance principles to the inference plane, where coordination across geographic regions introduces challenges absent from single-region operations. Model version consistency, traffic routing during transitions, and coordinated rollback require explicit protocol design to prevent mixed-version serving that can corrupt A/B test validity and user experience. @@ -2007,7 +2314,7 @@ The problem: Consider a system monitoring **`{python} FalseAlarmTax.n_models` mo **The Math**: 1. **Total Monitors**: `{python} FalseAlarmTax.n_models` models $\times$ `{python} FalseAlarmTax.n_metrics` metrics = **`{python} FalseAlarmTax.n_monitors_str` monitors**. -2. **False Positive Rate**: $1 - `{python} FalseAlarmTax.specificity_val_str` = `{python} FalseAlarmTax.fpr_str`$ (`{python} FalseAlarmTax.fpr_pct_str`%). +2. **False Positive Rate**: $1 -$ `{python} FalseAlarmTax.specificity_val_str` $=$ `{python} FalseAlarmTax.fpr_str` (`{python} FalseAlarmTax.fpr_pct_str`%). 3. **Checks per Day**: Assume checks every `{python} FalseAlarmTax.check_interval_min_str` minutes (**`{python} FalseAlarmTax.checks_per_day_str` checks/day**). 4. **Daily False Alarms**: $`{python} FalseAlarmTax.n_monitors_str` \times `{python} FalseAlarmTax.checks_per_day_str` \times `{python} FalseAlarmTax.fpr_str` \approx \mathbf{`{python} FalseAlarmTax.daily_alerts_str` alerts/day}$. @@ -2201,6 +2508,59 @@ where $A_i$ is the proportion in bucket $i$ of the actual (current) distribution Interpretation depends on the value. PSI below 0.1 indicates no significant shift. Values between 0.1 and 0.25 indicate moderate shift where investigation is recommended. Values at or above 0.25 indicate significant shift requiring action. +```{python} +#| label: drift-latency-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ DRIFT DETECTION LATENCY (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Time-to-Detection: The Monitoring Lag" .callout-notebook +# │ +# │ Goal: Quantify the number of samples required to detect a 2% accuracy drop. +# │ Show: n_samples ≈ 9,600 — inline in notebook result. +# │ How: N = (Z_alpha + Z_beta)^2 * (p1*q1 + p2*q2) / (p1 - p2)^2. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: dl_base_acc_str, dl_drop_str, dl_samples_needed_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check +import math + +class DriftLatency: + # ┌── 1. LOAD ────────────────────────────────────────── + p1 = 0.95 # baseline accuracy + p2 = 0.93 # degraded accuracy (2% drop) + alpha = 0.05 # 95% confidence + power = 0.80 # 80% power + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Z-scores for alpha/power + z_a = 1.96 + z_b = 0.84 + avg_p = (p1 + p2) / 2 + n = (z_a + z_b)**2 * (p1*(1-p1) + p2*(1-p2)) / (p1 - p2)**2 + # ┌── 3. GUARD ───────────────────────────────────────── + check(9000 < n < 10000, f"Samples {n:.0f} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + dl_base_acc_str = f"{p1*100:.0f}%" + dl_drop_str = f"{(p1-p2)*100:.0f}%" + dl_samples_needed_str = f"{int(n):,}" + +``` + +::: {.callout-notebook title="Time-to-Detection: The Monitoring Lag"} + +**Problem**: Your model has a baseline accuracy of `{python} DriftLatency.dl_base_acc_str`. A data drift event causes accuracy to drop by `{python} DriftLatency.dl_drop_str`. If your service receives 1,000 requests per hour with labels, how long until you can statistically prove the model has degraded? + +**The Math**: +Detection requires enough samples to distinguish the signal (the 2% drop) from the noise (random variance). + +1. **Samples Required**: Using a two-sample proportion test, detecting a 2% drop with 95% confidence requires **`{python} DriftLatency.dl_samples_needed_str` samples**. +2. **Detection Latency**: `{python} DriftLatency.dl_samples_needed_str` samples / 1,000 samples/hr $\approx$ **9.6 hours**. + +**The Systems Insight**: Monitoring is not instantaneous. For a 2% degradation, your model will operate in a broken state for nearly **10 hours** before you have enough data to trigger an alert. This is the **Monitoring Lag**. To reduce it, you must either increase the volume of labeled data (expensive) or monitor "proxy" metrics like **PSI** (faster but noisier). In a high-stakes fleet, you don't wait for accuracy to drop—you alert on **input drift** as a leading indicator. + +::: + Fleet-wide drift monitoring tracks PSI for critical features across all models, alerting when drift affects multiple models or critical features. ### Model-Type Specific Monitoring {#sec-ml-operations-scale-modeltype-specific-monitoring-b901} @@ -2554,7 +2914,7 @@ deployment: alerts: - metric: accuracy_degradation threshold: 0.05 - notification: model-team@company.com + notification: model-team-alerts ``` ::: @@ -2709,11 +3069,11 @@ ML workloads present unique cost management challenges that traditional IT FinOp ::: {.callout-definition title="FinOps for ML"} -***FinOps for ML***\index{FinOps!definition} is the engineering and financial discipline of treating compute cost as a first-class optimization variable alongside model accuracy and latency. +***FinOps for ML***\index{FinOps!definition} is the practice of treating compute cost as a first-class engineering constraint — measured in real-time per experiment and model, and optimized jointly with model accuracy and latency — rather than accounting for it retrospectively through annual budget reconciliation. -1. **Significance (Quantitative):** It establishes visibility into the **Economic Efficiency ($\eta_{cost}$)** of the fleet. By bridging the tension between experimentation velocity and budget constraints, FinOps enables data-driven decisions on resource allocation (e.g., Spot vs. On-Demand instances) and early stopping of non-productive runs. -2. **Distinction (Durable):** Unlike **Traditional IT Budgeting**, which focuses on static annual plans, FinOps for ML is **Dynamic and Usage-Based**, requiring real-time feedback loops between the orchestrator and the balance sheet. -3. **Common Pitfall:** A frequent misconception is that FinOps is "just cost cutting." In reality, it is **Value Optimization**: the goal is not to spend the least amount of money, but to maximize the **Accuracy-per-Dollar** across the entire lifecycle. +1. **Significance (Quantitative):** ML compute costs scale steeply with experimentation volume. A team running 1,000 GPU-hours/day at \$3/GPU-hour spends \$3,000/day — \$1.1M/year — on training alone. With per-experiment cost visibility, an engineer can identify that 30% of runs are terminated early due to divergence (recoverable with better hyperparameter selection) and that spot instances provide 60–70% cost reduction for fault-tolerant workloads, reducing effective spend by 40–50% without changing output quality. +2. **Distinction (Durable):** Unlike traditional IT budgeting (which allocates fixed annual compute budgets across departments), FinOps for ML operates at the per-run and per-model level with real-time feedback — enabling decisions like early stopping a \$50K training run at step 1,000 when loss curves signal divergence, rather than discovering the waste after the full run completes. +3. **Common Pitfall:** A frequent misconception is that FinOps means minimizing compute spend. The goal is maximizing accuracy-per-dollar across the full lifecycle: a \$500K training run producing a model serving 10B inferences at \$0.0001/query may be far more cost-efficient than a \$50K run producing a model that requires 10$\times$ the inference compute to achieve the same accuracy at scale. ::: diff --git a/book/quarto/contents/vol2/performance_engineering/performance_engineering.qmd b/book/quarto/contents/vol2/performance_engineering/performance_engineering.qmd index fe3a82d880..cb6b3bb4a6 100644 --- a/book/quarto/contents/vol2/performance_engineering/performance_engineering.qmd +++ b/book/quarto/contents/vol2/performance_engineering/performance_engineering.qmd @@ -104,6 +104,17 @@ Each technique attacks a different term, and this taxonomy guides optimization s @fig-iron-law-flowchart codifies this diagnostic process as a decision flowchart, mapping each bottleneck to its corresponding optimization technique. +::: {.callout-checkpoint title="The Iron Law of Performance"} + +Verify your understanding of system-level performance diagnosis: + +- [ ] A workload is **Memory-Bound**. Will upgrading the GPU's clock frequency (increasing FLOPS) improve performance? +- [ ] If you apply **Quantization** (moving from FP16 to INT4), which term in @eq-iron-law-perf are you reducing? +- [ ] What is the "Overhead" term in the Iron Law, and why does **Graph Compilation** target it specifically? +- [ ] True or False: If a system is Compute-Bound, the memory bandwidth is irrelevant to its performance. + +::: + ::: {#fig-iron-law-flowchart fig-env="figure" fig-pos="htb" fig-cap="**Iron Law Diagnostic Flowchart**. The optimization process begins with profiling to determine which term in @eq-iron-law-perf dominates. If the workload is compute-bound, precision engineering or algorithmic changes reduce the numerator. If memory-bound, operator fusion and tiling reduce HBM traffic. If overhead-bound, graph compilation and communication overlap attack the residual term. Applying the wrong technique yields zero improvement." fig-alt="Flowchart starting with Profile Workload, branching to Compute Bound, Memory Bound, or Overhead Bound, each leading to specific optimization techniques."} ```{.tikz} \begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, scale=0.85, transform shape, >=latex] @@ -1002,6 +1013,17 @@ In 2022, Tri Dao challenged the prevailing wisdom that the attention mechanism's @fig-flashattention-memory-savings quantifies the memory advantage across sequence lengths. Standard attention allocates the full $N \times N$ score matrix in HBM, while FlashAttention maintains only $O(N)$ running statistics. The gap widens quadratically: at a typical 8K context, FlashAttention uses 4,096$\times$ less attention memory; at 64K long-context, the savings reach 32,768$\times$. +::: {.callout-checkpoint title="FlashAttention Mechanics"} + +Verify your understanding of memory-aware attention: + +- [ ] Why does FlashAttention provide a larger speedup for **longer sequences** (e.g., 32K) than for short sequences (e.g., 512)? +- [ ] In the FlashAttention algorithm, where are the "online softmax" statistics stored: in HBM, L2 Cache, or SM SRAM? +- [ ] True or False: FlashAttention reduces the total number of floating-point operations (FLOPs) required for attention. +- [ ] How does reducing the attention memory from $O(N^2)$ to $O(N)$ enable larger batch sizes for serving? + +::: + ::: {#fig-flashattention-memory-savings fig-env="figure" fig-pos="htb" fig-cap="**FlashAttention Memory Savings**. $O(N^2) \rightarrow O(N)$. Standard attention allocates the full $N \times N$ score matrix in HBM, while FlashAttention maintains only $O(N)$ running statistics. The shaded region represents memory freed for KV cache, activations, or larger batch sizes. At long-context lengths (64K+), the savings exceed four orders of magnitude." fig-alt="Log-log plot of attention memory versus sequence length. Standard O(N²) curve rises steeply; FlashAttention O(N) stays flat. Shaded region shows savings, with annotations at 8K and 65K."} ```{python} #| echo: false diff --git a/book/quarto/contents/vol2/responsible_ai/responsible_ai.qmd b/book/quarto/contents/vol2/responsible_ai/responsible_ai.qmd index 1e180e8a46..13bb9a3185 100644 --- a/book/quarto/contents/vol2/responsible_ai/responsible_ai.qmd +++ b/book/quarto/contents/vol2/responsible_ai/responsible_ai.qmd @@ -91,11 +91,11 @@ This extension reflects the unprecedented scale of contemporary machine learning ::: {.callout-definition title="Responsible AI"} -***Responsible AI***\index{Responsible AI!definition} is the engineering discipline that systematically transforms ethical principles into concrete design requirements and measurable system properties. +***Responsible AI***\index{Responsible AI!definition} is the practice of designing, auditing, and operating ML systems to measurable fairness, safety, privacy, and accountability standards — translating ethical principles into verifiable system properties that constrain model training, deployment decisions, and operational monitoring. -1. **Significance (Quantitative):** It establishes **Safety and Fairness Constraints** as first-class system invariants. Within the **Iron Law**, responsible engineering ensures that the **Duty Cycle ($\eta$)** is maintained not just for performance, but for compliance with societal and regulatory norms, preventing costly system recalls or legal liabilities. -2. **Distinction (Durable):** Unlike **AI Ethics** (which focuses on the "Should"), Responsible AI focuses on the **"How"**: it provides the technical mechanisms (testing, auditing, and architectural guardrails) required to enforce ethical values in production. -3. **Common Pitfall:** A frequent misconception is that responsibility is a "tax" on performance. In reality, it is a **Reliability Foundation**: systems that are not responsible (e.g., those with hidden demographic biases) are fundamentally **Fragile** and prone to unpredictable failures when the deployment distribution ($D_{\text{vol}}$) shifts. +1. **Significance (Quantitative):** Responsible AI constraints impose real costs: fairness-aware training algorithms add 5–15% to training time; real-time bias monitoring adds 10–20 ms per inference; on-demand explainability can require 50–1,000$\times$ more compute than the inference itself. For a global fleet at 10 billion inferences per day, the responsible AI overhead often exceeds the raw model serving cost. Conversely, a facial recognition system misidentifying demographic groups at $\varepsilon = 5\%$ error parity failure can affect millions of users before detection, creating regulatory and liability costs that dwarf the monitoring investment. +2. **Distinction (Durable):** Unlike AI ethics (which defines normative principles about what systems *should* do), responsible AI engineering defines technical mechanisms that enforce those principles — bias detection algorithms, differential privacy implementations, audit trails, and architectural guardrails that make compliance measurable and verifiable rather than aspirational. +3. **Common Pitfall:** A frequent misconception is that responsible AI is a final compliance review applied to a finished model. Responsible constraints that are not designed in from the data collection stage typically require fundamental retraining to fix: a model trained on biased labels cannot be fairly calibrated by post-hoc threshold adjustment alone, as the learned representations themselves encode the bias. ::: @@ -1140,6 +1140,53 @@ Because these computational costs and architectural constraints heavily influenc Imagine a credit scoring model deployed nationally. How quickly would your team know if the model suddenly began rejecting qualified applicants from a specific zip code at twice the normal rate? Bias detection transforms theoretical fairness definitions into live, operational telemetry. Just as a Site Reliability Engineer monitors latency dashboards, a Responsible AI engineer monitors demographic parity dashboards, using slice-based analysis to instantly identify when the model begins failing specific subpopulations. +```{python} +#| label: bias-tax-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ ALGORITHMIC BIAS TAX (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Fairness-Efficiency Frontier" .callout-notebook +# │ +# │ Goal: Quantify the accuracy trade-off when enforcing demographic parity. +# │ Show: accuracy_loss ≈ 2.5% — inline in notebook result. +# │ How: Acc_fair = Acc_std - Bias_Penalty (Conceptual). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: ab_base_acc_str, ab_fair_acc_str, ab_gap_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class BiasTax: + # ┌── 1. LOAD ────────────────────────────────────────── + std_acc = 0.92 + fair_acc = 0.895 # Accuracy after enforcing Demographic Parity + # ┌── 2. EXECUTE ─────────────────────────────────────── + gap = std_acc - fair_acc + # ┌── 3. GUARD ───────────────────────────────────────── + check(0.02 < gap < 0.03, f"Gap {gap:.3f} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + ab_base_acc_str = f"{std_acc*100:.0f}%" + ab_fair_acc_str = f"{fair_acc*100:.1f}%" + ab_gap_str = f"{gap*100:.1f}%" + +``` + +::: {.callout-notebook title="The Fairness-Efficiency Frontier"} + +**Problem**: You are optimizing a hiring model. The "Unconstrained" model reaches `{python} BiasTax.ab_base_acc_str` accuracy but exhibits a 15% disparity between demographic groups. You apply a **Fairness Constraint** (Demographic Parity) to eliminate the disparity. What is the "Bias Tax" on your model's performance? + +**The Math**: +Enforcing group-level parity often requires shifting decision thresholds away from the global mathematical optimum. + +1. **Base Accuracy**: `{python} BiasTax.ab_base_acc_str`. +2. **Fairness-Constrained Accuracy**: **`{python} BiasTax.ab_fair_acc_str`**. +3. **The Bias Tax**: `{python} BiasTax.ab_base_acc_str` - `{python} BiasTax.ab_fair_acc_str` = **`{python} BiasTax.ab_gap_str`**. + +**The Systems Insight**: Fairness is an **Optimization Constraint**. Achieving social parity often costs **`{python} BiasTax.ab_gap_str` in accuracy**. In the Machine Learning Fleet, this is not a "failure" of the model; it is the **Price of Governance**. A model that is 92% accurate but structurally biased is not "better" than a 89.5% accurate model that is fair—it is simply incomplete. Responsible engineering means choosing the operating point on the **Fairness-Efficiency Frontier** that aligns with societal values, even if it leaves some technical performance on the table. + +::: + #### Bias Detection and Mitigation {#sec-responsible-ai-bias-detection-mitigation-9174} The fairness definitions examined in @sec-responsible-ai-fairness-machine-learning-2ba4 provide mathematical precision for what fairness means: demographic parity, equalized odds, and equality of opportunity are now precisely defined. Practitioners, however, face a practical challenge: how do you actually compute these metrics on production systems processing thousands of predictions per second? Manual calculation using the formulas above is infeasible at scale. This gap between definition and deployment motivates specialized tooling. @@ -1791,9 +1838,9 @@ Training a next-word predictor on sensitive messages with **DP-SGD** (Differenti The privacy parameter $\epsilon$ is the privacy budget. Lower $\epsilon$ means more privacy but more noise. -**Strong Privacy ($\epsilon = `{python} PrivacyPriceAnalysis.eps_strong`$):** Gradients are heavily clipped ($C = 1.0$, clipping 40% of updates) and noisy ($\sigma = 1.0$). The model requires 3$\times$ more epochs to converge. Training cost jumps from \$4.6M to approximately **\$`{python} PrivacyPriceAnalysis.cost_strong_str`M**. Accuracy drops **`{python} PrivacyPriceAnalysis.acc_drop_strong_str`%**. +**Strong Privacy ($\epsilon =$ `{python} PrivacyPriceAnalysis.eps_strong`):** Gradients are heavily clipped ($C = 1.0$, clipping 40% of updates) and noisy ($\sigma = 1.0$). The model requires 3$\times$ more epochs to converge. Training cost jumps from \$4.6M to approximately **\$`{python} PrivacyPriceAnalysis.cost_strong_str`M**. Accuracy drops **`{python} PrivacyPriceAnalysis.acc_drop_strong_str`%**. -**Moderate Privacy ($\epsilon = `{python} PrivacyPriceAnalysis.eps_mod`$, Apple's standard for keyboard predictions):** Less noise ($\sigma = 0.5$). Training overhead is **`{python} PrivacyPriceAnalysis.overhead_mod_str`%**. Accuracy drops **`{python} PrivacyPriceAnalysis.acc_drop_mod_str`%**. +**Moderate Privacy ($\epsilon =$ `{python} PrivacyPriceAnalysis.eps_mod`, Apple's standard for keyboard predictions):** Less noise ($\sigma = 0.5$). Training overhead is **`{python} PrivacyPriceAnalysis.overhead_mod_str`%**. Accuracy drops **`{python} PrivacyPriceAnalysis.acc_drop_mod_str`%**. **Weak Privacy ($\epsilon = 10$):** Minimal noise. Less than 1% accuracy loss. Limited formal guarantees. @@ -1949,7 +1996,7 @@ A user invokes GDPR Article 17 ("Right to Erasure") on a model trained on 1  **Baseline (Full Retraining):** For a 175B-parameter model at GPT-3 scale, retraining requires 1,024 A100 GPUs for approximately 34 days at a cost of roughly **\$`{python} UnlearningCostAnalysis.full_cost_m` million**. -**The Engineering Fix (SISA):** Sharded, Isolated, Sliced, and Aggregated training partitions data into $K = `{python} UnlearningCostAnalysis.n_shards`$ independent shards, training 100 sub-models. To delete one datum, retrain only the specific shard containing it (1% of data). New cost: **\$`{python} UnlearningCostAnalysis.sisa_cost_k_str`{,}000**. Time: approximately **`{python} UnlearningCostAnalysis.sisa_time_hr_str` hours**. +**The Engineering Fix (SISA):** Sharded, Isolated, Sliced, and Aggregated training partitions data into $K =$ `{python} UnlearningCostAnalysis.n_shards` independent shards, training 100 sub-models. To delete one datum, retrain only the specific shard containing it (1% of data). New cost: **\$`{python} UnlearningCostAnalysis.sisa_cost_k_str`{,}000**. Time: approximately **`{python} UnlearningCostAnalysis.sisa_time_hr_str` hours**. **The Trade-off:** Accuracy drops 3--7% because each sub-model sees less data. Inference slows because predictions must be aggregated across 100 sub-models. For a fleet receiving 1,000 deletion requests per day, SISA transforms unlearning from "economically impossible" to "manageable operational cost"---at the price of model quality. ::: diff --git a/book/quarto/contents/vol2/robust_ai/robust_ai.qmd b/book/quarto/contents/vol2/robust_ai/robust_ai.qmd index 058180e21b..62f98d563a 100644 --- a/book/quarto/contents/vol2/robust_ai/robust_ai.qmd +++ b/book/quarto/contents/vol2/robust_ai/robust_ai.qmd @@ -65,11 +65,11 @@ This imperative for fault tolerance establishes what we define as Robust AI: ::: {.callout-definition title="Robust AI"} -***Robust AI***\index{Robust AI!definition} is the engineering discipline of ensuring that machine learning systems maintain performance and reliability despite system errors, malicious inputs, or environmental shifts. +***Robust AI***\index{Robust AI!definition} is the measurable systems property that a model's predictions remain valid — within specified error bounds — under distribution shift, adversarial perturbation, and hardware or software faults, as opposed to the average-case accuracy achieved under ideal i.i.d. conditions. -1. **Significance (Quantitative):** It treats reliability as a **System Invariant**. It focuses on the **Worst-Case Error Bound** rather than the average-case accuracy, ensuring the system's **Validity Confidence** remains high across the entire deployment distribution ($D_{\text{vol}}$). -2. **Distinction (Durable):** Unlike **Standard Generalization** (performance on unseen i.i.d. data), Robustness addresses performance on **Non-i.i.d. Data**, including adversarial perturbations and out-of-distribution shifts. -3. **Common Pitfall:** A frequent misconception is that robustness is a "final check." In reality, it is an **Architectural Constraint**: a model that is not robust by design (e.g., through adversarial training or architectural priors) cannot easily be "fixed" with downstream monitoring. +1. **Significance (Quantitative):** Robustness is quantified by worst-case guarantees: a certified robust classifier guarantees accuracy above a threshold for all inputs within an $\ell_\infty$ ball of radius $\varepsilon$ around any test point. For image classification, $\varepsilon = 8/255$ (a perturbation invisible to humans) typically reduces accuracy by 30–60% in non-robust models. Distribution shift compounds this: a clinical NLP model trained on 2019 records and deployed in 2021 without retraining can see accuracy drop 15–25% as medical coding practices and terminology evolve. +2. **Distinction (Durable):** Unlike standard generalization (which measures average-case accuracy on held-out i.i.d. test data drawn from the same distribution as training), robustness measures worst-case performance on adversarial or out-of-distribution inputs — a distinction that matters because a model can achieve 95% i.i.d. test accuracy while failing completely on inputs that differ from training by amounts imperceptible to humans. +3. **Common Pitfall:** A frequent misconception is that robustness can be added as a post-hoc monitoring layer to any existing model. A model's robustness properties are determined primarily during training — models trained without adversarial examples or robustness objectives cannot achieve certified robustness through inference-time filtering alone, because the vulnerability is in the learned decision boundary, not in which inputs reach the model. ::: @@ -589,6 +589,56 @@ These three categories of challenges stem from different sources but share sever Detection and monitoring form the foundation of any robustness strategy. Hardware monitoring systems typically sample metrics at 1-10 Hz frequencies and detect temperature anomalies (±5°C from baseline), voltage fluctuations (±5% from nominal), and memory error rates exceeding 10^-12 errors per bit per hour. Adversarial input detection uses statistical tests with p-value thresholds of 0.01-0.05 and achieves 85-95% detection rates with false positive rates below 2%. Distribution monitoring using MMD[^fn-mmd-drift] tests processes 1,000-10,000 samples per evaluation and detects shifts with Cohen's d > 0.3 within 95% confidence intervals. +```{python} +#| label: adversarial-payback-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ ADVERSARIAL ATTACK PAYBACK (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Cost of Defense" .callout-notebook +# │ +# │ Goal: Quantify the compute overhead of adversarial training. +# │ Show: training_slowdown ≈ 8x — inline in notebook result. +# │ How: T_total = T_std + N_Steps * T_std (Conceptual PGD). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: ap_n_pgd_steps_str, ap_slowdown_str, ap_cert_acc_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class AdversarialPayback: + # ┌── 1. LOAD ────────────────────────────────────────── + n_pgd_steps = 7 # PGD-7 is standard for adversarial training + std_acc = 0.95 + robust_acc = 0.70 # Accuracy against worst-case attack + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Each training step requires generating an attack (7 forward/backward passes) + # total steps per batch = 1 (std) + 7 (attack) = 8 + slowdown = 1 + n_pgd_steps + # ┌── 3. GUARD ───────────────────────────────────────── + check(slowdown == 8, f"Slowdown {slowdown} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + ap_n_pgd_steps_str = f"{n_pgd_steps}" + ap_slowdown_str = f"{slowdown}" + ap_cert_acc_str = f"{robust_acc*100:.0f}%" + +``` + +::: {.callout-notebook title="The Cost of Defense"} + +**Problem**: You are training a robust classifier for an autonomous vehicle. You use **Adversarial Training** (PGD-7), which generates a worst-case attack for every training batch. How much does this "Robustness Tax" slow down your training run? + +**The Math**: +Generating an adversarial example requires $K$ additional gradient steps per training sample. + +1. **Forward/Backward Passes**: 1 (Standard) + 7 (Attack Generation) = **8 total passes**. +2. **Training Slowdown**: **`{python} AdversarialPayback.ap_slowdown_str`$\times$ slower**. +3. **Utility Cost**: Accuracy on clean data drops from 95% to **`{python} AdversarialPayback.ap_cert_acc_str`**. + +**The Systems Insight**: Robustness is an **Efficiency-Utility Trade-off**. To guarantee that your model won't misclassify a stop sign with a few stickers on it, you must pay **8$\times$ the training cost** and accept a **25% drop in general performance**. In the Machine Learning Fleet, "Robustness" is not a setting you turn on; it is a budget you spend. This is why most fleets use **Detection** (cheaper) rather than **Certification** (robust training) for all but the most safety-critical components. + +::: + [^fn-mmd-drift]: **Maximum Mean Discrepancy (MMD)**: A kernel-based statistical test measuring distance between two distributions in a reproducing kernel Hilbert space, without parametric assumptions. Unlike univariate tests (K-S, PSI) that require per-feature evaluation, MMD operates on joint distributions natively---critical for ML inputs where drift manifests in feature correlations, not individual features. The trade-off is compute: MMD scales $O(n^2)$ in sample size, making it impractical for real-time monitoring without subsampling or random feature approximations. \index{MMD!drift detection} Graceful degradation ensures that systems maintain core functionality even when operating under stress. Rather than catastrophic failure, robust systems exhibit predictable performance reduction that preserves critical capabilities. ECC memory systems recover from single-bit errors with 99.9% success rates while adding 12.5% bandwidth overhead. Model quantization from FP32 to INT8 reduces memory requirements by 75% and inference time by 2-4$\times$, trading 1-3% accuracy for continued operation under resource constraints. Ensemble fallback systems maintain 85-90% of peak performance when primary models fail, with switchover latency under 10 ms. @@ -664,6 +714,59 @@ The most pervasive robustness challenge addresses the natural evolution of real- Consider a medical diagnosis model trained on X-ray images from a modern hospital. When deployed in a rural clinic with older equipment, the model's accuracy plummets not because the underlying medical conditions have changed, but because the image characteristics differ. This exemplifies distribution shift: the world the model encounters differs from the world it learned from. +```{python} +#| label: distribution-shift-confidence-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ DISTRIBUTION SHIFT CONFIDENCE (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "Is the World Changing?" .callout-notebook +# │ +# │ Goal: Quantify the confidence of a distribution shift detection. +# │ Show: p_value ≈ 0.001 — inline in notebook result. +# │ How: P-value for shift detection (Conceptual MMD/PSI). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: ds_n_samples_str, ds_p_value_str, ds_shift_detected_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class ShiftConfidence: + # ┌── 1. LOAD ────────────────────────────────────────── + n_samples = 1000 + baseline_mean = 0.5 + current_mean = 0.55 # A 5% shift in feature distribution + std_dev = 0.3 + # ┌── 2. EXECUTE ─────────────────────────────────────── + # Z-score = (mean_diff) / (std / sqrt(N)) + z_score = (current_mean - baseline_mean) / (std_dev / (n_samples ** 0.5)) + # For Z=5.27, p-value is extremely small + p_value = 0.000001 + # ┌── 3. GUARD ───────────────────────────────────────── + check(z_score > 5, f"Z-score {z_score:.2f} too low for significant shift") + # ┌── 4. OUTPUT ──────────────────────────────────────── + ds_n_samples_str = f"{n_samples:,}" + ds_p_value_str = "< 0.001" + ds_shift_detected_str = "YES" + +``` + +::: {.callout-notebook title="Is the World Changing?"} + +**Problem**: Your model monitors a critical input feature. The baseline mean was `{python} ShiftConfidence.baseline_mean`. Over the last `{python} ShiftConfidence.ds_n_samples_str` requests, the mean has shifted to `{python} ShiftConfidence.current_mean`. Is this a random fluctuation or a real **Distribution Shift**? + +**The Math**: +Detection requires proving that the observed change is statistically unlikely under the baseline distribution. + +1. **Difference in Means**: 0.05. +2. **Standard Error**: 0.3 / $\sqrt{1000} \approx$ **0.009**. +3. **Statistical Significance**: The shift is **5.5 standard errors** away from the mean. +4. **P-Value**: **`{python} ShiftConfidence.ds_p_value_str`**. + +**The Systems Insight**: Statistical significance is the "Signal-to-Noise Ratio" of your monitoring system. A shift of 0.05 might seem "small," but with 1,000 samples, the probability of it being random noise is less than **0.1%**. In the Machine Learning Fleet, this is a **Confirmed Regression**. You must trigger an automated response: either falling back to a more robust model or initiating an emergency retraining run on the new data distribution. + +::: + ::: {.callout-note title="Figure: Types of Distribution Shift"} ```{.tikz} \begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, scale=0.7, node distance=6cm and 4cm] @@ -723,7 +826,7 @@ Covariate shift occurs when the input distribution changes while the relationshi ::: {.callout-note title="Decision Boundary Under Shift"} ```{.tikz} -\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, node distance=2cm, +\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, node distance=2cm, BlueLine/.style={line width=1.0pt, color=blue}, RedLine/.style={line width=1.0pt, color=red}, GreenL/.style={fill=green}, @@ -1450,7 +1553,7 @@ to[out=270,in=0](0.36,-0.15)to[out=160,in=340](0.2,-0.051)to cycle; \fill[draw=none,fill=BrownLine,even odd rule,xshift=3.8mm,yshift=2mm] \gear{11}{0.25}{0.21}{10}{1}{0.07}; \fill[draw=none,fill=BrownLine,even odd rule,xshift=0.6mm,yshift=5.8mm] coordinate(F) \gear{11}{0.25}{0.21}{10}{1}{0.07}; \end{scope} - + % Channels \begin{scope}[local bounding box=CHANEL2,shift={(0,3.5)}] \foreach \i/\sf in {5/0.7,6/0.8,7/0.9} { @@ -1463,26 +1566,26 @@ to[out=270,in=0](0.36,-0.15)to[out=160,in=340](0.2,-0.051)to cycle; \pic[shift={(2.55,0)}] at ({-\i*0.13}, {-0.13*\i}) {channel={scalefac=0.5,picname=\i-CH3,filllcolor=BrownLine,drawcolor=BrownLine}}; } \end{scope} - + % Brain \begin{scope}[local bounding box=BRAIN1,shift={(0,0)},scale=1, every node/.append style={transform shape}] \pic[shift={(0,0)}] at (0,0) {brain={scalefac=0.7,picname=1,filllcolor=OrangeL, Linewidth=0.75pt}}; \end{scope} - + % Persons \foreach \i in {1,2,3} { \pic[shift={(0,0)}] at (7-CH\i) {person={scalefac=0.3,picname=1,drawcolor=BrownLine,filllcolor=GreenL, Linewidth=0.75pt}}; } - + \draw[LineA] (BRAIN1) -- ++(90:1.32); \foreach \i in {1,2,3} { \draw[Line] (7-CH\i.south) -- (F); } - + \scoped[on background layer] \node[draw=BlueLine,inner xsep=3mm,inner ysep=2mm,fill=BlueL,fit=(BRAIN1)(7-CH1)(5-CH3),line width=0.75pt] (BB1) {}; \node[above=2pt of BB1,BlueLine] {Normal learning}; - + \path[RedLine] (7-CH1) -- ++(180:3) coordinate(DA); \path[RedLine] (DA) |- coordinate(LE) (GEAR1); \path[RedLine] (DA) |- coordinate(MA) (BRAIN1); @@ -1499,7 +1602,7 @@ to[out=270,in=0](0.36,-0.15)to[out=160,in=340](0.2,-0.051)to cycle; \fill[draw=none,fill=BrownLine,even odd rule,xshift=3.8mm,yshift=2mm] \gear{11}{0.25}{0.21}{10}{1}{0.07}; \fill[draw=none,fill=BrownLine,even odd rule,xshift=0.6mm,yshift=5.8mm] coordinate(F) \gear{11}{0.25}{0.21}{10}{1}{0.07}; \end{scope} - + % Channels \begin{scope}[local bounding box=CHANEL2,shift={(0,3.5)}] \foreach \i/\sf in {5/0.7,6/0.8,7/0.9} { @@ -1512,30 +1615,30 @@ to[out=270,in=0](0.36,-0.15)to[out=160,in=340](0.2,-0.051)to cycle; \pic[shift={(2.55,0)}] at ({-\i*0.13}, {-0.13*\i}) {channel={scalefac=0.5,picname=\i-CH3,filllcolor=BrownLine,drawcolor=BrownLine}}; } \end{scope} - + % Brain \begin{scope}[local bounding box=BRAIN1,shift={(0,0)},scale=1, every node/.append style={transform shape}] \pic[shift={(0,0)}] at (0,0) {brain={scalefac=0.7,picname=1,filllcolor=BrownL, Linewidth=0.75pt, filllcirclecolor=BrownL,drawcircle=BrownLine}}; \end{scope} - + % Skull \begin{scope}[local bounding box=SKULL1,shift={(0,0)},scale=1, every node/.append style={transform shape}] \pic[shift={(0,0)}] at (0,0) {skull={scalefac=0.8,picname=1,filllcolor=RedLine, Linewidth=0.75pt}}; \end{scope} - + % Persons \foreach \i in {1,3} { \pic[shift={(0,0)}] at (7-CH\i) {person={scalefac=0.3,picname=1,drawcolor=BrownLine,filllcolor=GreenL, Linewidth=0.75pt}}; } - + % Skull \pic[shift={(0,0)}] at (7-CH2) {skull={scalefac=0.8,picname=1,filllcolor=RedLine, Linewidth=0.75pt}}; - + \draw[LineA] (BRAIN1) -- ++(90:1.32); \foreach \i in {1,2,3} { \draw[Line] (7-CH\i.south) -- (F); } - + \scoped[on background layer] \node[draw=RedLine,inner xsep=3mm,inner ysep=2mm,fill=RedL,fit=(BRAIN1)(7-CH1)(5-CH3),line width=0.75pt] (BB1) {}; \node[above=2pt of BB1,RedLine] {Poisoning attack}; diff --git a/book/quarto/contents/vol2/security_privacy/security_privacy.qmd b/book/quarto/contents/vol2/security_privacy/security_privacy.qmd index dc8b84c98e..1de972bf01 100644 --- a/book/quarto/contents/vol2/security_privacy/security_privacy.qmd +++ b/book/quarto/contents/vol2/security_privacy/security_privacy.qmd @@ -120,11 +120,11 @@ Security in machine learning focuses on defending systems from adversarial behav ::: {.callout-definition title="Security"} -***Security***\index{Security!definition} is the protection of ML system assets (data, models, and infrastructure) from unauthorized access, manipulation, or disruption. +***Security***\index{Security!definition} is the set of system properties — confidentiality, integrity, and availability — that protect an ML system's data, model weights, and inference pipeline from intentional adversarial actions, spanning both the infrastructure layer (network intrusion, credential theft) and the algorithmic layer (model extraction, prompt injection, adversarial examples). -1. **Significance (Quantitative):** It ensures the **Integrity** of the system's learned behavior and the **Availability** of its services. Within the **Iron Law**, security failures (e.g., DDoS, model theft) can collapse the **Duty Cycle ($\eta$)** or compromise the proprietary value of the model operations ($O$). -2. **Distinction (Durable):** Unlike **General Robustness** (which handles stochastic environmental errors), Security addresses **Intentional Adversarial Threats** designed to exploit system vulnerabilities. -3. **Common Pitfall:** A frequent misconception is that security is "solved" by traditional IT firewalling. In reality, ML systems introduce a new **Attack Surface** at the algorithmic layer (e.g., prompt injection, adversarial examples) that traditional security tools cannot detect. +1. **Significance (Quantitative):** Security failures operate on both surfaces simultaneously. At the infrastructure layer, a stolen GPT-4-class model (training cost ~\$100M) represents direct IP loss. At the algorithmic layer, model extraction via black-box queries can reconstruct a functionally equivalent copy with as few as 10,000–100,000 API calls — orders of magnitude cheaper than training — bypassing the investment and competitive moat. Either failure collapses the business value of the $O$ (model operations) term in the Iron Law. +2. **Distinction (Durable):** Unlike general robustness (which addresses stochastic distribution shift from unintentional environmental change), security addresses intentional adversarial threats where an attacker actively maximizes the probability of a targeted failure — requiring worst-case rather than average-case analysis. +3. **Common Pitfall:** A frequent misconception is that traditional IT security (firewalls, access controls, encryption) adequately secures ML systems. ML introduces an algorithmic attack surface orthogonal to infrastructure: a properly authenticated API request containing an adversarial input or a prompt injection bypasses all network-layer defenses and manipulates model behavior through the model's own learned functions. ::: @@ -198,11 +198,57 @@ class DPCostAnalysis: 1. **Sensitivity ($S$)**: The maximum one person can change the sum is **\$`{python} DPCostAnalysis.sensitivity_str`**. 2. **Privacy Budget ($\epsilon$)**: `{python} DPCostAnalysis.epsilon`. -3. **Laplace Noise Scale ($b$)**: $S / \epsilon = `{python} DPCostAnalysis.sensitivity_str` / `{python} DPCostAnalysis.epsilon` = \mathbf{`{python} DPCostAnalysis.noise_scale_str`}$. +3. **Laplace Noise Scale ($b$)**: $S / \epsilon =$ `{python} DPCostAnalysis.sensitivity_str` / `{python} DPCostAnalysis.epsilon` = **`{python} DPCostAnalysis.noise_scale_str`**. 4. **Impact on Mean**: The noise added to the *sum* has magnitude $\approx `{python} DPCostAnalysis.noise_scale_str`. * Noise per person (average) = `{python} DPCostAnalysis.noise_scale_str` / `{python} DPCostAnalysis.n_employees` = **\$`{python} DPCostAnalysis.error_per_person_str`**. -**The Systems Conclusion**: Protecting one outlier introduces a **\$`{python} DPCostAnalysis.error_per_person_str` error** to the average. For a dataset of $N=`{python} DPCostAnalysis.n_small`, the error would be **\$`{python} DPCostAnalysis.error_small_str`**! Differential Privacy kills utility for small $N$. It only works at scale where $1/N$ dampens the noise. +**The Systems Conclusion**: Protecting one outlier introduces a **\$`{python} DPCostAnalysis.error_per_person_str` error** to the average. For a dataset of $N=$ `{python} DPCostAnalysis.n_small`, the error would be **\$`{python} DPCostAnalysis.error_small_str`**! Differential Privacy kills utility for small $N$. It only works at scale where $1/N$ dampens the noise. +::: + +```{python} +#| label: isolation-tax-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ MULTI-TENANT ISOLATION TAX (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Tax of Secure Multi-Tenancy" .callout-notebook +# │ +# │ Goal: Quantify the throughput loss from secure GPU partitioning (MIG/SR-IOV). +# │ Show: isolation_overhead_pct ≈ 15% — inline in notebook result. +# │ How: Throughput_shared = Throughput_peak * (1 - Isolation_Tax). +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: mt_peak_tokens_str, mt_secure_tokens_str, mt_overhead_pct_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class MultiTenantIsolation: + # ┌── 1. LOAD ────────────────────────────────────────── + peak_throughput = 1000 # tokens/sec on dedicated GPU + secure_throughput = 850 # after MIG partitioning + secure context switching + # ┌── 2. EXECUTE ─────────────────────────────────────── + overhead_pct = (1 - secure_throughput / peak_throughput) * 100 + # ┌── 3. GUARD ───────────────────────────────────────── + check(overhead_pct == 15.0, f"Overhead {overhead_pct}% unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + mt_peak_tokens_str = f"{peak_throughput:,}" + mt_secure_tokens_str = f"{secure_throughput:,}" + mt_overhead_pct_str = f"{overhead_pct:.0f}" + +``` + +::: {.callout-notebook title="The Tax of Secure Multi-Tenancy"} + +**Problem**: You are hosting two models on a single H100 using **Multi-Instance GPU (MIG)** to provide hardware-level isolation. On a dedicated GPU, your model achieves `{python} MultiTenantIsolation.mt_peak_tokens_str` tokens/sec. After enabling secure partitioning, it achieves `{python} MultiTenantIsolation.mt_secure_tokens_str` tokens/sec. What is the performance cost of security? + +**The Math**: +Isolation requires dedicated hardware resources (SRAM, cache) and adds context-switching overhead. + +1. **Throughput Loss**: `{python} MultiTenantIsolation.mt_peak_tokens_str` - `{python} MultiTenantIsolation.mt_secure_tokens_str` = **150 tokens/sec**. +2. **The Isolation Tax**: (150 / 1000) = **`{python} MultiTenantIsolation.mt_overhead_pct_str`%**. + +**The Systems Insight**: Security is a **Capacity Drain**. Providing a "Secure Enclave" for your model costs **`{python} MultiTenantIsolation.mt_overhead_pct_str`%** of your GPU's raw throughput. In the Machine Learning Fleet, multi-tenancy is an economic necessity, but it is not free. You must decide if your data sensitivity justifies losing 15% of your fleet's total capacity. For most public-facing APIs, this "Tax" is the price of preventing one tenant's prompt from leaking into another's response. + ::: ### Security versus Privacy {#sec-security-privacy-security-versus-privacy-e0b8} @@ -1084,7 +1130,7 @@ Model theft can target two distinct objectives: extracting exact model propertie ``` ::: -::: {.callout-war-story title="The GPT-2 Model Extraction"} +::: {.callout-war-story title="The BERT Model Extraction"} Researchers demonstrated that proprietary models behind APIs are vulnerable to functional extraction. By querying a victim BERT-based API with just 2 million carefully crafted inputs (costing roughly \$50 in query fees), they trained a "student" model that achieved >97% agreement with the victim on test tasks. This "model stealing" attack exploited the high-information signal returned by confidence scores and logits. It proved that API access alone is sufficient to replicate intellectual property, forcing providers to implement defensive measures like API rate limiting, query auditing, and output truncation to obscure decision boundaries. ::: @@ -1515,7 +1561,9 @@ Physical adversarial attacks against traffic signs reveal how small, targeted pe In 2017, researchers demonstrated this vulnerability in the physical world by placing small black and white stickers on stop signs [@eykholt2018robust]. @fig-adversarial-stickers shows the physical implementation: stickers occupying less than 10% of the sign's surface area, designed to be nearly imperceptible to the human eye, yet sufficient to shift the sign's representation across the model's decision boundary. When images of these modified stop signs were fed into standard traffic sign classification models, they were misclassified as speed limit signs over 85% of the time, while human observers identified the signs correctly in every trial. -![**Adversarial Stickers**: Nearly imperceptible stickers can trick machine learning models into misclassifying stop signs as speed limit signs over 85% of the time. This emphasizes the vulnerability of ML systems to adversarial attacks.](./images/png/stop_signs.png){#fig-adversarial-stickers fig-alt="Photo of stop signs with small black and white stickers applied. Original signs appear normal to humans but cause ML classifiers to misread them as speed limit signs."} +::: {#fig-adversarial-stickers fig-env="figure" fig-pos="htb" fig-cap="**Adversarial Stickers**: Nearly imperceptible stickers can trick machine learning models into misclassifying stop signs as speed limit signs over 85% of the time. This emphasizes the vulnerability of ML systems to adversarial attacks." fig-alt="Photo of stop signs with small black and white stickers applied. Original signs appear normal to humans but cause ML classifiers to misread them as speed limit signs."} +![](./images/png/stop_signs.png) +::: This demonstration showed how simple adversarial stickers could trick ML systems into misreading important road signs. If deployed realistically, these attacks could endanger public safety, causing autonomous vehicles to misinterpret stop signs as speed limits. Researchers warned this could potentially cause dangerous rolling stops or acceleration into intersections. @@ -1688,11 +1736,15 @@ For machine learning systems, these attacks pose several concrete risks. Fault i The practical viability of these attacks has been demonstrated through controlled experiments. One notable example is the work by @breier2018deeplaser, where researchers successfully used a laser fault injection attack on a deep neural network deployed on a microcontroller. @fig-laser-bitflip captures the mechanism: by heating specific transistors with focused laser pulses, they forced the hardware to skip execution steps, including a ReLU activation function. -![**Laser Fault Injection**: Focused laser pulses induce bit flips within microcontroller memory, enabling attackers to manipulate model execution and compromise system integrity. Researchers use this technique to simulate hardware errors, revealing vulnerabilities in embedded machine learning systems and informing the development of fault-tolerant designs.](images/png/laser_bitflip.png){#fig-laser-bitflip fig-pos=t! fig-alt="Diagram showing laser beam targeting microcontroller chip. Beam focuses on specific memory location, causing bit flip that changes stored value and corrupts neural network computation."} +::: {#fig-laser-bitflip fig-env="figure" fig-pos="htb" fig-cap="**Laser Fault Injection**: Focused laser pulses induce bit flips within microcontroller memory, enabling attackers to manipulate model execution and compromise system integrity. Researchers use this technique to simulate hardware errors, revealing vulnerabilities in embedded machine learning systems and informing the development of fault-tolerant designs." fig-alt="Diagram showing laser beam targeting microcontroller chip. Beam focuses on specific memory location, causing bit flip that changes stored value and corrupts neural network computation."} +![](images/png/laser_bitflip.png) +::: @fig-injection reveals the assembly-level consequences: a segment implementing the ReLU activation function that compares the most significant bit (MSB) of the accumulator to zero and uses a brge (branch if greater or equal) instruction to skip the assignment if the value is non-positive. The fault injection suppresses the branch, causing the processor to always execute the "else" block. As a result, the neuron's output is forcibly zeroed out, regardless of the input value. -![**Fault Injection Attack**: Manipulating assembly code bypasses safety checks, forcing a neuron's output to zero regardless of input and demonstrating a hardware vulnerability in machine learning systems.](images/png/fault-injection_demonstrated_with_assembly_code.png){#fig-injection width=75% fig-pos=b! fig-alt="Assembly code snippet showing ReLU implementation with branch instruction. Red annotation indicates fault injection point where branch is skipped, forcing neuron output to zero."} +::: {#fig-injection fig-env="figure" fig-pos="htb" fig-cap="**Fault Injection Attack**: Manipulating assembly code bypasses safety checks, forcing a neuron's output to zero regardless of input and demonstrating a hardware vulnerability in machine learning systems." fig-alt="Assembly code snippet showing ReLU implementation with branch instruction. Red annotation indicates fault injection point where branch is skipped, forcing neuron output to zero."} +![](images/png/fault-injection_demonstrated_with_assembly_code.png){width=75%} +::: Fault injection attacks can also be combined with side-channel analysis, where attackers first observe power or timing characteristics to infer model structure or data flow. This reconnaissance allows them to target specific layers or operations, such as activation functions or final decision layers, maximizing the impact of the injected faults. @@ -1718,15 +1770,21 @@ A useful example of this attack technique can be seen in a power analysis of a p @fig-encryption establishes the baseline: with the correct password entered, the red waveform captures the serial data stream, marking each byte as it is received, while the blue curve records the device's power consumption over time. When the full, correct password is supplied, the power profile remains stable and consistent across all five bytes, providing a clear reference for comparison with failed attempts. -![**Power Profile**: The device's power consumption remains stable during authentication when the correct password is entered, setting a baseline for comparison in subsequent figures through This figure. Source: colin o'flynn.](images/png/power_analysis_of_an_encryption_device_with_a_correct_password.png){#fig-encryption fig-alt="Oscilloscope trace with two signals. Red line shows serial data with five password bytes. Blue line shows steady power consumption throughout all five bytes, indicating successful authentication."} +::: {#fig-encryption fig-env="figure" fig-pos="htb" fig-cap="**Power Profile**: The device's power consumption remains stable during authentication when the correct password is entered, setting a baseline for comparison in subsequent figures. Source: Colin O'Flynn." fig-alt="Oscilloscope trace with two signals. Red line shows serial data with five password bytes. Blue line shows steady power consumption throughout all five bytes, indicating successful authentication."} +![](images/png/power_analysis_of_an_encryption_device_with_a_correct_password.png) +::: @fig-encryption2 demonstrates what happens when an incorrect password is entered: the power signature diverges. In this case, the first three bytes (`0x61, 0x52, 0x77`) are correct, so the power patterns closely match the correct password up to that point. However, when the fourth byte (`0x42`) is processed and found to be incorrect, the device halts authentication. This change is reflected in the sudden jump in the blue power line, indicating that the device has stopped processing and entered an error state. -![**Side-Channel Attack Vulnerability**: Power consumption patterns reveal cryptographic key information during authentication; consistent power usage indicates correct password bytes, while abrupt changes signal incorrect input and halted processing. Even without knowing the password, an attacker can infer it by analyzing the device's power usage during authentication attempts via this figure. Source: Colin O'Flynn.](images/png/power_analysis_of_an_encryption_device_with_a_partially_wrong_password.png){#fig-encryption2 fig-alt="Oscilloscope trace showing red serial data line and blue power line. Power remains stable through first three correct bytes, then jumps sharply at fourth incorrect byte, revealing partial password match through timing."} +::: {#fig-encryption2 fig-env="figure" fig-pos="htb" fig-cap="**Side-Channel Attack Vulnerability**: Power consumption patterns reveal cryptographic key information during authentication; consistent power usage indicates correct password bytes, while abrupt changes signal incorrect input and halted processing. Even without knowing the password, an attacker can infer it by analyzing the device's power usage during authentication attempts. Source: Colin O'Flynn." fig-alt="Oscilloscope trace showing red serial data line and blue power line. Power remains stable through first three correct bytes, then jumps sharply at fourth incorrect byte, revealing partial password match through timing."} +![](images/png/power_analysis_of_an_encryption_device_with_a_partially_wrong_password.png) +::: @fig-encryption3 shows the extreme case: with an entirely incorrect password (`0x30, 0x30, 0x30, 0x30, 0x30`), the device fails immediately. The device detects the mismatch after the first byte and halts processing much earlier. This is visible in the power profile, where the blue line exhibits a sharp jump following the first byte, reflecting the device's early termination of authentication. -![**Power Consumption Jump**: The blue line's sharp increase after processing the first byte indicates immediate authentication failure, highlighting how incorrect passwords are quickly detected through power usage. Source: Colin O'Flynn.](images/png/power_analysis_of_an_encryption_device_with_a_wrong_password.png){#fig-encryption3 fig-alt="Oscilloscope trace showing red serial data and blue power consumption. Blue line jumps sharply after first byte, indicating immediate authentication failure with completely wrong password."} +::: {#fig-encryption3 fig-env="figure" fig-pos="htb" fig-cap="**Power Consumption Jump**: The blue line's sharp increase after processing the first byte indicates immediate authentication failure, highlighting how incorrect passwords are quickly detected through power usage. Source: Colin O'Flynn." fig-alt="Oscilloscope trace showing red serial data and blue power consumption. Blue line jumps sharply after first byte, indicating immediate authentication failure with completely wrong password."} +![](images/png/power_analysis_of_an_encryption_device_with_a_wrong_password.png) +::: These examples demonstrate how attackers can exploit observable power consumption differences to reduce the search space and eventually recover secret data through brute-force analysis. By systematically measuring power consumption patterns and correlating them with different inputs, attackers can extract sensitive information that should remain hidden. @@ -1956,7 +2014,7 @@ To illustrate these offensive capabilities concretely, we examine a specific cas \addplot+[domain=-0.75:0.75, plotline] {6.5*exp(-x^2/2)} node[pos=0.66, outer sep=0pt, inner sep=0pt] (T1) {}; \addplot+[domain=-0.796:0.796, plotline] {6.75*exp(-x^2/2)} node[pos=0.63, outer sep=0pt, inner sep=0pt] (T2) {}; \addplot+[domain=-0.84:0.84, plotline] {7*exp(-x^2/2)} node[pos=0.6, outer sep=0pt, inner sep=0pt] (T3) {}; - + \draw[BlueLine, arrow] (T1) -- ++(0:9.5mm) node[right] {0000}; \draw[GreenLine, arrow] (T2) -- ++(0:9.9mm) node[right] {1111}; \draw[VioletLine, arrow] (T3) -- ++(0:10.4mm) node[right] {0101}; @@ -1987,7 +2045,9 @@ In their study, @scaaml_2019 trained a convolutional neural network to extract A @fig-stm32f-board shows the experimental hardware configuration: a ChipWhisperer setup with a custom STM32F target board that executes AES operations while allowing external equipment to monitor power consumption with high temporal precision. This setup demonstrates how even inexpensive, low-power embedded devices can leak information through side channels that modern machine learning models can learn to exploit. -![**STM32F415 Target Board**: Enables monitoring of power consumption during AES operations on the microcontroller, highlighting side-channel vulnerabilities that can be exploited by machine learning models.](images/png/stm32f_board.png){#fig-stm32f-board fig-alt="Photo of ChipWhisperer setup with STM32F415 target board. Green PCB with microcontroller chip connected to measurement probes for monitoring power consumption during cryptographic operations."} +::: {#fig-stm32f-board fig-env="figure" fig-pos="htb" fig-cap="**STM32F415 Target Board**: Enables monitoring of power consumption during AES operations on the microcontroller, highlighting side-channel vulnerabilities that can be exploited by machine learning models." fig-alt="Photo of ChipWhisperer setup with STM32F415 target board. Green PCB with microcontroller chip connected to measurement probes for monitoring power consumption during cryptographic operations."} +![](images/png/stm32f_board.png) +::: Subsequent work expanded on this approach by introducing long-range models capable of exploiting broader temporal dependencies in the traces, improving performance even under noise and desynchronization [@bursztein2023generic]. These developments highlight the potential for machine learning models to serve as offensive cryptanalysis tools, especially in the analysis of secure hardware. @@ -2290,7 +2350,9 @@ However, prompt filtering alone is insufficient for safety. Research has shown t @fig-adversarial-nibbler provides concrete examples revealing how innocuous prompts can trigger unsafe generations. Such examples highlight the limitations of pre-generation safety checks and reinforce the necessity of output-based monitoring as a second line of defense. This two-stage pipeline consisting of prompt filtering followed by post-hoc content analysis is essential for ensuring the safe deployment of generative models in open-ended or user-facing environments. -![**Adversarial Prompt Evasion**: Implicitly adversarial prompts bypass typical content filters by triggering unintended generations, revealing limitations of solely relying on pre-generation safety checks. These examples underscore the necessity of post-hoc content analysis as a complementary defense layer for robust generative AI systems.](images/png/adversarial_nibbler_example.png){#fig-adversarial-nibbler fig-alt="Grid of generated images from text-to-image model. Prompts appear innocuous but produce policy-violating outputs, demonstrating how adversarial prompts bypass content filters."} +::: {#fig-adversarial-nibbler fig-env="figure" fig-pos="htb" fig-cap="**Adversarial Prompt Evasion**: Implicitly adversarial prompts bypass typical content filters by triggering unintended generations, revealing limitations of solely relying on pre-generation safety checks. These examples underscore the necessity of post-hoc content analysis as a complementary defense layer for robust generative AI systems." fig-alt="Grid of generated images from text-to-image model. Prompts appear innocuous but produce policy-violating outputs, demonstrating how adversarial prompts bypass content filters."} +![](images/png/adversarial_nibbler_example.png) +::: In the domain of language generation, output monitoring plays a different but equally important role. Here, the goal is often to detect toxicity, hallucinated claims, or off-distribution responses. For example, a customer support chatbot may be monitored for keyword presence, tonal alignment, or semantic coherence. If a response contains profanity, unsupported assertions, or syntactically malformed text, the system may trigger a rephrasing, initiate a fallback to scripted templates, or halt the response altogether. @@ -2384,7 +2446,64 @@ Hardware security mechanisms introduce measurable overhead that must be factored Security features scale differently than computational resources. TEE memory limitations constrain model size regardless of available system memory. A quantized ResNet-18 model (47 MB) can operate within ARM TrustZone constraints, while ResNet-50 (176 MB) requires careful memory management or model partitioning. These constraints create architectural decisions that must be made early in system design. -Different threat models and protection levels require quantitative trade-off analysis. For ML workloads requiring cryptographic verification, AES-256 operations add 0.1--0.5 ms per inference depending on model size and hardware acceleration availability. Homomorphic encryption operations impose 100-100,000$\times$ computational overhead, with fully homomorphic encryption (FHE) at the higher end and somewhat homomorphic encryption (SHE) at the lower end, making them viable only for small models or offline scenarios where strong privacy guarantees justify the performance cost. +Different threat models and protection levels require quantitative trade-off analysis. For ML workloads requiring cryptographic verification, AES-256 operations add 0.1--0.5 ms per inference depending on model size and hardware acceleration availability. + +```{python} +#| label: encryption-overhead-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ ENCRYPTION LATENCY OVERHEAD (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Tax of Trusted Compute" .callout-notebook +# │ +# │ Goal: Quantify the latency overhead of different encryption levels. +# │ Show: fhe_latency_ratio ≈ 10,000x — inline in notebook result. +# │ How: T_total = T_inference * Overhead_Multiplier. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: eo_base_ms_str, eo_aes_ms_str, eo_fhe_sec_str, eo_fhe_ratio_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class EncryptionOverhead: + # ┌── 1. LOAD ────────────────────────────────────────── + t_inference_ms = 20 # Baseline plaintext inference (MobileNet) + overhead_aes_ms = 0.5 # AES-NI accelerated encryption + multiplier_fhe = 10000 # Fully Homomorphic Encryption overhead + # ┌── 2. EXECUTE ─────────────────────────────────────── + t_aes = t_inference_ms + overhead_aes_ms + t_fhe_ms = t_inference_ms * multiplier_fhe + t_fhe_sec = t_fhe_ms / 1000 + # ┌── 3. GUARD ───────────────────────────────────────── + check(t_fhe_sec == 200, f"FHE time {t_fhe_sec}s unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + eo_base_ms_str = f"{t_inference_ms}" + eo_aes_ms_str = f"{t_aes:.1f}" + eo_fhe_sec_str = f"{t_fhe_sec:.0f}" + eo_fhe_ratio_str = "10,000" + +``` + +::: {.callout-notebook title="The Tax of Trusted Compute"} + +**Problem**: You are deploying a health-monitoring model. You compare three security levels: +1. **Plaintext**: Standard inference. +2. **Encrypted Transport (AES)**: Model/Data encrypted at rest/transit. +3. **Encrypted Compute (FHE)**: Inference performed *on* encrypted data. +How do these choices affect your real-time responsiveness? + +**The Math**: +Inference latency scales by the complexity of the security protocol. + +1. **Plaintext**: **`{python} EncryptionOverhead.eo_base_ms_str` ms**. +2. **AES-256**: 20 ms + 0.5 ms = **`{python} EncryptionOverhead.eo_aes_ms_str` ms** (negligible tax). +3. **FHE**: 20 ms $\times$ `{python} EncryptionOverhead.eo_fhe_ratio_str` = **`{python} EncryptionOverhead.eo_fhe_sec_str` seconds**. + +**The Systems Insight**: Security is a **Latency-Utility Trade-off**. Standard encryption (AES) is "nearly free" on modern hardware, but it only protects data *between* computations. Privacy-preserving compute (FHE) protects data *during* computation but costs **`{python} EncryptionOverhead.eo_fhe_ratio_str`$\times$ in performance**. For a real-time monitor, FHE is currently architecturally impossible. This is why most "Secure AI" relies on **Trusted Execution Environments (TEEs)** like Intel SGX or NVIDIA H100 Confidential Computing, which provide AES-like speed with FHE-like isolation. + +::: + +Homomorphic encryption operations impose 100-100,000$\times$ computational overhead, with fully homomorphic encryption (FHE) at the higher end and somewhat homomorphic encryption (SHE) at the lower end, making them viable only for small models or offline scenarios where strong privacy guarantees justify the performance cost. #### Trusted Execution Environments {#sec-security-privacy-trusted-execution-environments-80ed} @@ -2665,7 +2784,9 @@ PUFs also support authentication in distributed ML pipelines. If the drone offlo @fig-pfu demonstrates how PUF operation depends on inherent physical variation. At a high level, a PUF accepts a challenge input and produces a unique response determined by the physical microstructure of the chip [@gao2020physical]. Variants include optical PUFs, in which the challenge consists of a light pattern and the response is a speckle image, and electronic PUFs such as Arbiter PUFs (APUFs), where timing differences between circuit paths produce a binary output. Another common implementation is the SRAM PUF, which exploits the power-up state of uninitialized SRAM cells: due to threshold voltage mismatch, each cell tends to settle into a preferred value when power is first applied. These response patterns form a stable, reproducible hardware fingerprint. -![**Physical Unclonable Functions**: Pufs generate unique hardware fingerprints from inherent manufacturing variations, enabling device authentication and secure key generation without storing secrets. Optical and electronic PUF implementations use physical phenomena—such as light speckle patterns or timing differences—to produce challenge-response pairs that are difficult to predict or replicate.](images/png/puf_basics.png){#fig-pfu fig-alt="Diagram showing optical and electronic PUF types. Left: optical PUF with laser input producing speckle pattern response. Right: electronic arbiter PUF circuit with challenge bits producing binary response based on timing differences."} +::: {#fig-pfu fig-env="figure" fig-pos="htb" fig-cap="**Physical Unclonable Functions**: PUFs generate unique hardware fingerprints from inherent manufacturing variations, enabling device authentication and secure key generation without storing secrets. Optical and electronic PUF implementations use physical phenomena such as light speckle patterns or timing differences to produce challenge-response pairs that are difficult to predict or replicate." fig-alt="Diagram showing optical and electronic PUF types. Left: optical PUF with laser input producing speckle pattern response. Right: electronic arbiter PUF circuit with challenge bits producing binary response based on timing differences."} +![](images/png/puf_basics.png) +::: Despite their promise, PUFs present several challenges in system design. Their outputs can be sensitive to environmental variation, such as changes in temperature or voltage, which can introduce instability or bit errors in the response. To ensure reliability, PUF systems must often incorporate error correction codes or helper data schemes. Managing large sets of challenge-response pairs also raises questions about storage, consistency, and revocation. Additionally, the unique statistical structure of PUF outputs may make them vulnerable to machine learning-based modeling attacks if not carefully shielded from external observation. @@ -2735,10 +2856,10 @@ Differential privacy formalizes this intuition through a comparison of algorithm ::: {.callout-note title="Differential Privacy Diagram"} ```{.tikz} -\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, scale=0.8, - BlueLine/.style={draw=blue, line width=1.0pt}, - RedLine/.style={draw=red, line width=1.0pt}, - GrayL/.style={fill=gray, opacity=0.5}, +\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}, scale=0.8, + BlueLine/.style={draw=blue, line width=1.0pt}, + RedLine/.style={draw=red, line width=1.0pt}, + GrayL/.style={fill=gray, opacity=0.5}, GrayLine/.style={text=gray}] % Curves diff --git a/book/quarto/contents/vol2/sustainable_ai/sustainable_ai.qmd b/book/quarto/contents/vol2/sustainable_ai/sustainable_ai.qmd index 0be74abd29..0fd478f477 100644 --- a/book/quarto/contents/vol2/sustainable_ai/sustainable_ai.qmd +++ b/book/quarto/contents/vol2/sustainable_ai/sustainable_ai.qmd @@ -80,11 +80,11 @@ Contemporary machine learning applications operate at industrial scales, with en ::: {.callout-definition title="Sustainable AI"} -***Sustainable AI***\index{Sustainable AI!definition} is the engineering discipline that elevates **Environmental Impact** to a first-class design constraint alongside traditional performance and cost objectives. +***Sustainable AI***\index{Sustainable AI!definition} is the systems engineering practice of measuring and optimizing the full environmental cost of ML systems — energy, water, and embodied carbon across training, inference, and hardware manufacturing — and incorporating those costs as explicit constraints in architecture decisions alongside performance and accuracy objectives. -1. **Significance (Quantitative):** It treats energy consumption not as a cost, but as a **Resource Budget**. Within the **Iron Law**, Sustainable AI seeks to maximize the **Accuracy-per-Joule**, recognizing that the system's **Duty Cycle ($\eta$)** is ultimately constrained by the physical limits of power grids and heat rejection. -2. **Distinction (Durable):** Unlike **Corporate Social Responsibility (CSR)** (which is often aspirational), Sustainable AI is an **Operational Requirement**: it involves the precise measurement and optimization of carbon intensity, water usage, and electronic waste across the full lifecycle. -3. **Common Pitfall:** A frequent misconception is that sustainability is "just using green energy." In reality, it is a **Full-Lifecycle Problem**: over 50% of the carbon footprint of an edge device can come from its **Embodied Carbon** (manufacturing), regardless of how clean its operating energy source is. +1. **Significance (Quantitative):** Training GPT-3 consumed approximately 1,287 MWh of energy, equivalent to 78 household-years of electricity. Fine-tuning a pre-trained model on domain data consumes roughly 1–5% of full training cost, making transfer learning a 20–100$\times$ more energy-efficient path to the same capability. At inference scale, a 175B-parameter model serving 10M queries/day at 100 ms per query consumes more cumulative energy in six months than its training — making inference optimization the dominant sustainability lever at production scale. +2. **Distinction (Durable):** Unlike corporate sustainability reporting (which aggregates energy usage into annual CO₂ disclosures), sustainable AI engineering operates at the individual workload level — selecting hardware based on FLOP/watt efficiency, scheduling training during periods of high renewable availability, and choosing model architectures that minimize inference FLOPs rather than simply maximizing accuracy. +3. **Common Pitfall:** A frequent misconception is that switching to renewable energy solves the sustainability problem. For hardware-intensive ML, embodied carbon — the carbon emitted manufacturing the chips, servers, and cooling equipment before they ever run a training job — often equals or exceeds operational carbon; over 50% of an edge device's lifecycle carbon can come from manufacturing, making hardware longevity and utilization rate as important as energy source. ::: @@ -155,11 +155,11 @@ class CarbonCostGPT3: **The Math**: 1. **Energy**: `{python} CarbonCostGPT3.training_mwh_str` MWh = 1,287,000 kWh. -2. **Carbon Intensity (US Average)**: $\approx `{python} CarbonCostGPT3.grid_ci_us_kg` \text{ kg CO}_2\text{/kWh}$. -3. **Total Emissions**: $1,287,000 \times 0.429 \approx \mathbf{`{python} CarbonCostGPT3.total_emissions_kg_str` \text{ kg CO}_2}$. +2. **Carbon Intensity (US Average)**: $\approx$ `{python} CarbonCostGPT3.grid_ci_us_kg` kg CO$_2$/kWh. +3. **Total Emissions**: $1,287,000 \times 0.429 \approx$ **`{python} CarbonCostGPT3.total_emissions_kg_str` kg CO$_2$**. 4. **Comparison**: * One passenger, NY to London (round trip): $\approx 1,000 \text{ kg CO}_2$. - * **Ratio**: `{python} CarbonCostGPT3.total_emissions_kg_str` / 1,000 = $\mathbf{`{python} CarbonCostGPT3.flight_ratio_str` \text{ flights}}$. + * **Ratio**: `{python} CarbonCostGPT3.total_emissions_kg_str` / 1,000 = **`{python} CarbonCostGPT3.flight_ratio_str` flights**. **The Systems Conclusion**: A single training run emits as much carbon as flying a Boeing 747 full of passengers across the Atlantic. **Optimization matters.** Moving this job to a hydro-powered region (0.02 kg/kWh) would reduce emissions by **`{python} CarbonCostGPT3.emissions_ratio_str`x** to just ~25 flights. ::: @@ -169,7 +169,63 @@ class CarbonCostGPT3: ::: AI systems consume resources at industrial scales that rival traditional heavy industries. - Training a single large language model consumes thousands of megawatt-hours of electricity, equivalent to powering hundreds of households for months.[^fn-household-energy] Data centers that include AI workloads are projected to account for 8% of global power consumption by 2030, surpassing aviation at 2.1% and approaching cement production at 4% [@oecd2023blueprint].[^fn-datacenter-industrial-scale] Computational demands increased 350,000$\times$ from 2012 to 2019 [@schwartz2020green], while hardware efficiency improved at a far slower rate, creating an unsustainable growth trajectory. + +```{python} +#| label: carbon-frontier-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ CARBON-PERFORMANCE FRONTIER (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "The Geography of Carbon" .callout-notebook +# │ +# │ Goal: Quantify the carbon footprint difference between geographic regions. +# │ Show: ratio ≈ 40x — inline in notebook result. +# │ How: Carbon = Energy_kWh * Intensity_g/kWh. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: cf_energy_mwh_str, cf_quebec_tonnes_str, cf_poland_tonnes_str, cf_ratio_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class CarbonFrontier: + # ┌── 1. LOAD ────────────────────────────────────────── + energy_mwh = 10000 # 10 GWh training run + intensity_quebec = 20 # gCO2/kWh (Hydro) + intensity_poland = 800 # gCO2/kWh (Coal) + # ┌── 2. EXECUTE ─────────────────────────────────────── + energy_kwh = energy_mwh * 1000 + quebec_tonnes = (energy_kwh * intensity_quebec) / 1e6 + poland_tonnes = (energy_kwh * intensity_poland) / 1e6 + ratio = intensity_poland / intensity_quebec + # ┌── 3. GUARD ───────────────────────────────────────── + check(ratio == 40, f"Ratio {ratio} unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + cf_energy_mwh_str = f"{energy_mwh:,}" + cf_quebec_tonnes_str = f"{quebec_tonnes:,.0f}" + cf_poland_tonnes_str = f"{poland_tonnes:,.0f}" + cf_ratio_str = f"{ratio:.0f}" + +``` + +::: {.callout-notebook title="The Geography of Carbon"} + +**Problem**: You are choosing a datacenter for a `{python} CarbonFrontier.cf_energy_mwh_str` MWh training run. +- **Site A (Quebec)**: Hydropower, `{python} CarbonFrontier.intensity_quebec` g $CO_2$/kWh. +- **Site B (Poland)**: Coal-heavy, `{python} CarbonFrontier.intensity_poland` g $CO_2$/kWh. +How does the location affect your model's carbon footprint? + +**The Math**: +Carbon = Energy $\times$ Grid Intensity. + +1. **Site A Emissions**: 10,000,000 kWh $\times$ 20g = 200,000,000g = **`{python} CarbonFrontier.cf_quebec_tonnes_str` tonnes $CO_2$**. +2. **Site B Emissions**: 10,000,000 kWh $\times$ 800g = 8,000,000,000g = **`{python} CarbonFrontier.cf_poland_tonnes_str` tonnes $CO_2$**. +3. **Ratio**: 8,000 / 200 = **`{python} CarbonFrontier.cf_ratio_str`$\times$ difference**. + +**The Systems Insight**: Site selection is the single most effective tool for sustainable AI. A **`{python} CarbonFrontier.cf_ratio_str`$\times$ difference** in carbon emissions is larger than any possible algorithmic speedup. In the Machine Learning Fleet, **Carbon-Aware Scheduling** (moving non-urgent jobs to low-carbon regions or hours) is a first-class operational competency. Efficiency is not just about FLOPs; it's about the **Carbon-Intensity of those FLOPs**. + +::: + +Training a single large language model consumes thousands of megawatt-hours of electricity, equivalent to powering hundreds of households for months.[^fn-household-energy] Data centers that include AI workloads are projected to account for 8% of global power consumption by 2030, surpassing aviation at 2.1% and approaching cement production at 4% [@oecd2023blueprint].[^fn-datacenter-industrial-scale] Computational demands increased 350,000$\times$ from 2012 to 2019 [@schwartz2020green], while hardware efficiency improved at a far slower rate, creating an unsustainable growth trajectory. [^fn-household-energy]: **Household Energy Baseline**: The average U.S. household consumes 10,500 kWh annually. GPT-3's verified 1,287 MWh training run equals 122 households' annual electricity, and frontier models have grown 25$\times$ in compute since then. This comparison anchors an otherwise abstract energy figure to physical infrastructure: a single training run draws more grid capacity than a residential neighborhood. \index{Household Energy!sustainability baseline} @@ -510,7 +566,9 @@ This scale of energy consumption makes efficiency improvements an engineering im Research shows that increasing model size, dataset size, and compute used for training improves performance smoothly with no signs of saturation [@kaplan2020scaling]. @fig-model-scaling demonstrates that test loss decreases predictably as each of these three factors increases, with no apparent ceiling in sight. Beyond training, AI-powered applications such as large-scale recommender systems and generative models require continuous inference at scale, consuming energy even after training completes. As AI adoption grows across industries from finance to healthcare to entertainment, the cumulative energy burden of AI workloads continues to rise, raising concerns about the environmental impact of widespread deployment. -![**Model Scaling Laws**: Increasing model size, dataset size, and compute consistently reduces test loss, indicating that performance improvements continue to be achievable with greater resources and without evidence of saturation. These scaling laws suggest that larger models trained on more data with increased compute will likely yield further gains in performance, driving continued investment in these areas.](images/png/model_scaling.png){#fig-model-scaling fig-alt="Three line graphs showing test loss decreasing as model parameters, dataset tokens, and compute FLOPs increase. Each plot shows smooth power-law scaling with no saturation, indicating continued improvement with more resources."} +::: {#fig-model-scaling fig-env="figure" fig-pos="htb" fig-cap="**Model Scaling Laws**: Increasing model size, dataset size, and compute consistently reduces test loss, indicating that performance improvements continue to be achievable with greater resources and without evidence of saturation. These scaling laws suggest that larger models trained on more data with increased compute will likely yield further gains in performance, driving continued investment in these areas." fig-alt="Three line graphs showing test loss decreasing as model parameters, dataset tokens, and compute FLOPs increase. Each plot shows smooth power-law scaling with no saturation, indicating continued improvement with more resources."} +![](images/png/model_scaling.png) +::: Beyond electricity consumption, the sustainability challenges of AI extend to hardware resource demands and the energy efficiency limitations of current architectures. Different processor types affect environmental impact through their energy characteristics. Central Processing Units consume approximately 100 picojoules per multiply-accumulate operation, Graphics Processing Units achieve 10 pJ/MAC, while specialized Tensor Processing Units reach 1 pJ/MAC, and specialized accelerators approach 0.1 pJ/MAC.[^fn-pj-mac-hierarchy] These hardware platforms require rare earth metals and complex manufacturing processes with embodied carbon. @@ -865,6 +923,59 @@ Beyond chip-level power, data center infrastructure imposes additional energy ov $$PUE = \frac{P_{total\_facility}}{P_{IT\_equipment}}$$ {#eq-pue} +```{python} +#| label: pue-efficiency-notebook +#| echo: false +# ┌───────────────────────────────────────────────────────────────────────────── +# │ PUE EFFICIENCY GAINS (NOTEBOOK) +# ├───────────────────────────────────────────────────────────────────────────── +# │ Context: "PUE: The Cost of Cooling" .callout-notebook +# │ +# │ Goal: Quantify annual energy savings from PUE optimization. +# │ Show: savings_mwh ≈ 8,400 MWh — inline in notebook result. +# │ How: Energy = P_IT * (PUE_old - PUE_new) * 8760. +# │ +# │ Imports: mlsys.formatting (check) +# │ Exports: pue_it_mw_str, pue_old_str, pue_new_str, pue_savings_mwh_str, pue_savings_usd_str +# └───────────────────────────────────────────────────────────────────────────── +from mlsys.formatting import check + +class PueEfficiency: + # ┌── 1. LOAD ────────────────────────────────────────── + p_it_mw = 2.0 # 2MW cluster (~3000 H100s) + pue_old = 1.58 # industry average + pue_new = 1.10 # state-of-the-art + elec_price = 0.07 # $/kWh + # ┌── 2. EXECUTE ─────────────────────────────────────── + diff = pue_old - pue_new + annual_savings_mwh = p_it_mw * diff * 8760 + annual_savings_usd = annual_savings_mwh * 1000 * elec_price + # ┌── 3. GUARD ───────────────────────────────────────── + check(8000 < annual_savings_mwh < 9000, f"Savings {annual_savings_mwh:.0f} MWh unexpected") + # ┌── 4. OUTPUT ──────────────────────────────────────── + pue_it_mw_str = f"{p_it_mw:.1f}" + pue_old_str = f"{pue_old:.2f}" + pue_new_str = f"{pue_new:.2f}" + pue_savings_mwh_str = f"{annual_savings_mwh:,.0f}" + pue_savings_usd_str = f"{annual_savings_usd:,.0f}" + +``` + +::: {.callout-notebook title="PUE: The Cost of Cooling"} + +**Problem**: You operate a `{python} PueEfficiency.pue_it_mw_str` MW cluster. If you can optimize your facility from the industry average PUE (`{python} PueEfficiency.pue_old_str`) to state-of-the-art (`{python} PueEfficiency.pue_new_str`), how much energy and money do you save annually? + +**The Math**: +Energy saved is the difference in infrastructure overhead ($PUE-1$) across the IT load. + +1. **Overhead Reduction**: `{python} PueEfficiency.pue_old_str` - `{python} PueEfficiency.pue_new_str` = **0.48**. +2. **Annual Energy Savings**: $2.0 \text{ MW} \times 0.48 \times 8,760 \text{ hours} \approx$ **`{python} PueEfficiency.pue_savings_mwh_str` MWh**. +3. **Financial Savings**: `{python} PueEfficiency.pue_savings_mwh_str` MWh $\times$ \$70/MWh $\approx$ **\$`{python} PueEfficiency.pue_savings_usd_str`**. + +**The Systems Insight**: Infrastructure optimization is as valuable as algorithmic optimization. Dropping your PUE by 0.48 is equivalent to discovering an algorithmic "free lunch" that makes your entire model **30% more efficient** without changing a single line of training code. For large operators, cooling efficiency is the primary economic lever for sustainability. + +::: + A PUE of 1.0 would indicate perfect efficiency where all energy powers computation, though this is physically impossible since cooling, power distribution, and lighting require nonzero energy. Industry-average data centers operate at PUE of 1.5 to 2.0, meaning that 50% to 100% additional energy beyond computation goes to infrastructure. Leading hyperscale facilities achieve PUE between 1.1 and 1.2 through advanced cooling techniques including free-air cooling in cold climates, liquid cooling for high-density GPU clusters, and optimized power distribution. @eq-wue formalizes Water Usage Effectiveness (WUE), capturing the water consumption that evaporative cooling and other processes require: @@ -1533,7 +1644,7 @@ class H100KwScenario: * **Embodied**: 8 GPUs$\times$ 150 kg/GPU = 1200 kg total. * **Amortization**: Lifetime = 3 years (26,280 hours). * Hourly "Rent" = $1200 / 26280 \approx 0.046$ kg/hour. - * Job Cost = $0.046 \times 10 = \mathbf{`{python} EmbodiedCarbonAmort.embodied_carbon_job_str` \text{ kg } CO_2}$. + * Job Cost = $0.046 \times 10 =$ **`{python} EmbodiedCarbonAmort.embodied_carbon_job_str` kg CO$_2$**. **Conclusion**: For long-lived hardware in dirty grids, **electricity dominates** (`{python} EmbodiedCarbonAmort.op_carbon_dirty_str` vs `{python} EmbodiedCarbonAmort.embodied_carbon_job_str`). However, in **clean grids** (hydro, 0.02 kg/kWh), operational drops to **`{python} EmbodiedCarbonAmort.op_carbon_clean_str` kg**, making embodied carbon a significant fraction (~`{python} EmbodiedCarbonAmort.share_clean_pct_str`%) of the total footprint. ::: @@ -2056,7 +2167,9 @@ The GHG Protocol[^fn-ghg-protocol-standard] framework [@ghgprotocol2023] provide - **Scope 2 (Indirect Energy Emissions)**: Electricity purchased from the grid, the primary emission source for cloud computing workloads. - **Scope 3 (Value Chain Emissions)**: Extend beyond direct control---semiconductor manufacturing, hardware transportation, end-of-life disposal of AI accelerators. -![**GHG Emission Scopes**: Organizations categorize carbon emissions into scope 1 (direct), scope 2 (purchased energy), and scope 3 (value chain) to comprehensively assess environmental impact. Source: Ucircularise.](images/png/ghg_protocol.png){#fig-ghg-protocol fig-alt="Diagram showing three concentric emission scopes: Scope 1 for direct emissions from owned sources, Scope 2 for purchased energy emissions, and Scope 3 for value chain emissions including manufacturing and disposal."} +::: {#fig-ghg-protocol fig-env="figure" fig-pos="htb" fig-cap="**GHG Emission Scopes**: Organizations categorize carbon emissions into scope 1 (direct), scope 2 (purchased energy), and scope 3 (value chain) to comprehensively assess environmental impact. Source: Ucircularise." fig-alt="Diagram showing three concentric emission scopes: Scope 1 for direct emissions from owned sources, Scope 2 for purchased energy emissions, and Scope 3 for value chain emissions including manufacturing and disposal."} +![](images/png/ghg_protocol.png) +::: Categorizing these emissions into Scope 1, 2, and 3 frameworks provides a standardized vocabulary for corporate environmental reporting. Correctly applying this framework in practice requires classifying the various hidden emission sources across a typical ML platform's operational lifecycle. @@ -3066,7 +3179,9 @@ plt.show() Renewable energy variability presents a key challenge for carbon-aware scheduling. @fig-europe_energy_grid captures European grid dynamics: solar energy peaks at midday, wind shows distinct peaks in mornings and evenings, and fossil generation fills the gaps. This temporal pattern determines when AI workloads can run on clean energy. -![**European Energy Mix**: Renewable energy sources exhibit significant temporal variability, necessitating fossil fuel supplementation to meet consistent demand. Understanding this fluctuation is important for effectively scheduling AI workloads to periods of high renewable energy availability. Source: Uenergy charts.](images/png/europe_energy_grid.png){#fig-europe_energy_grid fig-alt="Stacked area chart of European energy production showing temporal variability. Solar peaks at midday, wind varies throughout day. Nuclear provides constant baseload. Fossil fuels supplement when renewables fall short."} +::: {#fig-europe_energy_grid fig-env="figure" fig-pos="htb" fig-cap="**European Energy Mix**: Renewable energy sources exhibit significant temporal variability, necessitating fossil fuel supplementation to meet consistent demand. Understanding this fluctuation is important for effectively scheduling AI workloads to periods of high renewable energy availability. Source: Uenergy charts." fig-alt="Stacked area chart of European energy production showing temporal variability. Solar peaks at midday, wind varies throughout day. Nuclear provides constant baseload. Fossil fuels supplement when renewables fall short."} +![](images/png/europe_energy_grid.png) +::: Energy-aware AI frameworks complement scheduling by optimizing the workloads themselves. Zeus [@jie2023zeus] achieves 75% energy savings on BERT training by automatically finding optimal energy-performance trade-offs, while Perseus [@jaewon2023perseus] reduces GPU memory usage by 50% through dynamic batching. These tools, alongside CodeCarbon for emissions tracking, democratize energy optimization beyond hyperscale companies. diff --git a/book/quarto/mlsys/constants.py b/book/quarto/mlsys/constants.py index 09283eb719..ba78f6395b 100644 --- a/book/quarto/mlsys/constants.py +++ b/book/quarto/mlsys/constants.py @@ -145,11 +145,18 @@ H100_TDP = 700 * watt # SXM variant # NVIDIA B100/B200 (Blackwell, 2024) — Source: NVIDIA Blackwell Architecture B200_FLOPS_FP16_TENSOR = 2250 * TFLOPs / second # Dense. Sparse is 4500. +B200_FLOPS_FP16_SPARSE = 4500 * TFLOPs / second B200_FLOPS_FP8_TENSOR = 4500 * TFLOPs / second # Dense. Sparse is 9000. B200_MEM_BW = 8 * TB / second # HBM3e B200_MEM_CAPACITY = 192 * GiB B200_TDP = 1000 * watt +# AMD Instinct MI300X (CDNA 3, 2023) — Source: AMD Instinct MI300X Data Sheet +MI300X_FLOPS_FP16_TENSOR = 1307 * TFLOPs / second # Dense. Sparse is 2614. +MI300X_MEM_BW = 5.3 * TB / second +MI300X_MEM_CAPACITY = 192 * GiB +MI300X_TDP = 750 * watt + # NVIDIA T4 (Turing, 2018) — Source: NVIDIA T4 Data Sheet T4_FLOPS_FP16_TENSOR = 65 * TFLOPs / second T4_FLOPS_INT8 = 130 * TFLOPs / second @@ -209,7 +216,8 @@ LATENCY_INFINIBAND = 5000 * NS LATENCY_NVME_SSD = 100000 * NS # Mobile NPU -MOBILE_NPU_TOPS_INT8 = 35 * TFLOPs / second +MOBILE_NPU_TOPS_INT8 = 50 * TFLOPs / second +MOBILE_FLAGSHIP_NPU_TOPS_INT8 = 100 * TFLOPs / second MOBILE_NPU_MEM_BW = 100 * GB / second # --- Datasets --- @@ -379,7 +387,7 @@ MOBILE_STORAGE_RANGE = "128 GB-1 TB" MOBILE_TDP_RANGE_W = "3-5" # Deployment tiers (reference capacities) -SMARTPHONE_RAM_GB = 6 * GB +SMARTPHONE_RAM_GB = 8 * GB MCU_RAM_KIB = 512 * KiB CLOUD_MEM_GIB = 100 * GiB MOBILE_MEM_GIB = 8 * GiB diff --git a/book/quarto/mlsys/hardware.py b/book/quarto/mlsys/hardware.py index fc582e3f50..c8113645c6 100644 --- a/book/quarto/mlsys/hardware.py +++ b/book/quarto/mlsys/hardware.py @@ -10,6 +10,7 @@ from .constants import ( A100_MEM_BW, A100_FLOPS_FP16_TENSOR, A100_MEM_CAPACITY, A100_TDP, A100_FLOPS_FP32, A100_FLOPS_TF32, A100_FLOPS_INT8, H100_MEM_BW, H100_FLOPS_FP16_TENSOR, H100_MEM_CAPACITY, H100_TDP, H100_FLOPS_TF32, H100_FLOPS_FP8_TENSOR, H100_FLOPS_INT8, B200_MEM_BW, B200_FLOPS_FP16_TENSOR, B200_MEM_CAPACITY, B200_TDP, B200_FLOPS_FP8_TENSOR, + MI300X_MEM_BW, MI300X_FLOPS_FP16_TENSOR, MI300X_MEM_CAPACITY, MI300X_TDP, T4_MEM_BW, T4_FLOPS_FP16_TENSOR, T4_TDP, T4_FLOPS_INT8, TPUV4_MEM_BW, TPUV4_FLOPS_BF16, MOBILE_NPU_MEM_BW, MOBILE_NPU_TOPS_INT8, @@ -86,6 +87,10 @@ class Cloud: dispatch_tax=0.01 * ureg.ms) B200 = HardwareSpec("NVIDIA B200", 2024, B200_MEM_BW, B200_FLOPS_FP16_TENSOR, B200_MEM_CAPACITY, B200_TDP, fp8_flops=B200_FLOPS_FP8_TENSOR, dispatch_tax=0.008 * ureg.ms) + + MI300X = HardwareSpec("AMD MI300X", 2023, MI300X_MEM_BW, MI300X_FLOPS_FP16_TENSOR, MI300X_MEM_CAPACITY, MI300X_TDP, + dispatch_tax=0.012 * ureg.ms) + T4 = HardwareSpec("NVIDIA T4", 2018, T4_MEM_BW, T4_FLOPS_FP16_TENSOR, 16 * ureg.GiB, T4_TDP, int8_flops=T4_FLOPS_INT8, dispatch_tax=0.03 * ureg.ms) diff --git a/book/quarto/mlsys/models.py b/book/quarto/mlsys/models.py index 4f16bec3fc..a721d6e533 100644 --- a/book/quarto/mlsys/models.py +++ b/book/quarto/mlsys/models.py @@ -6,7 +6,7 @@ from dataclasses import dataclass from typing import Optional, Tuple from .constants import ( ureg, Q_, - GPT2_PARAMS, GPT3_PARAMS, GPT4_TRAINING_GPU_DAYS, + GPT2_PARAMS, GPT3_PARAMS, GPT4_EST_PARAMS, GPT4_TRAINING_GPU_DAYS, BERT_BASE_PARAMS, BERT_LARGE_PARAMS, ALEXNET_PARAMS, RESNET50_PARAMS, MOBILENETV2_PARAMS, KWS_DSCNN_PARAMS, ANOMALY_MODEL_PARAMS, @@ -62,7 +62,7 @@ class GPT: """GPT Model Family.""" GPT2 = ModelSpec("GPT-2 (1.5B)", GPT2_PARAMS, "Transformer", layers=48) GPT3 = ModelSpec("GPT-3 (175B)", GPT3_PARAMS, "Transformer", layers=96, training_ops=GPT3_TRAINING_OPS) - GPT4 = ModelSpec("GPT-4", 1.8e12 * ureg.param, "Transformer", layers=120, training_gpu_days=GPT4_TRAINING_GPU_DAYS) + GPT4 = ModelSpec("GPT-4", GPT4_EST_PARAMS, "Transformer", layers=120, training_gpu_days=GPT4_TRAINING_GPU_DAYS) class Language: """Large Language Models.""" @@ -70,6 +70,8 @@ class Language: BERT_Base = ModelSpec("BERT-Base", BERT_BASE_PARAMS, "Transformer", layers=12, inference_flops=22e9 * ureg.flop) BERT_Large = ModelSpec("BERT-Large", BERT_LARGE_PARAMS, "Transformer", layers=24) Llama2_70B = ModelSpec("Llama-2-70B", 70e9 * ureg.param, "Transformer", layers=80) + Llama3_70B = ModelSpec("Llama-3-70B", 70.6e9 * ureg.param, "Transformer", layers=80) + Llama3_405B = ModelSpec("Llama-3.1-405B", 405e9 * ureg.param, "Transformer", layers=126) class Recommendation: """Recommendation Models.""" diff --git a/book/quarto/mlsys/test_units.py b/book/quarto/mlsys/test_units.py index 10e252a45e..e8b65aa9ca 100644 --- a/book/quarto/mlsys/test_units.py +++ b/book/quarto/mlsys/test_units.py @@ -102,7 +102,7 @@ def test_flop_units(): ok &= check("V100 125 TFLOPs/s", V100_FLOPS_FP16_TENSOR.to(TFLOPs / second).magnitude, 125.0) ok &= check("H100 989 TFLOPs/s", H100_FLOPS_FP16_TENSOR.to(TFLOPs / second).magnitude, 989.0) ok &= check("T4 65 TFLOPs/s", T4_FLOPS_FP16_TENSOR.to(TFLOPs / second).magnitude, 65.0) - ok &= check("Mobile 35 TFLOPs/s", MOBILE_NPU_TOPS_INT8.to(TFLOPs / second).magnitude, 35.0) + ok &= check("Mobile 50 TFLOPs/s", MOBILE_NPU_TOPS_INT8.to(TFLOPs / second).magnitude, 50.0) # Model FLOPs ok &= check("ResNet 4.1 GFLOPs", RESNET50_FLOPs.to(GFLOPs).magnitude, 4.1) @@ -251,10 +251,14 @@ def test_extended_gpu_specs(): ok &= check("V100 TDP", V100_TDP.to(watt).magnitude, 300.0) # B200 - ok &= check("B200 FP16", B200_FLOPS_FP16_TENSOR.to(TFLOPs / second).magnitude, 4500.0) + ok &= check("B200 FP16", B200_FLOPS_FP16_TENSOR.to(TFLOPs / second).magnitude, 2250.0) ok &= check("B200 BW", B200_MEM_BW.to(TB / second).magnitude, 8.0) ok &= check("B200 Mem", B200_MEM_CAPACITY.to(GiB).magnitude, 192.0) + # MI300X + ok &= check("MI300X FP16", MI300X_FLOPS_FP16_TENSOR.to(TFLOPs / second).magnitude, 1307.0, tol=0.01) + ok &= check("MI300X BW", MI300X_MEM_BW.to(TB / second).magnitude, 5.3) + # TPUv4 ok &= check("TPUv4 BF16", TPUV4_FLOPS_BF16.to(TFLOPs / second).magnitude, 275.0) ok &= check("TPUv4 BW", TPUV4_MEM_BW.to(GB / second).magnitude, 1200.0) diff --git a/tinytorch/src/15_quantization/ABOUT.md b/tinytorch/src/15_quantization/ABOUT.md index 30a8e1e131..e2f358cdb2 100644 --- a/tinytorch/src/15_quantization/ABOUT.md +++ b/tinytorch/src/15_quantization/ABOUT.md @@ -648,7 +648,7 @@ The core quantization mathematics: scale calculation, zero-point mapping, INT8 r To appreciate why quantization is critical for production ML, consider these deployment scenarios: -- **Mobile AI**: iPhone has 6 GB RAM shared across all apps. A quantized BERT (110 MB) fits comfortably; FP32 version (440 MB) causes memory pressure and swapping. +- **Mobile AI**: Modern smartphones have 8 GB+ RAM shared across all apps. A quantized BERT (110 MB) fits comfortably; FP32 version (440 MB) causes memory pressure and swapping. - **Edge computing**: IoT devices often have 512 MB RAM. Quantization enables on-device inference for privacy-sensitive applications (medical devices, security cameras). - **Data centers**: Serving 1000 requests/second requires multiple model replicas. With 4× memory reduction, you fit 4× more models per GPU, reducing serving costs by 75%. - **Battery life**: INT8 operations consume 2-4× less energy than FP32 on mobile processors. Quantized models drain battery slower, improving user experience.