Caution

This volume is under active development. I am writing and revising chapters continuously. Diagrams, figures, and cross-references are being created and updated throughout. What you see here is a work in progress, not a finished product. I share it openly because I believe in transparent development. Expect the content to evolve significantly before the Summer 2027 release.

About This Volume

Volume II moves from introductory single-machine ML systems to fleets of machines connected by high-speed networks. It covers the mathematical and algorithmic demand for scale, how to build the physical infrastructure that meets it, how to serve models to billions of users, and how to do all of this safely and responsibly.

The assumed foundation is local model execution, memory hierarchy, training and inference, and accelerator basics. The advanced material explains how those constraints change when many nodes coordinate through network fabrics, message passing, and fault tolerance across racks and data centers. Security, privacy, environmental limits, and governance enter as design requirements throughout; the final part develops their enforcement mechanisms.

What You Will Learn

Part	Focus	What you will learn
I. The Fleet	Physical infrastructure	How to design compute clusters, high-speed network fabrics, and scalable data storage systems that form the foundation of distributed ML
II. Distributed ML	The logic of distributed coordination	How to partition workloads with 3D parallelism, coordinate collective communication primitives, survive inevitable hardware failures, and orchestrate shared clusters
III. The Fleet in Production	Serving and operations	How to measure how much of the fleet becomes useful work, serve models under tail-latency SLOs, deploy to edge devices, and manage production MLOps lifecycles
IV. The Responsible Fleet	Governance and boundaries	How to secure ML systems against adversaries, evaluate and improve empirical robustness under drift, operate within planetary carbon limits, and govern models responsibly

Chapter Map

#	Chapter	Directory	Core question
1	Introduction	`01_introduction/`	Why do single-machine systems break down at production scale?
2	Compute Infrastructure	`02_compute_infrastructure/`	How do you build and provision GPU clusters and thermal envelopes?
3	Network Fabrics	`03_network_fabrics/`	How does data move between machines across high-speed fabrics?
4	Data Storage	`04_data_storage/`	How do you feed training pipelines and persist checkpoints without starving accelerators?
5	Collective Communication	`05_collective_communication/`	How do machines coordinate and exchange state during distributed training?
6	Distributed Training	`06_distributed_training/`	How do you partition models across thousands of accelerators?
7	Fault Tolerance	`07_fault_tolerance/`	Why is hardware failure routine, and how do you recover without losing progress?
8	Fleet Orchestration	`08_fleet_orchestration/`	How do you schedule, pack, and isolate multi-tenant workloads across shared clusters?
9	Fleet Efficiency	`09_performance_engineering/`	How much of an allocated fleet becomes training progress and served work, and how is the lost share recovered?
10	Serving at Scale	`10_inference/`	How do you serve models to millions of concurrent requests under latency SLOs?
11	Edge Intelligence	`11_edge_intelligence/`	How do you adapt and coordinate models on resource-constrained edge devices?
12	ML Operations at Scale	`12_ops_scale/`	How do you monitor, version, and operate fleets of production models?
13	Security & Privacy	`13_security_privacy/`	How do you defend ML systems against adversarial attacks and preserve privacy?
14	Robust AI	`14_robust_ai/`	How do you make ML systems reliable under distribution drift and silent failures?
15	Sustainable AI	`15_sustainable_ai/`	How do you measure and reduce the energy and carbon footprint of the fleet?
16	Responsible AI	`16_responsible_ai/`	How do you govern ML systems fairly, transparently, and accountably?
17	Conclusion	`17_conclusion/`	Why must the unit of machine learning systems design become the fleet?

Prerequisites

Volume II is a standalone book. It assumes an introductory ML systems course or equivalent experience with single-machine execution, memory hierarchy, training and inference pipelines, and hardware acceleration. No other book is required. Appendix A recaps the single-machine performance model (the iron law, arithmetic intensity, and the Roofline Model), and the book introduces its own C³ taxonomy for reasoning about fleets.

Working in the Open

I develop this volume in the open because I believe it produces a better textbook. Every commit is visible, every editorial decision is traceable. If something looks rough, that is because you are watching the book being written.

If you notice an error, have a suggestion, or want to propose a topic, please open an issue or start a discussion. Feedback on structure, missing topics, examples, and clarity is especially valuable at this stage. The book is better for every reader who engages with it.

Links

Resource	Link
Read online	mlsysbook.ai/vol2
Volume I	books/vol1/
Main README	Repository root
Full textbook	mlsysbook.ai/book
Discussions	GitHub Discussions