mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-07-16 11:12:00 -05:00
Add workforce gap framing to introduction with supporting citations
Reframe introduction to emphasize ML systems engineering workforce shortage as core problem: - Open with 3.2:1 supply/demand ratio and 150K global practitioners - Position tacit knowledge (not algorithmic ML) as bottleneck for systems engineering - Contrast automation of model design vs. manual judgment for memory/performance tradeoffs - Add workforce citations: Robert Half 2024 talent gap, Keller Executive Search 2025 AI gap Introduction now grounds TinyTorch's pedagogical approach in workforce development necessity. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -173,13 +173,11 @@ The 20-module curriculum (60--80 hours) provides complete open-source implementa
|
||||
% Main content
|
||||
\section{Introduction}
|
||||
|
||||
Machine learning systems have emerged as a distinct discipline requiring specialized education—just as computers became sufficiently complex to warrant computer engineering curricula integrating hardware and software. This integration is not merely additive but transformative: students who understand \emph{how} frameworks work internally can debug gradient flow issues, diagnose memory bottlenecks, and make architecture-performance tradeoffs that algorithm-only education cannot provide.
|
||||
The machine learning field faces a critical bottleneck: demand for ML systems engineers outstrips supply by over 3:1, with only 150,000 skilled practitioners worldwide serving an industry where AI job postings have grown 78\% year-over-year while the talent pool expanded just 24\%~\citep{roberthalf2024talent,keller2025ai}. Forty to fifty percent of executives cite this talent shortage as their primary barrier to AI deployment~\citep{keller2025ai}. Unlike algorithmic ML—where automated tools increasingly handle model architecture search and hyperparameter tuning—systems engineering remains bottlenecked by tacit knowledge: understanding \emph{why} Adam requires 2$\times$ optimizer state memory, \emph{when} attention's $O(N^2)$ scaling becomes prohibitive, \emph{how} to navigate accuracy-latency-memory tradeoffs. These engineering judgment calls resist automation because they depend on mental models of framework internals, traditionally acquired through years of production debugging rather than formal coursework.
|
||||
|
||||
\textbf{Learning Outcomes}: By building PyTorch's core components from scratch, students gain (1) \textbf{framework internals knowledge}—understanding why \texttt{loss.backward()} traverses computational graphs, why Adam's optimizer state requires 2$\times$ parameter memory (momentum + variance buffers, resulting in 4$\times$ total for training when including weights and gradients), and why attention mechanisms scale $O(N^2)$ with sequence length; (2) \textbf{debugging skills}—diagnosing gradient flow breaks, memory profiling OOM errors, and tracing performance bottlenecks through implementation understanding; and (3) \textbf{systems thinking}—calculating memory footprints before operations, reasoning about computational complexity, and navigating accuracy-speed-memory tradeoffs in optimization.
|
||||
Educational ML frameworks largely teach algorithm implementation without systems depth, or systems concepts without hands-on framework construction. Students learn to \emph{use} PyTorch's APIs without understanding \emph{how} \texttt{loss.backward()} traverses computational graphs, \emph{why} optimizers consume memory beyond parameters, or \emph{when} architectural choices impact deployment feasibility. This knowledge gap directly contributes to the workforce shortage: graduates can train models but struggle with memory profiling, performance optimization, and production system design—the tacit knowledge that separates framework users from systems engineers.
|
||||
|
||||
Traditional ML coursework teaches students to \emph{use} frameworks like PyTorch as black boxes. Students learn to call \texttt{loss.backward()} without understanding computational graph traversal, instantiate optimizers without knowing their memory costs, and deploy models without profiling resource requirements. TinyTorch inverts this: students build the internals first, gaining transparency that transfers to production framework usage.
|
||||
|
||||
We present TinyTorch, a hands-on companion to the \emph{Machine Learning Systems} textbook~\citep{reddi2024mlsysbook}, providing practical implementation experience alongside theoretical foundations. While the textbook establishes principles of ML systems engineering, TinyTorch enables students to build those systems from scratch. The curriculum inverts traditional pedagogy: instead of using frameworks as abstractions, students construct the internals themselves, implementing every component from tensors to transformers using only NumPy. \Cref{fig:code-comparison} illustrates this transformation from framework users to framework engineers.
|
||||
We present TinyTorch, a 20-module curriculum where students build PyTorch's core components from scratch—tensors, autograd, optimizers, CNNs, transformers, and production optimization techniques—using only NumPy. As a hands-on companion to the \emph{Machine Learning Systems} textbook~\citep{reddi2024mlsysbook}, TinyTorch transforms the tacit knowledge bottleneck into explicit pedagogy: students don't just learn \emph{that} Adam requires 4$\times$ training memory, they \emph{implement} momentum and variance buffers and measure the memory cost directly. By building framework internals, students develop the mental models that enable systems engineering judgment—the durable skills that complement, rather than compete with, automated tools. \Cref{fig:code-comparison} illustrates this transformation from framework users to framework engineers.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
|
||||
@@ -497,6 +497,21 @@
|
||||
url = {http://cs231n.stanford.edu/},
|
||||
}
|
||||
|
||||
@misc{keller2025ai,
|
||||
author = {{Keller Executive Search}},
|
||||
title = {AI \& Machine-Learning Talent Gap 2025},
|
||||
year = {2025},
|
||||
url = {https://www.kellerexecutivesearch.com/intelligence/ai-machine-learning-talent-gap-2025/}
|
||||
}
|
||||
|
||||
@misc{roberthalf2024talent,
|
||||
author = {{Robert Half}},
|
||||
title = {New Robert Half Research Reveals Severity of the Technology Skills Gap Amid Talent Shortage},
|
||||
year = {2024},
|
||||
month = {May},
|
||||
url = {https://press.roberthalf.com/2024-05-08-New-Robert-Half-Research-Reveals-Severity-of-the-Technology-Skills-Gap-Amid-Talent-Shortage}
|
||||
}
|
||||
|
||||
@misc{chen2022dlsyscourse,
|
||||
author = {Chen, Tianqi and Zheng, Zico},
|
||||
title = {CS 10-414/614: Deep Learning Systems},
|
||||
|
||||
Reference in New Issue
Block a user