Improve paragraph transitions throughout paper

- Future Directions: add numbering and connecting phrases between extensions
- Limitations: add category intro and transition words
- Pedagogical Scope: clarify two-principle structure
- Conclusion: add 'First/Second/Third' and audience transition

tinytorch/paper/paper.tex

@@ -1025,25 +1025,25 @@ This section reflects on TinyTorch's design through three lenses: pedagogical sc
 \subsection{Pedagogical Scope as Design Decision}
 \label{subsec:scope}
 
-TinyTorch's single node, CPU-only, framework-internals-focused scope represents deliberate pedagogical constraint, not technical limitation. This scoping embodies three design principles:
+TinyTorch's single node, CPU-only, framework-internals-focused scope represents deliberate pedagogical constraint, not technical limitation. This scoping embodies two core design principles.
 
-\textbf{Accessibility over performance}: Pure Python eliminates GPU dependency, prioritizing equitable access over execution speed (pedagogical transparency detailed in \Cref{sec:systems}). GPU access remains inequitably distributed: cloud credits favor well-funded institutions, personal GPUs favor affluent students. The 100--10,000$\times$ slowdown versus PyTorch is acceptable when the pedagogical goal is understanding internals, not training production models.
+\textbf{Accessibility over performance.} The first principle prioritizes equitable access. Pure Python eliminates GPU dependency, enabling participation regardless of hardware resources (pedagogical transparency detailed in \Cref{sec:systems}). GPU access remains inequitably distributed: cloud credits favor well-funded institutions, personal GPUs favor affluent students. The 100--10,000$\times$ slowdown versus PyTorch is acceptable when the pedagogical goal is understanding internals, not training production models.
 
-\textbf{Incremental complexity management}: GPU programming introduces memory hierarchy (registers, shared memory, global memory), kernel launch semantics, race conditions, and hardware-specific tuning. Teaching GPU programming simultaneously with autograd would violate cognitive load constraints. TinyTorch enables "framework internals now, hardware optimization later" learning pathway. Students completing TinyTorch should pursue GPU acceleration through PyTorch tutorials, NVIDIA Deep Learning Institute courses, or advanced ML systems courses, building on internals understanding to comprehend optimization techniques.
+\textbf{Incremental complexity management.} The second principle manages cognitive load through deliberate staging. GPU programming introduces memory hierarchy (registers, shared memory, global memory), kernel launch semantics, race conditions, and hardware-specific tuning. Teaching GPU programming simultaneously with autograd would violate cognitive load constraints. TinyTorch enables a ``framework internals now, hardware optimization later'' learning pathway. Students completing TinyTorch should pursue GPU acceleration through PyTorch tutorials, NVIDIA Deep Learning Institute courses, or advanced ML systems courses, building on internals understanding to comprehend optimization techniques.
 
-Similarly, distributed training (data parallelism, model parallelism, gradient synchronization) and production deployment (model serving, compilation, MLOps) introduce substantial additional complexity orthogonal to framework understanding. These topics remain important but beyond current pedagogical scope. Future extensions could address distributed systems through simulation-based pedagogy (\Cref{sec:future-work}), maintaining accessibility while teaching concepts.
+The same principle applies to distributed and production systems. Distributed training (data parallelism, model parallelism, gradient synchronization) and production deployment (model serving, compilation, MLOps) introduce substantial additional complexity orthogonal to framework understanding. These topics remain important but beyond current pedagogical scope. Future extensions could address distributed systems through simulation-based pedagogy (\Cref{sec:future-work}), maintaining accessibility while teaching concepts.
 
 \subsection{Limitations}
 
-TinyTorch's current implementation contains gaps requiring future work.
+TinyTorch's current implementation contains gaps requiring future work. We organize these into four categories.
 
-\textbf{Experimentation constraints}: The performance trade-off discussed above (\Cref{subsec:scope}) limits practical experimentation. Students complete milestones (65--75\% CIFAR-10 accuracy, transformer text generation) but cannot iterate rapidly on architecture search or hyperparameter tuning.
+\textbf{Experimentation constraints.} The performance trade-off discussed above (\Cref{subsec:scope}) limits practical experimentation. Students complete milestones (65--75\% CIFAR-10 accuracy, transformer text generation) but cannot iterate rapidly on architecture search or hyperparameter tuning. This trade-off is intentional (understanding over speed), but instructors should set appropriate expectations about what TinyTorch enables versus production frameworks.
 
-\textbf{Energy consumption measurement}: While TinyTorch covers optimization techniques with significant energy implications (quantization achieving 4$\times$ compression, pruning enabling 10$\times$ model shrinkage), the curriculum does not explicitly measure or quantify energy consumption. Students understand that quantization reduces model size and pruning decreases computation, but may not connect these optimizations to concrete energy savings (joules/inference, watt-hours/training epoch). Future iterations could integrate energy profiling libraries to make sustainability an explicit learning objective alongside memory and latency optimization, particularly relevant for edge deployment.
+\textbf{Energy consumption measurement.} Related to experimentation constraints, TinyTorch covers optimization techniques with significant energy implications (quantization achieving 4$\times$ compression, pruning enabling 10$\times$ model shrinkage) but does not explicitly measure or quantify energy consumption. Students understand that quantization reduces model size and pruning decreases computation, but may not connect these optimizations to concrete energy savings (joules/inference, watt-hours/training epoch). Future iterations could integrate energy profiling libraries to make sustainability an explicit learning objective alongside memory and latency optimization, particularly relevant for edge deployment.
 
-\textbf{Language and accessibility}: Materials exist exclusively in English. Modular structure facilitates translation; community contributions welcome. Code examples omit type annotations (e.g., \texttt{def forward(self, x: Tensor) -> Tensor:}) to reduce visual complexity for students learning ML concepts simultaneously. While this prioritizes pedagogical clarity, it means students don't practice type-driven development increasingly standard in production ML codebases. Future iterations could introduce type hints progressively: omitting them in early Modules (01--05), then adding them in optimization Modules (14--18) where interface contracts become critical.
+\textbf{Language and accessibility.} Beyond technical limitations, materials exist exclusively in English. The modular structure facilitates translation, and community contributions are welcome. Additionally, code examples omit type annotations (e.g., \texttt{def forward(self, x: Tensor) -> Tensor:}) to reduce visual complexity for students learning ML concepts simultaneously. While this prioritizes pedagogical clarity, it means students don't practice type-driven development increasingly standard in production ML codebases. Future iterations could introduce type hints progressively: omitting them in early Modules (01--05), then adding them in optimization Modules (14--18) where interface contracts become critical.
 
-\textbf{Forward dependency prevention}: A recurring curriculum maintenance challenge is forward dependency creep: advanced concepts leaking into foundational modules through helper functions, error messages, or test cases that assume knowledge students haven't yet acquired. For example, an error message in Module 03 (Layers) that mentions ``computational graph'' assumes Module 06 (Autograd) knowledge. Maintaining pedagogical ordering requires vigilance during curriculum updates; future work could automate this validation through CI/CD checks that flag cross-module dependencies violating prerequisite ordering.
+\textbf{Forward dependency prevention.} Finally, a recurring curriculum maintenance challenge is forward dependency creep: advanced concepts leaking into foundational modules through helper functions, error messages, or test cases that assume knowledge students haven't yet acquired. For example, an error message in Module 03 (Layers) that mentions ``computational graph'' assumes Module 06 (Autograd) knowledge. Maintaining pedagogical ordering requires vigilance during curriculum updates; future work could automate this validation through CI/CD checks that flag cross-module dependencies violating prerequisite ordering.
 
 \section{Future Directions}
 \label{sec:future-work}
@@ -1056,41 +1056,40 @@ While TinyTorch's design is grounded in established learning theory (cognitive l
 
 \subsection{Curriculum Evolution}
 
-TinyTorch deliberately focuses on single node ML systems fundamentals: understanding what happens inside \texttt{loss.backward()} on one machine, how memory flows through computational graphs, and why optimizer state consumes resources. This single node focus provides the foundation for understanding distributed systems, where the same operations must coordinate across multiple nodes with communication overhead, synchronization barriers, and failure modes that compound single node complexity. Future curriculum extensions would maintain TinyTorch's core principle (understanding through transparent implementation) while expanding to multi node systems coverage through simulation based pedagogy.
+TinyTorch deliberately focuses on single node ML systems fundamentals: understanding what happens inside \texttt{loss.backward()} on one machine, how memory flows through computational graphs, and why optimizer state consumes resources. This single node focus provides the foundation for understanding distributed systems, where the same operations must coordinate across multiple nodes with communication overhead, synchronization barriers, and failure modes that compound single node complexity. Future curriculum extensions would maintain TinyTorch's core principle (understanding through transparent implementation) while expanding to multi node systems coverage through simulation based pedagogy. We outline five specific directions below.
 
-\textbf{Performance Analysis Through Analytical Models.} Future extensions could enable students to compare TinyTorch CPU implementations against PyTorch GPU equivalents through roofline modeling~\citep{williams2009roofline}. Rather than writing CUDA code, students would profile existing implementations to understand memory hierarchy differences, parallelism benefits, and compute versus memory bottlenecks. The roofline approach maintains TinyTorch's accessibility (no GPU hardware required) while preparing students for GPU programming by teaching first-principles performance analysis.
+\textbf{Performance Analysis Through Analytical Models.} A first extension would enable students to compare TinyTorch CPU implementations against PyTorch GPU equivalents through roofline modeling~\citep{williams2009roofline}. Rather than writing CUDA code, students would profile existing implementations to understand memory hierarchy differences, parallelism benefits, and compute versus memory bottlenecks. The roofline approach maintains TinyTorch's accessibility (no GPU hardware required) while preparing students for GPU programming by teaching first-principles performance analysis.
 
-\textbf{Hardware Simulation Integration.}
-TinyTorch's current profiling infrastructure—memory tracking (tracemalloc), FLOP counting, and performance benchmarking—provides algorithmic-level performance analysis. A natural extension would integrate architecture simulators (e.g., scale-sim~\citep{samajdar2018scale}, timeloop~\citep{parashar2019timeloop}, astra-sim~\citep{kannan2022astrasim}) to connect high-level ML operations with cycle-accurate hardware models. This layered approach mirrors real ML systems engineering: students first understand algorithmic complexity and memory patterns in TinyTorch, then trace those operations down to microarchitectural performance in simulators. Such integration would complete the educational arc from algorithmic implementation $\rightarrow$ systems profiling $\rightarrow$ hardware realization, maintaining TinyTorch's accessibility while preparing students for hardware-aware optimization through measurement-driven analysis.
+\textbf{Hardware Simulation Integration.} Building on performance analysis, a second extension would connect algorithmic understanding to hardware realities. TinyTorch's current profiling infrastructure (memory tracking via tracemalloc, FLOP counting, and performance benchmarking) provides algorithmic-level analysis. Integrating architecture simulators (e.g., scale-sim~\citep{samajdar2018scale}, timeloop~\citep{parashar2019timeloop}, astra-sim~\citep{kannan2022astrasim}) would connect high-level ML operations with cycle-accurate hardware models. This layered approach mirrors real ML systems engineering: students first understand algorithmic complexity and memory patterns in TinyTorch, then trace those operations down to microarchitectural performance in simulators. Such integration would complete the educational arc from algorithmic implementation $\rightarrow$ systems profiling $\rightarrow$ hardware realization, maintaining TinyTorch's accessibility while preparing students for hardware-aware optimization.
 
-\textbf{Energy and Power Profiling.} Edge deployment and sustainable ML~\citep{strubell2019energy,patterson2021carbon} require understanding energy consumption. Future extensions could integrate power profiling tools enabling students to measure energy costs (joules per inference, watt-hours per training epoch) alongside latency and memory. This connects existing optimization techniques (quantization, pruning) taught in Modules 15--18 to concrete sustainability metrics, particularly relevant for edge AI~\citep{banbury2021benchmarking} where battery life constrains deployment.
+\textbf{Energy and Power Profiling.} A third extension addresses sustainability. Edge deployment and sustainable ML~\citep{strubell2019energy,patterson2021carbon} require understanding energy consumption. Integrating power profiling tools would enable students to measure energy costs (joules per inference, watt-hours per training epoch) alongside latency and memory. This connects existing optimization techniques (quantization, pruning) taught in Modules 15--18 to concrete sustainability metrics, particularly relevant for edge AI~\citep{banbury2021benchmarking} where battery life constrains deployment.
 
-\textbf{Distributed Training Through Simulation.} Understanding distributed training communication patterns requires simulation-based pedagogy rather than multi-GPU clusters. Future extensions could leverage simulators introduced above to explore multi-node concepts: gradient synchronization overhead, scalability analysis across worker counts, network topology impact on communication patterns, and pipeline parallelism trade-offs. Students who master single-node fundamentals in TinyTorch would then explore how those same operations (forward pass, backward pass, optimizer step) change when distributed across nodes with communication latency and bandwidth constraints.
+\textbf{Distributed Training Through Simulation.} Fourth, understanding distributed training communication patterns requires simulation-based pedagogy rather than multi-GPU clusters. Leveraging the simulators introduced above, students could explore multi-node concepts: gradient synchronization overhead, scalability analysis across worker counts, network topology impact on communication patterns, and pipeline parallelism trade-offs. Students who master single-node fundamentals in TinyTorch would then explore how those same operations (forward pass, backward pass, optimizer step) change when distributed across nodes with communication latency and bandwidth constraints.
 
-\textbf{Architecture Extensions.} Potential additions (graph neural networks, diffusion models, reinforcement learning) must justify inclusion through systems pedagogy rather than completeness. The question is not ``Can TinyTorch implement this?'' but rather ``Does implementing this teach fundamental systems concepts unavailable through existing modules?'' Graph convolutions might teach sparse tensor operations; diffusion models might illuminate iterative refinement trade-offs. The curriculum remains intentionally incomplete as a production framework: completeness lies in foundational systems thinking applicable across all ML architectures.
+\textbf{Architecture Extensions.} Finally, potential architecture additions (graph neural networks, diffusion models, reinforcement learning) must justify inclusion through systems pedagogy rather than completeness. The question is not ``Can TinyTorch implement this?'' but rather ``Does implementing this teach fundamental systems concepts unavailable through existing modules?'' Graph convolutions might teach sparse tensor operations; diffusion models might illuminate iterative refinement trade-offs. The curriculum remains intentionally incomplete as a production framework: completeness lies in foundational systems thinking applicable across all ML architectures.
 
 \subsection{Community and Sustainability}
 
-As part of the ML Systems Book ecosystem (\texttt{mlsysbook.ai}), TinyTorch benefits from broader educational infrastructure while the open-source model (MIT license) enables collaborative refinement across institutions: instructor discussion forums for pedagogical exchange, shared teaching resources, and empirical validation of learning outcomes.
+Beyond curriculum evolution, TinyTorch's long-term impact depends on community adoption and collaborative refinement. As part of the ML Systems Book ecosystem (\texttt{mlsysbook.ai}), TinyTorch benefits from broader educational infrastructure while the open-source model (MIT license) enables collaborative refinement across institutions: instructor discussion forums for pedagogical exchange, shared teaching resources, and empirical validation of learning outcomes.
 
-As described earlier, Module 20 (Capstone) culminates the curriculum with competitive systems engineering challenges. Inspired by MLPerf benchmarking~\citep{mattson2020mlperf,reddi2020mlperf}, students optimize their implementations across accuracy, speed, compression, and efficiency dimensions, comparing results globally through standardized benchmarking infrastructure. This competitive element reinforces systems thinking: optimization requires measurement-driven decisions (profiling bottlenecks), principled tradeoffs (accuracy versus compression), and reproducible methodology (standardized metrics collection). The focus remains pedagogical—understanding \emph{why} optimizations work—rather than achieving state-of-the-art performance, but the competitive framing increases engagement and mirrors real ML engineering workflows.
+The curriculum's capstone reinforces this community focus. Module 20 culminates with competitive systems engineering challenges inspired by MLPerf benchmarking~\citep{mattson2020mlperf,reddi2020mlperf}, where students optimize their implementations across accuracy, speed, compression, and efficiency dimensions, comparing results globally through standardized benchmarking infrastructure. This competitive element reinforces systems thinking: optimization requires measurement-driven decisions (profiling bottlenecks), principled tradeoffs (accuracy versus compression), and reproducible methodology (standardized metrics collection). The focus remains pedagogical (understanding \emph{why} optimizations work) rather than achieving state-of-the-art performance, but the competitive framing increases engagement and mirrors real ML engineering workflows.
 
 \section{Conclusion}
 \label{sec:conclusion}
 
 Machine learning education faces a fundamental choice: teach students to \emph{use} frameworks as black boxes, or teach them to \emph{understand} what happens inside \texttt{loss.backward()}. TinyTorch demonstrates that deep systems understanding is accessible without GPU clusters or distributed infrastructure. \emph{Building systems creates irreversible understanding}: once you implement autograd, you cannot unsee the computational graph; once you profile memory allocation, you cannot unknow the costs. This accessibility matters: students worldwide can develop framework internals knowledge on modest hardware, transforming production debugging from trial-and-error into systematic engineering.
 
-Three pedagogical contributions enable this transformation. Progressive disclosure manages complexity through gradual feature activation: students work with unified Tensor implementations that gain capabilities across modules rather than replacing code mid-semester. Systems-first integration embeds memory profiling from Module 01, preventing ``algorithms without costs'' learning where students optimize accuracy while ignoring deployment constraints. Historical milestone validation proves correctness through recreating nearly 70 years of ML breakthroughs (1958--2025, from Perceptron through Transformers), making abstract implementations concrete through reproducing published results.
+Three pedagogical contributions enable this transformation. First, progressive disclosure manages complexity through gradual feature activation: students work with unified Tensor implementations that gain capabilities across modules rather than replacing code mid-semester. Second, systems-first integration embeds memory profiling from Module 01, preventing ``algorithms without costs'' learning where students optimize accuracy while ignoring deployment constraints. Third, historical milestone validation proves correctness through recreating nearly 70 years of ML breakthroughs (1958--2025, from Perceptron through Transformers), making abstract implementations concrete through reproducing published results.
 
-\textbf{For ML practitioners}: Building TinyTorch's 20 modules transforms how you debug production failures. When PyTorch training crashes with OOM errors, you understand memory allocation across parameters, optimizer states, and activation tensors. When gradient explosions occur, you recognize backpropagation numerical instability from implementing it yourself. When choosing between Adam and SGD under memory constraints, you know the 4$\times$ total memory multiplier from building both optimizers. This systems knowledge transfers directly to production framework usage: you become an engineer who understands \emph{why} frameworks behave as they do, not just \emph{what} they do.
+These contributions have distinct implications for different audiences. \textbf{For ML practitioners}, building TinyTorch's 20 modules transforms how you debug production failures. When PyTorch training crashes with OOM errors, you understand memory allocation across parameters, optimizer states, and activation tensors. When gradient explosions occur, you recognize backpropagation numerical instability from implementing it yourself. When choosing between Adam and SGD under memory constraints, you know the 4$\times$ total memory multiplier from building both optimizers. This systems knowledge transfers directly to production framework usage: you become an engineer who understands \emph{why} frameworks behave as they do, not just \emph{what} they do.
 
-\textbf{For CS education researchers}: TinyTorch provides replicable infrastructure for testing pedagogical hypotheses about ML systems education. Does progressive disclosure reduce cognitive load compared to teaching autograd as separate framework? Does systems-first integration improve production readiness versus algorithms-only instruction? Do historical milestones increase engagement and retention? The curriculum embodies design patterns amenable to controlled empirical investigation. Open-source release with detailed validation roadmap enables multi-institutional studies to establish evidence-based best practices for teaching framework internals.
+\textbf{For CS education researchers}, TinyTorch provides replicable infrastructure for testing pedagogical hypotheses about ML systems education. Does progressive disclosure reduce cognitive load compared to teaching autograd as separate framework? Does systems-first integration improve production readiness versus algorithms-only instruction? Do historical milestones increase engagement and retention? The curriculum embodies design patterns amenable to controlled empirical investigation. Open-source release with detailed validation roadmap enables multi-institutional studies to establish evidence-based best practices for teaching framework internals.
 
-\textbf{For educators and bootcamp instructors}: TinyTorch supports flexible integration: self-paced learning requiring zero infrastructure (students run locally on laptops), institutional courses with automated NBGrader assessment, or industry team onboarding for ML engineers transitioning from application development to systems work. The modular structure enables selective adoption: foundation tier only (Modules 01--08, teaching core concepts), architecture focus (adding CNNs and Transformers through Module 13), or complete systems coverage (all 20 modules including optimization and deployment). No GPU access required, no cloud credits needed, no infrastructure barriers.
+\textbf{For educators and bootcamp instructors}, TinyTorch supports flexible integration: self-paced learning requiring zero infrastructure (students run locally on laptops), institutional courses with automated NBGrader assessment, or industry team onboarding for ML engineers transitioning from application development to systems work. The modular structure enables selective adoption: foundation tier only (Modules 01--08, teaching core concepts), architecture focus (adding CNNs and Transformers through Module 13), or complete systems coverage (all 20 modules including optimization and deployment). No GPU access required, no cloud credits needed, no infrastructure barriers.
 
 The complete codebase, curriculum materials, and assessment infrastructure are openly available at \texttt{mlsysbook.ai/tinytorch} (or \texttt{tinytorch.ai}) under permissive open-source licensing. We invite the global ML education community to adopt TinyTorch in courses, contribute curriculum improvements, translate materials for international accessibility, fork for domain-specific variants (robotics, edge AI), and empirically evaluate whether implementation-based pedagogy achieves its promise.
 
-Sutton's Bitter Lesson teaches that general methods leveraging computation ultimately triumph, but someone must build the systems that enable that computation to scale. TinyTorch prepares students to be those engineers, the AI engineers who bridge the gap between what ML research makes possible and what reliable production systems require. They are practitioners who understand not just \emph{what} ML systems do, but \emph{why} they work and \emph{how} to make them scale.
+Returning to where we began: Sutton's Bitter Lesson teaches that general methods leveraging computation ultimately triumph, but someone must build the systems that enable that computation to scale. TinyTorch prepares students to be those engineers: the AI engineers who bridge the gap between what ML research makes possible and what reliable production systems require. They are practitioners who understand not just \emph{what} ML systems do, but \emph{why} they work and \emph{how} to make them scale.
 
 \section*{Acknowledgments}