mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-11-09 10:39:41 -06:00
Vijay Janapa Reddi
972f5ea84b
dev's Classic Expository Style pass over the websites is the base; re-applied from this branch only where a fact was missing or wrong: real volume titles and status, no stale counts or timing claims, the four-volume teaching guide and slide pages. Lecture lists regenerated by tools/lectures.py; the PPTX note appears on all four volume pages.
89 lines
3.8 KiB
Plaintext
89 lines
3.8 KiB
Plaintext
---
|
|
title: "Learning Resources"
|
|
---
|
|
|
|
**TinyTorch teaches you to *build* ML systems. These resources help you understand the *why* behind what you are building.**
|
|
|
|
|
|
## Companion Textbooks
|
|
|
|
### TinyTorch: From Tensors to Transformers
|
|
The companion textbook (available at [mlsysbook.ai/tinytorch/](https://mlsysbook.ai/tinytorch/))
|
|
|
|
<div class="resource-note resource-note-orange">
|
|
<p>
|
|
<strong>The reference textbook.</strong> This Lab Guide provides hands-on modules and exercise notebooks; the companion textbook provides reference implementations, formal systems invariants, memory traces, napkin-math budgets, and production bridges.
|
|
</p>
|
|
</div>
|
|
|
|
**What it teaches**: Full framework architecture from first principles: CPython tensor internals, DAG reverse-mode tape autograd, AdamW moment buffers, BPE tokenization pipelines, multi-head attention math, KV caching state machines, INT8 quantization grids, and compilation bridges to PyTorch Inductor and OpenAI Triton.
|
|
|
|
**When to use it**: Consult whenever you want to inspect a complete reference solution, trace an exact execution path through memory, or understand how a subsystem connects to production GPU infrastructure.
|
|
|
|
### Machine Learning Systems
|
|
**[mlsysbook.ai](https://mlsysbook.ai)** by Prof. Vijay Janapa Reddi (Harvard University)
|
|
|
|
<div class="resource-note resource-note-blue">
|
|
<p>
|
|
<strong>The parent ML Systems curriculum.</strong> The ML Systems book provides the broad theoretical depth and production context behind modern machine learning infrastructure.
|
|
</p>
|
|
</div>
|
|
|
|
**What it teaches**: Systems engineering for production ML, including memory hierarchies, hardware accelerators, distributed training algorithms, and deployment strategies.
|
|
|
|
**When to use it**: Read in parallel with TinyTorch to understand the high-level landscape of production ML infrastructure.
|
|
|
|
|
|
|
|
## Related Academic Courses
|
|
|
|
- **[CS 329S: Machine Learning Systems Design](https://stanford-cs329s.github.io/)** (Stanford)
|
|
*Production ML systems and deployment*
|
|
|
|
- **[TinyML and Efficient Deep Learning](https://efficientml.ai)** (MIT 6.5940)
|
|
*Edge computing, model compression, and efficient ML*
|
|
|
|
- **[CS 249r: Tiny Machine Learning](https://sites.google.com/g.harvard.edu/tinyml/home)** (Harvard)
|
|
*TinyML systems and resource-constrained ML*
|
|
|
|
- **[CS 231n: Convolutional Neural Networks](http://cs231n.stanford.edu/)** (Stanford)
|
|
*Computer vision - complements TinyTorch Module 09*
|
|
|
|
- **[CS 224n: Natural Language Processing](http://web.stanford.edu/class/cs224n/)** (Stanford)
|
|
*Transformers and NLP - complements TinyTorch Modules 10-13*
|
|
|
|
|
|
## Other Textbooks
|
|
|
|
- **[Deep Learning](https://www.deeplearningbook.org/)** by Goodfellow, Bengio, Courville
|
|
*Mathematical foundations behind what you implement in TinyTorch*
|
|
|
|
- **[Hands-On Machine Learning](https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/)** by Aurélien Géron
|
|
*Practical implementations using established frameworks*
|
|
|
|
|
|
## Minimal Frameworks
|
|
|
|
**Alternative approaches to building ML from scratch:**
|
|
|
|
- **[micrograd](https://github.com/karpathy/micrograd)** by Andrej Karpathy
|
|
*Autograd in 100 lines. A short introduction to read before TinyTorch.*
|
|
|
|
- **[nanoGPT](https://github.com/karpathy/nanoGPT)** by Andrej Karpathy
|
|
*Minimalist GPT implementation. Complements TinyTorch Modules 12-13.*
|
|
|
|
- **[tinygrad](https://github.com/tinygrad/tinygrad)** by George Hotz
|
|
*Performance-focused educational framework with GPU acceleration.*
|
|
|
|
|
|
## Production Framework Internals
|
|
|
|
- **[PyTorch Internals](http://blog.ezyang.com/2019/05/pytorch-internals/)** by Edward Yang
|
|
*How PyTorch works internally*
|
|
|
|
- **[PyTorch: Extending PyTorch](https://pytorch.org/docs/stable/notes/extending.html)**
|
|
*Custom operators and autograd functions*
|
|
|
|
|
|
**Ready to start?** See the **[Quick Start](getting-started.qmd)** to set up in about five minutes and begin Module 01.
|