github-starred/TinyTorch
2
0
Fork 0
mirror of https://github.com/MLSysBook/TinyTorch.git synced 2026-05-07 15:44:01 -05:00
Code Issues 7 Packages Projects Releases Wiki Activity
Files
2f23f757e71dfe052d33418baf66dfb2cb6e3210
TinyTorch/book/_build/html/intro.html
Vijay Janapa Reddi 92781736a1 Restructure TinyTorch: Move TinyGPT to examples, improve testing framework 
Major changes:
- Moved TinyGPT from Module 16 to examples/tinygpt (capstone demo)
- Fixed Module 10 (optimizers) and Module 11 (training) bugs
- All 16 modules now passing tests (100% health)
- Added comprehensive testing with 'tito test --comprehensive'
- Renamed example files for clarity (train_xor_network.py, etc.)
- Created working TinyGPT example structure
- Updated documentation to reflect 15 core modules + examples
- Added KISS principle and testing framework documentation
2025-09-22 09:37:18 -04:00

893 lines
46 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en" data-content_root="./" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta content="TinyTorch: Build your own ML framework from scratch" property="og:title" />
<meta content="Learn ML systems by building them. From computer vision to language models. Comprehensive educational framework for understanding ML systems engineering." property="og:description" />
<meta content="https://mlsysbook.github.io/TinyTorch/" property="og:url" />
<meta content="website" property="og:type" />
<meta content="https://mlsysbook.github.io/TinyTorch/logo.png" property="og:image" />
<meta content="TinyTorch Course" property="og:site_name" />
<meta content="summary_large_image" name="twitter:card" />
<meta content="TinyTorch: Build your own ML framework" name="twitter:title" />
<meta content="TinyTorch is a minimalist framework for building machine learning systems from scratch—from vision to language." name="twitter:description" />
<meta content="https://mlsysbook.github.io/TinyTorch/logo.png" name="twitter:image" />
<title>TinyTorch: Build Your Own ML Framework from First Principles &#8212; TinyTorch</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!-- Loaded before other Sphinx assets -->
<link href="_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
<link rel="stylesheet" type="text/css" href="_static/custom.css?v=5a689c52" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
<script src="_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/documentation_options.js?v=9eb32ce0"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/clipboard.min.js?v=a7894cd8"></script>
<script src="_static/copybutton.js?v=f281be69"></script>
<script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js?v=4a39c7ea"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script src="_static/design-tabs.js?v=f930bc37"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script async="async" src="_static/sphinx-thebe.js?v=c100c467"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script type="module" src="https://cdn.jsdelivr.net/npm/mermaid@10.6.1/dist/mermaid.esm.min.mjs"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/@mermaid-js/layout-elk@0.1.4/dist/mermaid-layout-elk.esm.min.mjs"></script>
<script type="module">import mermaid from "https://cdn.jsdelivr.net/npm/mermaid@10.6.1/dist/mermaid.esm.min.mjs";import elkLayouts from "https://cdn.jsdelivr.net/npm/@mermaid-js/layout-elk@0.1.4/dist/mermaid-layout-elk.esm.min.mjs";mermaid.registerLayoutLoaders(elkLayouts);mermaid.initialize({startOnLoad:false});</script>
<script src="https://cdn.jsdelivr.net/npm/d3@7.9.0/dist/d3.min.js"></script>
<script type="module">
import mermaid from "https://cdn.jsdelivr.net/npm/mermaid@10.6.1/dist/mermaid.esm.min.mjs";
window.addEventListener("load", () => mermaid.run());
</script>
<script>DOCUMENTATION_OPTIONS.pagename = 'intro';</script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Quick Exploration Path" href="usage-paths/quick-exploration.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<input type="checkbox"
class="sidebar-toggle"
id="pst-primary-sidebar-checkbox"/>
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
<input type="checkbox"
class="sidebar-toggle"
id="pst-secondary-sidebar-checkbox"/>
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
<div class="search-button__wrapper">
<div class="search-button__overlay"></div>
<div class="search-button__search-container">
<form class="bd-search d-flex align-items-center"
action="search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
id="search-input"
placeholder="Search this book..."
aria-label="Search this book..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form></div>
</div>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<div class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<a class="navbar-brand logo" href="#">
<img src="_static/logo.png" class="logo__image only-light" alt="TinyTorch - Home"/>
<script>document.write(`<img src="_static/logo.png" class="logo__image only-dark" alt="TinyTorch - Home"/>`);</script>
</a></div>
<div class="sidebar-primary-item">
<script>
document.write(`
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
`);
</script></div>
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
<div class="bd-toc-item navbar-nav active">
<ul class="nav bd-sidenav bd-sidenav__home-link">
<li class="toctree-l1 current active">
<a class="reference internal" href="#">
TinyTorch: Build Your Own ML Framework from First Principles
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Usage Paths</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="usage-paths/quick-exploration.html">Quick Exploration</a></li>
<li class="toctree-l1"><a class="reference internal" href="usage-paths/serious-development.html">Serious Development</a></li>
<li class="toctree-l1"><a class="reference internal" href="usage-paths/classroom-use.html">Classroom Use</a></li>
<li class="toctree-l1"><a class="reference internal" href="instructor-guide.html">Instructor Guide</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Foundation</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapters/01-setup.html">0. Setup</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/02-tensor.html">1. Tensors</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/03-activations.html">2. Activations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Building Blocks</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapters/04-layers.html">3. Layers</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/05-dense.html">4. Dense</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/06-spatial.html">5. Spatial</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/07-attention.html">6. Attention</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Training Systems</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapters/08-dataloader.html">7. DataLoader</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/09-autograd.html">8. Autograd</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/10-optimizers.html">9. Optimizers</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/11-training.html">10. Training</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Inference &amp; Serving</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapters/12-compression.html">11. Compression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/13-kernels.html">12. Kernels</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/14-benchmarking.html">13. Benchmarking</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapters/15-mlops.html">14. MLOps</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Language Models</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapters/16-tinygpt.html">15. TinyGPT</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Additional Resources</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="vision.html">TinyTorch Vision</a></li>
<li class="toctree-l1"><a class="reference internal" href="kiss-principle.html">KISS Principle</a></li>
<li class="toctree-l1"><a class="reference internal" href="checkpoint-system.html">Checkpoint System</a></li>
<li class="toctree-l1"><a class="reference internal" href="testing-framework.html">Testing Framework</a></li>
<li class="toctree-l1"><a class="reference internal" href="resources.html">Learning Resources</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
</div>
<div id="rtd-footer-container"></div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="sbt-scroll-pixel-helper"></div>
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-bars"></span>
</button></div>
</div>
<div class="header-article-items__end">
<div class="header-article-item">
<div class="article-header-buttons">
<div class="dropdown dropdown-source-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Source repositories">
<i class="fab fa-github"></i>
</button>
<ul class="dropdown-menu">
<li><a href="https://github.com/mlsysbook/TinyTorch" target="_blank"
class="btn btn-sm btn-source-repository-button dropdown-item"
title="Source repository"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fab fa-github"></i>
</span>
<span class="btn__text-container">Repository</span>
</a>
</li>
<li><a href="https://github.com/mlsysbook/TinyTorch/edit/main/book/intro.md" target="_blank"
class="btn btn-sm btn-source-edit-button dropdown-item"
title="Suggest edit"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-pencil-alt"></i>
</span>
<span class="btn__text-container">Suggest edit</span>
</a>
</li>
<li><a href="https://github.com/mlsysbook/TinyTorch/issues/new?title=Issue%20on%20page%20%2Fintro.html&body=Your%20issue%20content%20here." target="_blank"
class="btn btn-sm btn-source-issues-button dropdown-item"
title="Open an issue"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-lightbulb"></i>
</span>
<span class="btn__text-container">Open issue</span>
</a>
</li>
</ul>
</div>
<div class="dropdown dropdown-download-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
<i class="fas fa-download"></i>
</button>
<ul class="dropdown-menu">
<li><a href="_sources/intro.md" target="_blank"
class="btn btn-sm btn-download-source-button dropdown-item"
title="Download source file"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file"></i>
</span>
<span class="btn__text-container">.md</span>
</a>
</li>
<li>
<button onclick="window.print()"
class="btn btn-sm btn-download-pdf-button dropdown-item"
title="Print to PDF"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file-pdf"></i>
</span>
<span class="btn__text-container">.pdf</span>
</button>
</li>
</ul>
</div>
<button onclick="toggleFullScreen()"
class="btn btn-sm btn-fullscreen-button"
title="Fullscreen mode"
data-bs-placement="bottom" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-expand"></i>
</span>
</button>
<script>
document.write(`
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
</button>
`);
</script>
<script>
document.write(`
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
</button>
`);
</script>
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-list"></span>
</button>
</div></div>
</div>
</div>
</div>
<div id="jb-print-docs-body" class="onlyprint">
<h1>TinyTorch: Build Your Own ML Framework from First Principles</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-core-difference">The Core Difference</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-philosophy-build-use-reflect">Learning Philosophy: Build, Use, Reflect</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#example-activation-functions">Example: Activation Functions</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#who-this-is-for">👥 Who This Is For</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#perfect-for">🎯 Perfect For:</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#prerequisites">📚 Prerequisites:</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#career-impact">🚀 Career Impact:</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#course-journey-16-modules-foundation-to-framework">📚 Course Journey: 16 Modules - Foundation to Framework</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#complete-system-integration">🔗 Complete System Integration</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ready-to-start">Ready to Start?</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#quick-taste-try-module-1-right-now">Quick Taste: Try Module 1 Right Now</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#acknowledgments">Acknowledgments</a></li>
</ul>
</nav>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<section id="tinytorch-build-your-own-ml-framework-from-first-principles">
<h1>TinyTorch: Build Your Own ML Framework from First Principles<a class="headerlink" href="#tinytorch-build-your-own-ml-framework-from-first-principles" title="Link to this heading">#</a></h1>
<p><strong>Most ML education teaches you to <em>use</em> frameworks. TinyTorch teaches you to <em>build</em> them.</strong></p>
<p>Tiny🔥Torch is a minimalist framework for building machine learning systems from scratch—from tensors to systems. Instead of relying on PyTorch or TensorFlow, you implement everything yourself—tensors, autograd, optimizers, even MLOps tooling.</p>
<p><strong>The Vision: Train ML Systems Engineers, Not Just ML Users</strong></p>
<p>This hands-on approach builds the deep systems intuition that separates ML engineers from ML users. You’ll understand not just <em>what</em> neural networks do, but <em>how</em> they work under the hood, <em>why</em> certain design choices matter in production, and <em>when</em> to make trade-offs between memory, speed, and accuracy.</p>
<div class="tip admonition">
<p class="admonition-title">What You’ll Build</p>
<p><strong>A complete ML framework from scratch</strong>: your own production-ready toolkit that can:</p>
<ul class="simple">
<li><p><strong>Train neural networks to 55%+ accuracy on CIFAR-10</strong> (solid, reliable performance!)</p></li>
<li><p>Implement automatic differentiation from first principles</p></li>
<li><p>Build complete training loops with real datasets</p></li>
<li><p>Handle full ML pipeline from data loading to evaluation</p></li>
<li><p><strong>Master XOR problem with 100% accuracy</strong> using your own autograd</p></li>
</ul>
<p><strong>Result:</strong> You become the expert others ask about “how ML frameworks actually work” and “why autograd enables all modern deep learning.” All 16 modules pass comprehensive tests with 100% health status.</p>
</div>
<p><em>Understanding how to build ML systems makes you a more effective ML engineer.</em></p>
<div class="note admonition">
<p class="admonition-title">The Perfect Learning Combination</p>
<p>TinyTorch was designed as the hands-on lab companion to <a class="reference external" href="https://mlsysbook.ai"><strong>Machine Learning Systems</strong></a> by <a class="reference external" href="https://vijay.seas.harvard.edu">Prof. Vijay Janapa Reddi</a> (Harvard). The book teaches you ML systems <strong>theory and principles</strong> - TinyTorch lets you <strong>implement and experience</strong> those concepts firsthand. Together, they provide complete ML systems mastery.</p>
</div>
<hr class="docutils" />
<section id="the-core-difference">
<h2>The Core Difference<a class="headerlink" href="#the-core-difference" title="Link to this heading">#</a></h2>
<p>Most ML courses focus on algorithms and theory. You learn <em>what</em> neural networks do and <em>why</em> they work, but you import everything:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">Traditional</span> <span class="n">ML</span> <span class="n">Course</span><span class="p">:</span> <span class="n">TinyTorch</span> <span class="n">Approach</span><span class="p">:</span>
<span class="err">├──</span> <span class="kn">import</span><span class="w"> </span><span class="nn">torch</span> <span class="err">├──</span> <span class="k">class</span><span class="w"> </span><span class="nc">Tensor</span><span class="p">:</span>
<span class="err">├──</span> <span class="n">model</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="err">│</span> <span class="k">def</span><span class="w"> </span><span class="fm">__add__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="o">...</span>
<span class="err">├──</span> <span class="n">loss</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">MSELoss</span><span class="p">()</span> <span class="err">│</span> <span class="k">def</span><span class="w"> </span><span class="nf">backward</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span>
<span class="err">└──</span> <span class="n">optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span> <span class="err">├──</span> <span class="k">class</span><span class="w"> </span><span class="nc">Linear</span><span class="p">:</span>
<span class="err">│</span> <span class="k">def</span><span class="w"> </span><span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="err">│</span> <span class="k">return</span> <span class="n">x</span> <span class="o">@</span> <span class="bp">self</span><span class="o">.</span><span class="n">weight</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bias</span>
<span class="err">├──</span> <span class="k">def</span><span class="w"> </span><span class="nf">mse_loss</span><span class="p">(</span><span class="n">pred</span><span class="p">,</span> <span class="n">target</span><span class="p">):</span>
<span class="err">│</span> <span class="k">return</span> <span class="p">((</span><span class="n">pred</span> <span class="o">-</span> <span class="n">target</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="err">├──</span> <span class="k">class</span><span class="w"> </span><span class="nc">SGD</span><span class="p">:</span>
<span class="err">│</span> <span class="k">def</span><span class="w"> </span><span class="nf">step</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="err">└──</span> <span class="n">param</span><span class="o">.</span><span class="n">data</span> <span class="o">-=</span> <span class="n">lr</span> <span class="o">*</span> <span class="n">param</span><span class="o">.</span><span class="n">grad</span>
<span class="n">Transform</span> <span class="kn">from</span><span class="w"> </span><span class="s2">&quot;How does this work?&quot;</span> <span class="n">to</span> <span class="s2">&quot;I implemented every line!&quot;</span>
</pre></div>
</div>
<p>TinyTorch focuses on implementation and systems thinking. You learn <em>how</em> to build working systems with progressive scaffolding, production ready practices, and comprehensive course infrastructure that bridges the gap between learning and building.</p>
<p><strong>What Makes This Different: Systems-First Thinking</strong></p>
<p>Traditional ML courses teach algorithms. TinyTorch teaches <strong>ML systems engineering</strong>:</p>
<ul class="simple">
<li><p><strong>Memory Management</strong>: Why Adam uses 3× more memory than SGD and when that matters</p></li>
<li><p><strong>Performance Analysis</strong>: How attention mechanisms scale O(N²) and limit context length</p></li>
<li><p><strong>Production Trade-offs</strong>: When to use gradient accumulation vs larger GPUs</p></li>
<li><p><strong>Hardware Awareness</strong>: How cache misses make naive convolution 100× slower</p></li>
<li><p><strong>System Design</strong>: How autograd graphs consume memory and enable gradient checkpointing</p></li>
</ul>
<p><strong>Result</strong>: You become the engineer who designs ML systems, not just uses them.</p>
</section>
<hr class="docutils" />
<section id="learning-philosophy-build-use-reflect">
<h2>Learning Philosophy: Build, Use, Reflect<a class="headerlink" href="#learning-philosophy-build-use-reflect" title="Link to this heading">#</a></h2>
<p>Every component follows the same powerful learning cycle:</p>
<section id="example-activation-functions">
<h3>Example: Activation Functions<a class="headerlink" href="#example-activation-functions" title="Link to this heading">#</a></h3>
<p><strong>Build:</strong> Implement ReLU from scratch</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">relu</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="c1"># YOU implement this function</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">maximum</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="c1"># Your solution</span>
</pre></div>
</div>
<p><strong>Use:</strong> Immediately use your own code</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">tinytorch.core.activations</span><span class="w"> </span><span class="kn">import</span> <span class="n">ReLU</span> <span class="c1"># YOUR implementation!</span>
<span class="n">layer</span> <span class="o">=</span> <span class="n">ReLU</span><span class="p">()</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">layer</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">input_tensor</span><span class="p">)</span> <span class="c1"># Your code working!</span>
</pre></div>
</div>
<p><strong>Reflect:</strong> See it working in real networks</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># Your ReLU is now part of a real neural network</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">([</span>
<span class="n">Dense</span><span class="p">(</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">),</span>
<span class="n">ReLU</span><span class="p">(),</span> <span class="c1"># &lt;-- Your implementation</span>
<span class="n">Dense</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span>
<span class="p">])</span>
</pre></div>
</div>
<p>This pattern repeats for every component: tensors, layers, optimizers, even MLOps systems. You build it, use it immediately, then reflect on how it fits into larger systems.</p>
<p><strong>🎯 Track Your Capabilities</strong></p>
<p>TinyTorch uses a <a class="reference internal" href="checkpoint-system.html"><span class="std std-doc">checkpoint system</span></a> to track your progress through <strong>ML systems engineering capabilities</strong>:</p>
<ul class="simple">
<li><p><strong>Foundation</strong> → Core ML primitives and setup</p></li>
<li><p><strong>Architecture</strong> → Neural network building</p></li>
<li><p><strong>Training</strong> → Model training pipeline</p></li>
<li><p><strong>Inference</strong> → Deployment and optimization</p></li>
<li><p><strong>Serving</strong> → Complete system integration</p></li>
</ul>
<p>Use <code class="docutils literal notranslate"><span class="pre">tito</span> <span class="pre">checkpoint</span> <span class="pre">status</span></code> to see your progress anytime!</p>
<p><strong>🎯 Beyond Code: Systems Intuition</strong></p>
<p>Each module includes <strong>ML Systems Thinking</strong> sections that connect your implementations to production reality:</p>
<ul class="simple">
<li><p><em>“How does your tensor implementation compare to PyTorch’s memory management?”</em></p></li>
<li><p><em>“When would you choose SGD over Adam in production training?”</em></p></li>
<li><p><em>“How do frameworks handle the quadratic memory scaling of attention?”</em></p></li>
<li><p><em>“What happens to your autograd implementation under distributed training?”</em></p></li>
</ul>
<p>These aren’t just academic questions - they’re the system-level challenges that ML engineers solve every day.</p>
</section>
</section>
<hr class="docutils" />
<section id="who-this-is-for">
<h2>👥 Who This Is For<a class="headerlink" href="#who-this-is-for" title="Link to this heading">#</a></h2>
<section id="perfect-for">
<h3>🎯 Perfect For:<a class="headerlink" href="#perfect-for" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>CS students</strong> who want to understand ML systems beyond high-level APIs</p></li>
<li><p><strong>Software engineers</strong> transitioning to ML engineering roles</p></li>
<li><p><strong>ML practitioners</strong> who want to optimize and debug production systems</p></li>
<li><p><strong>Researchers</strong> who need to implement custom operations and architectures</p></li>
<li><p><strong>Anyone curious</strong> about how PyTorch/TensorFlow actually work under the hood</p></li>
</ul>
</section>
<section id="prerequisites">
<h3>📚 Prerequisites:<a class="headerlink" href="#prerequisites" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>Python programming</strong> (comfortable with classes, functions, basic NumPy)</p></li>
<li><p><strong>Linear algebra basics</strong> (matrix multiplication, gradients)</p></li>
<li><p><strong>Learning mindset</strong> - we’ll teach you everything else!</p></li>
</ul>
</section>
<section id="career-impact">
<h3>🚀 Career Impact:<a class="headerlink" href="#career-impact" title="Link to this heading">#</a></h3>
<p>After TinyTorch, you’ll be the person your team asks:</p>
<ul class="simple">
<li><p><em>“Why is this training so slow?”</em> (You’ll know how to profile and optimize)</p></li>
<li><p><em>“Can we fit this model in GPU memory?”</em> (You’ll understand memory trade-offs)</p></li>
<li><p><em>“What’s the best optimizer for this problem?”</em> (You’ll know the system implications)</p></li>
</ul>
</section>
</section>
<hr class="docutils" />
<section id="course-journey-16-modules-foundation-to-framework">
<h2>📚 Course Journey: 16 Modules - Foundation to Framework<a class="headerlink" href="#course-journey-16-modules-foundation-to-framework" title="Link to this heading">#</a></h2>
<div class="note admonition">
<p class="admonition-title">Foundation</p>
<p><strong>0. Setup</strong> • <strong>1. Tensors</strong> • <strong>2. Activations</strong></p>
<p>Development workflow, multi-dimensional arrays, and mathematical functions that enable learning.</p>
</div>
<div class="note admonition">
<p class="admonition-title">Building Blocks</p>
<p><strong>3. Layers</strong> • <strong>4. Dense</strong> • <strong>5. Spatial</strong> • <strong>6. Attention</strong></p>
<p>Dense layers, sequential networks, convolutional operations, and self-attention mechanisms with memory analysis.</p>
</div>
<div class="note admonition">
<p class="admonition-title">Training Systems</p>
<p><strong>7. DataLoader</strong> • <strong>8. Autograd</strong> • <strong>9. Optimizers</strong> • <strong>10. Training</strong></p>
<p>CIFAR-10 loading, automatic differentiation with graph management, SGD/Adam with memory profiling, and complete training orchestration.</p>
</div>
<div class="note admonition">
<p class="admonition-title">Production Systems</p>
<p><strong>11. Compression</strong> • <strong>12. Kernels</strong> • <strong>13. Benchmarking</strong> • <strong>14. MLOps</strong></p>
<p>Model optimization, high-performance operations, systematic evaluation, and production monitoring with real deployment patterns.</p>
</div>
<div class="note admonition">
<p class="admonition-title">Framework Generalization</p>
<p><strong>15. TinyGPT</strong></p>
<p>Demonstrate framework universality: GPT-style transformers, character tokenization, autoregressive generation with 95% component reuse from your ML systems foundation.</p>
</div>
</section>
<hr class="docutils" />
<section id="complete-system-integration">
<h2>🔗 Complete System Integration<a class="headerlink" href="#complete-system-integration" title="Link to this heading">#</a></h2>
<p><strong>This isn’t 16 separate exercises.</strong> Every component you build integrates into one fully functional ML framework with universal foundations:</p>
<div class="important admonition">
<p class="admonition-title">🎯 How It All Connects</p>
<pre class="mermaid">
flowchart TD
Z[00_introduction&lt;br/&gt;🎯 System Overview] --&gt; A[01_setup&lt;br/&gt;Setup &amp; Environment]
A --&gt; B[02_tensor&lt;br/&gt;Core Tensor Operations]
B --&gt; C[03_activations&lt;br/&gt;ReLU, Sigmoid, Tanh]
B --&gt; I[09_autograd&lt;br/&gt;Automatic Differentiation]
C --&gt; D[04_layers&lt;br/&gt;Dense Layers]
D --&gt; E[05_dense&lt;br/&gt;Sequential Networks]
E --&gt; F[06_spatial&lt;br/&gt;Convolutional Networks]
E --&gt; G[07_attention&lt;br/&gt;Self-Attention]
B --&gt; H[08_dataloader&lt;br/&gt;Data Loading]
I --&gt; J[10_optimizers&lt;br/&gt;SGD &amp; Adam]
H --&gt; K[11_training&lt;br/&gt;Training Loops]
E --&gt; K
F --&gt; K
G --&gt; K
J --&gt; K
K --&gt; L[12_compression&lt;br/&gt;Model Optimization]
K --&gt; M[13_kernels&lt;br/&gt;High-Performance Ops]
K --&gt; N[14_benchmarking&lt;br/&gt;Performance Analysis]
K --&gt; O[15_mlops&lt;br/&gt;Production Monitoring]
L --&gt; P[16_tinygpt&lt;br/&gt;🔥 Language Models]
G --&gt; P
J --&gt; P
K --&gt; P
</pre></div>
<p><strong>Result:</strong> Every component you build converges into TinyGPT - proving your framework is complete and production-ready.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>
### 🔥 TinyGPT: The Complete Framework in Action
After building all the components, TinyGPT is your **capstone demonstration** - showing how everything clicks together into a working system.
**What TinyGPT Proves:**
- **Component Integration**: Your tensors, layers, autograd, and optimizers work together seamlessly
- **Universal Foundations**: The same mathematical primitives power any neural architecture
- **Framework Completeness**: You built a production-ready ML framework from scratch
- **Systems Mastery**: You understand how every piece fits together under the hood
**The Achievement:** Build a complete GPT-style language model using only components you implemented. This proves your framework is real, complete, and ready for any ML task.
---
## Choose Your Learning Path
```{admonition} Three Ways to Engage with TinyTorch
:class: important
### [Quick Exploration](usage-paths/quick-exploration.md) *(5 minutes)*
*&quot;I want to see what this is about&quot;*
- Click and run code immediately in your browser (Binder)
- No installation or setup required
- Implement ReLU, tensors, neural networks interactively
- Perfect for getting a feel for the course
### [Serious Development](usage-paths/serious-development.md) *(8+ weeks)*
*&quot;I want to build this myself&quot;*
- Fork the repo and work locally with full development environment
- Build complete ML framework from scratch with `tito` CLI
- 16 progressive assignments from setup to language models
- Professional development workflow with automated testing
### [Classroom Use](usage-paths/classroom-use.md) *(Instructors)*
*&quot;I want to teach this course&quot;*
- Complete course infrastructure with NBGrader integration
- Automated grading for comprehensive testing
- Flexible pacing (8-16 weeks) with proven pedagogical structure
- Turn-key solution for ML systems education
</pre></div>
</div>
</section>
<hr class="docutils" />
<section id="ready-to-start">
<h2>Ready to Start?<a class="headerlink" href="#ready-to-start" title="Link to this heading">#</a></h2>
<section id="quick-taste-try-module-1-right-now">
<h3>Quick Taste: Try Module 1 Right Now<a class="headerlink" href="#quick-taste-try-module-1-right-now" title="Link to this heading">#</a></h3>
<p>Want to see what TinyTorch feels like? <strong><a class="reference internal" href="chapters/01-setup.html"><span class="std std-doc">Launch the Setup chapter</span></a></strong> in Binder and implement your first TinyTorch function in 2 minutes!</p>
</section>
</section>
<hr class="docutils" />
<section id="acknowledgments">
<h2>Acknowledgments<a class="headerlink" href="#acknowledgments" title="Link to this heading">#</a></h2>
<p>TinyTorch originated from CS249r: Tiny Machine Learning Systems at Harvard University. We’re inspired by projects like <a class="reference external" href="https://github.com/geohot/tinygrad">tinygrad</a>, <a class="reference external" href="https://github.com/karpathy/micrograd">micrograd</a>, and <a class="reference external" href="https://minitorch.github.io/">MiniTorch</a> that demonstrate the power of minimal implementations.</p>
</section>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
<div class="toctree-wrapper compound">
</div>
</section>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
name: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="right-next"
href="usage-paths/quick-exploration.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Quick Exploration Path</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> Contents
</div>
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-core-difference">The Core Difference</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-philosophy-build-use-reflect">Learning Philosophy: Build, Use, Reflect</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#example-activation-functions">Example: Activation Functions</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#who-this-is-for">👥 Who This Is For</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#perfect-for">🎯 Perfect For:</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#prerequisites">📚 Prerequisites:</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#career-impact">🚀 Career Impact:</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#course-journey-16-modules-foundation-to-framework">📚 Course Journey: 16 Modules - Foundation to Framework</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#complete-system-integration">🔗 Complete System Integration</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ready-to-start">Ready to Start?</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#quick-taste-try-module-1-right-now">Quick Taste: Try Module 1 Right Now</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#acknowledgments">Acknowledgments</a></li>
</ul>
</nav></div>
</div></div>
</div>
<footer class="bd-footer-content">
<div class="bd-footer-content__inner container">
<div class="footer-item">
<p class="component-author">
By Prof. Vijay Janapa Reddi (Harvard University)
</p>
</div>
<div class="footer-item">
<p class="copyright">
© Copyright 2025.
<br/>
</p>
</div>
<div class="footer-item">
</div>
<div class="footer-item">
</div>
</div>
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
<footer class="bd-footer">
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink