Files
TinyTorch/book/_build/html/usage-paths/serious-development.html
Vijay Janapa Reddi 92781736a1 Restructure TinyTorch: Move TinyGPT to examples, improve testing framework
Major changes:
- Moved TinyGPT from Module 16 to examples/tinygpt (capstone demo)
- Fixed Module 10 (optimizers) and Module 11 (training) bugs
- All 16 modules now passing tests (100% health)
- Added comprehensive testing with 'tito test --comprehensive'
- Renamed example files for clarity (train_xor_network.py, etc.)
- Created working TinyGPT example structure
- Updated documentation to reflect 15 core modules + examples
- Added KISS principle and testing framework documentation
2025-09-22 09:37:18 -04:00

909 lines
41 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en" data-content_root="../" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Serious Development Path &#8212; TinyTorch</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!-- Loaded before other Sphinx assets -->
<link href="../_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="../_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="../_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="../_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="../_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="../_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="../_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="../_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="../_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="../_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="../_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="../_static/sphinx-design.min.css?v=95c83b7e" />
<link rel="stylesheet" type="text/css" href="../_static/custom.css?v=5a689c52" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
<link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
<script src="../_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="../_static/documentation_options.js?v=9eb32ce0"></script>
<script src="../_static/doctools.js?v=9a2dae69"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/clipboard.min.js?v=a7894cd8"></script>
<script src="../_static/copybutton.js?v=f281be69"></script>
<script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="../_static/togglebutton.js?v=4a39c7ea"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script src="../_static/design-tabs.js?v=f930bc37"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script async="async" src="../_static/sphinx-thebe.js?v=c100c467"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script>DOCUMENTATION_OPTIONS.pagename = 'usage-paths/serious-development';</script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Classroom Use Overview" href="classroom-use.html" />
<link rel="prev" title="Quick Exploration Path" href="quick-exploration.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<input type="checkbox"
class="sidebar-toggle"
id="pst-primary-sidebar-checkbox"/>
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
<input type="checkbox"
class="sidebar-toggle"
id="pst-secondary-sidebar-checkbox"/>
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
<div class="search-button__wrapper">
<div class="search-button__overlay"></div>
<div class="search-button__search-container">
<form class="bd-search d-flex align-items-center"
action="../search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
id="search-input"
placeholder="Search this book..."
aria-label="Search this book..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form></div>
</div>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<div class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<a class="navbar-brand logo" href="../intro.html">
<img src="../_static/logo.png" class="logo__image only-light" alt="TinyTorch - Home"/>
<script>document.write(`<img src="../_static/logo.png" class="logo__image only-dark" alt="TinyTorch - Home"/>`);</script>
</a></div>
<div class="sidebar-primary-item">
<script>
document.write(`
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
`);
</script></div>
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
<div class="bd-toc-item navbar-nav active">
<ul class="nav bd-sidenav bd-sidenav__home-link">
<li class="toctree-l1">
<a class="reference internal" href="../intro.html">
TinyTorch: Build Your Own ML Framework from First Principles
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Usage Paths</span></p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="quick-exploration.html">Quick Exploration</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Serious Development</a></li>
<li class="toctree-l1"><a class="reference internal" href="classroom-use.html">Classroom Use</a></li>
<li class="toctree-l1"><a class="reference internal" href="../instructor-guide.html">Instructor Guide</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Foundation</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../chapters/01-setup.html">0. Setup</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/02-tensor.html">1. Tensors</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/03-activations.html">2. Activations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Building Blocks</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../chapters/04-layers.html">3. Layers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/05-dense.html">4. Dense</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/06-spatial.html">5. Spatial</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/07-attention.html">6. Attention</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Training Systems</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../chapters/08-dataloader.html">7. DataLoader</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/09-autograd.html">8. Autograd</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/10-optimizers.html">9. Optimizers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/11-training.html">10. Training</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Inference &amp; Serving</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../chapters/12-compression.html">11. Compression</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/13-kernels.html">12. Kernels</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/14-benchmarking.html">13. Benchmarking</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/15-mlops.html">14. MLOps</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Language Models</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../chapters/16-tinygpt.html">15. TinyGPT</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Additional Resources</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../vision.html">TinyTorch Vision</a></li>
<li class="toctree-l1"><a class="reference internal" href="../kiss-principle.html">KISS Principle</a></li>
<li class="toctree-l1"><a class="reference internal" href="../checkpoint-system.html">Checkpoint System</a></li>
<li class="toctree-l1"><a class="reference internal" href="../testing-framework.html">Testing Framework</a></li>
<li class="toctree-l1"><a class="reference internal" href="../resources.html">Learning Resources</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
</div>
<div id="rtd-footer-container"></div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="sbt-scroll-pixel-helper"></div>
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-bars"></span>
</button></div>
</div>
<div class="header-article-items__end">
<div class="header-article-item">
<div class="article-header-buttons">
<div class="dropdown dropdown-source-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Source repositories">
<i class="fab fa-github"></i>
</button>
<ul class="dropdown-menu">
<li><a href="https://github.com/mlsysbook/TinyTorch" target="_blank"
class="btn btn-sm btn-source-repository-button dropdown-item"
title="Source repository"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fab fa-github"></i>
</span>
<span class="btn__text-container">Repository</span>
</a>
</li>
<li><a href="https://github.com/mlsysbook/TinyTorch/edit/main/book/usage-paths/serious-development.md" target="_blank"
class="btn btn-sm btn-source-edit-button dropdown-item"
title="Suggest edit"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-pencil-alt"></i>
</span>
<span class="btn__text-container">Suggest edit</span>
</a>
</li>
<li><a href="https://github.com/mlsysbook/TinyTorch/issues/new?title=Issue%20on%20page%20%2Fusage-paths/serious-development.html&body=Your%20issue%20content%20here." target="_blank"
class="btn btn-sm btn-source-issues-button dropdown-item"
title="Open an issue"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-lightbulb"></i>
</span>
<span class="btn__text-container">Open issue</span>
</a>
</li>
</ul>
</div>
<div class="dropdown dropdown-download-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
<i class="fas fa-download"></i>
</button>
<ul class="dropdown-menu">
<li><a href="../_sources/usage-paths/serious-development.md" target="_blank"
class="btn btn-sm btn-download-source-button dropdown-item"
title="Download source file"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file"></i>
</span>
<span class="btn__text-container">.md</span>
</a>
</li>
<li>
<button onclick="window.print()"
class="btn btn-sm btn-download-pdf-button dropdown-item"
title="Print to PDF"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file-pdf"></i>
</span>
<span class="btn__text-container">.pdf</span>
</button>
</li>
</ul>
</div>
<button onclick="toggleFullScreen()"
class="btn btn-sm btn-fullscreen-button"
title="Fullscreen mode"
data-bs-placement="bottom" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-expand"></i>
</span>
</button>
<script>
document.write(`
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
</button>
`);
</script>
<script>
document.write(`
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
</button>
`);
</script>
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-list"></span>
</button>
</div></div>
</div>
</div>
</div>
<div id="jb-print-docs-body" class="onlyprint">
<h1>Serious Development Path</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#what-youll-build">What Youll Build</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#quick-start-5-minutes">Quick Start (5 minutes)</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-1-get-the-code">Step 1: Get the Code</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-2-setup-environment">Step 2: Setup Environment</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-3-start-building">Step 3: Start Building</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-4-build-test-export-use">Step 4: Build → Test → Export → Use</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-path-progressive-complexity">Learning Path (Progressive Complexity)</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#foundation-weeks-1-2">Foundation (Weeks 1-2)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#building-blocks-weeks-3-4">🧱 Building Blocks (Weeks 3-4)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#training-systems-weeks-5-6">Training Systems (Weeks 5-6)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#production-performance-weeks-7-8">Production &amp; Performance (Weeks 7-8)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#development-workflow">Development Workflow</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#the-tito-cli-system">The <code class="docutils literal notranslate"><span class="pre">tito</span></code> CLI System</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#quality-assurance">Quality Assurance</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#proven-student-outcomes">Proven Student Outcomes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-this-approach-works">Why This Approach Works</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#build-use-understand">Build → Use → Understand</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#real-data-real-systems">Real Data, Real Systems</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#immediate-feedback">Immediate Feedback</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ready-to-start">Ready to Start?</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#choose-your-module">Choose Your Module</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#get-help">Get Help</a></li>
</ul>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<section id="serious-development-path">
<h1>Serious Development Path<a class="headerlink" href="#serious-development-path" title="Link to this heading">#</a></h1>
<p><strong>Perfect for:</strong> “I want to build this myself” • “This is my class assignment” • “I want to understand ML frameworks deeply”</p>
<hr class="docutils" />
<section id="what-youll-build">
<h2>What Youll Build<a class="headerlink" href="#what-youll-build" title="Link to this heading">#</a></h2>
<p>A complete ML framework from scratch, including:</p>
<ul class="simple">
<li><p><strong>Your own tensor library</strong> with operations and autograd</p></li>
<li><p><strong>Neural network components</strong> (layers, activations, optimizers)</p></li>
<li><p><strong>Training systems</strong> that work on real datasets (CIFAR-10)</p></li>
<li><p><strong>Production features</strong> (compression, monitoring, benchmarking)</p></li>
<li><p><strong>Language models</strong> that extend your vision framework to TinyGPT</p></li>
</ul>
<p><strong>End result:</strong> A working ML framework that powers both computer vision AND language models.</p>
</section>
<hr class="docutils" />
<section id="quick-start-5-minutes">
<h2>Quick Start (5 minutes)<a class="headerlink" href="#quick-start-5-minutes" title="Link to this heading">#</a></h2>
<section id="step-1-get-the-code">
<h3>Step 1: Get the Code<a class="headerlink" href="#step-1-get-the-code" title="Link to this heading">#</a></h3>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/your-org/tinytorch.git
<span class="nb">cd</span><span class="w"> </span>TinyTorch
</pre></div>
</div>
</section>
<section id="step-2-setup-environment">
<h3>Step 2: Setup Environment<a class="headerlink" href="#step-2-setup-environment" title="Link to this heading">#</a></h3>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Activate virtual environment </span>
<span class="nb">source</span><span class="w"> </span>bin/activate-tinytorch.sh
<span class="c1"># Install dependencies</span>
make<span class="w"> </span>install
<span class="c1"># Verify everything works</span>
tito<span class="w"> </span>system<span class="w"> </span>doctor
</pre></div>
</div>
</section>
<section id="step-3-start-building">
<h3>Step 3: Start Building<a class="headerlink" href="#step-3-start-building" title="Link to this heading">#</a></h3>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Open first assignment</span>
<span class="nb">cd</span><span class="w"> </span>modules/source/01_setup
jupyter<span class="w"> </span>lab<span class="w"> </span>setup_dev.py
</pre></div>
</div>
</section>
<section id="step-4-build-test-export-use">
<h3>Step 4: Build → Test → Export → Use<a class="headerlink" href="#step-4-build-test-export-use" title="Link to this heading">#</a></h3>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># After implementing code in the notebook:</span>
tito<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="c1"># Export your code to tinytorch package</span>
tito<span class="w"> </span><span class="nb">test</span><span class="w"> </span>setup<span class="w"> </span><span class="c1"># Test your implementation</span>
<span class="c1"># Now use YOUR own code:</span>
python<span class="w"> </span>-c<span class="w"> </span><span class="s2">&quot;from tinytorch.core.setup import hello_tinytorch; hello_tinytorch()&quot;</span>
<span class="c1"># 🔥 TinyTorch! Built by: [Your Name]</span>
</pre></div>
</div>
</section>
</section>
<hr class="docutils" />
<section id="learning-path-progressive-complexity">
<h2>Learning Path (Progressive Complexity)<a class="headerlink" href="#learning-path-progressive-complexity" title="Link to this heading">#</a></h2>
<section id="foundation-weeks-1-2">
<h3>Foundation (Weeks 1-2)<a class="headerlink" href="#foundation-weeks-1-2" title="Link to this heading">#</a></h3>
<p>Build the core infrastructure:</p>
<p><strong>Module 01: Setup &amp; CLI</strong></p>
<ul class="simple">
<li><p>Professional development workflow with <code class="docutils literal notranslate"><span class="pre">tito</span></code> CLI</p></li>
<li><p>Understanding package architecture and exports</p></li>
<li><p>Quality assurance with automated testing</p></li>
</ul>
<p><strong>Module 01: Tensors</strong></p>
<ul class="simple">
<li><p>Multi-dimensional arrays and operations</p></li>
<li><p>Memory management and data types</p></li>
<li><p>Foundation for all ML operations</p></li>
</ul>
<p><strong>Module 02: Activations</strong></p>
<ul class="simple">
<li><p>ReLU, Sigmoid, Tanh, Softmax functions</p></li>
<li><p>Understanding nonlinearity in neural networks</p></li>
<li><p>Mathematical foundations of deep learning</p></li>
</ul>
</section>
<hr class="docutils" />
<section id="building-blocks-weeks-3-4">
<h3>🧱 Building Blocks (Weeks 3-4)<a class="headerlink" href="#building-blocks-weeks-3-4" title="Link to this heading">#</a></h3>
<p>Create neural network components:</p>
<p><strong>Module 03: Layers</strong></p>
<ul class="simple">
<li><p>Dense (linear) layers with matrix multiplication</p></li>
<li><p>Weight initialization strategies</p></li>
<li><p>Building blocks that stack together</p></li>
</ul>
<p><strong>Module 04: Networks</strong></p>
<ul class="simple">
<li><p>Sequential model architecture</p></li>
<li><p>Composition patterns and forward propagation</p></li>
<li><p>Creating complete neural networks</p></li>
</ul>
<p><strong>Module 05: CNNs</strong></p>
<ul class="simple">
<li><p>Convolutional operations for computer vision</p></li>
<li><p>Understanding spatial processing</p></li>
<li><p>Building blocks for image classification</p></li>
</ul>
</section>
<hr class="docutils" />
<section id="training-systems-weeks-5-6">
<h3>Training Systems (Weeks 5-6)<a class="headerlink" href="#training-systems-weeks-5-6" title="Link to this heading">#</a></h3>
<p>Complete training infrastructure:</p>
<p><strong>Module 06: DataLoader</strong></p>
<ul class="simple">
<li><p>Efficient data loading and preprocessing</p></li>
<li><p>Real dataset handling (CIFAR-10)</p></li>
<li><p>Batching, shuffling, and memory management</p></li>
</ul>
<p><strong>Module 07: Autograd</strong></p>
<ul class="simple">
<li><p>Automatic differentiation engine</p></li>
<li><p>Computational graphs and backpropagation</p></li>
<li><p>The magic that makes training possible</p></li>
</ul>
<p><strong>Module 08: Optimizers</strong></p>
<ul class="simple">
<li><p>SGD, Adam, and learning rate scheduling</p></li>
<li><p>Understanding gradient descent variants</p></li>
<li><p>Convergence and training dynamics</p></li>
</ul>
<p><strong>Module 09: Training</strong></p>
<ul class="simple">
<li><p>Complete training loops and loss functions</p></li>
<li><p>Model evaluation and metrics</p></li>
<li><p>Checkpointing and persistence</p></li>
</ul>
</section>
<hr class="docutils" />
<section id="production-performance-weeks-7-8">
<h3>Production &amp; Performance (Weeks 7-8)<a class="headerlink" href="#production-performance-weeks-7-8" title="Link to this heading">#</a></h3>
<p>Real-world deployment:</p>
<p><strong>Module 10: Compression</strong></p>
<ul class="simple">
<li><p>Model pruning and quantization</p></li>
<li><p>Reducing model size by 75%+</p></li>
<li><p>Deployment optimization</p></li>
</ul>
<p><strong>Module 11: Kernels</strong></p>
<ul class="simple">
<li><p>High-performance custom operations</p></li>
<li><p>Hardware-aware optimization</p></li>
<li><p>Understanding framework internals</p></li>
</ul>
<p><strong>Module 12: Benchmarking</strong></p>
<ul class="simple">
<li><p>Systematic performance measurement</p></li>
<li><p>Statistical validation and reporting</p></li>
<li><p>MLPerf-style evaluation</p></li>
</ul>
<p><strong>Module 13: MLOps</strong></p>
<ul class="simple">
<li><p>Production deployment and monitoring</p></li>
<li><p>Continuous learning and model updates</p></li>
<li><p>Complete production pipeline</p></li>
</ul>
<p><strong>Module 16: TinyGPT 🔥</strong></p>
<ul class="simple">
<li><p>Extend vision framework to language models</p></li>
<li><p>GPT-style transformers with 95% component reuse</p></li>
<li><p>Autoregressive text generation</p></li>
<li><p>Framework generalization mastery</p></li>
</ul>
</section>
</section>
<hr class="docutils" />
<section id="development-workflow">
<h2>Development Workflow<a class="headerlink" href="#development-workflow" title="Link to this heading">#</a></h2>
<section id="the-tito-cli-system">
<h3>The <code class="docutils literal notranslate"><span class="pre">tito</span></code> CLI System<a class="headerlink" href="#the-tito-cli-system" title="Link to this heading">#</a></h3>
<p>TinyTorch includes a complete CLI for professional development:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># System management</span>
tito<span class="w"> </span>system<span class="w"> </span>doctor<span class="w"> </span><span class="c1"># Check environment health</span>
tito<span class="w"> </span>system<span class="w"> </span>info<span class="w"> </span><span class="c1"># Show module status</span>
<span class="c1"># Module development </span>
tito<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="c1"># Export dev code to package</span>
tito<span class="w"> </span><span class="nb">test</span><span class="w"> </span>setup<span class="w"> </span><span class="c1"># Test specific module</span>
tito<span class="w"> </span><span class="nb">test</span><span class="w"> </span>--all<span class="w"> </span><span class="c1"># Test everything</span>
<span class="c1"># NBGrader integration</span>
tito<span class="w"> </span>nbgrader<span class="w"> </span>generate<span class="w"> </span>setup<span class="w"> </span><span class="c1"># Create assignments</span>
tito<span class="w"> </span>nbgrader<span class="w"> </span>release<span class="w"> </span>setup<span class="w"> </span><span class="c1"># Release to students</span>
tito<span class="w"> </span>nbgrader<span class="w"> </span>autograde<span class="w"> </span>setup<span class="w"> </span><span class="c1"># Auto-grade submissions</span>
</pre></div>
</div>
</section>
<section id="quality-assurance">
<h3>Quality Assurance<a class="headerlink" href="#quality-assurance" title="Link to this heading">#</a></h3>
<p>Every module includes comprehensive testing:</p>
<ul class="simple">
<li><p><strong>100+ automated tests</strong> ensure correctness</p></li>
<li><p><strong>Inline tests</strong> provide immediate feedback</p></li>
<li><p><strong>Integration tests</strong> verify cross-module functionality</p></li>
<li><p><strong>Performance benchmarks</strong> track optimization</p></li>
</ul>
</section>
</section>
<hr class="docutils" />
<section id="proven-student-outcomes">
<h2>Proven Student Outcomes<a class="headerlink" href="#proven-student-outcomes" title="Link to this heading">#</a></h2>
<div class="success admonition">
<p class="admonition-title">Real Results</p>
<p><strong>After 6-8 weeks, students consistently:</strong></p>
<p>✅ Build multi-layer perceptrons that classify CIFAR-10 images<br />
✅ Implement automatic differentiation from scratch<br />
✅ Create custom optimizers (SGD, Adam) that converge reliably<br />
✅ Optimize models with pruning and quantization<br />
✅ Deploy production ML systems with monitoring<br />
✅ Understand framework internals better than most ML engineers<br />
🔥 <strong>Extend their vision framework to language models with 95% reuse</strong></p>
<p><strong>Test Coverage:</strong> 200+ tests across all modules ensure student implementations work</p>
</div>
</section>
<hr class="docutils" />
<section id="why-this-approach-works">
<h2>Why This Approach Works<a class="headerlink" href="#why-this-approach-works" title="Link to this heading">#</a></h2>
<section id="build-use-understand">
<h3>Build → Use → Understand<a class="headerlink" href="#build-use-understand" title="Link to this heading">#</a></h3>
<p>Every component follows this pattern:</p>
<ol class="arabic simple">
<li><p><strong>🔧 Build</strong>: Implement <code class="docutils literal notranslate"><span class="pre">ReLU()</span></code> from scratch</p></li>
<li><p><strong>🚀 Use</strong>: <code class="docutils literal notranslate"><span class="pre">from</span> <span class="pre">tinytorch.core.activations</span> <span class="pre">import</span> <span class="pre">ReLU</span></code> - your code!</p></li>
<li><p><strong>💡 Understand</strong>: See how it enables complex pattern learning</p></li>
</ol>
</section>
<section id="real-data-real-systems">
<h3>Real Data, Real Systems<a class="headerlink" href="#real-data-real-systems" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p>Work with CIFAR-10 (not toy datasets)</p></li>
<li><p>Production-style code organization</p></li>
<li><p>Performance and engineering considerations</p></li>
<li><p>Professional development practices</p></li>
</ul>
</section>
<section id="immediate-feedback">
<h3>Immediate Feedback<a class="headerlink" href="#immediate-feedback" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p>Code works immediately after implementation</p></li>
<li><p>Visual progress indicators and success messages</p></li>
<li><p>Comprehensive error handling and guidance</p></li>
<li><p>Professional-quality development experience</p></li>
</ul>
</section>
</section>
<hr class="docutils" />
<section id="ready-to-start">
<h2>Ready to Start?<a class="headerlink" href="#ready-to-start" title="Link to this heading">#</a></h2>
<section id="choose-your-module">
<h3>Choose Your Module<a class="headerlink" href="#choose-your-module" title="Link to this heading">#</a></h3>
<p><strong>New to ML frameworks?</strong> → Start with <a class="reference internal" href="../chapters/01-setup.html"><span class="std std-doc">Setup</span></a>
<strong>Have ML experience?</strong> → Jump to <a class="reference internal" href="../chapters/02-tensor.html"><span class="std std-doc">Tensors</span></a>
<strong>Want to see the vision?</strong> → Try <a class="reference internal" href="../chapters/03-activations.html"><span class="std std-doc">Activations</span></a></p>
</section>
<section id="get-help">
<h3>Get Help<a class="headerlink" href="#get-help" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>💬 Discussions</strong>: GitHub Discussions for questions</p></li>
<li><p><strong>🐛 Issues</strong>: Report bugs or suggest improvements</p></li>
<li><p><strong>📧 Support</strong>: Direct contact with TinyTorch team</p></li>
</ul>
<hr class="docutils" />
<p><em>🎉 Ready to build your own ML framework? Your unified vision+language framework is 8 weeks away!</em></p>
</section>
</section>
</section>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
name: "python3",
path: "./usage-paths"
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="left-prev"
href="quick-exploration.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Quick Exploration Path</p>
</div>
</a>
<a class="right-next"
href="classroom-use.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Classroom Use Overview</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> Contents
</div>
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#what-youll-build">What Youll Build</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#quick-start-5-minutes">Quick Start (5 minutes)</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-1-get-the-code">Step 1: Get the Code</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-2-setup-environment">Step 2: Setup Environment</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-3-start-building">Step 3: Start Building</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#step-4-build-test-export-use">Step 4: Build → Test → Export → Use</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-path-progressive-complexity">Learning Path (Progressive Complexity)</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#foundation-weeks-1-2">Foundation (Weeks 1-2)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#building-blocks-weeks-3-4">🧱 Building Blocks (Weeks 3-4)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#training-systems-weeks-5-6">Training Systems (Weeks 5-6)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#production-performance-weeks-7-8">Production &amp; Performance (Weeks 7-8)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#development-workflow">Development Workflow</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#the-tito-cli-system">The <code class="docutils literal notranslate"><span class="pre">tito</span></code> CLI System</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#quality-assurance">Quality Assurance</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#proven-student-outcomes">Proven Student Outcomes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-this-approach-works">Why This Approach Works</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#build-use-understand">Build → Use → Understand</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#real-data-real-systems">Real Data, Real Systems</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#immediate-feedback">Immediate Feedback</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ready-to-start">Ready to Start?</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#choose-your-module">Choose Your Module</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#get-help">Get Help</a></li>
</ul>
</li>
</ul>
</nav></div>
</div></div>
</div>
<footer class="bd-footer-content">
<div class="bd-footer-content__inner container">
<div class="footer-item">
<p class="component-author">
By Prof. Vijay Janapa Reddi (Harvard University)
</p>
</div>
<div class="footer-item">
<p class="copyright">
© Copyright 2025.
<br/>
</p>
</div>
<div class="footer-item">
</div>
<div class="footer-item">
</div>
</div>
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="../_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="../_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
<footer class="bd-footer">
</footer>
</body>
</html>