cs249r_book/tinytorch/site/intro.md

TinyTorch

<div style="background: linear-gradient(135deg, #fef3c7 0%, #fde68a 100%); border: 2px solid #f59e0b; border-radius: 0.75rem; padding: 1.25rem 1.5rem; margin: 0 0 2rem 0; text-align: center;">
  <div style="display: flex; align-items: center; justify-content: center; gap: 0.5rem; margin-bottom: 0.5rem;">
    <span style="font-size: 1.25rem;">🚧</span>
    <span style="font-weight: 700; color: #92400e; font-size: 1.1rem;">Preview Release</span>
  </div>
  <p style="margin: 0; color: #78350f; font-size: 0.95rem; line-height: 1.5;">
    TinyTorch is functional but evolving. We're sharing early to shape the direction with community input.<br>
    <strong>Classroom ready: Summer/Fall 2026</strong> · <a href="https://github.com/harvard-edge/cs249r_book/discussions/1076" style="color: #b45309; font-weight: 600;">Share your feedback →</a>
  </p>
</div>

<p style="text-align: center; font-size: 2.5rem; margin: 0rem 0 0.5rem 0; font-weight: 700;">
Build Your Own ML Framework
</p>

Hands-on labs for the Machine Learning Systems textbook

Don't import it. Build it.

Build a complete machine learning (ML) framework from tensors to systems—understand how PyTorch, TensorFlow, and JAX really work under the hood.

<style>
.vision-box {
  max-width: 720px;
  margin: 0 auto 2.5rem auto;
  background: linear-gradient(135deg, #1e293b 0%, #0f172a 100%);
  border-radius: 12px;
  padding: 2rem;
  border: 1px solid rgba(249, 115, 22, 0.3);
  box-shadow: 0 8px 24px rgba(0,0,0,0.2);
}
.vision-title {
  color: #f97316;
  font-size: 1.1rem;
  font-weight: 600;
  margin: 0 0 1rem 0;
  text-align: center;
}
.vision-grid {
  display: grid;
  grid-template-columns: repeat(2, 1fr);
  gap: 1rem;
}
.vision-item {
  display: flex;
  align-items: flex-start;
  gap: 0.75rem;
}
.vision-icon {
  font-size: 1.25rem;
  flex-shrink: 0;
}
.vision-text {
  color: #e2e8f0;
  font-size: 0.9rem;
  line-height: 1.5;
}
.vision-text strong {
  color: #fbbf24;
}
@media (max-width: 600px) {
  .vision-grid { grid-template-columns: 1fr; }
}
</style>

<div class="vision-box">
  <p class="vision-title">The "AI Bricks" Approach</p>
  <div class="vision-grid">
    <div class="vision-item">
      <span class="vision-icon">🔧</span>
      <span class="vision-text"><strong>Build each piece</strong> — Tensors, autograd, optimizers, attention. No magic imports.</span>
    </div>
    <div class="vision-item">
      <span class="vision-icon"></span>
      <span class="vision-text"><strong>Recreate history</strong> — Perceptron (1957) to Transformers (2017) to MLPerf (Present).</span>
    </div>
    <div class="vision-item">
      <span class="vision-icon">⚡</span>
      <span class="vision-text"><strong>Understand systems</strong> — Memory, compute, optimization trade-offs.</span>
    </div>
    <div class="vision-item">
      <span class="vision-icon">🎯</span>
      <span class="vision-text"><strong>End goal</strong> — A complete ML framework that trains and optimizes models.</span>
    </div>
  </div>
</div>

Share Your Feedback → Early Adopter Setup

Getting Started

TinyTorch is organized into four progressive tiers that take you from mathematical foundations to production-ready systems. Each tier builds on the previous one, teaching you not just how to code ML components, but how they work together as a complete system.

Foundation (Modules 01-07)

Build the mathematical core that makes neural networks learn.

Unlocks: Perceptron (1957) • XOR Crisis (1969) • MLP (1986)

Architecture (Modules 08-13)

Build modern neural architectures—from computer vision to language models.

Unlocks: CNN Revolution (1998) • Transformer Era (2017)

Optimization (Modules 14-19)

Transform research prototypes into production-ready systems.

Unlocks: MLPerf Benchmarks (2018–Present) • 8-16× compression • 12-40× speedup

Torch Olympics (Module 20)

The ultimate test: Build a complete, competition-ready ML system.

Capstone: Vision • Language • Speed • Compression tracks

The Big Picture • Getting started guide • Join the community

Recreate ML History

Walk through ML history by rebuilding its greatest breakthroughs with YOUR TinyTorch implementations. Click each milestone to see what you'll build and how it shaped modern AI.

<div class="ml-timeline-container">
    <div class="ml-timeline-line"></div>

    <div class="ml-timeline-item left perceptron">
        <div class="ml-timeline-dot"></div>
        <div class="ml-timeline-content">
            <div class="ml-timeline-year">1957</div>
            <div class="ml-timeline-title">The Perceptron</div>
            <div class="ml-timeline-desc">The first trainable neural network</div>
            <div class="ml-timeline-tech">Input → Linear → Sigmoid → Output</div>
        </div>
    </div>

    <div class="ml-timeline-item right xor">
        <div class="ml-timeline-dot"></div>
        <div class="ml-timeline-content">
            <div class="ml-timeline-year">1969</div>
            <div class="ml-timeline-title">XOR Crisis</div>
            <div class="ml-timeline-desc">Minsky & Papert expose limits of single-layer networks</div>
            <div class="ml-timeline-tech">Input → Linear → Sigmoid → FAIL!</div>
        </div>
    </div>

    <div class="ml-timeline-item left mlp">
        <div class="ml-timeline-dot"></div>
        <div class="ml-timeline-content">
            <div class="ml-timeline-year">1986</div>
            <div class="ml-timeline-title">MLP Revival</div>
            <div class="ml-timeline-desc">Backpropagation enables deep learning (95%+ MNIST)</div>
            <div class="ml-timeline-tech">Images → Flatten → Linear → ... → Classes</div>
        </div>
    </div>

    <div class="ml-timeline-item right cnn">
        <div class="ml-timeline-dot"></div>
        <div class="ml-timeline-content">
            <div class="ml-timeline-year">1998</div>
            <div class="ml-timeline-title">CNN Revolution 🎯</div>
            <div class="ml-timeline-desc">Spatial intelligence unlocks computer vision (75%+ CIFAR-10)</div>
            <div class="ml-timeline-tech">Images → Conv → Pool → ... → Classes</div>
        </div>
    </div>

    <div class="ml-timeline-item left transformer">
        <div class="ml-timeline-dot"></div>
        <div class="ml-timeline-content">
            <div class="ml-timeline-year">2017</div>
            <div class="ml-timeline-title">Transformer Era</div>
            <div class="ml-timeline-desc">Attention launches the LLM revolution</div>
            <div class="ml-timeline-tech">Tokens → Attention → FFN → Output</div>
        </div>
    </div>

    <div class="ml-timeline-item right olympics">
        <div class="ml-timeline-dot"></div>
        <div class="ml-timeline-content">
            <div class="ml-timeline-year">2018–Present</div>
            <div class="ml-timeline-title">MLPerf Benchmarks</div>
            <div class="ml-timeline-desc">Production optimization (8-16× smaller, 12-40× faster)</div>
            <div class="ml-timeline-tech">Profile → Compress → Accelerate</div>
        </div>
    </div>
</div>

View complete milestone details to see full technical requirements and learning objectives.

Why Build Instead of Use?

Understanding the difference between using a framework and building one is the difference between being limited by tools and being empowered to create them.

Traditional ML Education

import torch
model = torch.nn.Linear(784, 10)
output = model(input)
# When this breaks, you're stuck

Problem: OOM errors, NaN losses, slow training—you can't debug what you don't understand.

TinyTorch Approach

from tinytorch import Linear  # YOUR code
model = Linear(784, 10)       # YOUR implementation
output = model(input)
# You know exactly how this works

Advantage: You understand memory layouts, gradient flows, and performance bottlenecks because you implemented them.

Systems Thinking: TinyTorch emphasizes understanding how components interact—memory hierarchies, computational complexity, and optimization trade-offs—not just isolated algorithms. Every module connects mathematical theory to systems understanding.

See Preface for the full origin story and pedagogical approach.

The Build → Use → Reflect Approach

Every module follows a proven learning cycle that builds deep understanding:

:align: center
:caption: "**Build-Use-Reflect Learning Cycle.** Every module follows this proven pattern: implement from scratch, apply to real problems, then answer systems thinking questions."
graph LR
    B[Build<br/>Implement from scratch] --> U[Use<br/>Real data, real problems]
    U --> R[Reflect<br/>Systems thinking questions]
    R --> B

    style B fill:#FFC107,color:#000
    style U fill:#4CAF50,color:#fff
    style R fill:#2196F3,color:#fff

1. Build: Implement each component yourself—tensors, autograd, optimizers, attention
2. Use: Apply your implementations to real problems—MNIST, CIFAR-10, text generation
3. Reflect: Answer systems thinking questions—memory usage, scaling behavior, trade-offs

This approach develops not just coding ability, but systems engineering intuition essential for production ML.

Is This For You?

Perfect if you want to debug ML systems, implement custom operations, or understand how PyTorch actually works.

Prerequisites: Python + basic linear algebra. No prior ML experience required.

Join the Community

See learners building ML systems worldwide

Add yourself to the map • Share your progress • Connect with builders

Join the Discussion → ✉ Subscribe

Next Steps: Quick Start (15 min) • The Big Picture • Community