mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-05-31 21:37:54 -05:00
5386b58e07
Major Educational Framework Enhancements: • Deploy interactive NBGrader text response questions across ALL modules • Replace passive question lists with active 150-300 word student responses • Enable comprehensive ML Systems learning assessment and grading TinyGPT Integration (Module 16): • Complete TinyGPT implementation showing 70% component reuse from TinyTorch • Demonstrates vision-to-language framework generalization principles • Full transformer architecture with attention, tokenization, and generation • Shakespeare demo showing autoregressive text generation capabilities Module Structure Standardization: • Fix section ordering across all modules: Tests → Questions → Summary • Ensure Module Summary is always the final section for consistency • Standardize comprehensive testing patterns before educational content Interactive Question Implementation: • 3 focused questions per module replacing 10-15 passive questions • NBGrader integration with manual grading workflow for text responses • Questions target ML Systems thinking: scaling, deployment, optimization • Cumulative knowledge building across the 16-module progression Technical Infrastructure: • TPM agent for coordinated multi-agent development workflows • Enhanced documentation with pedagogical design principles • Updated book structure to include TinyGPT as capstone demonstration • Comprehensive QA validation of all module structures Framework Design Insights: • Mathematical unity: Dense layers power both vision and language models • Attention as key innovation for sequential relationship modeling • Production-ready patterns: training loops, optimization, evaluation • System-level thinking: memory, performance, scaling considerations Educational Impact: • Transform passive learning to active engagement through written responses • Enable instructors to assess deep ML Systems understanding • Provide clear progression from foundations to complete language models • Demonstrate real-world framework design principles and trade-offs
🔥 Module: Tensor
📊 Module Info
- Difficulty: ⭐⭐ Intermediate
- Time Estimate: 4-6 hours
- Prerequisites: Setup module
- Next Steps: Activations, Layers
Build the foundation of TinyTorch! This module implements the core Tensor class - the fundamental data structure that powers all neural networks and machine learning operations.
🎯 Learning Objectives
By the end of this module, you will:
- Understand what tensors are and why they're essential for ML
- Implement a complete Tensor class with core operations
- Handle tensor shapes, data types, and memory management efficiently
- Implement element-wise operations and reductions with proper broadcasting
- Have a solid foundation for building neural networks
🧠 Build → Use → Understand
- Build: Complete Tensor class with arithmetic operations, shape management, and reductions
- Use: Create tensors, perform operations, and validate with real data
- Understand: How tensors serve as the foundation for all neural network computations
📚 What You'll Build
Core Tensor Class
# Creating tensors
x = Tensor([[1.0, 2.0], [3.0, 4.0]])
y = Tensor([[0.5, 1.5], [2.5, 3.5]])
# Properties
print(x.shape) # (2, 2)
print(x.size) # 4
print(x.dtype) # float64
# Element-wise operations
z = x + y # Addition
w = x * y # Multiplication
p = x ** 2 # Exponentiation
# Shape manipulation
reshaped = x.reshape(4, 1) # (4, 1)
transposed = x.T # (2, 2) transposed
# Reductions
total = x.sum() # Scalar sum
means = x.mean(axis=0) # Mean along axis
Essential Operations
- Arithmetic: Addition, subtraction, multiplication, division, powers
- Shape management: Reshape, transpose, broadcasting rules
- Reductions: Sum, mean, min, max along any axis
- Memory handling: Efficient data storage and copying
🚀 Getting Started
Prerequisites Check
tito test --module setup # Should pass ✅
Development Workflow
# Navigate to tensor module
cd modules/source/02_tensor
# Open development file
jupyter notebook tensor_dev.ipynb
# OR edit directly: code tensor_dev.py
Step-by-Step Implementation
- Basic Tensor class - Constructor and properties
- Shape management - Understanding tensor dimensions
- Arithmetic operations - Addition, multiplication, etc.
- Utility methods - Reshape, transpose, sum, mean
- Error handling - Robust edge case management
🧪 Testing Your Implementation
Inline Testing
# Test in the notebook or Python REPL
x = Tensor([[1.0, 2.0], [3.0, 4.0]])
print(f"Shape: {x.shape}") # Should be (2, 2)
print(f"Sum: {x.sum()}") # Should be 10.0
Module Tests
# Export your tensor implementation
tito export
# Test your implementation
tito test --module tensor
Manual Verification
# Create and test tensors
from tinytorch.core.tensor import Tensor
x = Tensor([1, 2, 3, 4, 5])
y = Tensor([2, 4, 6, 8, 10])
# Test operations
assert (x + y).data.tolist() == [3, 6, 9, 12, 15]
assert (x * 2).data.tolist() == [2, 4, 6, 8, 10]
print("✅ Basic operations working!")
🎯 Key Concepts
Tensors as Universal Data Structures
- Scalars: 0-dimensional tensors (single numbers)
- Vectors: 1-dimensional tensors (arrays)
- Matrices: 2-dimensional tensors (common in ML)
- Higher dimensions: Images (3D), video (4D), etc.
Why Tensors Matter in ML
- Neural networks: All computations operate on tensors
- GPU acceleration: operates on tensor primitives
- Broadcasting: Efficient operations across different shapes
- Vectorization: Process entire datasets simultaneously
Real-World Connections
- PyTorch/TensorFlow: Your implementation mirrors production frameworks
- NumPy: Foundation for scientific computing (we build similar abstractions)
- Production systems: Understanding tensors is essential for ML engineering
Memory and Performance
- Data layout: How tensors store data efficiently
- Broadcasting: Smart operations without data copying
- View vs Copy: Understanding memory management
🎉 Ready to Build?
The tensor module is where TinyTorch really begins. You're about to create the fundamental building block that will power neural networks, training loops, and production ML systems.
Take your time, test thoroughly, and enjoy building something that really works! 🔥