Vijay Janapa Reddi
97e0563614
- Implement tito benchmark baseline and capstone commands - Add SPEC-style normalization for baseline benchmarks - Implement tito community join, update, leave, stats, profile commands - Use project-local storage (.tinytorch/) for user data - Add privacy-by-design with explicit consent prompts - Update site documentation for community and benchmark features - Add Marimo integration for online notebooks - Clean up redundant milestone setup exploration docs - Finalize baseline design: fast setup validation (~1 second) with normalized results
3.3 KiB
Expert Opinion Request: Setup Validation Approach
Question for ML Systems Experts
Context: We're building TinyTorch, an educational ML framework where students build ML components from scratch (tensors, autograd, optimizers, CNNs, transformers, etc.).
Decision Point: How should we validate setup and create baseline results?
Two Approaches
Approach 1: Quick Baseline Benchmark (Current)
What: Run lightweight benchmarks (tensor ops, matrix multiply, forward pass) - ~1 second
Pros:
- ✅ Fast setup validation
- ✅ Doesn't require student code
- ✅ Normalized to reference system (SPEC-style)
- ✅ Simple and reliable
Cons:
- ❌ Limited validation (just basic ops)
- ❌ Not comprehensive
- ❌ Doesn't test full ML workflows
Approach 2: Milestone-Based Validation (Proposed)
What: Run full milestone scripts with reference implementation fallback (PyTorch if tinytorch.* unavailable)
Pros:
- ✅ Comprehensive validation (full ML workflows)
- ✅ Meaningful baseline results (real milestone performance)
- ✅ Better "Hello World" moment (students see what they'll build)
- ✅ Fair comparison (everyone runs same reference)
Cons:
- ⚠️ More complex (requires fallback logic)
- ⚠️ Takes longer (minutes vs seconds)
- ⚠️ Requires modifying milestones
Technical Implementation
Reference Fallback Approach:
# In milestone scripts
try:
from tinytorch import Tensor, Linear, ReLU
implementation = "student"
except ImportError:
import torch
Tensor = torch.Tensor
Linear = torch.nn.Linear
ReLU = torch.nn.ReLU
implementation = "reference"
Results:
- Setup: "Reference baseline: 95% accuracy"
- Later: "Your code: 92% accuracy (vs reference: 95%)"
Questions for Experts
-
Setup Validation: Should setup validation be quick (basic ops) or comprehensive (full workflows)?
-
Reference Implementation: Is it appropriate to use PyTorch as reference fallback in educational frameworks?
-
Baseline Results: Should baseline be environment-only or framework-level (milestone results)?
-
Student Experience: What creates better "Hello World" moment - quick validation or seeing real results?
-
Best Practices: What do successful educational ML frameworks (Fast.ai, PyTorch Lightning tutorials) do?
-
Normalization: Should we normalize milestone results to reference system (like SPEC)?
-
Complexity Trade-off: Is added complexity worth comprehensive validation?
Our Context
- Educational Focus: Students build everything from scratch
- 20 Modules: Progressive complexity (tensors → transformers)
- 6 Milestones: Historical recreations (1957-2018)
- Community Goal: Students feel part of global cohort
What We're Seeking
Expert opinion on:
- Which approach is better for educational frameworks?
- Is reference fallback appropriate?
- Should setup be quick or comprehensive?
- What creates best student experience?
Recommendations on:
- Best practices from industry (MLPerf, SPEC)
- What successful educational frameworks do
- How to balance simplicity vs comprehensiveness
Contact
We'd love feedback from:
- MLPerf/SPEC benchmark experts
- Educational ML framework developers
- ML systems engineers with educational experience
Thank you for your expertise!