TinyTorch/binder/COMMUNITY_BENCHMARK_DESIGN.md

Community Benchmark & "Hello World" Experience Design

Goal: First Success Moment

Create an immediate "wow, I did it!" moment where students:

1. ✅ Clone and setup TinyTorch
2. ✅ Run all tests (validate installation)
3. ✅ Run milestones (validate their implementation)
4. 🎉 Get benchmark score and join the community

User Journey Flow

Clone & Setup
    ↓
tito system doctor (verify setup)
    ↓
tito milestone validate --all (run all milestones)
    ↓
tito benchmark baseline (generate benchmark score)
    ↓
🎉 "Welcome to TinyTorch Community!"
    ↓
[Optional] Upload to leaderboard

Implementation Design

1. Baseline Benchmark Command

Command: tito benchmark baseline

What it does:

• Runs a set of lightweight benchmarks (not full module 20)
• Tests basic operations: tensor creation, matrix multiplication, simple forward pass
• Measures: execution time, memory usage, basic throughput
• Generates JSON with results

When to run:

• After tito system doctor passes
• After tito milestone validate --all passes
• Can be run anytime to check baseline

2. Benchmark JSON Structure

{
  "timestamp": "2024-11-20T10:30:00Z",
  "version": "1.0.0",
  "system": {
    "platform": "darwin",
    "python_version": "3.11.0",
    "numpy_version": "1.24.0",
    "cpu_count": 8,
    "memory_gb": 16
  },
  "baseline_benchmarks": {
    "tensor_creation": {
      "time_ms": 0.5,
      "memory_mb": 0.1
    },
    "matrix_multiply": {
      "time_ms": 2.3,
      "throughput_ops_per_sec": 434.78
    },
    "simple_forward_pass": {
      "time_ms": 5.2,
      "memory_mb": 2.5
    }
  },
  "milestone_status": {
    "milestone_01_perceptron": "passed",
    "milestone_02_xor": "passed",
    "milestone_03_mlp": "passed"
  },
  "setup_validated": true,
  "all_tests_passed": true
}

3. Upload/Submission System

Command: tito benchmark submit [--public]

What it does:

• Uploads benchmark JSON to server
• Gets back: community rank, percentile, badge
• Optional: make public on leaderboard

Server endpoint (to be created):

• POST /api/benchmarks/submit
• Returns: { "rank": 1234, "percentile": 75, "badge": "🚀 First Steps" }

4. Community Leaderboard

Features:

• Public leaderboard (optional participation)
• Shows: rank, percentile, system info, timestamp
• Filterable by: system type, date, milestone status
• Badges: "🚀 First Steps", "⚡ Fast Setup", "🏆 All Milestones"

5. "Hello World" Experience

After tito benchmark baseline:

🎉 Congratulations! You've successfully set up TinyTorch!

📊 Your Baseline Performance:
   • Tensor Operations: ⚡ Fast (0.5ms)
   • Matrix Multiply: ⚡ Fast (2.3ms)
   • Forward Pass: ⚡ Fast (5.2ms)

✅ Milestones Validated: 3/6 passed

🌍 Join the Community:
   Run 'tito benchmark submit' to share your results
   and see how you compare to others worldwide!

📈 Your Score: 85/100
   You're in the top 25% of TinyTorch users!

🚀 Next Steps:
   • Continue building modules
   • Run 'tito benchmark baseline' anytime
   • Complete all milestones for full score

Implementation Steps

Phase 1: Baseline Benchmark (Core)

1. Create tito/commands/benchmark.py:

   • tito benchmark baseline - Run benchmarks, generate JSON
   • tito benchmark submit - Upload to server (optional)

2. Benchmark Suite:

   • Lightweight tests (don't require all modules)
   • Basic tensor operations
   • Simple forward pass
   • Memory profiling

3. JSON Generation:

   • Save to benchmarks/baseline_YYYYMMDD_HHMMSS.json
   • Include system info, benchmark results, milestone status

Phase 2: Server Integration

1. API Endpoint:

   • Simple REST API
   • Accepts benchmark JSON
   • Returns rank/percentile/badge
   • Stores in database

2. Leaderboard:

   • Public web page
   • Shows rankings
   • Filterable/searchable

Phase 3: Community Features

1. Badges:

   • "🚀 First Steps" - Completed baseline
   • "⚡ Fast Setup" - Top 10% performance
   • "🏆 All Milestones" - All milestones passed
   • "🌍 Community Member" - Submitted to leaderboard

2. Sharing:

   • Generate shareable image/card
   • "I just set up TinyTorch! Score: 85/100"
   • Link to leaderboard

Technical Considerations

Benchmark Design

Keep it lightweight:

• Don't require all modules
• Use basic operations only
• Fast execution (< 30 seconds)
• Works after setup + milestone validation

What to benchmark:

• Tensor creation speed
• Matrix multiplication throughput
• Simple forward pass (2-layer network)
• Memory efficiency
• Basic autograd operations

Privacy & Opt-in

• Default: Benchmarks saved locally only
• Optional: --public flag to share
• Anonymized: System info only (no personal data)
• Consent: Clear messaging about what's shared

Server Architecture

Simple approach:

• Static JSON file storage (GitHub Pages?)
• Or simple API (Flask/FastAPI)
• Database: SQLite or PostgreSQL
• Leaderboard: Static site generator

More advanced:

• Real-time leaderboard
• User accounts (optional)
• Historical tracking
• Regional comparisons

User Experience Flow

First Time Setup

# 1. Clone and setup
git clone https://github.com/mlsysbook/TinyTorch.git
cd TinyTorch
./setup-environment.sh
source activate.sh

# 2. Verify setup
tito system doctor
# ✅ All checks passed!

# 3. Run milestones (if modules completed)
tito milestone validate --all
# ✅ Milestone 01: Perceptron - PASSED
# ✅ Milestone 02: XOR - PASSED
# ✅ Milestone 03: MLP - PASSED

# 4. Generate baseline benchmark
tito benchmark baseline
# 🎉 Congratulations! You've successfully set up TinyTorch!
# 📊 Your Baseline Performance: 85/100
# 🌍 Run 'tito benchmark submit' to join the community!

# 5. (Optional) Submit to leaderboard
tito benchmark submit --public
# ✅ Submitted! You're rank #1234 (top 25%)
# 🔗 View leaderboard: https://tinytorch.ai/leaderboard

Benefits

1. Immediate Gratification: "I did it!" moment
2. Community Feeling: Part of something bigger
3. Motivation: See how they compare
4. Validation: Confirms setup worked
5. Progress Tracking: Can re-run anytime

Next Steps

1. Design benchmark suite (what to test)
2. Implement tito benchmark baseline command
3. Create JSON schema
4. Design server API (or use GitHub Pages)
5. Build leaderboard page
6. Add badges/sharing features

This creates a "hello world" experience that makes students feel successful and part of the community immediately!