mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-04-27 23:57:31 -05:00
832c569cad
Added all module development files to modules/XX_name/ directories:
Module notebooks and scripts:
- 18 modules with .ipynb and .py files (01-20, excluding some gaps)
- Moved from modules/source/ to direct module directories
- Includes tensor, autograd, layers, transformers, optimization modules
Module README files:
- Added README.md for modules with additional documentation
- Complements ABOUT.md files added earlier
This completes the module restructuring:
- Before: modules/source/XX_name/*_dev.{py,ipynb}
- After: modules/XX_name/*_dev.{py,ipynb}
All development happens directly in numbered module directories now.
1084 lines
44 KiB
Plaintext
1084 lines
44 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aabba6c2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| default_exp competition.submit"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b5222d75",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"# Module 20: TinyMLPerf Competition - Your Capstone Challenge\n",
|
|
"\n",
|
|
"Welcome to the capstone! You've built an entire ML system from scratch (M01-13) and learned optimization techniques (M14-19). Now it's time to compete and show what you can do! 🏅\n",
|
|
"\n",
|
|
"## 🔗 Your Journey\n",
|
|
"```\n",
|
|
"Modules 01-13: Build ML System (tensors → transformers)\n",
|
|
"Modules 14-18: Learn Optimization Techniques \n",
|
|
"Module 19: Learn Benchmarking\n",
|
|
"Module 20: Compete in TinyMLPerf! 🏅\n",
|
|
"```\n",
|
|
"\n",
|
|
"## 🏅 TinyMLPerf: Two Ways to Compete\n",
|
|
"\n",
|
|
"Inspired by industry-standard MLPerf (which you learned about in Module 19), TinyMLPerf offers **two competition tracks**:\n",
|
|
"\n",
|
|
"### 🔒 Closed Division - \"Optimization Challenge\"\n",
|
|
"**What you do:**\n",
|
|
"- Start with provided baseline model (everyone gets the same)\n",
|
|
"- Apply optimization techniques from Modules 14-18\n",
|
|
"- Compete on: Who optimizes best?\n",
|
|
"\n",
|
|
"**Best for:** Most students - clear rules, fair comparison\n",
|
|
"**Focus:** Your optimization skills\n",
|
|
"\n",
|
|
"### 🔓 Open Division - \"Innovation Challenge\" \n",
|
|
"**What you do:**\n",
|
|
"- Modify anything! Improve your implementations from M01-19\n",
|
|
"- Design better architectures\n",
|
|
"- Novel approaches encouraged\n",
|
|
"\n",
|
|
"**Best for:** Advanced students who want more creative freedom\n",
|
|
"**Focus:** Your systems innovations\n",
|
|
"\n",
|
|
"## Competition Categories (Both Divisions)\n",
|
|
"- 🏃 **Latency Sprint**: Fastest inference\n",
|
|
"- 🏋️ **Memory Challenge**: Smallest model\n",
|
|
"- 🎯 **Accuracy Contest**: Best accuracy within constraints\n",
|
|
"- 🏋️♂️ **All-Around**: Best balanced performance\n",
|
|
"- 🚀 **Extreme Push**: Most aggressive optimization\n",
|
|
"\n",
|
|
"## What This Module Provides\n",
|
|
"1. **Validation**: Check your TinyTorch works\n",
|
|
"2. **Baseline**: Starting point for Closed Division\n",
|
|
"3. **Examples**: See both tracks in action\n",
|
|
"4. **Template**: Your competition workspace\n",
|
|
"\n",
|
|
"Pick your track, optimize, and compete! 🔥"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8bbad866",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 📦 Where This Code Lives in the Final Package\n",
|
|
"\n",
|
|
"**Learning Side:** You work in `modules/20_competition/competition_dev.py` \n",
|
|
"**Building Side:** Code exports to `tinytorch.competition.submit`\n",
|
|
"\n",
|
|
"```python\n",
|
|
"# Validation and baseline tools:\n",
|
|
"from tinytorch.competition.submit import validate_installation, generate_baseline\n",
|
|
"\n",
|
|
"# Competition helpers:\n",
|
|
"from tinytorch.competition.submit import load_baseline_model, generate_submission\n",
|
|
"```\n",
|
|
"\n",
|
|
"**Why this matters:**\n",
|
|
"- **Validation:** Ensures your TinyTorch installation works correctly\n",
|
|
"- **Baseline:** Establishes reference performance for fair comparison\n",
|
|
"- **Competition:** Provides standardized framework for submissions\n",
|
|
"- **Integration:** Brings together all 19 modules into one complete workflow"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a56c298b",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"# 1. Pick Your Track & Validate\n",
|
|
"\n",
|
|
"Before competing, choose your track and make sure your TinyTorch installation works!\n",
|
|
"\n",
|
|
"## Two Tracks, Two Styles\n",
|
|
"\n",
|
|
"### 🔒 Closed Division - \"The Optimization Challenge\"\n",
|
|
"- Everyone starts with the same baseline model\n",
|
|
"- Apply techniques from Modules 14-18 (quantization, pruning, etc.)\n",
|
|
"- Fair comparison: who optimizes best?\n",
|
|
"- **Choose this if:** You want clear rules and direct competition\n",
|
|
"\n",
|
|
"### 🔓 Open Division - \"The Innovation Challenge\"\n",
|
|
"- Modify anything! Improve YOUR TinyTorch implementations\n",
|
|
"- Better Conv2d? Faster matmul? Novel architecture? All allowed!\n",
|
|
"- Compete on innovation and creativity\n",
|
|
"- **Choose this if:** You want freedom to explore and innovate\n",
|
|
"\n",
|
|
"**Can I do both?** Absolutely! Submit to both tracks.\n",
|
|
"\n",
|
|
"**Which is \"better\"?** Neither - they test different skills:\n",
|
|
"- Closed = Optimization mastery\n",
|
|
"- Open = Systems innovation\n",
|
|
"\n",
|
|
"## Quick Validation\n",
|
|
"\n",
|
|
"Before competing, let's verify everything works:\n",
|
|
"- ✅ All modules imported successfully\n",
|
|
"- ✅ Optimization techniques available\n",
|
|
"- ✅ Benchmarking tools ready"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4748e00b",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"import numpy as np\n",
|
|
"import json\n",
|
|
"import time\n",
|
|
"from pathlib import Path\n",
|
|
"from typing import Dict, List, Tuple, Any, Optional\n",
|
|
"from tinytorch.benchmarking.benchmark import Benchmark, calculate_normalized_scores\n",
|
|
"from tinytorch.profiling.profiler import Profiler\n",
|
|
"\n",
|
|
"def validate_installation() -> Dict[str, bool]:\n",
|
|
" \"\"\"\n",
|
|
" Validate TinyTorch installation and return status of each component.\n",
|
|
" \n",
|
|
" Returns:\n",
|
|
" Dictionary mapping module names to validation status (True = working)\n",
|
|
" \n",
|
|
" Example:\n",
|
|
" >>> status = validate_installation()\n",
|
|
" >>> print(status)\n",
|
|
" {'tensor': True, 'autograd': True, 'layers': True, ...}\n",
|
|
" \"\"\"\n",
|
|
" validation_results = {}\n",
|
|
" \n",
|
|
" print(\"🔧 Validating TinyTorch Installation...\")\n",
|
|
" print(\"=\" * 60)\n",
|
|
" \n",
|
|
" # Core modules (M01-13)\n",
|
|
" core_modules = [\n",
|
|
" (\"tensor\", \"tinytorch.core.tensor\", \"Tensor\"),\n",
|
|
" (\"autograd\", \"tinytorch.core.autograd\", \"enable_autograd\"),\n",
|
|
" (\"layers\", \"tinytorch.core.layers\", \"Linear\"),\n",
|
|
" (\"activations\", \"tinytorch.core.activations\", \"ReLU\"),\n",
|
|
" (\"losses\", \"tinytorch.core.training\", \"MSELoss\"),\n",
|
|
" (\"optimizers\", \"tinytorch.core.optimizers\", \"SGD\"),\n",
|
|
" (\"spatial\", \"tinytorch.core.spatial\", \"Conv2d\"),\n",
|
|
" (\"attention\", \"tinytorch.core.attention\", \"MultiHeadAttention\"),\n",
|
|
" (\"transformers\", \"tinytorch.models.transformer\", \"GPT\"),\n",
|
|
" ]\n",
|
|
" \n",
|
|
" for name, module_path, class_name in core_modules:\n",
|
|
" try:\n",
|
|
" exec(f\"from {module_path} import {class_name}\")\n",
|
|
" validation_results[name] = True\n",
|
|
" print(f\"✅ {name.capitalize()}: Working\")\n",
|
|
" except Exception as e:\n",
|
|
" validation_results[name] = False\n",
|
|
" print(f\"❌ {name.capitalize()}: Failed - {str(e)}\")\n",
|
|
" \n",
|
|
" # Optimization modules (M14-18)\n",
|
|
" opt_modules = [\n",
|
|
" (\"kv_caching\", \"tinytorch.generation.kv_cache\", \"enable_kv_cache\"),\n",
|
|
" (\"profiling\", \"tinytorch.profiling.profiler\", \"Profiler\"),\n",
|
|
" (\"quantization\", \"tinytorch.optimization.quantization\", \"quantize_model\"),\n",
|
|
" (\"compression\", \"tinytorch.optimization.compression\", \"magnitude_prune\"),\n",
|
|
" ]\n",
|
|
" \n",
|
|
" for name, module_path, func_name in opt_modules:\n",
|
|
" try:\n",
|
|
" exec(f\"from {module_path} import {func_name}\")\n",
|
|
" validation_results[name] = True\n",
|
|
" print(f\"✅ {name.replace('_', ' ').capitalize()}: Working\")\n",
|
|
" except Exception as e:\n",
|
|
" validation_results[name] = False\n",
|
|
" print(f\"❌ {name.replace('_', ' ').capitalize()}: Failed - {str(e)}\")\n",
|
|
" \n",
|
|
" # Benchmarking (M19)\n",
|
|
" try:\n",
|
|
" from tinytorch.benchmarking.benchmark import Benchmark, OlympicEvent\n",
|
|
" validation_results[\"benchmarking\"] = True\n",
|
|
" print(f\"✅ Benchmarking: Working\")\n",
|
|
" except Exception as e:\n",
|
|
" validation_results[\"benchmarking\"] = False\n",
|
|
" print(f\"❌ Benchmarking: Failed - {str(e)}\")\n",
|
|
" \n",
|
|
" print(\"=\" * 60)\n",
|
|
" \n",
|
|
" # Summary\n",
|
|
" total = len(validation_results)\n",
|
|
" working = sum(validation_results.values())\n",
|
|
" \n",
|
|
" if working == total:\n",
|
|
" print(f\"🎉 Perfect! All {total}/{total} modules working!\")\n",
|
|
" print(\"✅ You're ready to compete in TorchPerf Olympics!\")\n",
|
|
" else:\n",
|
|
" print(f\"⚠️ {working}/{total} modules working\")\n",
|
|
" print(f\"❌ {total - working} modules need attention\")\n",
|
|
" print(\"\\nPlease run: pip install -e . (in TinyTorch root)\")\n",
|
|
" \n",
|
|
" return validation_results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "190e1466",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"# 2. The Baseline (For Closed Division)\n",
|
|
"\n",
|
|
"If you're competing in **Closed Division**, everyone starts with this baseline model. If you're in **Open Division**, you can skip this or use it as a reference!\n",
|
|
"\n",
|
|
"## Baseline Model: Simple CNN on CIFAR-10\n",
|
|
"\n",
|
|
"We provide a simple CNN as the starting point for Closed Division:\n",
|
|
"- **Architecture:** Conv → Pool → Conv → Pool → FC → FC\n",
|
|
"- **Dataset:** CIFAR-10 (standardized test set)\n",
|
|
"- **Metrics:** Accuracy, latency, memory (we'll measure together)\n",
|
|
"\n",
|
|
"**Closed Division:** Optimize THIS model using M14-18 techniques\n",
|
|
"**Open Division:** Build/modify whatever you want!\n",
|
|
"\n",
|
|
"### Baseline Components\n",
|
|
"\n",
|
|
"1. **Model:** Standard CNN (no optimizations)\n",
|
|
"2. **Metrics:** Accuracy, latency, memory, parameters\n",
|
|
"3. **Test Data:** CIFAR-10 test set (standardized)\n",
|
|
"4. **Hardware:** Your local machine (reported for reproducibility)\n",
|
|
"\n",
|
|
"The baseline establishes what \"unoptimized\" looks like. Your job: beat it!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ff944a6c",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def load_baseline_model(model_name: str = \"cifar10_cnn\"):\n",
|
|
" \"\"\"\n",
|
|
" Load a baseline model for TorchPerf Olympics competition.\n",
|
|
" \n",
|
|
" Args:\n",
|
|
" model_name: Name of baseline model to load\n",
|
|
" - \"cifar10_cnn\": Simple CNN for CIFAR-10 classification\n",
|
|
" \n",
|
|
" Returns:\n",
|
|
" Baseline model instance\n",
|
|
" \n",
|
|
" Example:\n",
|
|
" >>> model = load_baseline_model(\"cifar10_cnn\")\n",
|
|
" >>> print(f\"Parameters: {sum(p.size for p in model.parameters())}\")\n",
|
|
" \"\"\"\n",
|
|
" from tinytorch.core.layers import Linear\n",
|
|
" from tinytorch.core.spatial import Conv2d, MaxPool2d, Flatten\n",
|
|
" from tinytorch.core.activations import ReLU\n",
|
|
" \n",
|
|
" if model_name == \"cifar10_cnn\":\n",
|
|
" # Simple CNN: Conv -> Pool -> Conv -> Pool -> FC -> FC\n",
|
|
" class BaselineCNN:\n",
|
|
" def __init__(self):\n",
|
|
" self.name = \"Baseline_CIFAR10_CNN\"\n",
|
|
" \n",
|
|
" # Convolutional layers\n",
|
|
" self.conv1 = Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)\n",
|
|
" self.relu1 = ReLU()\n",
|
|
" self.pool1 = MaxPool2d(kernel_size=2, stride=2)\n",
|
|
" \n",
|
|
" self.conv2 = Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)\n",
|
|
" self.relu2 = ReLU()\n",
|
|
" self.pool2 = MaxPool2d(kernel_size=2, stride=2)\n",
|
|
" \n",
|
|
" # Fully connected layers\n",
|
|
" self.flatten = Flatten()\n",
|
|
" self.fc1 = Linear(64 * 8 * 8, 128)\n",
|
|
" self.relu3 = ReLU()\n",
|
|
" self.fc2 = Linear(128, 10) # 10 classes for CIFAR-10\n",
|
|
" \n",
|
|
" def forward(self, x):\n",
|
|
" # Forward pass\n",
|
|
" x = self.conv1.forward(x)\n",
|
|
" x = self.relu1.forward(x)\n",
|
|
" x = self.pool1.forward(x)\n",
|
|
" \n",
|
|
" x = self.conv2.forward(x)\n",
|
|
" x = self.relu2.forward(x)\n",
|
|
" x = self.pool2.forward(x)\n",
|
|
" \n",
|
|
" x = self.flatten.forward(x)\n",
|
|
" x = self.fc1.forward(x)\n",
|
|
" x = self.relu3.forward(x)\n",
|
|
" x = self.fc2.forward(x)\n",
|
|
" \n",
|
|
" return x\n",
|
|
" \n",
|
|
" def __call__(self, x):\n",
|
|
" return self.forward(x)\n",
|
|
" \n",
|
|
" return BaselineCNN()\n",
|
|
" else:\n",
|
|
" raise ValueError(f\"Unknown baseline model: {model_name}\")\n",
|
|
"\n",
|
|
"def generate_baseline(model_name: str = \"cifar10_cnn\", quick: bool = True) -> Dict[str, Any]:\n",
|
|
" \"\"\"\n",
|
|
" Generate baseline performance metrics for a model.\n",
|
|
" \n",
|
|
" Args:\n",
|
|
" model_name: Name of baseline model\n",
|
|
" quick: If True, use quick estimates instead of full benchmarks\n",
|
|
" \n",
|
|
" Returns:\n",
|
|
" Baseline scorecard with metrics\n",
|
|
" \n",
|
|
" Example:\n",
|
|
" >>> baseline = generate_baseline(\"cifar10_cnn\", quick=True)\n",
|
|
" >>> print(f\"Baseline latency: {baseline['latency_ms']}ms\")\n",
|
|
" \"\"\"\n",
|
|
" print(\"📊 Generating Baseline Scorecard...\")\n",
|
|
" print(\"=\" * 60)\n",
|
|
" \n",
|
|
" # Load model\n",
|
|
" model = load_baseline_model(model_name)\n",
|
|
" print(f\"✅ Loaded baseline model: {model.name}\")\n",
|
|
" \n",
|
|
" # Count parameters\n",
|
|
" def count_parameters(model):\n",
|
|
" total = 0\n",
|
|
" for attr_name in dir(model):\n",
|
|
" attr = getattr(model, attr_name)\n",
|
|
" if hasattr(attr, 'weights') and attr.weights is not None:\n",
|
|
" total += attr.weights.size\n",
|
|
" if hasattr(attr, 'bias') and attr.bias is not None:\n",
|
|
" total += attr.bias.size\n",
|
|
" return total\n",
|
|
" \n",
|
|
" params = count_parameters(model)\n",
|
|
" memory_mb = params * 4 / (1024 * 1024) # Assuming float32\n",
|
|
" \n",
|
|
" if quick:\n",
|
|
" # Quick estimates for fast validation\n",
|
|
" print(\"⚡ Using quick estimates (set quick=False for full benchmark)\")\n",
|
|
" \n",
|
|
" baseline = {\n",
|
|
" \"model\": model_name,\n",
|
|
" \"accuracy\": 85.0, # Typical for this architecture\n",
|
|
" \"latency_ms\": 45.2,\n",
|
|
" \"memory_mb\": memory_mb,\n",
|
|
" \"parameters\": params,\n",
|
|
" \"mode\": \"quick_estimate\"\n",
|
|
" }\n",
|
|
" else:\n",
|
|
" # Full benchmark (requires more time)\n",
|
|
" from tinytorch.benchmarking.benchmark import Benchmark\n",
|
|
" \n",
|
|
" print(\"🔬 Running full benchmark (this may take a minute)...\")\n",
|
|
" \n",
|
|
" benchmark = Benchmark([model], [{\"name\": \"baseline\"}], \n",
|
|
" warmup_runs=5, measurement_runs=20)\n",
|
|
" \n",
|
|
" # Measure latency\n",
|
|
" input_shape = (1, 3, 32, 32) # CIFAR-10 input\n",
|
|
" latency_results = benchmark.run_latency_benchmark(input_shape=input_shape)\n",
|
|
" latency_ms = list(latency_results.values())[0].mean * 1000\n",
|
|
" \n",
|
|
" baseline = {\n",
|
|
" \"model\": model_name,\n",
|
|
" \"accuracy\": 85.0, # Would need actual test set evaluation\n",
|
|
" \"latency_ms\": latency_ms,\n",
|
|
" \"memory_mb\": memory_mb,\n",
|
|
" \"parameters\": params,\n",
|
|
" \"mode\": \"full_benchmark\"\n",
|
|
" }\n",
|
|
" \n",
|
|
" # Display baseline\n",
|
|
" print(\"\\n📋 BASELINE SCORECARD\")\n",
|
|
" print(\"=\" * 60)\n",
|
|
" print(f\"Model: {baseline['model']}\")\n",
|
|
" print(f\"Accuracy: {baseline['accuracy']:.1f}%\")\n",
|
|
" print(f\"Latency: {baseline['latency_ms']:.1f}ms\")\n",
|
|
" print(f\"Memory: {baseline['memory_mb']:.2f}MB\")\n",
|
|
" print(f\"Parameters: {baseline['parameters']:,}\")\n",
|
|
" print(\"=\" * 60)\n",
|
|
" print(\"📌 This is your starting point. Optimize to compete!\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" return baseline"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fdef4b17",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"# 3. Complete Example - See Both Tracks in Action\n",
|
|
"\n",
|
|
"Let's see complete examples for BOTH competition tracks!\n",
|
|
"\n",
|
|
"## Example 1: Closed Division - Optimization Master\n",
|
|
"\n",
|
|
"**Goal:** Compete in All-Around category using provided baseline\n",
|
|
"\n",
|
|
"**Strategy:**\n",
|
|
"1. Load baseline CNN\n",
|
|
"2. Apply quantization (INT8) → 4x memory reduction\n",
|
|
"3. Apply pruning (60%) → Speed boost\n",
|
|
"4. Benchmark and submit\n",
|
|
"\n",
|
|
"**Why this order?** Quantize first preserves more accuracy than pruning first.\n",
|
|
"\n",
|
|
"## Example 2: Open Division - Innovation Master\n",
|
|
"\n",
|
|
"**Goal:** Beat everyone with a novel approach\n",
|
|
"\n",
|
|
"**Strategy:**\n",
|
|
"1. Improve YOUR Conv2d implementation (faster algorithm)\n",
|
|
"2. OR design a better architecture (MobileNet-style)\n",
|
|
"3. OR novel quantization (mixed precision per layer)\n",
|
|
"4. Benchmark and submit\n",
|
|
"\n",
|
|
"**Freedom:** Modify anything in your TinyTorch implementation!\n",
|
|
"\n",
|
|
"Let's see the Closed Division example in detail below:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4a5e4560",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def worked_example_optimization():\n",
|
|
" \"\"\"\n",
|
|
" Complete worked example showing full optimization workflow.\n",
|
|
" \n",
|
|
" This demonstrates:\n",
|
|
" - Loading baseline model\n",
|
|
" - Applying multiple optimization techniques\n",
|
|
" - Benchmarking systematically\n",
|
|
" - Generating submission\n",
|
|
" \n",
|
|
" Students should study this and adapt for their own strategies!\n",
|
|
" \"\"\"\n",
|
|
" print(\"🏅 WORKED EXAMPLE: Complete Optimization Workflow\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" print(\"Target: All-Around Event (balanced performance)\")\n",
|
|
" print(\"Strategy: Quantization (INT8) → Pruning (60%)\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Step 1: Load Baseline\n",
|
|
" print(\"📦 Step 1: Load Baseline Model\")\n",
|
|
" print(\"-\" * 70)\n",
|
|
" baseline = load_baseline_model(\"cifar10_cnn\")\n",
|
|
" baseline_metrics = generate_baseline(\"cifar10_cnn\", quick=True)\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Step 2: Apply Quantization\n",
|
|
" print(\"🔧 Step 2: Apply INT8 Quantization (Module 17)\")\n",
|
|
" print(\"-\" * 70)\n",
|
|
" print(\"💡 Why quantize? Reduces memory 4x (FP32 → INT8)\")\n",
|
|
" \n",
|
|
" # For demonstration, we'll simulate quantization\n",
|
|
" # In real competition, students would use:\n",
|
|
" # from tinytorch.optimization.quantization import quantize_model\n",
|
|
" # optimized = quantize_model(baseline, bits=8)\n",
|
|
" \n",
|
|
" print(\"✅ Quantized model (simulated)\")\n",
|
|
" print(\" - Memory: 12.4MB → 3.1MB (4x reduction)\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Step 3: Apply Pruning\n",
|
|
" print(\"✂️ Step 3: Apply Magnitude Pruning (Module 18)\")\n",
|
|
" print(\"-\" * 70)\n",
|
|
" print(\"💡 Why prune? Removes 60% of weights for faster inference\")\n",
|
|
" \n",
|
|
" # For demonstration, we'll simulate pruning\n",
|
|
" # In real competition, students would use:\n",
|
|
" # from tinytorch.optimization.compression import magnitude_prune\n",
|
|
" # optimized = magnitude_prune(optimized, sparsity=0.6)\n",
|
|
" \n",
|
|
" print(\"✅ Pruned model (simulated)\")\n",
|
|
" print(\" - Active parameters: 3.2M → 1.28M (60% removed)\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Step 4: Benchmark Results\n",
|
|
" print(\"📊 Step 4: Benchmark Optimized Model (Module 19)\")\n",
|
|
" print(\"-\" * 70)\n",
|
|
" \n",
|
|
" # Simulated optimized metrics\n",
|
|
" optimized_metrics = {\n",
|
|
" \"model\": \"Optimized_CIFAR10_CNN\",\n",
|
|
" \"accuracy\": 83.5, # Slight drop from aggressive optimization\n",
|
|
" \"latency_ms\": 22.1,\n",
|
|
" \"memory_mb\": 1.24, # 4x quantization + 60% pruning\n",
|
|
" \"parameters\": 1280000,\n",
|
|
" \"techniques\": [\"quantization_int8\", \"magnitude_prune_0.6\"]\n",
|
|
" }\n",
|
|
" \n",
|
|
" print(\"Baseline vs Optimized:\")\n",
|
|
" print(f\" Accuracy: {baseline_metrics['accuracy']:.1f}% → {optimized_metrics['accuracy']:.1f}% (-1.5pp)\")\n",
|
|
" print(f\" Latency: {baseline_metrics['latency_ms']:.1f}ms → {optimized_metrics['latency_ms']:.1f}ms (2.0x faster ✅)\")\n",
|
|
" print(f\" Memory: {baseline_metrics['memory_mb']:.2f}MB → {optimized_metrics['memory_mb']:.2f}MB (10.0x smaller ✅)\")\n",
|
|
" print(f\" Parameters: {baseline_metrics['parameters']:,} → {optimized_metrics['parameters']:,} (60% fewer ✅)\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Step 5: Generate Submission\n",
|
|
" print(\"📤 Step 5: Generate Competition Submission\")\n",
|
|
" print(\"-\" * 70)\n",
|
|
" \n",
|
|
" submission = {\n",
|
|
" \"event\": \"all_around\",\n",
|
|
" \"athlete_name\": \"Example_Submission\",\n",
|
|
" \"baseline\": baseline_metrics,\n",
|
|
" \"optimized\": optimized_metrics,\n",
|
|
" \"improvements\": {\n",
|
|
" \"accuracy_drop\": -1.5,\n",
|
|
" \"latency_speedup\": 2.0,\n",
|
|
" \"memory_reduction\": 10.0\n",
|
|
" },\n",
|
|
" \"techniques_applied\": [\"quantization_int8\", \"magnitude_prune_0.6\"],\n",
|
|
" \"technique_order\": \"quantize_first_then_prune\"\n",
|
|
" }\n",
|
|
" \n",
|
|
" print(\"✅ Submission generated!\")\n",
|
|
" print(f\" Event: {submission['event']}\")\n",
|
|
" print(f\" Techniques: {', '.join(submission['techniques_applied'])}\")\n",
|
|
" print()\n",
|
|
" print(\"=\" * 70)\n",
|
|
" print(\"🎯 This is the complete workflow!\")\n",
|
|
" print(\" Now it's your turn to implement your own optimization strategy.\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" \n",
|
|
" return submission"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b013b5eb",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"# 4. Your Turn - Pick Your Track!\n",
|
|
"\n",
|
|
"Now it's time to compete! Choose your track and implement your strategy.\n",
|
|
"\n",
|
|
"## Choose Your Track\n",
|
|
"\n",
|
|
"### 🔒 Closed Division Template\n",
|
|
"**If you choose Closed Division:**\n",
|
|
"1. Pick a category (Latency Sprint, Memory Challenge, etc.)\n",
|
|
"2. Design your optimization strategy\n",
|
|
"3. Implement in `optimize_for_competition()` below\n",
|
|
"4. Use techniques from Modules 14-18 only\n",
|
|
"5. Generate submission\n",
|
|
"\n",
|
|
"**Good for:** Clear path, fair comparison, most students\n",
|
|
"\n",
|
|
"### 🔓 Open Division Template \n",
|
|
"**If you choose Open Division:**\n",
|
|
"1. Pick a category\n",
|
|
"2. Modify YOUR TinyTorch implementations (go edit earlier modules!)\n",
|
|
"3. OR design novel architectures\n",
|
|
"4. Re-export with `tito export` and benchmark\n",
|
|
"5. Generate submission\n",
|
|
"\n",
|
|
"**Good for:** Creative freedom, systems innovation, advanced students\n",
|
|
"\n",
|
|
"## Competition Categories (Pick ONE)\n",
|
|
"- 🏃 **Latency Sprint:** Fastest inference\n",
|
|
"- 🏋️ **Memory Challenge:** Smallest model\n",
|
|
"- 🎯 **Accuracy Contest:** Best accuracy within constraints\n",
|
|
"- 🏋️♂️ **All-Around:** Best balanced performance\n",
|
|
"- 🚀 **Extreme Push:** Most aggressive optimization\n",
|
|
"\n",
|
|
"## Template Below\n",
|
|
"\n",
|
|
"Use the `optimize_for_competition()` function to implement your strategy:\n",
|
|
"- **Closed Division:** Apply M14-18 techniques\n",
|
|
"- **Open Division:** Do whatever you want, document it!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d51c16c8",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def optimize_for_competition(baseline_model, event: str = \"all_around\", division: str = \"closed\"):\n",
|
|
" \"\"\"\n",
|
|
" 🏅 YOUR COMPETITION ENTRY - IMPLEMENT YOUR STRATEGY HERE!\n",
|
|
" \n",
|
|
" Args:\n",
|
|
" baseline_model: Starting model (use for Closed, optional for Open)\n",
|
|
" event: Category you're competing in\n",
|
|
" - \"latency_sprint\": Minimize latency\n",
|
|
" - \"memory_challenge\": Minimize memory\n",
|
|
" - \"accuracy_contest\": Maximize accuracy\n",
|
|
" - \"all_around\": Best balance\n",
|
|
" - \"extreme_push\": Most aggressive\n",
|
|
" division: \"closed\" or \"open\" - which track you chose\n",
|
|
" \n",
|
|
" Returns:\n",
|
|
" Your optimized model\n",
|
|
" \n",
|
|
" 🔒 CLOSED DIVISION Example:\n",
|
|
" from tinytorch.optimization.quantization import quantize_model\n",
|
|
" from tinytorch.optimization.compression import magnitude_prune\n",
|
|
" \n",
|
|
" optimized = baseline_model\n",
|
|
" optimized = quantize_model(optimized, bits=8)\n",
|
|
" optimized = magnitude_prune(optimized, sparsity=0.7)\n",
|
|
" return optimized\n",
|
|
" \n",
|
|
" 🔓 OPEN DIVISION Example:\n",
|
|
" # Build your own model OR\n",
|
|
" # Use your improved implementations from earlier modules\n",
|
|
" # (after you've modified and re-exported them)\n",
|
|
" \n",
|
|
" from tinytorch.models import YourCustomArchitecture\n",
|
|
" optimized = YourCustomArchitecture()\n",
|
|
" return optimized\n",
|
|
" \"\"\"\n",
|
|
" \n",
|
|
" print(f\"🏅 YOUR OPTIMIZATION STRATEGY FOR: {event}\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" \n",
|
|
" # Start with baseline\n",
|
|
" optimized_model = baseline_model\n",
|
|
" \n",
|
|
" # ============================================================\n",
|
|
" # YOUR CODE BELOW - Apply optimization techniques here!\n",
|
|
" # ============================================================\n",
|
|
" \n",
|
|
" # TODO: Students implement their optimization strategy\n",
|
|
" #\n",
|
|
" # Example strategies by event:\n",
|
|
" #\n",
|
|
" # Latency Sprint (speed priority):\n",
|
|
" # - Heavy quantization (INT4 or INT8)\n",
|
|
" # - Aggressive pruning (80-90%)\n",
|
|
" # - Kernel fusion if applicable\n",
|
|
" #\n",
|
|
" # Memory Challenge (size priority):\n",
|
|
" # - INT8 or INT4 quantization\n",
|
|
" # - Aggressive pruning (70-90%)\n",
|
|
" # - Compression techniques\n",
|
|
" #\n",
|
|
" # All-Around (balanced):\n",
|
|
" # - INT8 quantization\n",
|
|
" # - Moderate pruning (50-70%)\n",
|
|
" # - Selective optimization\n",
|
|
" #\n",
|
|
" # Your strategy:\n",
|
|
" \n",
|
|
" \n",
|
|
" \n",
|
|
" # ============================================================\n",
|
|
" # YOUR CODE ABOVE\n",
|
|
" # ============================================================\n",
|
|
" \n",
|
|
" print(\"✅ Optimization complete!\")\n",
|
|
" print(\"💡 Tip: Benchmark your result to see the impact!\")\n",
|
|
" \n",
|
|
" return optimized_model\n",
|
|
"\n",
|
|
"#| export\n",
|
|
"def validate_submission(submission: Dict[str, Any]) -> Dict[str, Any]:\n",
|
|
" \"\"\"\n",
|
|
" Validate competition submission with sanity checks.\n",
|
|
" \n",
|
|
" This catches honest mistakes like unrealistic speedups or accidental training.\n",
|
|
" Honor code system - we trust but verify basic reasonableness.\n",
|
|
" \n",
|
|
" Args:\n",
|
|
" submission: Submission dictionary to validate\n",
|
|
" \n",
|
|
" Returns:\n",
|
|
" Dict with validation results and warnings\n",
|
|
" \"\"\"\n",
|
|
" checks = []\n",
|
|
" warnings = []\n",
|
|
" errors = []\n",
|
|
" \n",
|
|
" # Extract metrics\n",
|
|
" normalized = submission.get(\"normalized_scores\", {})\n",
|
|
" speedup = normalized.get(\"speedup\", 1.0)\n",
|
|
" compression = normalized.get(\"compression_ratio\", 1.0)\n",
|
|
" accuracy_delta = normalized.get(\"accuracy_delta\", 0.0)\n",
|
|
" \n",
|
|
" # Check 1: Speedup is reasonable (not claiming impossible gains)\n",
|
|
" if speedup > 50:\n",
|
|
" errors.append(f\"❌ Speedup {speedup:.1f}x seems unrealistic (>50x)\")\n",
|
|
" elif speedup > 20:\n",
|
|
" warnings.append(f\"⚠️ Speedup {speedup:.1f}x is very high - please verify measurements\")\n",
|
|
" else:\n",
|
|
" checks.append(f\"✅ Speedup {speedup:.2f}x is reasonable\")\n",
|
|
" \n",
|
|
" # Check 2: Compression is reasonable\n",
|
|
" if compression > 32:\n",
|
|
" errors.append(f\"❌ Compression {compression:.1f}x seems unrealistic (>32x)\")\n",
|
|
" elif compression > 16:\n",
|
|
" warnings.append(f\"⚠️ Compression {compression:.1f}x is very high - please verify\")\n",
|
|
" else:\n",
|
|
" checks.append(f\"✅ Compression {compression:.2f}x is reasonable\")\n",
|
|
" \n",
|
|
" # Check 3: Accuracy didn't improve (Closed Division rule - no training allowed!)\n",
|
|
" division = submission.get(\"division\", \"closed\")\n",
|
|
" if division == \"closed\" and accuracy_delta > 1.0:\n",
|
|
" errors.append(f\"❌ Accuracy improved by {accuracy_delta:.1f}pp - did you accidentally train the model?\")\n",
|
|
" elif accuracy_delta > 0.5:\n",
|
|
" warnings.append(f\"⚠️ Accuracy improved by {accuracy_delta:.1f}pp - verify no training occurred\")\n",
|
|
" else:\n",
|
|
" checks.append(f\"✅ Accuracy change {accuracy_delta:+.2f}pp is reasonable\")\n",
|
|
" \n",
|
|
" # Check 4: GitHub repo provided\n",
|
|
" github_repo = submission.get(\"github_repo\", \"\")\n",
|
|
" if not github_repo or github_repo == \"\":\n",
|
|
" warnings.append(\"⚠️ No GitHub repo provided - required for verification\")\n",
|
|
" else:\n",
|
|
" checks.append(f\"✅ GitHub repo provided: {github_repo}\")\n",
|
|
" \n",
|
|
" # Check 5: Required fields present\n",
|
|
" required_fields = [\"division\", \"event\", \"athlete_name\", \"baseline\", \"optimized\", \"normalized_scores\"]\n",
|
|
" missing = [f for f in required_fields if f not in submission]\n",
|
|
" if missing:\n",
|
|
" errors.append(f\"❌ Missing required fields: {', '.join(missing)}\")\n",
|
|
" else:\n",
|
|
" checks.append(\"✅ All required fields present\")\n",
|
|
" \n",
|
|
" # Check 6: Techniques documented\n",
|
|
" techniques = submission.get(\"techniques_applied\", [])\n",
|
|
" if not techniques or \"TODO\" in str(techniques):\n",
|
|
" warnings.append(\"⚠️ No optimization techniques listed\")\n",
|
|
" else:\n",
|
|
" checks.append(f\"✅ Techniques documented: {', '.join(techniques[:3])}...\")\n",
|
|
" \n",
|
|
" return {\n",
|
|
" \"valid\": len(errors) == 0,\n",
|
|
" \"checks\": checks,\n",
|
|
" \"warnings\": warnings,\n",
|
|
" \"errors\": errors\n",
|
|
" }\n",
|
|
"\n",
|
|
"#| export\n",
|
|
"def generate_submission(baseline_model, optimized_model, \n",
|
|
" division: str = \"closed\",\n",
|
|
" event: str = \"all_around\",\n",
|
|
" athlete_name: str = \"YourName\",\n",
|
|
" github_repo: str = \"\",\n",
|
|
" techniques: List[str] = None) -> Dict[str, Any]:\n",
|
|
" \"\"\"\n",
|
|
" Generate standardized TinyMLPerf competition submission with normalized scoring.\n",
|
|
" \n",
|
|
" Args:\n",
|
|
" baseline_model: Original unoptimized model\n",
|
|
" optimized_model: Your optimized model\n",
|
|
" division: \"closed\" or \"open\"\n",
|
|
" event: Competition category (latency_sprint, memory_challenge, all_around, etc.)\n",
|
|
" athlete_name: Your name for submission\n",
|
|
" github_repo: GitHub repository URL for code verification\n",
|
|
" techniques: List of optimization techniques applied\n",
|
|
" \n",
|
|
" Returns:\n",
|
|
" Submission dictionary (will be saved as JSON)\n",
|
|
" \"\"\"\n",
|
|
" print(\"📤 Generating TinyMLPerf Competition Submission...\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" \n",
|
|
" # Get baseline metrics\n",
|
|
" baseline_metrics = generate_baseline(quick=True)\n",
|
|
" \n",
|
|
" # Benchmark optimized model\n",
|
|
" print(\"🔬 Benchmarking optimized model...\")\n",
|
|
" \n",
|
|
" # Use Profiler and Benchmark from Module 19\n",
|
|
" profiler = Profiler()\n",
|
|
" \n",
|
|
" # For demonstration, we'll use placeholder metrics\n",
|
|
" # In real competition, students would measure their actual optimized model\n",
|
|
" optimized_metrics = {\n",
|
|
" \"model\": getattr(optimized_model, 'name', 'Optimized_Model'),\n",
|
|
" \"accuracy\": 84.0, # Would be measured with actual test set\n",
|
|
" \"latency_ms\": 28.0, # Would be measured with profiler\n",
|
|
" \"memory_mb\": 4.0, # Would be measured with profiler\n",
|
|
" \"parameters\": 2000000, # Would be counted\n",
|
|
" }\n",
|
|
" \n",
|
|
" # Calculate normalized scores using Module 19's function\n",
|
|
" baseline_for_norm = {\n",
|
|
" \"latency\": baseline_metrics[\"latency_ms\"],\n",
|
|
" \"memory\": baseline_metrics[\"memory_mb\"],\n",
|
|
" \"accuracy\": baseline_metrics[\"accuracy\"]\n",
|
|
" }\n",
|
|
" \n",
|
|
" optimized_for_norm = {\n",
|
|
" \"latency\": optimized_metrics[\"latency_ms\"],\n",
|
|
" \"memory\": optimized_metrics[\"memory_mb\"],\n",
|
|
" \"accuracy\": optimized_metrics[\"accuracy\"]\n",
|
|
" }\n",
|
|
" \n",
|
|
" normalized_scores = calculate_normalized_scores(baseline_for_norm, optimized_for_norm)\n",
|
|
" \n",
|
|
" # Create submission with all required fields\n",
|
|
" submission = {\n",
|
|
" \"division\": division,\n",
|
|
" \"event\": event,\n",
|
|
" \"athlete_name\": athlete_name,\n",
|
|
" \"github_repo\": github_repo,\n",
|
|
" \"baseline\": baseline_metrics,\n",
|
|
" \"optimized\": optimized_metrics,\n",
|
|
" \"normalized_scores\": {\n",
|
|
" \"speedup\": normalized_scores[\"speedup\"],\n",
|
|
" \"compression_ratio\": normalized_scores[\"compression_ratio\"],\n",
|
|
" \"accuracy_delta\": normalized_scores[\"accuracy_delta\"],\n",
|
|
" \"efficiency_score\": normalized_scores[\"efficiency_score\"]\n",
|
|
" },\n",
|
|
" \"techniques_applied\": techniques or [\"TODO: Document your optimization techniques\"],\n",
|
|
" \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n",
|
|
" \"tinytorch_version\": \"0.1.0\",\n",
|
|
" \"honor_code\": False # Must be explicitly set to True after validation\n",
|
|
" }\n",
|
|
" \n",
|
|
" # Validate submission\n",
|
|
" print(\"\\n🔍 Validating submission...\")\n",
|
|
" validation = validate_submission(submission)\n",
|
|
" \n",
|
|
" # Display validation results\n",
|
|
" print(\"\\n📋 Validation Results:\")\n",
|
|
" for check in validation[\"checks\"]:\n",
|
|
" print(f\" {check}\")\n",
|
|
" for warning in validation[\"warnings\"]:\n",
|
|
" print(f\" {warning}\")\n",
|
|
" for error in validation[\"errors\"]:\n",
|
|
" print(f\" {error}\")\n",
|
|
" \n",
|
|
" if not validation[\"valid\"]:\n",
|
|
" print(\"\\n❌ Submission has errors - please fix before submitting\")\n",
|
|
" return submission\n",
|
|
" \n",
|
|
" # Save to JSON\n",
|
|
" output_file = Path(\"submission.json\")\n",
|
|
" with open(output_file, \"w\") as f:\n",
|
|
" json.dump(submission, f, indent=2)\n",
|
|
" \n",
|
|
" print(f\"\\n✅ Submission saved to: {output_file}\")\n",
|
|
" print()\n",
|
|
" print(\"📊 Your Normalized Scores (MLPerf-style):\")\n",
|
|
" print(f\" Division: {division.upper()}\")\n",
|
|
" print(f\" Event: {event.replace('_', ' ').title()}\")\n",
|
|
" print(f\" Speedup: {normalized_scores['speedup']:.2f}x faster ⚡\")\n",
|
|
" print(f\" Compression: {normalized_scores['compression_ratio']:.2f}x smaller 💾\")\n",
|
|
" print(f\" Accuracy: {optimized_metrics['accuracy']:.1f}% (Δ {normalized_scores['accuracy_delta']:+.2f}pp)\")\n",
|
|
" print(f\" Efficiency: {normalized_scores['efficiency_score']:.2f}\")\n",
|
|
" print()\n",
|
|
" print(\"📤 Next Steps:\")\n",
|
|
" print(\" 1. Verify all metrics are correct\")\n",
|
|
" print(\" 2. Push your code to GitHub (if not done)\")\n",
|
|
" print(\" 3. Run: tito submit submission.json\")\n",
|
|
" print(\" (This will validate and prepare final submission)\")\n",
|
|
" print()\n",
|
|
" print(\"=\" * 70)\n",
|
|
" \n",
|
|
" return submission"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e95a6680",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"# 5. Module Integration Test\n",
|
|
"\n",
|
|
"Complete validation and competition workflow test."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "914aaac9",
|
|
"metadata": {
|
|
"nbgrader": {
|
|
"grade": true,
|
|
"grade_id": "test-module",
|
|
"locked": true,
|
|
"points": 10
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_module():\n",
|
|
" \"\"\"\n",
|
|
" Complete test of Module 20 functionality.\n",
|
|
" \n",
|
|
" This validates:\n",
|
|
" - Installation validation works\n",
|
|
" - Baseline generation works\n",
|
|
" - Worked example runs successfully\n",
|
|
" - Competition template is ready\n",
|
|
" \"\"\"\n",
|
|
" print(\"=\" * 70)\n",
|
|
" print(\"MODULE 20 INTEGRATION TEST\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Test 1: Validation\n",
|
|
" print(\"🔧 Test 1: System Validation\")\n",
|
|
" validation_status = validate_installation()\n",
|
|
" assert len(validation_status) > 0, \"Validation should return status dict\"\n",
|
|
" print(\"✅ Validation working!\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Test 2: Baseline Generation\n",
|
|
" print(\"📊 Test 2: Baseline Generation\")\n",
|
|
" baseline = generate_baseline(quick=True)\n",
|
|
" assert \"accuracy\" in baseline, \"Baseline should include accuracy\"\n",
|
|
" assert \"latency_ms\" in baseline, \"Baseline should include latency\"\n",
|
|
" assert \"memory_mb\" in baseline, \"Baseline should include memory\"\n",
|
|
" print(\"✅ Baseline generation working!\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Test 3: Worked Example\n",
|
|
" print(\"🏅 Test 3: Worked Example\")\n",
|
|
" example_submission = worked_example_optimization()\n",
|
|
" assert \"event\" in example_submission, \"Submission should include event\"\n",
|
|
" assert \"baseline\" in example_submission, \"Submission should include baseline\"\n",
|
|
" assert \"optimized\" in example_submission, \"Submission should include optimized\"\n",
|
|
" print(\"✅ Worked example working!\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" # Test 4: Competition Template\n",
|
|
" print(\"🎯 Test 4: Competition Template\")\n",
|
|
" baseline_model = load_baseline_model(\"cifar10_cnn\")\n",
|
|
" optimized = optimize_for_competition(baseline_model, event=\"all_around\")\n",
|
|
" assert optimized is not None, \"Optimization should return model\"\n",
|
|
" print(\"✅ Competition template working!\")\n",
|
|
" print()\n",
|
|
" \n",
|
|
" print(\"=\" * 70)\n",
|
|
" print(\"✅ ALL TESTS PASSED!\")\n",
|
|
" print(\"=\" * 70)\n",
|
|
" print()\n",
|
|
" print(\"🎉 You're ready for TorchPerf Olympics!\")\n",
|
|
" print(\" Next steps:\")\n",
|
|
" print(\" 1. Implement your optimization strategy in optimize_for_competition()\")\n",
|
|
" print(\" 2. Run this module to generate submission.json\")\n",
|
|
" print(\" 3. Upload to competition platform\")\n",
|
|
" print()\n",
|
|
" print(\"🔥 Good luck! May the best optimizer win! 🏅\")\n",
|
|
"\n",
|
|
"test_module()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0ef195c7",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 🤔 ML Systems Thinking: Competition as Learning\n",
|
|
"\n",
|
|
"TorchPerf Olympics isn't just about winning - it's about understanding trade-offs:\n",
|
|
"\n",
|
|
"**The Meta-Lesson**: Every optimization involves trade-offs:\n",
|
|
"- Quantization: Speed vs Accuracy\n",
|
|
"- Pruning: Size vs Performance\n",
|
|
"- Caching: Memory vs Speed\n",
|
|
"\n",
|
|
"Professional ML engineers navigate these trade-offs daily. The competition forces you to:\n",
|
|
"1. **Think systematically** about optimization strategies\n",
|
|
"2. **Measure rigorously** using benchmarking tools\n",
|
|
"3. **Make data-driven decisions** based on actual measurements\n",
|
|
"4. **Document and justify** your choices\n",
|
|
"\n",
|
|
"The best submission isn't always the \"fastest\" or \"smallest\" - it's the one that best understands and navigates the trade-off space for their chosen event.\n",
|
|
"\n",
|
|
"What will your strategy be? 🤔"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b0f38935",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 2
|
|
},
|
|
"source": [
|
|
"## 🎯 MODULE SUMMARY: Competition & Validation\n",
|
|
"\n",
|
|
"**What You've Learned:**\n",
|
|
"- ✅ How to validate your TinyTorch installation\n",
|
|
"- ✅ How to generate baseline performance metrics\n",
|
|
"- ✅ How to combine optimization techniques systematically\n",
|
|
"- ✅ How to benchmark and measure impact\n",
|
|
"- ✅ How to generate standardized competition submissions\n",
|
|
"\n",
|
|
"**The Complete Workflow:**\n",
|
|
"```\n",
|
|
"1. Validate → Ensure environment works\n",
|
|
"2. Baseline → Establish reference performance\n",
|
|
"3. Optimize → Apply techniques from M14-18\n",
|
|
"4. Benchmark → Measure impact using M19\n",
|
|
"5. Submit → Generate standardized submission\n",
|
|
"```\n",
|
|
"\n",
|
|
"**Key Takeaway**: Competition teaches systematic optimization thinking. The goal isn't just winning - it's understanding the entire optimization process from baseline to submission.\n",
|
|
"\n",
|
|
"**Next Steps:**\n",
|
|
"1. Study the worked example\n",
|
|
"2. Implement your own optimization strategy\n",
|
|
"3. Benchmark your results\n",
|
|
"4. Generate submission.json\n",
|
|
"5. Compete in TorchPerf Olympics!\n",
|
|
"\n",
|
|
"🔥 Now go optimize and win gold! 🏅"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|