{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aabba6c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| default_exp competition.submit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5222d75",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "# Module 20: TinyMLPerf Competition - Your Capstone Challenge\n",
    "\n",
    "Welcome to the capstone! You've built an entire ML system from scratch (M01-13) and learned optimization techniques (M14-19). Now it's time to compete and show what you can do! 🏅\n",
    "\n",
    "## 🔗 Your Journey\n",
    "```\n",
    "Modules 01-13: Build ML System (tensors → transformers)\n",
    "Modules 14-18: Learn Optimization Techniques  \n",
    "Module 19:     Learn Benchmarking\n",
    "Module 20:     Compete in TinyMLPerf! 🏅\n",
    "```\n",
    "\n",
    "## 🏅 TinyMLPerf: Two Ways to Compete\n",
    "\n",
    "Inspired by industry-standard MLPerf (which you learned about in Module 19), TinyMLPerf offers **two competition tracks**:\n",
    "\n",
    "### 🔒 Closed Division - \"Optimization Challenge\"\n",
    "**What you do:**\n",
    "- Start with provided baseline model (everyone gets the same)\n",
    "- Apply optimization techniques from Modules 14-18\n",
    "- Compete on: Who optimizes best?\n",
    "\n",
    "**Best for:** Most students - clear rules, fair comparison\n",
    "**Focus:** Your optimization skills\n",
    "\n",
    "### 🔓 Open Division - \"Innovation Challenge\"  \n",
    "**What you do:**\n",
    "- Modify anything! Improve your implementations from M01-19\n",
    "- Design better architectures\n",
    "- Novel approaches encouraged\n",
    "\n",
    "**Best for:** Advanced students who want more creative freedom\n",
    "**Focus:** Your systems innovations\n",
    "\n",
    "## Competition Categories (Both Divisions)\n",
    "- 🏃 **Latency Sprint**: Fastest inference\n",
    "- 🏋️ **Memory Challenge**: Smallest model\n",
    "- 🎯 **Accuracy Contest**: Best accuracy within constraints\n",
    "- 🏋️‍♂️ **All-Around**: Best balanced performance\n",
    "- 🚀 **Extreme Push**: Most aggressive optimization\n",
    "\n",
    "## What This Module Provides\n",
    "1. **Validation**: Check your TinyTorch works\n",
    "2. **Baseline**: Starting point for Closed Division\n",
    "3. **Examples**: See both tracks in action\n",
    "4. **Template**: Your competition workspace\n",
    "\n",
    "Pick your track, optimize, and compete! 🔥"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bbad866",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 📦 Where This Code Lives in the Final Package\n",
    "\n",
    "**Learning Side:** You work in `modules/20_competition/competition_dev.py`  \n",
    "**Building Side:** Code exports to `tinytorch.competition.submit`\n",
    "\n",
    "```python\n",
    "# Validation and baseline tools:\n",
    "from tinytorch.competition.submit import validate_installation, generate_baseline\n",
    "\n",
    "# Competition helpers:\n",
    "from tinytorch.competition.submit import load_baseline_model, generate_submission\n",
    "```\n",
    "\n",
    "**Why this matters:**\n",
    "- **Validation:** Ensures your TinyTorch installation works correctly\n",
    "- **Baseline:** Establishes reference performance for fair comparison\n",
    "- **Competition:** Provides standardized framework for submissions\n",
    "- **Integration:** Brings together all 19 modules into one complete workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a56c298b",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "# 1. Pick Your Track & Validate\n",
    "\n",
    "Before competing, choose your track and make sure your TinyTorch installation works!\n",
    "\n",
    "## Two Tracks, Two Styles\n",
    "\n",
    "### 🔒 Closed Division - \"The Optimization Challenge\"\n",
    "- Everyone starts with the same baseline model\n",
    "- Apply techniques from Modules 14-18 (quantization, pruning, etc.)\n",
    "- Fair comparison: who optimizes best?\n",
    "- **Choose this if:** You want clear rules and direct competition\n",
    "\n",
    "### 🔓 Open Division - \"The Innovation Challenge\"\n",
    "- Modify anything! Improve YOUR TinyTorch implementations\n",
    "- Better Conv2d? Faster matmul? Novel architecture? All allowed!\n",
    "- Compete on innovation and creativity\n",
    "- **Choose this if:** You want freedom to explore and innovate\n",
    "\n",
    "**Can I do both?** Absolutely! Submit to both tracks.\n",
    "\n",
    "**Which is \"better\"?** Neither - they test different skills:\n",
    "- Closed = Optimization mastery\n",
    "- Open = Systems innovation\n",
    "\n",
    "## Quick Validation\n",
    "\n",
    "Before competing, let's verify everything works:\n",
    "- ✅ All modules imported successfully\n",
    "- ✅ Optimization techniques available\n",
    "- ✅ Benchmarking tools ready"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4748e00b",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "import numpy as np\n",
    "import json\n",
    "import time\n",
    "from pathlib import Path\n",
    "from typing import Dict, List, Tuple, Any, Optional\n",
    "from tinytorch.benchmarking.benchmark import Benchmark, calculate_normalized_scores\n",
    "from tinytorch.profiling.profiler import Profiler\n",
    "\n",
    "def validate_installation() -> Dict[str, bool]:\n",
    "    \"\"\"\n",
    "    Validate TinyTorch installation and return status of each component.\n",
    "    \n",
    "    Returns:\n",
    "        Dictionary mapping module names to validation status (True = working)\n",
    "    \n",
    "    Example:\n",
    "        >>> status = validate_installation()\n",
    "        >>> print(status)\n",
    "        {'tensor': True, 'autograd': True, 'layers': True, ...}\n",
    "    \"\"\"\n",
    "    validation_results = {}\n",
    "    \n",
    "    print(\"🔧 Validating TinyTorch Installation...\")\n",
    "    print(\"=\" * 60)\n",
    "    \n",
    "    # Core modules (M01-13)\n",
    "    core_modules = [\n",
    "        (\"tensor\", \"tinytorch.core.tensor\", \"Tensor\"),\n",
    "        (\"autograd\", \"tinytorch.core.autograd\", \"enable_autograd\"),\n",
    "        (\"layers\", \"tinytorch.core.layers\", \"Linear\"),\n",
    "        (\"activations\", \"tinytorch.core.activations\", \"ReLU\"),\n",
    "        (\"losses\", \"tinytorch.core.training\", \"MSELoss\"),\n",
    "        (\"optimizers\", \"tinytorch.core.optimizers\", \"SGD\"),\n",
    "        (\"spatial\", \"tinytorch.core.spatial\", \"Conv2d\"),\n",
    "        (\"attention\", \"tinytorch.core.attention\", \"MultiHeadAttention\"),\n",
    "        (\"transformers\", \"tinytorch.models.transformer\", \"GPT\"),\n",
    "    ]\n",
    "    \n",
    "    for name, module_path, class_name in core_modules:\n",
    "        try:\n",
    "            exec(f\"from {module_path} import {class_name}\")\n",
    "            validation_results[name] = True\n",
    "            print(f\"✅ {name.capitalize()}: Working\")\n",
    "        except Exception as e:\n",
    "            validation_results[name] = False\n",
    "            print(f\"❌ {name.capitalize()}: Failed - {str(e)}\")\n",
    "    \n",
    "    # Optimization modules (M14-18)\n",
    "    opt_modules = [\n",
    "        (\"kv_caching\", \"tinytorch.generation.kv_cache\", \"enable_kv_cache\"),\n",
    "        (\"profiling\", \"tinytorch.profiling.profiler\", \"Profiler\"),\n",
    "        (\"quantization\", \"tinytorch.optimization.quantization\", \"quantize_model\"),\n",
    "        (\"compression\", \"tinytorch.optimization.compression\", \"magnitude_prune\"),\n",
    "    ]\n",
    "    \n",
    "    for name, module_path, func_name in opt_modules:\n",
    "        try:\n",
    "            exec(f\"from {module_path} import {func_name}\")\n",
    "            validation_results[name] = True\n",
    "            print(f\"✅ {name.replace('_', ' ').capitalize()}: Working\")\n",
    "        except Exception as e:\n",
    "            validation_results[name] = False\n",
    "            print(f\"❌ {name.replace('_', ' ').capitalize()}: Failed - {str(e)}\")\n",
    "    \n",
    "    # Benchmarking (M19)\n",
    "    try:\n",
    "        from tinytorch.benchmarking.benchmark import Benchmark, OlympicEvent\n",
    "        validation_results[\"benchmarking\"] = True\n",
    "        print(f\"✅ Benchmarking: Working\")\n",
    "    except Exception as e:\n",
    "        validation_results[\"benchmarking\"] = False\n",
    "        print(f\"❌ Benchmarking: Failed - {str(e)}\")\n",
    "    \n",
    "    print(\"=\" * 60)\n",
    "    \n",
    "    # Summary\n",
    "    total = len(validation_results)\n",
    "    working = sum(validation_results.values())\n",
    "    \n",
    "    if working == total:\n",
    "        print(f\"🎉 Perfect! All {total}/{total} modules working!\")\n",
    "        print(\"✅ You're ready to compete in TorchPerf Olympics!\")\n",
    "    else:\n",
    "        print(f\"⚠️  {working}/{total} modules working\")\n",
    "        print(f\"❌ {total - working} modules need attention\")\n",
    "        print(\"\\nPlease run: pip install -e . (in TinyTorch root)\")\n",
    "    \n",
    "    return validation_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "190e1466",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "# 2. The Baseline (For Closed Division)\n",
    "\n",
    "If you're competing in **Closed Division**, everyone starts with this baseline model. If you're in **Open Division**, you can skip this or use it as a reference!\n",
    "\n",
    "## Baseline Model: Simple CNN on CIFAR-10\n",
    "\n",
    "We provide a simple CNN as the starting point for Closed Division:\n",
    "- **Architecture:** Conv → Pool → Conv → Pool → FC → FC\n",
    "- **Dataset:** CIFAR-10 (standardized test set)\n",
    "- **Metrics:** Accuracy, latency, memory (we'll measure together)\n",
    "\n",
    "**Closed Division:** Optimize THIS model using M14-18 techniques\n",
    "**Open Division:** Build/modify whatever you want!\n",
    "\n",
    "### Baseline Components\n",
    "\n",
    "1. **Model:** Standard CNN (no optimizations)\n",
    "2. **Metrics:** Accuracy, latency, memory, parameters\n",
    "3. **Test Data:** CIFAR-10 test set (standardized)\n",
    "4. **Hardware:** Your local machine (reported for reproducibility)\n",
    "\n",
    "The baseline establishes what \"unoptimized\" looks like. Your job: beat it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff944a6c",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "def load_baseline_model(model_name: str = \"cifar10_cnn\"):\n",
    "    \"\"\"\n",
    "    Load a baseline model for TorchPerf Olympics competition.\n",
    "    \n",
    "    Args:\n",
    "        model_name: Name of baseline model to load\n",
    "            - \"cifar10_cnn\": Simple CNN for CIFAR-10 classification\n",
    "    \n",
    "    Returns:\n",
    "        Baseline model instance\n",
    "    \n",
    "    Example:\n",
    "        >>> model = load_baseline_model(\"cifar10_cnn\")\n",
    "        >>> print(f\"Parameters: {sum(p.size for p in model.parameters())}\")\n",
    "    \"\"\"\n",
    "    from tinytorch.core.layers import Linear\n",
    "    from tinytorch.core.spatial import Conv2d, MaxPool2d, Flatten\n",
    "    from tinytorch.core.activations import ReLU\n",
    "    \n",
    "    if model_name == \"cifar10_cnn\":\n",
    "        # Simple CNN: Conv -> Pool -> Conv -> Pool -> FC -> FC\n",
    "        class BaselineCNN:\n",
    "            def __init__(self):\n",
    "                self.name = \"Baseline_CIFAR10_CNN\"\n",
    "                \n",
    "                # Convolutional layers\n",
    "                self.conv1 = Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)\n",
    "                self.relu1 = ReLU()\n",
    "                self.pool1 = MaxPool2d(kernel_size=2, stride=2)\n",
    "                \n",
    "                self.conv2 = Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)\n",
    "                self.relu2 = ReLU()\n",
    "                self.pool2 = MaxPool2d(kernel_size=2, stride=2)\n",
    "                \n",
    "                # Fully connected layers\n",
    "                self.flatten = Flatten()\n",
    "                self.fc1 = Linear(64 * 8 * 8, 128)\n",
    "                self.relu3 = ReLU()\n",
    "                self.fc2 = Linear(128, 10)  # 10 classes for CIFAR-10\n",
    "            \n",
    "            def forward(self, x):\n",
    "                # Forward pass\n",
    "                x = self.conv1.forward(x)\n",
    "                x = self.relu1.forward(x)\n",
    "                x = self.pool1.forward(x)\n",
    "                \n",
    "                x = self.conv2.forward(x)\n",
    "                x = self.relu2.forward(x)\n",
    "                x = self.pool2.forward(x)\n",
    "                \n",
    "                x = self.flatten.forward(x)\n",
    "                x = self.fc1.forward(x)\n",
    "                x = self.relu3.forward(x)\n",
    "                x = self.fc2.forward(x)\n",
    "                \n",
    "                return x\n",
    "            \n",
    "            def __call__(self, x):\n",
    "                return self.forward(x)\n",
    "        \n",
    "        return BaselineCNN()\n",
    "    else:\n",
    "        raise ValueError(f\"Unknown baseline model: {model_name}\")\n",
    "\n",
    "def generate_baseline(model_name: str = \"cifar10_cnn\", quick: bool = True) -> Dict[str, Any]:\n",
    "    \"\"\"\n",
    "    Generate baseline performance metrics for a model.\n",
    "    \n",
    "    Args:\n",
    "        model_name: Name of baseline model\n",
    "        quick: If True, use quick estimates instead of full benchmarks\n",
    "    \n",
    "    Returns:\n",
    "        Baseline scorecard with metrics\n",
    "    \n",
    "    Example:\n",
    "        >>> baseline = generate_baseline(\"cifar10_cnn\", quick=True)\n",
    "        >>> print(f\"Baseline latency: {baseline['latency_ms']}ms\")\n",
    "    \"\"\"\n",
    "    print(\"📊 Generating Baseline Scorecard...\")\n",
    "    print(\"=\" * 60)\n",
    "    \n",
    "    # Load model\n",
    "    model = load_baseline_model(model_name)\n",
    "    print(f\"✅ Loaded baseline model: {model.name}\")\n",
    "    \n",
    "    # Count parameters\n",
    "    def count_parameters(model):\n",
    "        total = 0\n",
    "        for attr_name in dir(model):\n",
    "            attr = getattr(model, attr_name)\n",
    "            if hasattr(attr, 'weights') and attr.weights is not None:\n",
    "                total += attr.weights.size\n",
    "            if hasattr(attr, 'bias') and attr.bias is not None:\n",
    "                total += attr.bias.size\n",
    "        return total\n",
    "    \n",
    "    params = count_parameters(model)\n",
    "    memory_mb = params * 4 / (1024 * 1024)  # Assuming float32\n",
    "    \n",
    "    if quick:\n",
    "        # Quick estimates for fast validation\n",
    "        print(\"⚡ Using quick estimates (set quick=False for full benchmark)\")\n",
    "        \n",
    "        baseline = {\n",
    "            \"model\": model_name,\n",
    "            \"accuracy\": 85.0,  # Typical for this architecture\n",
    "            \"latency_ms\": 45.2,\n",
    "            \"memory_mb\": memory_mb,\n",
    "            \"parameters\": params,\n",
    "            \"mode\": \"quick_estimate\"\n",
    "        }\n",
    "    else:\n",
    "        # Full benchmark (requires more time)\n",
    "        from tinytorch.benchmarking.benchmark import Benchmark\n",
    "        \n",
    "        print(\"🔬 Running full benchmark (this may take a minute)...\")\n",
    "        \n",
    "        benchmark = Benchmark([model], [{\"name\": \"baseline\"}], \n",
    "                            warmup_runs=5, measurement_runs=20)\n",
    "        \n",
    "        # Measure latency\n",
    "        input_shape = (1, 3, 32, 32)  # CIFAR-10 input\n",
    "        latency_results = benchmark.run_latency_benchmark(input_shape=input_shape)\n",
    "        latency_ms = list(latency_results.values())[0].mean * 1000\n",
    "        \n",
    "        baseline = {\n",
    "            \"model\": model_name,\n",
    "            \"accuracy\": 85.0,  # Would need actual test set evaluation\n",
    "            \"latency_ms\": latency_ms,\n",
    "            \"memory_mb\": memory_mb,\n",
    "            \"parameters\": params,\n",
    "            \"mode\": \"full_benchmark\"\n",
    "        }\n",
    "    \n",
    "    # Display baseline\n",
    "    print(\"\\n📋 BASELINE SCORECARD\")\n",
    "    print(\"=\" * 60)\n",
    "    print(f\"Model:          {baseline['model']}\")\n",
    "    print(f\"Accuracy:       {baseline['accuracy']:.1f}%\")\n",
    "    print(f\"Latency:        {baseline['latency_ms']:.1f}ms\")\n",
    "    print(f\"Memory:         {baseline['memory_mb']:.2f}MB\")\n",
    "    print(f\"Parameters:     {baseline['parameters']:,}\")\n",
    "    print(\"=\" * 60)\n",
    "    print(\"📌 This is your starting point. Optimize to compete!\")\n",
    "    print()\n",
    "    \n",
    "    return baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdef4b17",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "# 3. Complete Example - See Both Tracks in Action\n",
    "\n",
    "Let's see complete examples for BOTH competition tracks!\n",
    "\n",
    "## Example 1: Closed Division - Optimization Master\n",
    "\n",
    "**Goal:** Compete in All-Around category using provided baseline\n",
    "\n",
    "**Strategy:**\n",
    "1. Load baseline CNN\n",
    "2. Apply quantization (INT8) → 4x memory reduction\n",
    "3. Apply pruning (60%) → Speed boost\n",
    "4. Benchmark and submit\n",
    "\n",
    "**Why this order?** Quantize first preserves more accuracy than pruning first.\n",
    "\n",
    "## Example 2: Open Division - Innovation Master\n",
    "\n",
    "**Goal:** Beat everyone with a novel approach\n",
    "\n",
    "**Strategy:**\n",
    "1. Improve YOUR Conv2d implementation (faster algorithm)\n",
    "2. OR design a better architecture (MobileNet-style)\n",
    "3. OR novel quantization (mixed precision per layer)\n",
    "4. Benchmark and submit\n",
    "\n",
    "**Freedom:** Modify anything in your TinyTorch implementation!\n",
    "\n",
    "Let's see the Closed Division example in detail below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a5e4560",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "def worked_example_optimization():\n",
    "    \"\"\"\n",
    "    Complete worked example showing full optimization workflow.\n",
    "    \n",
    "    This demonstrates:\n",
    "    - Loading baseline model\n",
    "    - Applying multiple optimization techniques\n",
    "    - Benchmarking systematically\n",
    "    - Generating submission\n",
    "    \n",
    "    Students should study this and adapt for their own strategies!\n",
    "    \"\"\"\n",
    "    print(\"🏅 WORKED EXAMPLE: Complete Optimization Workflow\")\n",
    "    print(\"=\" * 70)\n",
    "    print(\"Target: All-Around Event (balanced performance)\")\n",
    "    print(\"Strategy: Quantization (INT8) → Pruning (60%)\")\n",
    "    print(\"=\" * 70)\n",
    "    print()\n",
    "    \n",
    "    # Step 1: Load Baseline\n",
    "    print(\"📦 Step 1: Load Baseline Model\")\n",
    "    print(\"-\" * 70)\n",
    "    baseline = load_baseline_model(\"cifar10_cnn\")\n",
    "    baseline_metrics = generate_baseline(\"cifar10_cnn\", quick=True)\n",
    "    print()\n",
    "    \n",
    "    # Step 2: Apply Quantization\n",
    "    print(\"🔧 Step 2: Apply INT8 Quantization (Module 17)\")\n",
    "    print(\"-\" * 70)\n",
    "    print(\"💡 Why quantize? Reduces memory 4x (FP32 → INT8)\")\n",
    "    \n",
    "    # For demonstration, we'll simulate quantization\n",
    "    # In real competition, students would use:\n",
    "    # from tinytorch.optimization.quantization import quantize_model\n",
    "    # optimized = quantize_model(baseline, bits=8)\n",
    "    \n",
    "    print(\"✅ Quantized model (simulated)\")\n",
    "    print(\"   - Memory: 12.4MB → 3.1MB (4x reduction)\")\n",
    "    print()\n",
    "    \n",
    "    # Step 3: Apply Pruning\n",
    "    print(\"✂️  Step 3: Apply Magnitude Pruning (Module 18)\")\n",
    "    print(\"-\" * 70)\n",
    "    print(\"💡 Why prune? Removes 60% of weights for faster inference\")\n",
    "    \n",
    "    # For demonstration, we'll simulate pruning\n",
    "    # In real competition, students would use:\n",
    "    # from tinytorch.optimization.compression import magnitude_prune\n",
    "    # optimized = magnitude_prune(optimized, sparsity=0.6)\n",
    "    \n",
    "    print(\"✅ Pruned model (simulated)\")\n",
    "    print(\"   - Active parameters: 3.2M → 1.28M (60% removed)\")\n",
    "    print()\n",
    "    \n",
    "    # Step 4: Benchmark Results\n",
    "    print(\"📊 Step 4: Benchmark Optimized Model (Module 19)\")\n",
    "    print(\"-\" * 70)\n",
    "    \n",
    "    # Simulated optimized metrics\n",
    "    optimized_metrics = {\n",
    "        \"model\": \"Optimized_CIFAR10_CNN\",\n",
    "        \"accuracy\": 83.5,  # Slight drop from aggressive optimization\n",
    "        \"latency_ms\": 22.1,\n",
    "        \"memory_mb\": 1.24,  # 4x quantization + 60% pruning\n",
    "        \"parameters\": 1280000,\n",
    "        \"techniques\": [\"quantization_int8\", \"magnitude_prune_0.6\"]\n",
    "    }\n",
    "    \n",
    "    print(\"Baseline vs Optimized:\")\n",
    "    print(f\"  Accuracy:    {baseline_metrics['accuracy']:.1f}% → {optimized_metrics['accuracy']:.1f}% (-1.5pp)\")\n",
    "    print(f\"  Latency:     {baseline_metrics['latency_ms']:.1f}ms → {optimized_metrics['latency_ms']:.1f}ms (2.0x faster ✅)\")\n",
    "    print(f\"  Memory:      {baseline_metrics['memory_mb']:.2f}MB → {optimized_metrics['memory_mb']:.2f}MB (10.0x smaller ✅)\")\n",
    "    print(f\"  Parameters:  {baseline_metrics['parameters']:,} → {optimized_metrics['parameters']:,} (60% fewer ✅)\")\n",
    "    print()\n",
    "    \n",
    "    # Step 5: Generate Submission\n",
    "    print(\"📤 Step 5: Generate Competition Submission\")\n",
    "    print(\"-\" * 70)\n",
    "    \n",
    "    submission = {\n",
    "        \"event\": \"all_around\",\n",
    "        \"athlete_name\": \"Example_Submission\",\n",
    "        \"baseline\": baseline_metrics,\n",
    "        \"optimized\": optimized_metrics,\n",
    "        \"improvements\": {\n",
    "            \"accuracy_drop\": -1.5,\n",
    "            \"latency_speedup\": 2.0,\n",
    "            \"memory_reduction\": 10.0\n",
    "        },\n",
    "        \"techniques_applied\": [\"quantization_int8\", \"magnitude_prune_0.6\"],\n",
    "        \"technique_order\": \"quantize_first_then_prune\"\n",
    "    }\n",
    "    \n",
    "    print(\"✅ Submission generated!\")\n",
    "    print(f\"   Event: {submission['event']}\")\n",
    "    print(f\"   Techniques: {', '.join(submission['techniques_applied'])}\")\n",
    "    print()\n",
    "    print(\"=\" * 70)\n",
    "    print(\"🎯 This is the complete workflow!\")\n",
    "    print(\"   Now it's your turn to implement your own optimization strategy.\")\n",
    "    print(\"=\" * 70)\n",
    "    \n",
    "    return submission"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b013b5eb",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "# 4. Your Turn - Pick Your Track!\n",
    "\n",
    "Now it's time to compete! Choose your track and implement your strategy.\n",
    "\n",
    "## Choose Your Track\n",
    "\n",
    "### 🔒 Closed Division Template\n",
    "**If you choose Closed Division:**\n",
    "1. Pick a category (Latency Sprint, Memory Challenge, etc.)\n",
    "2. Design your optimization strategy\n",
    "3. Implement in `optimize_for_competition()` below\n",
    "4. Use techniques from Modules 14-18 only\n",
    "5. Generate submission\n",
    "\n",
    "**Good for:** Clear path, fair comparison, most students\n",
    "\n",
    "### 🔓 Open Division Template  \n",
    "**If you choose Open Division:**\n",
    "1. Pick a category\n",
    "2. Modify YOUR TinyTorch implementations (go edit earlier modules!)\n",
    "3. OR design novel architectures\n",
    "4. Re-export with `tito export` and benchmark\n",
    "5. Generate submission\n",
    "\n",
    "**Good for:** Creative freedom, systems innovation, advanced students\n",
    "\n",
    "## Competition Categories (Pick ONE)\n",
    "- 🏃 **Latency Sprint:** Fastest inference\n",
    "- 🏋️ **Memory Challenge:** Smallest model\n",
    "- 🎯 **Accuracy Contest:** Best accuracy within constraints\n",
    "- 🏋️‍♂️ **All-Around:** Best balanced performance\n",
    "- 🚀 **Extreme Push:** Most aggressive optimization\n",
    "\n",
    "## Template Below\n",
    "\n",
    "Use the `optimize_for_competition()` function to implement your strategy:\n",
    "- **Closed Division:** Apply M14-18 techniques\n",
    "- **Open Division:** Do whatever you want, document it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d51c16c8",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "def optimize_for_competition(baseline_model, event: str = \"all_around\", division: str = \"closed\"):\n",
    "    \"\"\"\n",
    "    🏅 YOUR COMPETITION ENTRY - IMPLEMENT YOUR STRATEGY HERE!\n",
    "    \n",
    "    Args:\n",
    "        baseline_model: Starting model (use for Closed, optional for Open)\n",
    "        event: Category you're competing in\n",
    "            - \"latency_sprint\": Minimize latency\n",
    "            - \"memory_challenge\": Minimize memory\n",
    "            - \"accuracy_contest\": Maximize accuracy\n",
    "            - \"all_around\": Best balance\n",
    "            - \"extreme_push\": Most aggressive\n",
    "        division: \"closed\" or \"open\" - which track you chose\n",
    "    \n",
    "    Returns:\n",
    "        Your optimized model\n",
    "    \n",
    "    🔒 CLOSED DIVISION Example:\n",
    "        from tinytorch.optimization.quantization import quantize_model\n",
    "        from tinytorch.optimization.compression import magnitude_prune\n",
    "        \n",
    "        optimized = baseline_model\n",
    "        optimized = quantize_model(optimized, bits=8)\n",
    "        optimized = magnitude_prune(optimized, sparsity=0.7)\n",
    "        return optimized\n",
    "    \n",
    "    🔓 OPEN DIVISION Example:\n",
    "        # Build your own model OR\n",
    "        # Use your improved implementations from earlier modules\n",
    "        # (after you've modified and re-exported them)\n",
    "        \n",
    "        from tinytorch.models import YourCustomArchitecture\n",
    "        optimized = YourCustomArchitecture()\n",
    "        return optimized\n",
    "    \"\"\"\n",
    "    \n",
    "    print(f\"🏅 YOUR OPTIMIZATION STRATEGY FOR: {event}\")\n",
    "    print(\"=\" * 70)\n",
    "    \n",
    "    # Start with baseline\n",
    "    optimized_model = baseline_model\n",
    "    \n",
    "    # ============================================================\n",
    "    # YOUR CODE BELOW - Apply optimization techniques here!\n",
    "    # ============================================================\n",
    "    \n",
    "    # TODO: Students implement their optimization strategy\n",
    "    #\n",
    "    # Example strategies by event:\n",
    "    #\n",
    "    # Latency Sprint (speed priority):\n",
    "    #   - Heavy quantization (INT4 or INT8)\n",
    "    #   - Aggressive pruning (80-90%)\n",
    "    #   - Kernel fusion if applicable\n",
    "    #\n",
    "    # Memory Challenge (size priority):\n",
    "    #   - INT8 or INT4 quantization\n",
    "    #   - Aggressive pruning (70-90%)\n",
    "    #   - Compression techniques\n",
    "    #\n",
    "    # All-Around (balanced):\n",
    "    #   - INT8 quantization\n",
    "    #   - Moderate pruning (50-70%)\n",
    "    #   - Selective optimization\n",
    "    #\n",
    "    # Your strategy:\n",
    "    \n",
    "    \n",
    "    \n",
    "    # ============================================================\n",
    "    # YOUR CODE ABOVE\n",
    "    # ============================================================\n",
    "    \n",
    "    print(\"✅ Optimization complete!\")\n",
    "    print(\"💡 Tip: Benchmark your result to see the impact!\")\n",
    "    \n",
    "    return optimized_model\n",
    "\n",
    "#| export\n",
    "def validate_submission(submission: Dict[str, Any]) -> Dict[str, Any]:\n",
    "    \"\"\"\n",
    "    Validate competition submission with sanity checks.\n",
    "    \n",
    "    This catches honest mistakes like unrealistic speedups or accidental training.\n",
    "    Honor code system - we trust but verify basic reasonableness.\n",
    "    \n",
    "    Args:\n",
    "        submission: Submission dictionary to validate\n",
    "        \n",
    "    Returns:\n",
    "        Dict with validation results and warnings\n",
    "    \"\"\"\n",
    "    checks = []\n",
    "    warnings = []\n",
    "    errors = []\n",
    "    \n",
    "    # Extract metrics\n",
    "    normalized = submission.get(\"normalized_scores\", {})\n",
    "    speedup = normalized.get(\"speedup\", 1.0)\n",
    "    compression = normalized.get(\"compression_ratio\", 1.0)\n",
    "    accuracy_delta = normalized.get(\"accuracy_delta\", 0.0)\n",
    "    \n",
    "    # Check 1: Speedup is reasonable (not claiming impossible gains)\n",
    "    if speedup > 50:\n",
    "        errors.append(f\"❌ Speedup {speedup:.1f}x seems unrealistic (>50x)\")\n",
    "    elif speedup > 20:\n",
    "        warnings.append(f\"⚠️  Speedup {speedup:.1f}x is very high - please verify measurements\")\n",
    "    else:\n",
    "        checks.append(f\"✅ Speedup {speedup:.2f}x is reasonable\")\n",
    "    \n",
    "    # Check 2: Compression is reasonable\n",
    "    if compression > 32:\n",
    "        errors.append(f\"❌ Compression {compression:.1f}x seems unrealistic (>32x)\")\n",
    "    elif compression > 16:\n",
    "        warnings.append(f\"⚠️  Compression {compression:.1f}x is very high - please verify\")\n",
    "    else:\n",
    "        checks.append(f\"✅ Compression {compression:.2f}x is reasonable\")\n",
    "    \n",
    "    # Check 3: Accuracy didn't improve (Closed Division rule - no training allowed!)\n",
    "    division = submission.get(\"division\", \"closed\")\n",
    "    if division == \"closed\" and accuracy_delta > 1.0:\n",
    "        errors.append(f\"❌ Accuracy improved by {accuracy_delta:.1f}pp - did you accidentally train the model?\")\n",
    "    elif accuracy_delta > 0.5:\n",
    "        warnings.append(f\"⚠️  Accuracy improved by {accuracy_delta:.1f}pp - verify no training occurred\")\n",
    "    else:\n",
    "        checks.append(f\"✅ Accuracy change {accuracy_delta:+.2f}pp is reasonable\")\n",
    "    \n",
    "    # Check 4: GitHub repo provided\n",
    "    github_repo = submission.get(\"github_repo\", \"\")\n",
    "    if not github_repo or github_repo == \"\":\n",
    "        warnings.append(\"⚠️  No GitHub repo provided - required for verification\")\n",
    "    else:\n",
    "        checks.append(f\"✅ GitHub repo provided: {github_repo}\")\n",
    "    \n",
    "    # Check 5: Required fields present\n",
    "    required_fields = [\"division\", \"event\", \"athlete_name\", \"baseline\", \"optimized\", \"normalized_scores\"]\n",
    "    missing = [f for f in required_fields if f not in submission]\n",
    "    if missing:\n",
    "        errors.append(f\"❌ Missing required fields: {', '.join(missing)}\")\n",
    "    else:\n",
    "        checks.append(\"✅ All required fields present\")\n",
    "    \n",
    "    # Check 6: Techniques documented\n",
    "    techniques = submission.get(\"techniques_applied\", [])\n",
    "    if not techniques or \"TODO\" in str(techniques):\n",
    "        warnings.append(\"⚠️  No optimization techniques listed\")\n",
    "    else:\n",
    "        checks.append(f\"✅ Techniques documented: {', '.join(techniques[:3])}...\")\n",
    "    \n",
    "    return {\n",
    "        \"valid\": len(errors) == 0,\n",
    "        \"checks\": checks,\n",
    "        \"warnings\": warnings,\n",
    "        \"errors\": errors\n",
    "    }\n",
    "\n",
    "#| export\n",
    "def generate_submission(baseline_model, optimized_model, \n",
    "                       division: str = \"closed\",\n",
    "                       event: str = \"all_around\",\n",
    "                       athlete_name: str = \"YourName\",\n",
    "                       github_repo: str = \"\",\n",
    "                       techniques: List[str] = None) -> Dict[str, Any]:\n",
    "    \"\"\"\n",
    "    Generate standardized TinyMLPerf competition submission with normalized scoring.\n",
    "    \n",
    "    Args:\n",
    "        baseline_model: Original unoptimized model\n",
    "        optimized_model: Your optimized model\n",
    "        division: \"closed\" or \"open\"\n",
    "        event: Competition category (latency_sprint, memory_challenge, all_around, etc.)\n",
    "        athlete_name: Your name for submission\n",
    "        github_repo: GitHub repository URL for code verification\n",
    "        techniques: List of optimization techniques applied\n",
    "    \n",
    "    Returns:\n",
    "        Submission dictionary (will be saved as JSON)\n",
    "    \"\"\"\n",
    "    print(\"📤 Generating TinyMLPerf Competition Submission...\")\n",
    "    print(\"=\" * 70)\n",
    "    \n",
    "    # Get baseline metrics\n",
    "    baseline_metrics = generate_baseline(quick=True)\n",
    "    \n",
    "    # Benchmark optimized model\n",
    "    print(\"🔬 Benchmarking optimized model...\")\n",
    "    \n",
    "    # Use Profiler and Benchmark from Module 19\n",
    "    profiler = Profiler()\n",
    "    \n",
    "    # For demonstration, we'll use placeholder metrics\n",
    "    # In real competition, students would measure their actual optimized model\n",
    "    optimized_metrics = {\n",
    "        \"model\": getattr(optimized_model, 'name', 'Optimized_Model'),\n",
    "        \"accuracy\": 84.0,  # Would be measured with actual test set\n",
    "        \"latency_ms\": 28.0,  # Would be measured with profiler\n",
    "        \"memory_mb\": 4.0,  # Would be measured with profiler\n",
    "        \"parameters\": 2000000,  # Would be counted\n",
    "    }\n",
    "    \n",
    "    # Calculate normalized scores using Module 19's function\n",
    "    baseline_for_norm = {\n",
    "        \"latency\": baseline_metrics[\"latency_ms\"],\n",
    "        \"memory\": baseline_metrics[\"memory_mb\"],\n",
    "        \"accuracy\": baseline_metrics[\"accuracy\"]\n",
    "    }\n",
    "    \n",
    "    optimized_for_norm = {\n",
    "        \"latency\": optimized_metrics[\"latency_ms\"],\n",
    "        \"memory\": optimized_metrics[\"memory_mb\"],\n",
    "        \"accuracy\": optimized_metrics[\"accuracy\"]\n",
    "    }\n",
    "    \n",
    "    normalized_scores = calculate_normalized_scores(baseline_for_norm, optimized_for_norm)\n",
    "    \n",
    "    # Create submission with all required fields\n",
    "    submission = {\n",
    "        \"division\": division,\n",
    "        \"event\": event,\n",
    "        \"athlete_name\": athlete_name,\n",
    "        \"github_repo\": github_repo,\n",
    "        \"baseline\": baseline_metrics,\n",
    "        \"optimized\": optimized_metrics,\n",
    "        \"normalized_scores\": {\n",
    "            \"speedup\": normalized_scores[\"speedup\"],\n",
    "            \"compression_ratio\": normalized_scores[\"compression_ratio\"],\n",
    "            \"accuracy_delta\": normalized_scores[\"accuracy_delta\"],\n",
    "            \"efficiency_score\": normalized_scores[\"efficiency_score\"]\n",
    "        },\n",
    "        \"techniques_applied\": techniques or [\"TODO: Document your optimization techniques\"],\n",
    "        \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n",
    "        \"tinytorch_version\": \"0.1.0\",\n",
    "        \"honor_code\": False  # Must be explicitly set to True after validation\n",
    "    }\n",
    "    \n",
    "    # Validate submission\n",
    "    print(\"\\n🔍 Validating submission...\")\n",
    "    validation = validate_submission(submission)\n",
    "    \n",
    "    # Display validation results\n",
    "    print(\"\\n📋 Validation Results:\")\n",
    "    for check in validation[\"checks\"]:\n",
    "        print(f\"  {check}\")\n",
    "    for warning in validation[\"warnings\"]:\n",
    "        print(f\"  {warning}\")\n",
    "    for error in validation[\"errors\"]:\n",
    "        print(f\"  {error}\")\n",
    "    \n",
    "    if not validation[\"valid\"]:\n",
    "        print(\"\\n❌ Submission has errors - please fix before submitting\")\n",
    "        return submission\n",
    "    \n",
    "    # Save to JSON\n",
    "    output_file = Path(\"submission.json\")\n",
    "    with open(output_file, \"w\") as f:\n",
    "        json.dump(submission, f, indent=2)\n",
    "    \n",
    "    print(f\"\\n✅ Submission saved to: {output_file}\")\n",
    "    print()\n",
    "    print(\"📊 Your Normalized Scores (MLPerf-style):\")\n",
    "    print(f\"  Division:        {division.upper()}\")\n",
    "    print(f\"  Event:           {event.replace('_', ' ').title()}\")\n",
    "    print(f\"  Speedup:         {normalized_scores['speedup']:.2f}x faster ⚡\")\n",
    "    print(f\"  Compression:     {normalized_scores['compression_ratio']:.2f}x smaller 💾\")\n",
    "    print(f\"  Accuracy:        {optimized_metrics['accuracy']:.1f}% (Δ {normalized_scores['accuracy_delta']:+.2f}pp)\")\n",
    "    print(f\"  Efficiency:      {normalized_scores['efficiency_score']:.2f}\")\n",
    "    print()\n",
    "    print(\"📤 Next Steps:\")\n",
    "    print(\"  1. Verify all metrics are correct\")\n",
    "    print(\"  2. Push your code to GitHub (if not done)\")\n",
    "    print(\"  3. Run: tito submit submission.json\")\n",
    "    print(\"     (This will validate and prepare final submission)\")\n",
    "    print()\n",
    "    print(\"=\" * 70)\n",
    "    \n",
    "    return submission"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e95a6680",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "# 5. Module Integration Test\n",
    "\n",
    "Complete validation and competition workflow test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "914aaac9",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "test-module",
     "locked": true,
     "points": 10
    }
   },
   "outputs": [],
   "source": [
    "def test_module():\n",
    "    \"\"\"\n",
    "    Complete test of Module 20 functionality.\n",
    "    \n",
    "    This validates:\n",
    "    - Installation validation works\n",
    "    - Baseline generation works\n",
    "    - Worked example runs successfully\n",
    "    - Competition template is ready\n",
    "    \"\"\"\n",
    "    print(\"=\" * 70)\n",
    "    print(\"MODULE 20 INTEGRATION TEST\")\n",
    "    print(\"=\" * 70)\n",
    "    print()\n",
    "    \n",
    "    # Test 1: Validation\n",
    "    print(\"🔧 Test 1: System Validation\")\n",
    "    validation_status = validate_installation()\n",
    "    assert len(validation_status) > 0, \"Validation should return status dict\"\n",
    "    print(\"✅ Validation working!\")\n",
    "    print()\n",
    "    \n",
    "    # Test 2: Baseline Generation\n",
    "    print(\"📊 Test 2: Baseline Generation\")\n",
    "    baseline = generate_baseline(quick=True)\n",
    "    assert \"accuracy\" in baseline, \"Baseline should include accuracy\"\n",
    "    assert \"latency_ms\" in baseline, \"Baseline should include latency\"\n",
    "    assert \"memory_mb\" in baseline, \"Baseline should include memory\"\n",
    "    print(\"✅ Baseline generation working!\")\n",
    "    print()\n",
    "    \n",
    "    # Test 3: Worked Example\n",
    "    print(\"🏅 Test 3: Worked Example\")\n",
    "    example_submission = worked_example_optimization()\n",
    "    assert \"event\" in example_submission, \"Submission should include event\"\n",
    "    assert \"baseline\" in example_submission, \"Submission should include baseline\"\n",
    "    assert \"optimized\" in example_submission, \"Submission should include optimized\"\n",
    "    print(\"✅ Worked example working!\")\n",
    "    print()\n",
    "    \n",
    "    # Test 4: Competition Template\n",
    "    print(\"🎯 Test 4: Competition Template\")\n",
    "    baseline_model = load_baseline_model(\"cifar10_cnn\")\n",
    "    optimized = optimize_for_competition(baseline_model, event=\"all_around\")\n",
    "    assert optimized is not None, \"Optimization should return model\"\n",
    "    print(\"✅ Competition template working!\")\n",
    "    print()\n",
    "    \n",
    "    print(\"=\" * 70)\n",
    "    print(\"✅ ALL TESTS PASSED!\")\n",
    "    print(\"=\" * 70)\n",
    "    print()\n",
    "    print(\"🎉 You're ready for TorchPerf Olympics!\")\n",
    "    print(\"   Next steps:\")\n",
    "    print(\"   1. Implement your optimization strategy in optimize_for_competition()\")\n",
    "    print(\"   2. Run this module to generate submission.json\")\n",
    "    print(\"   3. Upload to competition platform\")\n",
    "    print()\n",
    "    print(\"🔥 Good luck! May the best optimizer win! 🏅\")\n",
    "\n",
    "test_module()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef195c7",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🤔 ML Systems Thinking: Competition as Learning\n",
    "\n",
    "TorchPerf Olympics isn't just about winning - it's about understanding trade-offs:\n",
    "\n",
    "**The Meta-Lesson**: Every optimization involves trade-offs:\n",
    "- Quantization: Speed vs Accuracy\n",
    "- Pruning: Size vs Performance\n",
    "- Caching: Memory vs Speed\n",
    "\n",
    "Professional ML engineers navigate these trade-offs daily. The competition forces you to:\n",
    "1. **Think systematically** about optimization strategies\n",
    "2. **Measure rigorously** using benchmarking tools\n",
    "3. **Make data-driven decisions** based on actual measurements\n",
    "4. **Document and justify** your choices\n",
    "\n",
    "The best submission isn't always the \"fastest\" or \"smallest\" - it's the one that best understands and navigates the trade-off space for their chosen event.\n",
    "\n",
    "What will your strategy be? 🤔"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f38935",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 2
   },
   "source": [
    "## 🎯 MODULE SUMMARY: Competition & Validation\n",
    "\n",
    "**What You've Learned:**\n",
    "- ✅ How to validate your TinyTorch installation\n",
    "- ✅ How to generate baseline performance metrics\n",
    "- ✅ How to combine optimization techniques systematically\n",
    "- ✅ How to benchmark and measure impact\n",
    "- ✅ How to generate standardized competition submissions\n",
    "\n",
    "**The Complete Workflow:**\n",
    "```\n",
    "1. Validate  → Ensure environment works\n",
    "2. Baseline  → Establish reference performance\n",
    "3. Optimize  → Apply techniques from M14-18\n",
    "4. Benchmark → Measure impact using M19\n",
    "5. Submit    → Generate standardized submission\n",
    "```\n",
    "\n",
    "**Key Takeaway**: Competition teaches systematic optimization thinking. The goal isn't just winning - it's understanding the entire optimization process from baseline to submission.\n",
    "\n",
    "**Next Steps:**\n",
    "1. Study the worked example\n",
    "2. Implement your own optimization strategy\n",
    "3. Benchmark your results\n",
    "4. Generate submission.json\n",
    "5. Compete in TorchPerf Olympics!\n",
    "\n",
    "🔥 Now go optimize and win gold! 🏅"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
