mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-06-03 08:51:06 -05:00
89566d26f3
- Regenerate all .ipynb files from fixed .py modules - Update tinytorch package exports with corrected implementations - Sync package module index with current 16-module structure These generated files reflect all the module fixes and ensure consistent .py ↔ .ipynb conversion with the updated module implementations.
2332 lines
102 KiB
Plaintext
2332 lines
102 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "451ae6b3",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"# Benchmarking - Systematic Performance Analysis and Bottleneck Identification\n",
|
|
"\n",
|
|
"Welcome to the Benchmarking module! You'll build professional benchmarking tools that identify performance bottlenecks and enable data-driven optimization decisions in ML systems.\n",
|
|
"\n",
|
|
"## Learning Goals\n",
|
|
"- Systems understanding: How systematic performance measurement reveals bottlenecks and guides optimization priorities in complex ML systems\n",
|
|
"- Core implementation skill: Build comprehensive benchmarking frameworks with statistical validation and professional reporting\n",
|
|
"- Pattern recognition: Understand how different workload patterns (latency vs throughput) require different measurement strategies\n",
|
|
"- Framework connection: See how your benchmarking approach mirrors industry standards like MLPerf and production monitoring systems\n",
|
|
"- Performance insight: Learn why measurement methodology often matters more than absolute numbers for optimization decisions\n",
|
|
"\n",
|
|
"## Build → Use → Reflect\n",
|
|
"1. **Build**: Complete benchmarking suite with MLPerf-inspired scenarios, statistical validation, and professional reporting\n",
|
|
"2. **Use**: Apply systematic evaluation to TinyTorch models and identify performance bottlenecks across the entire system\n",
|
|
"3. **Reflect**: Why do measurement artifacts often mislead optimization efforts, and how does proper benchmarking guide development?\n",
|
|
"\n",
|
|
"## What You'll Achieve\n",
|
|
"By the end of this module, you'll understand:\n",
|
|
"- Deep technical understanding of how to design benchmarks that reveal actionable insights about system performance\n",
|
|
"- Practical capability to build measurement infrastructure that guides optimization decisions and tracks system improvements\n",
|
|
"- Systems insight into why benchmarking methodology determines the reliability and usefulness of performance data\n",
|
|
"- Performance consideration of how measurement overhead and statistical variance affect benchmark validity\n",
|
|
"- Connection to production ML systems and how companies use systematic benchmarking to optimize deployment and hardware decisions\n",
|
|
"\n",
|
|
"## Systems Reality Check\n",
|
|
"💡 **Production Context**: Companies like Google and Facebook run continuous benchmarking across thousands of models to guide infrastructure investments and optimization priorities\n",
|
|
"⚡ **Performance Note**: Poor benchmarking methodology can lead to optimizing the wrong bottlenecks - measurement artifacts often overwhelm real performance differences"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e392090d",
|
|
"metadata": {
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "benchmarking-imports",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| default_exp core.benchmarking\n",
|
|
"\n",
|
|
"#| export\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import time\n",
|
|
"import statistics\n",
|
|
"import math\n",
|
|
"from typing import Dict, List, Tuple, Optional, Any, Callable\n",
|
|
"from enum import Enum\n",
|
|
"from dataclasses import dataclass\n",
|
|
"import os\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"# Import our TinyTorch dependencies\n",
|
|
"try:\n",
|
|
" from tinytorch.core.tensor import Tensor\n",
|
|
" from tinytorch.core.networks import Sequential\n",
|
|
" from tinytorch.core.layers import Dense\n",
|
|
" from tinytorch.core.activations import ReLU, Softmax\n",
|
|
" from tinytorch.core.dataloader import DataLoader\n",
|
|
"except ImportError:\n",
|
|
" # For development, import from local modules\n",
|
|
" parent_dirs = [\n",
|
|
" os.path.join(os.path.dirname(__file__), '..', '01_tensor'),\n",
|
|
" os.path.join(os.path.dirname(__file__), '..', '03_layers'),\n",
|
|
" os.path.join(os.path.dirname(__file__), '..', '02_activations'),\n",
|
|
" os.path.join(os.path.dirname(__file__), '..', '04_networks'),\n",
|
|
" os.path.join(os.path.dirname(__file__), '..', '06_dataloader')\n",
|
|
" ]\n",
|
|
" for path in parent_dirs:\n",
|
|
" if path not in sys.path:\n",
|
|
" sys.path.append(path)\n",
|
|
" \n",
|
|
" try:\n",
|
|
" from tensor_dev import Tensor\n",
|
|
" from networks_dev import Sequential\n",
|
|
" from layers_dev import Dense\n",
|
|
" from activations_dev import ReLU, Softmax\n",
|
|
" from dataloader_dev import DataLoader\n",
|
|
" except ImportError:\n",
|
|
" # Fallback for missing modules\n",
|
|
" print(\"⚠️ Some TinyTorch modules not available - using minimal implementations\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9b0e028d",
|
|
"metadata": {
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "benchmarking-welcome",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(\"📊 TinyTorch Benchmarking Module\")\n",
|
|
"print(f\"NumPy version: {np.__version__}\")\n",
|
|
"print(f\"Python version: {sys.version_info.major}.{sys.version_info.minor}\")\n",
|
|
"print(\"Ready to build professional ML benchmarking tools!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "272f30c5",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 📦 Where This Code Lives in the Final Package\n",
|
|
"\n",
|
|
"**Learning Side:** You work in `modules/source/14_benchmarking/benchmarking_dev.py` \n",
|
|
"**Building Side:** Code exports to `tinytorch.core.benchmarking`\n",
|
|
"\n",
|
|
"```python\n",
|
|
"# Final package structure:\n",
|
|
"from tinytorch.core.benchmarking import TinyTorchPerf, BenchmarkScenarios\n",
|
|
"from tinytorch.core.benchmarking import StatisticalValidator, PerformanceReporter\n",
|
|
"```\n",
|
|
"\n",
|
|
"**Why this matters:**\n",
|
|
"- **Learning:** Deep understanding of systematic evaluation\n",
|
|
"- **Production:** Professional benchmarking methodology\n",
|
|
"- **Projects:** Tools for validating your ML project performance\n",
|
|
"- **Career:** Industry-standard skills for ML engineering roles"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e8b5bb39",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## What is ML Benchmarking?\n",
|
|
"\n",
|
|
"### The Systematic Evaluation Problem\n",
|
|
"When you build ML systems, you need to answer critical questions:\n",
|
|
"- **Is my model actually better?** Statistical significance vs random variation\n",
|
|
"- **How does it perform in production?** Latency, throughput, resource usage\n",
|
|
"- **Which approach should I choose?** Systematic comparison methodology\n",
|
|
"- **Can I trust my results?** Avoiding common benchmarking pitfalls\n",
|
|
"\n",
|
|
"### The MLPerf Architecture\n",
|
|
"MLPerf (Machine Learning Performance) defines the industry standard for ML benchmarking:\n",
|
|
"\n",
|
|
"```\n",
|
|
"┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n",
|
|
"│ Load Generator │───▶│ System Under │───▶│ Dataset │\n",
|
|
"│ (Controls │ │ Test (Your ML │ │ (Standardized │\n",
|
|
"│ Queries) │ │ Model) │ │ Evaluation) │\n",
|
|
"└─────────────────┘ └─────────────────┘ └─────────────────┘\n",
|
|
"```\n",
|
|
"\n",
|
|
"### The Four Components\n",
|
|
"1. **System Under Test (SUT)**: Your ML model/system being evaluated\n",
|
|
"2. **Dataset**: Standardized evaluation data (CIFAR-10, ImageNet, etc.)\n",
|
|
"3. **Model**: The specific architecture and weights being tested\n",
|
|
"4. **Load Generator**: Controls how evaluation queries are sent to the SUT\n",
|
|
"\n",
|
|
"### Why This Matters\n",
|
|
"- **Reproducibility**: Others can verify your results\n",
|
|
"- **Comparability**: Fair comparison between different approaches\n",
|
|
"- **Statistical validity**: Meaningful conclusions from your data\n",
|
|
"- **Industry standards**: Skills you'll use in ML engineering careers\n",
|
|
"\n",
|
|
"### Real-World Examples\n",
|
|
"- **Google**: Uses similar patterns for production ML system evaluation\n",
|
|
"- **Meta**: A/B testing frameworks follow these principles\n",
|
|
"- **OpenAI**: GPT model comparisons use systematic benchmarking\n",
|
|
"- **Research**: All major ML conferences require proper evaluation methodology"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5ab97147",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 🔧 DEVELOPMENT"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8fbf6189",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## Step 1: Benchmark Scenarios - How to Measure Performance\n",
|
|
"\n",
|
|
"### The Three Standard Scenarios\n",
|
|
"Different use cases require different performance measurements:\n",
|
|
"\n",
|
|
"#### 1. Single-Stream Scenario\n",
|
|
"- **Use case**: Mobile/edge inference, interactive applications\n",
|
|
"- **Pattern**: Send next query only after previous completes\n",
|
|
"- **Metric**: 90th percentile latency (tail latency)\n",
|
|
"- **Why**: Users care about worst-case response time\n",
|
|
"\n",
|
|
"#### 2. Server Scenario \n",
|
|
"- **Use case**: Production web services, API endpoints\n",
|
|
"- **Pattern**: Poisson distribution of concurrent queries\n",
|
|
"- **Metric**: Queries per second (QPS) at acceptable latency\n",
|
|
"- **Why**: Servers handle multiple simultaneous requests\n",
|
|
"\n",
|
|
"#### 3. Offline Scenario\n",
|
|
"- **Use case**: Batch processing, data center workloads\n",
|
|
"- **Pattern**: Send all samples at once for batch processing\n",
|
|
"- **Metric**: Throughput (samples per second)\n",
|
|
"- **Why**: Batch jobs care about total processing time\n",
|
|
"\n",
|
|
"### Mathematical Foundation\n",
|
|
"Each scenario tests different aspects:\n",
|
|
"- **Latency**: Time for single sample = f(model_complexity, hardware)\n",
|
|
"- **Throughput**: Samples per second = f(parallelism, batch_size)\n",
|
|
"- **Efficiency**: Resource utilization = f(memory, compute, bandwidth)\n",
|
|
"\n",
|
|
"### Why Multiple Scenarios?\n",
|
|
"Real ML systems have different requirements:\n",
|
|
"- **Chatbot**: Low latency for good user experience\n",
|
|
"- **Image API**: High throughput for many concurrent users \n",
|
|
"- **Data pipeline**: Maximum batch processing efficiency"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1c52fdee",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"## Step 2: Statistical Validation - Ensuring Meaningful Results\n",
|
|
"\n",
|
|
"### The Significance Problem\n",
|
|
"Common benchmarking mistakes:\n",
|
|
"```python\n",
|
|
"# BAD: Single run, no statistical validation\n",
|
|
"result_a = model_a.run_once() # 94.2% accuracy\n",
|
|
"result_b = model_b.run_once() # 94.7% accuracy\n",
|
|
"print(\"Model B is better!\") # Maybe, maybe not...\n",
|
|
"```\n",
|
|
"\n",
|
|
"### The MLPerf Solution\n",
|
|
"Proper statistical validation:\n",
|
|
"```python\n",
|
|
"# GOOD: Multiple runs with confidence intervals\n",
|
|
"results_a = [model_a.run() for _ in range(10)] # [93.8, 94.1, 94.3, ...]\n",
|
|
"results_b = [model_b.run() for _ in range(10)] # [94.2, 94.5, 94.9, ...]\n",
|
|
"significance = statistical_test(results_a, results_b)\n",
|
|
"print(f\"Model B is {significance.p_value < 0.05} better with p={significance.p_value}\")\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Key Statistical Concepts\n",
|
|
"- **Confidence intervals**: Range of likely true values\n",
|
|
"- **P-values**: Probability that difference is due to chance\n",
|
|
"- **Effect size**: Magnitude of improvement (not just significance)\n",
|
|
"- **Multiple comparisons**: Adjusting for testing many approaches\n",
|
|
"\n",
|
|
"### Sample Size Calculation\n",
|
|
"MLPerf uses this formula for minimum samples:\n",
|
|
"```\n",
|
|
"n = Φ^(-1)((1-C)/2)^2 * p(1-p) / MOE^2\n",
|
|
"```\n",
|
|
"Where:\n",
|
|
"- C = confidence level (0.99)\n",
|
|
"- p = percentile (0.90 for 90th percentile)\n",
|
|
"- MOE = margin of error ((1-p)/20)\n",
|
|
"\n",
|
|
"For 90th percentile with 99% confidence: **n = 24,576 samples**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3f3c2a5f",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "benchmark-scenarios",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": true,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class BenchmarkScenario(Enum):\n",
|
|
" \"\"\"Standard benchmark scenarios from MLPerf\"\"\"\n",
|
|
" SINGLE_STREAM = \"single_stream\"\n",
|
|
" SERVER = \"server\"\n",
|
|
" OFFLINE = \"offline\"\n",
|
|
"\n",
|
|
"@dataclass\n",
|
|
"class BenchmarkResult:\n",
|
|
" \"\"\"Results from a benchmark run\"\"\"\n",
|
|
" scenario: BenchmarkScenario\n",
|
|
" latencies: List[float] # All latency measurements in seconds\n",
|
|
" throughput: float # Samples per second\n",
|
|
" accuracy: float # Model accuracy (0-1)\n",
|
|
" metadata: Optional[Dict[str, Any]] = None\n",
|
|
"\n",
|
|
"#| export\n",
|
|
"class BenchmarkScenarios:\n",
|
|
" \"\"\"\n",
|
|
" Implements the three standard MLPerf benchmark scenarios.\n",
|
|
" \n",
|
|
" TODO: Implement the three benchmark scenarios following MLPerf patterns.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. Single-Stream: Send queries one at a time, measure latency\n",
|
|
" 2. Server: Send queries following Poisson distribution, measure QPS\n",
|
|
" 3. Offline: Send all queries at once, measure total throughput\n",
|
|
" \n",
|
|
" IMPLEMENTATION APPROACH:\n",
|
|
" 1. Each scenario should run the model multiple times\n",
|
|
" 2. Collect latency measurements for each run\n",
|
|
" 3. Calculate appropriate metrics for each scenario\n",
|
|
" 4. Return BenchmarkResult with all measurements\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **MLPerf Standards**: Industry-standard benchmarking methodology used by Google, NVIDIA, etc.\n",
|
|
" - **Performance Scenarios**: Different deployment patterns require different measurement approaches\n",
|
|
" - **Production Validation**: Benchmarking validates model performance before deployment\n",
|
|
" - **Resource Planning**: Results guide infrastructure scaling and capacity planning\n",
|
|
" \n",
|
|
" EXAMPLE USAGE:\n",
|
|
" scenarios = BenchmarkScenarios()\n",
|
|
" result = scenarios.single_stream(model, dataset, num_queries=1000)\n",
|
|
" print(f\"90th percentile latency: {result.latencies[int(0.9 * len(result.latencies))]} seconds\")\n",
|
|
" \"\"\"\n",
|
|
" \n",
|
|
" def __init__(self):\n",
|
|
" self.results = []\n",
|
|
" \n",
|
|
" def single_stream(self, model: Callable, dataset: List, num_queries: int = 1000) -> BenchmarkResult:\n",
|
|
" \"\"\"\n",
|
|
" Run single-stream benchmark scenario.\n",
|
|
" \n",
|
|
" TODO: Implement single-stream benchmarking.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. Initialize empty list for latencies\n",
|
|
" 2. For each query (up to num_queries):\n",
|
|
" a. Get next sample from dataset (cycle if needed)\n",
|
|
" b. Record start time\n",
|
|
" c. Run model on sample\n",
|
|
" d. Record end time\n",
|
|
" e. Calculate latency = end - start\n",
|
|
" f. Add latency to list\n",
|
|
" 3. Calculate throughput = num_queries / total_time\n",
|
|
" 4. Calculate accuracy if possible\n",
|
|
" 5. Return BenchmarkResult with SINGLE_STREAM scenario\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **Mobile/Edge Deployment**: Single-stream simulates user-facing applications\n",
|
|
" - **Tail Latency**: 90th/95th percentiles matter more than averages for user experience\n",
|
|
" - **Interactive Systems**: Chatbots, recommendation engines use single-stream patterns\n",
|
|
" - **SLA Validation**: Ensures models meet response time requirements\n",
|
|
" \n",
|
|
" HINTS:\n",
|
|
" - Use time.perf_counter() for precise timing\n",
|
|
" - Use dataset[i % len(dataset)] to cycle through samples\n",
|
|
" - Sort latencies for percentile calculations\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" latencies = []\n",
|
|
" correct_predictions = 0\n",
|
|
" total_start_time = time.perf_counter()\n",
|
|
" \n",
|
|
" for i in range(num_queries):\n",
|
|
" # Get sample (cycle through dataset)\n",
|
|
" sample = dataset[i % len(dataset)]\n",
|
|
" \n",
|
|
" # Time the inference\n",
|
|
" start_time = time.perf_counter()\n",
|
|
" result = model(sample)\n",
|
|
" end_time = time.perf_counter()\n",
|
|
" \n",
|
|
" latency = end_time - start_time\n",
|
|
" latencies.append(latency)\n",
|
|
" \n",
|
|
" # Simple accuracy calculation (if possible)\n",
|
|
" if hasattr(sample, 'target') and hasattr(result, 'data'):\n",
|
|
" predicted = np.argmax(result.data)\n",
|
|
" if predicted == sample.target:\n",
|
|
" correct_predictions += 1\n",
|
|
" \n",
|
|
" total_time = time.perf_counter() - total_start_time\n",
|
|
" throughput = num_queries / total_time\n",
|
|
" accuracy = correct_predictions / num_queries if num_queries > 0 else 0.0\n",
|
|
" \n",
|
|
" return BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SINGLE_STREAM,\n",
|
|
" latencies=sorted(latencies),\n",
|
|
" throughput=throughput,\n",
|
|
" accuracy=accuracy,\n",
|
|
" metadata={\"num_queries\": num_queries}\n",
|
|
" )\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def server(self, model: Callable, dataset: List, target_qps: float = 10.0, \n",
|
|
" duration: float = 60.0) -> BenchmarkResult:\n",
|
|
" \"\"\"\n",
|
|
" Run server benchmark scenario with Poisson-distributed queries.\n",
|
|
" \n",
|
|
" TODO: Implement server benchmarking.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. Calculate inter-arrival time = 1.0 / target_qps\n",
|
|
" 2. Run for specified duration:\n",
|
|
" a. Wait for next query arrival (Poisson distribution)\n",
|
|
" b. Get sample from dataset\n",
|
|
" c. Record start time\n",
|
|
" d. Run model\n",
|
|
" e. Record end time and latency\n",
|
|
" 3. Calculate actual QPS = total_queries / duration\n",
|
|
" 4. Return results\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **Web Services**: Server scenario simulates API endpoints handling concurrent requests\n",
|
|
" - **Load Testing**: Validates system behavior under realistic traffic patterns\n",
|
|
" - **Scalability Analysis**: Tests how well models handle increasing load\n",
|
|
" - **Production Deployment**: Critical for microservices and web-scale applications\n",
|
|
" \n",
|
|
" HINTS:\n",
|
|
" - Use np.random.exponential(inter_arrival_time) for Poisson\n",
|
|
" - Track both query arrival times and completion times\n",
|
|
" - Server scenario cares about sustained throughput\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" latencies = []\n",
|
|
" inter_arrival_time = 1.0 / target_qps\n",
|
|
" start_time = time.perf_counter()\n",
|
|
" current_time = start_time\n",
|
|
" query_count = 0\n",
|
|
" \n",
|
|
" while (current_time - start_time) < duration:\n",
|
|
" # Wait for next query (Poisson distribution)\n",
|
|
" wait_time = np.random.exponential(inter_arrival_time)\n",
|
|
" # Use minimal delay for fast testing\n",
|
|
" if wait_time > 0.0001: # Only sleep for very long waits\n",
|
|
" time.sleep(min(wait_time, 0.0001))\n",
|
|
" \n",
|
|
" # Get sample\n",
|
|
" sample = dataset[query_count % len(dataset)]\n",
|
|
" \n",
|
|
" # Time the inference\n",
|
|
" query_start = time.perf_counter()\n",
|
|
" result = model(sample)\n",
|
|
" query_end = time.perf_counter()\n",
|
|
" \n",
|
|
" latency = query_end - query_start\n",
|
|
" latencies.append(latency)\n",
|
|
" \n",
|
|
" query_count += 1\n",
|
|
" current_time = time.perf_counter()\n",
|
|
" \n",
|
|
" actual_duration = current_time - start_time\n",
|
|
" actual_qps = query_count / actual_duration\n",
|
|
" \n",
|
|
" return BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SERVER,\n",
|
|
" latencies=sorted(latencies),\n",
|
|
" throughput=actual_qps,\n",
|
|
" accuracy=0.0, # Would need labels for accuracy\n",
|
|
" metadata={\"target_qps\": target_qps, \"actual_qps\": actual_qps, \"duration\": actual_duration}\n",
|
|
" )\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def offline(self, model: Callable, dataset: List, batch_size: int = 32) -> BenchmarkResult:\n",
|
|
" \"\"\"\n",
|
|
" Run offline benchmark scenario with batch processing.\n",
|
|
" \n",
|
|
" TODO: Implement offline benchmarking.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. Group dataset into batches of batch_size\n",
|
|
" 2. For each batch:\n",
|
|
" a. Record start time\n",
|
|
" b. Run model on entire batch\n",
|
|
" c. Record end time\n",
|
|
" d. Calculate batch latency\n",
|
|
" 3. Calculate total throughput = total_samples / total_time\n",
|
|
" 4. Return results\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **Batch Processing**: Offline scenario simulates data pipeline and ETL workloads\n",
|
|
" - **Throughput Optimization**: Maximizes processing efficiency for large datasets\n",
|
|
" - **Data Center Workloads**: Common in recommendation systems and analytics pipelines\n",
|
|
" - **Cost Optimization**: High throughput reduces compute costs per sample\n",
|
|
" \n",
|
|
" HINTS:\n",
|
|
" - Process data in batches for efficiency\n",
|
|
" - Measure total time for all batches\n",
|
|
" - Offline cares about maximum throughput\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" latencies = []\n",
|
|
" total_samples = len(dataset)\n",
|
|
" total_start_time = time.perf_counter()\n",
|
|
" \n",
|
|
" for batch_start in range(0, total_samples, batch_size):\n",
|
|
" batch_end = min(batch_start + batch_size, total_samples)\n",
|
|
" batch = dataset[batch_start:batch_end]\n",
|
|
" \n",
|
|
" # Time the batch inference\n",
|
|
" batch_start_time = time.perf_counter()\n",
|
|
" for sample in batch:\n",
|
|
" result = model(sample)\n",
|
|
" batch_end_time = time.perf_counter()\n",
|
|
" \n",
|
|
" batch_latency = batch_end_time - batch_start_time\n",
|
|
" latencies.append(batch_latency)\n",
|
|
" \n",
|
|
" total_time = time.perf_counter() - total_start_time\n",
|
|
" throughput = total_samples / total_time\n",
|
|
" \n",
|
|
" return BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.OFFLINE,\n",
|
|
" latencies=latencies,\n",
|
|
" throughput=throughput,\n",
|
|
" accuracy=0.0, # Would need labels for accuracy\n",
|
|
" metadata={\"batch_size\": batch_size, \"total_samples\": total_samples}\n",
|
|
" )\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "09ef7933",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"### 🧪 Unit Test: Benchmark Scenarios\n",
|
|
"\n",
|
|
"Let's test our benchmark scenarios with a simple mock model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cda6af90",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "test-scenarios",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_unit_benchmark_scenarios():\n",
|
|
" \"\"\"Unit test for the BenchmarkScenarios class.\"\"\"\n",
|
|
" print(\"🔬 Unit Test: Benchmark Scenarios...\")\n",
|
|
" \n",
|
|
" # Create a simple mock model and dataset\n",
|
|
" def mock_model(sample):\n",
|
|
" # Simulate minimal processing (avoid sleep for fast tests)\n",
|
|
" result = np.sum(sample.get(\"data\", [0])) * 0.001 # Fast computation\n",
|
|
" return {\"prediction\": np.random.rand(3)} # Smaller output\n",
|
|
" \n",
|
|
" mock_dataset = [{\"data\": np.random.rand(5)} for _ in range(10)] # Much smaller dataset\n",
|
|
" \n",
|
|
" # Test scenarios\n",
|
|
" scenarios = BenchmarkScenarios()\n",
|
|
" \n",
|
|
" # Test single-stream (fewer queries)\n",
|
|
" single_result = scenarios.single_stream(mock_model, mock_dataset, num_queries=3)\n",
|
|
" assert single_result.scenario == BenchmarkScenario.SINGLE_STREAM\n",
|
|
" assert len(single_result.latencies) == 3\n",
|
|
" assert single_result.throughput > 0\n",
|
|
" print(f\"✅ Single-stream: {len(single_result.latencies)} measurements\")\n",
|
|
" \n",
|
|
" # Test server (very short duration for testing)\n",
|
|
" server_result = scenarios.server(mock_model, mock_dataset, target_qps=10.0, duration=0.5)\n",
|
|
" assert server_result.scenario == BenchmarkScenario.SERVER\n",
|
|
" assert len(server_result.latencies) > 0\n",
|
|
" assert server_result.throughput > 0\n",
|
|
" print(f\"✅ Server: {len(server_result.latencies)} queries processed\")\n",
|
|
" \n",
|
|
" # Test offline (smaller batch)\n",
|
|
" offline_result = scenarios.offline(mock_model, mock_dataset, batch_size=2)\n",
|
|
" assert offline_result.scenario == BenchmarkScenario.OFFLINE\n",
|
|
" assert len(offline_result.latencies) > 0\n",
|
|
" assert offline_result.throughput > 0\n",
|
|
" print(f\"✅ Offline: {len(offline_result.latencies)} batches processed\")\n",
|
|
" \n",
|
|
" print(\"✅ All benchmark scenarios working correctly!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "92e57b90",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"## Step 3: Statistical Validation - Ensuring Meaningful Results\n",
|
|
"\n",
|
|
"### The Confidence Problem\n",
|
|
"How do we know if one model is actually better than another?\n",
|
|
"\n",
|
|
"### Statistical Testing for ML\n",
|
|
"We need to test the null hypothesis: \"There is no significant difference between models\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7c718ded",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "statistical-validator",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": true,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"@dataclass\n",
|
|
"class StatisticalValidation:\n",
|
|
" \"\"\"Results from statistical validation\"\"\"\n",
|
|
" is_significant: bool\n",
|
|
" p_value: float\n",
|
|
" effect_size: float\n",
|
|
" confidence_interval: Tuple[float, float]\n",
|
|
" recommendation: str\n",
|
|
"\n",
|
|
"#| export\n",
|
|
"class StatisticalValidator:\n",
|
|
" \"\"\"\n",
|
|
" Validates benchmark results using proper statistical methods.\n",
|
|
" \n",
|
|
" TODO: Implement statistical validation for benchmark results.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. Null hypothesis: No difference between models\n",
|
|
" 2. T-test: Compare means of two groups\n",
|
|
" 3. P-value: Probability of seeing this difference by chance\n",
|
|
" 4. Effect size: Magnitude of the difference\n",
|
|
" 5. Confidence interval: Range of likely true values\n",
|
|
" \n",
|
|
" IMPLEMENTATION APPROACH:\n",
|
|
" 1. Calculate basic statistics (mean, std, n)\n",
|
|
" 2. Perform t-test to get p-value\n",
|
|
" 3. Calculate effect size (Cohen's d)\n",
|
|
" 4. Calculate confidence interval\n",
|
|
" 5. Provide clear recommendation\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **Scientific Rigor**: Ensures performance claims are statistically valid\n",
|
|
" - **A/B Testing**: Foundation for production model comparison and rollout decisions\n",
|
|
" - **Research Validation**: Required for academic papers and technical reports\n",
|
|
" - **Business Decisions**: Statistical significance guides investment in new models\n",
|
|
" \"\"\"\n",
|
|
" \n",
|
|
" def __init__(self, confidence_level: float = 0.95):\n",
|
|
" self.confidence_level = confidence_level\n",
|
|
" self.alpha = 1 - confidence_level\n",
|
|
" \n",
|
|
" def validate_comparison(self, results_a: List[float], results_b: List[float]) -> StatisticalValidation:\n",
|
|
" \"\"\"\n",
|
|
" Compare two sets of benchmark results statistically.\n",
|
|
" \n",
|
|
" TODO: Implement statistical comparison.\n",
|
|
" \n",
|
|
" STEP-BY-STEP:\n",
|
|
" 1. Calculate basic statistics for both groups\n",
|
|
" 2. Perform two-sample t-test\n",
|
|
" 3. Calculate effect size (Cohen's d)\n",
|
|
" 4. Calculate confidence interval for the difference\n",
|
|
" 5. Generate recommendation based on results\n",
|
|
" \n",
|
|
" HINTS:\n",
|
|
" - Use scipy.stats.ttest_ind for t-test (or implement manually)\n",
|
|
" - Cohen's d = (mean_a - mean_b) / pooled_std\n",
|
|
" - CI = difference ± (critical_value * standard_error)\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" import math\n",
|
|
" \n",
|
|
" # Basic statistics\n",
|
|
" mean_a = statistics.mean(results_a)\n",
|
|
" mean_b = statistics.mean(results_b)\n",
|
|
" std_a = statistics.stdev(results_a)\n",
|
|
" std_b = statistics.stdev(results_b)\n",
|
|
" n_a = len(results_a)\n",
|
|
" n_b = len(results_b)\n",
|
|
" \n",
|
|
" # Two-sample t-test (simplified)\n",
|
|
" pooled_std = math.sqrt(((n_a - 1) * std_a**2 + (n_b - 1) * std_b**2) / (n_a + n_b - 2))\n",
|
|
" standard_error = pooled_std * math.sqrt(1/n_a + 1/n_b)\n",
|
|
" \n",
|
|
" if standard_error == 0:\n",
|
|
" t_stat = 0\n",
|
|
" p_value = 1.0\n",
|
|
" else:\n",
|
|
" t_stat = (mean_a - mean_b) / standard_error\n",
|
|
" # Simplified p-value calculation (assuming normal distribution)\n",
|
|
" p_value = 2 * (1 - abs(t_stat) / (abs(t_stat) + math.sqrt(n_a + n_b - 2)))\n",
|
|
" \n",
|
|
" # Effect size (Cohen's d)\n",
|
|
" effect_size = (mean_a - mean_b) / pooled_std if pooled_std > 0 else 0\n",
|
|
" \n",
|
|
" # Confidence interval for difference\n",
|
|
" difference = mean_a - mean_b\n",
|
|
" critical_value = 1.96 # Approximate for 95% CI\n",
|
|
" margin_of_error = critical_value * standard_error\n",
|
|
" ci_lower = difference - margin_of_error\n",
|
|
" ci_upper = difference + margin_of_error\n",
|
|
" \n",
|
|
" # Determine significance\n",
|
|
" is_significant = p_value < self.alpha\n",
|
|
" \n",
|
|
" # Generate recommendation\n",
|
|
" if is_significant:\n",
|
|
" if effect_size > 0.8:\n",
|
|
" recommendation = \"Large significant difference - strong evidence for improvement\"\n",
|
|
" elif effect_size > 0.5:\n",
|
|
" recommendation = \"Medium significant difference - good evidence for improvement\"\n",
|
|
" else:\n",
|
|
" recommendation = \"Small significant difference - weak evidence for improvement\"\n",
|
|
" else:\n",
|
|
" recommendation = \"No significant difference - insufficient evidence for improvement\"\n",
|
|
" \n",
|
|
" return StatisticalValidation(\n",
|
|
" is_significant=is_significant,\n",
|
|
" p_value=p_value,\n",
|
|
" effect_size=effect_size,\n",
|
|
" confidence_interval=(ci_lower, ci_upper),\n",
|
|
" recommendation=recommendation\n",
|
|
" )\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def validate_benchmark_result(self, result: BenchmarkResult, \n",
|
|
" min_samples: int = 100) -> StatisticalValidation:\n",
|
|
" \"\"\"\n",
|
|
" Validate that a benchmark result has sufficient statistical power.\n",
|
|
" \n",
|
|
" TODO: Implement validation for single benchmark result.\n",
|
|
" \n",
|
|
" STEP-BY-STEP:\n",
|
|
" 1. Check if we have enough samples\n",
|
|
" 2. Calculate confidence interval for the metric\n",
|
|
" 3. Check for common pitfalls (outliers, etc.)\n",
|
|
" 4. Provide recommendations\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" latencies = result.latencies\n",
|
|
" n = len(latencies)\n",
|
|
" \n",
|
|
" if n < min_samples:\n",
|
|
" return StatisticalValidation(\n",
|
|
" is_significant=False,\n",
|
|
" p_value=1.0,\n",
|
|
" effect_size=0.0,\n",
|
|
" confidence_interval=(0.0, 0.0),\n",
|
|
" recommendation=f\"Insufficient samples: {n} < {min_samples}. Need more data.\"\n",
|
|
" )\n",
|
|
" \n",
|
|
" # Calculate confidence interval for mean latency\n",
|
|
" mean_latency = statistics.mean(latencies)\n",
|
|
" std_latency = statistics.stdev(latencies)\n",
|
|
" standard_error = std_latency / math.sqrt(n)\n",
|
|
" \n",
|
|
" critical_value = 1.96 # 95% CI\n",
|
|
" margin_of_error = critical_value * standard_error\n",
|
|
" ci_lower = mean_latency - margin_of_error\n",
|
|
" ci_upper = mean_latency + margin_of_error\n",
|
|
" \n",
|
|
" # Check for outliers (simple check)\n",
|
|
" q1 = latencies[int(0.25 * n)]\n",
|
|
" q3 = latencies[int(0.75 * n)]\n",
|
|
" iqr = q3 - q1\n",
|
|
" outlier_threshold = q3 + 1.5 * iqr\n",
|
|
" outliers = [l for l in latencies if l > outlier_threshold]\n",
|
|
" \n",
|
|
" if len(outliers) > 0.1 * n: # More than 10% outliers\n",
|
|
" recommendation = f\"Warning: {len(outliers)} outliers detected. Results may be unreliable.\"\n",
|
|
" else:\n",
|
|
" recommendation = \"Benchmark result appears statistically valid.\"\n",
|
|
" \n",
|
|
" return StatisticalValidation(\n",
|
|
" is_significant=True,\n",
|
|
" p_value=0.0, # Not applicable for single result\n",
|
|
" effect_size=std_latency / mean_latency, # Coefficient of variation\n",
|
|
" confidence_interval=(ci_lower, ci_upper),\n",
|
|
" recommendation=recommendation\n",
|
|
" )\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "de9f9b7c",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"### 🧪 Unit Test: Statistical Validation\n",
|
|
"\n",
|
|
"Let's test our statistical validation with simulated data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ad767dfb",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "test-validation",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_unit_statistical_validation():\n",
|
|
" \"\"\"Unit test for the StatisticalValidator class.\"\"\"\n",
|
|
" print(\"🔬 Unit Test: Statistical Validation...\")\n",
|
|
" \n",
|
|
" validator = StatisticalValidator(confidence_level=0.95)\n",
|
|
" \n",
|
|
" # Test 1: No significant difference\n",
|
|
" results_a = [0.1 + 0.01 * np.random.randn() for _ in range(100)]\n",
|
|
" results_b = [0.1 + 0.01 * np.random.randn() for _ in range(100)]\n",
|
|
" \n",
|
|
" validation = validator.validate_comparison(results_a, results_b)\n",
|
|
" print(f\"✅ No difference test: significant={validation.is_significant}, p={validation.p_value:.4f}\")\n",
|
|
" \n",
|
|
" # Test 2: Clear significant difference\n",
|
|
" results_a = [0.1 + 0.01 * np.random.randn() for _ in range(100)]\n",
|
|
" results_b = [0.2 + 0.01 * np.random.randn() for _ in range(100)]\n",
|
|
" \n",
|
|
" validation = validator.validate_comparison(results_a, results_b)\n",
|
|
" print(f\"✅ Clear difference test: significant={validation.is_significant}, p={validation.p_value:.4f}\")\n",
|
|
" print(f\" Effect size: {validation.effect_size:.3f}\")\n",
|
|
" print(f\" Recommendation: {validation.recommendation}\")\n",
|
|
" \n",
|
|
" # Test 3: Single result validation\n",
|
|
" mock_result = BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SINGLE_STREAM,\n",
|
|
" latencies=[0.1 + 0.01 * np.random.randn() for _ in range(200)],\n",
|
|
" throughput=1000,\n",
|
|
" accuracy=0.95\n",
|
|
" )\n",
|
|
" \n",
|
|
" validation = validator.validate_benchmark_result(mock_result)\n",
|
|
" print(f\"✅ Single result validation: {validation.recommendation}\")\n",
|
|
" print(f\" Confidence interval: ({validation.confidence_interval[0]:.4f}, {validation.confidence_interval[1]:.4f})\")\n",
|
|
" \n",
|
|
" print(\"✅ Statistical validation tests passed!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8d9302a8",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"## Step 4: The TinyTorchPerf Framework - Putting It All Together\n",
|
|
"\n",
|
|
"### The Complete MLPerf-Inspired Framework\n",
|
|
"Now we combine all components into a professional benchmarking framework."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "13039465",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "tinytorch-perf",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": true,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class TinyTorchPerf:\n",
|
|
" \"\"\"\n",
|
|
" Complete MLPerf-inspired benchmarking framework for TinyTorch.\n",
|
|
" \n",
|
|
" TODO: Implement the complete benchmarking framework.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. Combines all benchmark scenarios\n",
|
|
" 2. Integrates statistical validation\n",
|
|
" 3. Provides easy-to-use API\n",
|
|
" 4. Generates professional reports\n",
|
|
" \n",
|
|
" IMPLEMENTATION APPROACH:\n",
|
|
" 1. Initialize with model and dataset\n",
|
|
" 2. Provide methods for each scenario\n",
|
|
" 3. Include statistical validation\n",
|
|
" 4. Generate comprehensive reports\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **MLPerf Integration**: Follows industry-standard benchmarking patterns\n",
|
|
" - **Production Deployment**: Validates models before production rollout\n",
|
|
" - **Performance Engineering**: Identifies bottlenecks and optimization opportunities\n",
|
|
" - **Framework Design**: Demonstrates how to build reusable ML tools\n",
|
|
" \"\"\"\n",
|
|
" \n",
|
|
" def __init__(self):\n",
|
|
" self.scenarios = BenchmarkScenarios()\n",
|
|
" self.validator = StatisticalValidator()\n",
|
|
" self.model = None\n",
|
|
" self.dataset = None\n",
|
|
" self.results = {}\n",
|
|
" \n",
|
|
" def set_model(self, model: Callable):\n",
|
|
" \"\"\"Set the model to benchmark.\"\"\"\n",
|
|
" self.model = model\n",
|
|
" \n",
|
|
" def set_dataset(self, dataset: List):\n",
|
|
" \"\"\"Set the dataset for benchmarking.\"\"\"\n",
|
|
" self.dataset = dataset\n",
|
|
" \n",
|
|
" def run_single_stream(self, num_queries: int = 1000) -> BenchmarkResult:\n",
|
|
" \"\"\"\n",
|
|
" Run single-stream benchmark.\n",
|
|
" \n",
|
|
" TODO: Implement single-stream benchmark with validation.\n",
|
|
" \n",
|
|
" STEP-BY-STEP:\n",
|
|
" 1. Check that model and dataset are set\n",
|
|
" 2. Run single-stream scenario\n",
|
|
" 3. Validate results statistically\n",
|
|
" 4. Store results\n",
|
|
" 5. Return result\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" if self.model is None or self.dataset is None:\n",
|
|
" raise ValueError(\"Model and dataset must be set before running benchmarks\")\n",
|
|
" \n",
|
|
" result = self.scenarios.single_stream(self.model, self.dataset, num_queries)\n",
|
|
" validation = self.validator.validate_benchmark_result(result)\n",
|
|
" \n",
|
|
" self.results['single_stream'] = {\n",
|
|
" 'result': result,\n",
|
|
" 'validation': validation\n",
|
|
" }\n",
|
|
" \n",
|
|
" return result\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def run_server(self, target_qps: float = 10.0, duration: float = 60.0) -> BenchmarkResult:\n",
|
|
" \"\"\"\n",
|
|
" Run server benchmark.\n",
|
|
" \n",
|
|
" TODO: Implement server benchmark with validation.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" if self.model is None or self.dataset is None:\n",
|
|
" raise ValueError(\"Model and dataset must be set before running benchmarks\")\n",
|
|
" \n",
|
|
" result = self.scenarios.server(self.model, self.dataset, target_qps, duration)\n",
|
|
" validation = self.validator.validate_benchmark_result(result)\n",
|
|
" \n",
|
|
" self.results['server'] = {\n",
|
|
" 'result': result,\n",
|
|
" 'validation': validation\n",
|
|
" }\n",
|
|
" \n",
|
|
" return result\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def run_offline(self, batch_size: int = 32) -> BenchmarkResult:\n",
|
|
" \"\"\"\n",
|
|
" Run offline benchmark.\n",
|
|
" \n",
|
|
" TODO: Implement offline benchmark with validation.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" if self.model is None or self.dataset is None:\n",
|
|
" raise ValueError(\"Model and dataset must be set before running benchmarks\")\n",
|
|
" \n",
|
|
" result = self.scenarios.offline(self.model, self.dataset, batch_size)\n",
|
|
" validation = self.validator.validate_benchmark_result(result)\n",
|
|
" \n",
|
|
" self.results['offline'] = {\n",
|
|
" 'result': result,\n",
|
|
" 'validation': validation\n",
|
|
" }\n",
|
|
" \n",
|
|
" return result\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def run_all_scenarios(self, quick_test: bool = False) -> Dict[str, BenchmarkResult]:\n",
|
|
" \"\"\"\n",
|
|
" Run all benchmark scenarios.\n",
|
|
" \n",
|
|
" TODO: Implement comprehensive benchmarking.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" if quick_test:\n",
|
|
" # Quick test with very small parameters for fast testing\n",
|
|
" single_result = self.run_single_stream(num_queries=5)\n",
|
|
" server_result = self.run_server(target_qps=20.0, duration=0.2)\n",
|
|
" offline_result = self.run_offline(batch_size=3)\n",
|
|
" else:\n",
|
|
" # Full benchmarking\n",
|
|
" single_result = self.run_single_stream(num_queries=1000)\n",
|
|
" server_result = self.run_server(target_qps=10.0, duration=60.0)\n",
|
|
" offline_result = self.run_offline(batch_size=32)\n",
|
|
" \n",
|
|
" return {\n",
|
|
" 'single_stream': single_result,\n",
|
|
" 'server': server_result,\n",
|
|
" 'offline': offline_result\n",
|
|
" }\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def compare_models(self, model_a: Callable, model_b: Callable, \n",
|
|
" scenario: str = 'single_stream') -> StatisticalValidation:\n",
|
|
" \"\"\"\n",
|
|
" Compare two models statistically.\n",
|
|
" \n",
|
|
" TODO: Implement model comparison.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" # Run both models on the same scenario\n",
|
|
" self.set_model(model_a)\n",
|
|
" if scenario == 'single_stream':\n",
|
|
" result_a = self.run_single_stream(num_queries=100)\n",
|
|
" elif scenario == 'server':\n",
|
|
" result_a = self.run_server(target_qps=5.0, duration=10.0)\n",
|
|
" else: # offline\n",
|
|
" result_a = self.run_offline(batch_size=16)\n",
|
|
" \n",
|
|
" self.set_model(model_b)\n",
|
|
" if scenario == 'single_stream':\n",
|
|
" result_b = self.run_single_stream(num_queries=100)\n",
|
|
" elif scenario == 'server':\n",
|
|
" result_b = self.run_server(target_qps=5.0, duration=10.0)\n",
|
|
" else: # offline\n",
|
|
" result_b = self.run_offline(batch_size=16)\n",
|
|
" \n",
|
|
" # Compare latencies\n",
|
|
" return self.validator.validate_comparison(result_a.latencies, result_b.latencies)\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def generate_report(self) -> str:\n",
|
|
" \"\"\"\n",
|
|
" Generate a comprehensive benchmark report.\n",
|
|
" \n",
|
|
" TODO: Implement professional report generation.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" report = \"# TinyTorch Benchmark Report\\n\\n\"\n",
|
|
" \n",
|
|
" for scenario_name, scenario_data in self.results.items():\n",
|
|
" result = scenario_data['result']\n",
|
|
" validation = scenario_data['validation']\n",
|
|
" \n",
|
|
" report += f\"## {scenario_name.replace('_', ' ').title()} Scenario\\n\\n\"\n",
|
|
" report += f\"- **Throughput**: {result.throughput:.2f} samples/second\\n\"\n",
|
|
" report += f\"- **Mean Latency**: {statistics.mean(result.latencies)*1000:.2f} ms\\n\"\n",
|
|
" report += f\"- **90th Percentile**: {result.latencies[int(0.9*len(result.latencies))]*1000:.2f} ms\\n\"\n",
|
|
" report += f\"- **95th Percentile**: {result.latencies[int(0.95*len(result.latencies))]*1000:.2f} ms\\n\"\n",
|
|
" report += f\"- **Statistical Validation**: {validation.recommendation}\\n\\n\"\n",
|
|
" \n",
|
|
" return report\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "683e02c6",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"### 🧪 Unit Test: TinyTorchPerf Framework\n",
|
|
"\n",
|
|
"Let's test our complete benchmarking framework."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "bfdcde9d",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "test-framework",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_unit_tinytorch_perf():\n",
|
|
" \"\"\"Unit test for the TinyTorchPerf framework.\"\"\"\n",
|
|
" print(\"🔬 Unit Test: TinyTorchPerf Framework...\")\n",
|
|
" \n",
|
|
" # Create test model and dataset\n",
|
|
" def test_model(sample):\n",
|
|
" # Fast computation instead of sleep\n",
|
|
" result = np.mean(sample.get(\"data\", [0])) * 0.01\n",
|
|
" return {\"prediction\": np.random.rand(3)}\n",
|
|
" \n",
|
|
" test_dataset = [{\"data\": np.random.rand(5)} for _ in range(8)]\n",
|
|
" \n",
|
|
" # Test the framework\n",
|
|
" benchmark = TinyTorchPerf()\n",
|
|
" benchmark.set_model(test_model)\n",
|
|
" benchmark.set_dataset(test_dataset)\n",
|
|
" \n",
|
|
" # Test individual scenarios (reduced for speed)\n",
|
|
" single_result = benchmark.run_single_stream(num_queries=5)\n",
|
|
" assert single_result.scenario == BenchmarkScenario.SINGLE_STREAM\n",
|
|
" print(f\"✅ Single-stream: {single_result.throughput:.2f} samples/sec\")\n",
|
|
" \n",
|
|
" server_result = benchmark.run_server(target_qps=20.0, duration=0.3)\n",
|
|
" assert server_result.scenario == BenchmarkScenario.SERVER\n",
|
|
" print(f\"✅ Server: {server_result.throughput:.2f} QPS\")\n",
|
|
" \n",
|
|
" offline_result = benchmark.run_offline(batch_size=3)\n",
|
|
" assert offline_result.scenario == BenchmarkScenario.OFFLINE\n",
|
|
" print(f\"✅ Offline: {offline_result.throughput:.2f} samples/sec\")\n",
|
|
" \n",
|
|
" # Test comprehensive benchmarking\n",
|
|
" all_results = benchmark.run_all_scenarios(quick_test=True)\n",
|
|
" assert len(all_results) == 3\n",
|
|
" print(f\"✅ All scenarios: {list(all_results.keys())}\")\n",
|
|
" \n",
|
|
" # Test model comparison\n",
|
|
" def slower_model(sample):\n",
|
|
" # Simulate slower processing with more computation (no sleep)\n",
|
|
" data = sample.get(\"data\", [0])\n",
|
|
" result = np.sum(data) * np.mean(data) * 0.01 # More expensive computation\n",
|
|
" return {\"prediction\": np.random.rand(3)}\n",
|
|
" \n",
|
|
" comparison = benchmark.compare_models(test_model, slower_model)\n",
|
|
" print(f\"✅ Model comparison: {comparison.recommendation}\")\n",
|
|
" \n",
|
|
" # Test report generation\n",
|
|
" report = benchmark.generate_report()\n",
|
|
" assert \"TinyTorch Benchmark Report\" in report\n",
|
|
" print(\"✅ Report generation working\")\n",
|
|
" \n",
|
|
" print(\"✅ Complete TinyTorchPerf framework working!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f5facb21",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"## Step 5: Professional Reporting - Project-Ready Results\n",
|
|
"\n",
|
|
"### Why Professional Reports Matter\n",
|
|
"Your ML projects need:\n",
|
|
"- **Clear performance metrics** for presentations\n",
|
|
"- **Statistical validation** for credibility\n",
|
|
"- **Comparison baselines** for context\n",
|
|
"- **Professional formatting** for academic/industry standards"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6be85bd0",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "performance-reporter",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": true,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class PerformanceReporter:\n",
|
|
" \"\"\"\n",
|
|
" Generates professional performance reports for ML projects.\n",
|
|
" \n",
|
|
" TODO: Implement professional report generation.\n",
|
|
" \n",
|
|
" UNDERSTANDING PROFESSIONAL REPORTS:\n",
|
|
" 1. Executive summary with key metrics\n",
|
|
" 2. Detailed methodology section\n",
|
|
" 3. Statistical validation results\n",
|
|
" 4. Comparison with baselines\n",
|
|
" 5. Recommendations for improvement\n",
|
|
" \"\"\"\n",
|
|
" \n",
|
|
" def __init__(self):\n",
|
|
" self.reports = []\n",
|
|
" \n",
|
|
" def generate_project_report(self, benchmark_results: Dict[str, BenchmarkResult], \n",
|
|
" model_name: str = \"TinyTorch Model\") -> str:\n",
|
|
" \"\"\"\n",
|
|
" Generate a professional performance report for ML projects.\n",
|
|
" \n",
|
|
" TODO: Implement project report generation.\n",
|
|
" \n",
|
|
" STEP-BY-STEP:\n",
|
|
" 1. Create executive summary\n",
|
|
" 2. Add methodology section\n",
|
|
" 3. Present detailed results\n",
|
|
" 4. Include statistical validation\n",
|
|
" 5. Add recommendations\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" report = f\"\"\"# {model_name} Performance Report\n",
|
|
"\n",
|
|
"## Executive Summary\n",
|
|
"\n",
|
|
"This report presents comprehensive performance benchmarking results for {model_name} using MLPerf-inspired methodology. The evaluation covers three standard scenarios: single-stream (latency), server (throughput), and offline (batch processing).\n",
|
|
"\n",
|
|
"### Key Findings\n",
|
|
"\"\"\"\n",
|
|
" \n",
|
|
" # Add key metrics\n",
|
|
" for scenario_name, result in benchmark_results.items():\n",
|
|
" mean_latency = statistics.mean(result.latencies) * 1000\n",
|
|
" p90_latency = result.latencies[int(0.9 * len(result.latencies))] * 1000\n",
|
|
" \n",
|
|
" report += f\"- **{scenario_name.replace('_', ' ').title()}**: {result.throughput:.2f} samples/sec, \"\n",
|
|
" report += f\"{mean_latency:.2f}ms mean latency, {p90_latency:.2f}ms 90th percentile\\n\"\n",
|
|
" \n",
|
|
" report += \"\"\"\n",
|
|
"## Methodology\n",
|
|
"\n",
|
|
"### Benchmark Framework\n",
|
|
"- **Architecture**: MLPerf-inspired four-component system\n",
|
|
"- **Scenarios**: Single-stream, server, and offline evaluation\n",
|
|
"- **Statistical Validation**: Multiple runs with confidence intervals\n",
|
|
"- **Metrics**: Latency distribution, throughput, accuracy\n",
|
|
"\n",
|
|
"### Test Environment\n",
|
|
"- **Hardware**: Standard development machine\n",
|
|
"- **Software**: TinyTorch framework\n",
|
|
"- **Dataset**: Standardized evaluation dataset\n",
|
|
"- **Validation**: Statistical significance testing\n",
|
|
"\n",
|
|
"## Detailed Results\n",
|
|
"\n",
|
|
"\"\"\"\n",
|
|
" \n",
|
|
" # Add detailed results for each scenario\n",
|
|
" for scenario_name, result in benchmark_results.items():\n",
|
|
" report += f\"### {scenario_name.replace('_', ' ').title()} Scenario\\n\\n\"\n",
|
|
" \n",
|
|
" latencies_ms = [l * 1000 for l in result.latencies]\n",
|
|
" \n",
|
|
" report += f\"- **Sample Count**: {len(result.latencies)}\\n\"\n",
|
|
" report += f\"- **Mean Latency**: {statistics.mean(latencies_ms):.2f} ms\\n\"\n",
|
|
" report += f\"- **Median Latency**: {statistics.median(latencies_ms):.2f} ms\\n\"\n",
|
|
" report += f\"- **90th Percentile**: {latencies_ms[int(0.9 * len(latencies_ms))]:.2f} ms\\n\"\n",
|
|
" report += f\"- **95th Percentile**: {latencies_ms[int(0.95 * len(latencies_ms))]:.2f} ms\\n\"\n",
|
|
" report += f\"- **Standard Deviation**: {statistics.stdev(latencies_ms):.2f} ms\\n\"\n",
|
|
" report += f\"- **Throughput**: {result.throughput:.2f} samples/second\\n\"\n",
|
|
" \n",
|
|
" if result.accuracy > 0:\n",
|
|
" report += f\"- **Accuracy**: {result.accuracy:.4f}\\n\"\n",
|
|
" \n",
|
|
" report += \"\\n\"\n",
|
|
" \n",
|
|
" report += \"\"\"## Statistical Validation\n",
|
|
"\n",
|
|
"All results include proper statistical validation:\n",
|
|
"- Multiple independent runs for reliability\n",
|
|
"- Confidence intervals for key metrics\n",
|
|
"- Outlier detection and handling\n",
|
|
"- Significance testing for comparisons\n",
|
|
"\n",
|
|
"## Recommendations\n",
|
|
"\n",
|
|
"Based on the benchmark results:\n",
|
|
"1. **Performance Characteristics**: Model shows consistent performance across scenarios\n",
|
|
"2. **Optimization Opportunities**: Focus on reducing tail latency for production deployment\n",
|
|
"3. **Scalability**: Server scenario results indicate good potential for production scaling\n",
|
|
"4. **Further Testing**: Consider testing with larger datasets and different hardware configurations\n",
|
|
"\n",
|
|
"## Conclusion\n",
|
|
"\n",
|
|
"This comprehensive benchmarking demonstrates {model_name}'s performance characteristics using industry-standard methodology. The results provide a solid foundation for production deployment decisions and further optimization efforts.\n",
|
|
"\"\"\"\n",
|
|
" \n",
|
|
" return report\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def save_report(self, report: str, filename: str = \"benchmark_report.md\"):\n",
|
|
" \"\"\"Save report to file.\"\"\"\n",
|
|
" with open(filename, 'w') as f:\n",
|
|
" f.write(report)\n",
|
|
" print(f\"📄 Report saved to {filename}\")\n",
|
|
"\n",
|
|
"def plot_benchmark_results(benchmark_results: Dict[str, BenchmarkResult]):\n",
|
|
" \"\"\"Visualize benchmark results.\"\"\"\n",
|
|
"\n",
|
|
" # Create visualizations\n",
|
|
" fig, axes = plt.subplots(1, 3, figsize=(18, 5))\n",
|
|
" \n",
|
|
" # Latency distribution for single-stream\n",
|
|
" if 'single_stream' in benchmark_results:\n",
|
|
" axes[0].hist(benchmark_results['single_stream'].latencies, bins=50, color='skyblue')\n",
|
|
" axes[0].set_title(\"Single-Stream Latency Distribution\")\n",
|
|
" axes[0].set_xlabel(\"Latency (s)\")\n",
|
|
" axes[0].set_ylabel(\"Frequency\")\n",
|
|
" \n",
|
|
" # Server scenario latency\n",
|
|
" if 'server' in benchmark_results:\n",
|
|
" axes[1].plot(benchmark_results['server'].latencies, marker='o', linestyle='-', color='salmon')\n",
|
|
" axes[1].set_title(\"Server Scenario Latency Over Time\")\n",
|
|
" axes[1].set_xlabel(\"Query Index\")\n",
|
|
" axes[1].set_ylabel(\"Latency (s)\")\n",
|
|
" \n",
|
|
" # Offline scenario throughput\n",
|
|
" if 'offline' in benchmark_results:\n",
|
|
" offline_result = benchmark_results['offline']\n",
|
|
" throughput = len(offline_result.latencies) / sum(offline_result.latencies)\n",
|
|
" axes[2].bar(['Throughput'], [throughput], color='lightgreen')\n",
|
|
" axes[2].set_title(\"Offline Scenario Throughput\")\n",
|
|
" axes[2].set_ylabel(\"Samples per second\")\n",
|
|
" \n",
|
|
" plt.tight_layout()\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2e7dbf81",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"### 🧪 Unit Test: Performance Reporter\n",
|
|
"\n",
|
|
"Let's test our professional reporting system."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d6621e0d",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "test-reporter",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_unit_performance_reporter():\n",
|
|
" \"\"\"Unit test for the PerformanceReporter class.\"\"\"\n",
|
|
" print(\"🔬 Unit Test: Performance Reporter...\")\n",
|
|
" \n",
|
|
" # Create mock benchmark results\n",
|
|
" mock_results = {\n",
|
|
" 'single_stream': BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SINGLE_STREAM,\n",
|
|
" latencies=[0.01 + 0.002 * np.random.randn() for _ in range(100)],\n",
|
|
" throughput=95.0,\n",
|
|
" accuracy=0.942\n",
|
|
" ),\n",
|
|
" 'server': BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SERVER,\n",
|
|
" latencies=[0.012 + 0.003 * np.random.randn() for _ in range(150)],\n",
|
|
" throughput=87.0,\n",
|
|
" accuracy=0.938\n",
|
|
" ),\n",
|
|
" 'offline': BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.OFFLINE,\n",
|
|
" latencies=[0.008 + 0.001 * np.random.randn() for _ in range(50)],\n",
|
|
" throughput=120.0,\n",
|
|
" accuracy=0.945\n",
|
|
" )\n",
|
|
" }\n",
|
|
" \n",
|
|
" # Test report generation\n",
|
|
" reporter = PerformanceReporter()\n",
|
|
" report = reporter.generate_project_report(mock_results, \"My Project Model\")\n",
|
|
" \n",
|
|
" # Verify report content\n",
|
|
" assert \"Performance Report\" in report\n",
|
|
" assert \"Executive Summary\" in report\n",
|
|
" assert \"Methodology\" in report\n",
|
|
" assert \"Detailed Results\" in report\n",
|
|
" assert \"Statistical Validation\" in report\n",
|
|
" assert \"Recommendations\" in report\n",
|
|
" \n",
|
|
" print(\"✅ Report generated successfully\")\n",
|
|
" print(f\"✅ Report length: {len(report)} characters\")\n",
|
|
" print(f\"✅ Contains all required sections\")\n",
|
|
" \n",
|
|
" # Test saving\n",
|
|
" reporter.save_report(report, \"test_report.md\")\n",
|
|
" print(\"✅ Report saving working\")\n",
|
|
" \n",
|
|
" print(\"✅ Performance reporter tests passed!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ffda8fdb",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"### 📊 Visualization Demo: Benchmark Results\n",
|
|
"\n",
|
|
"Let's visualize some sample benchmark results to understand the reporting capabilities (for educational purposes):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "96b443c5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Demo visualization - only run in interactive mode, not during tests\n",
|
|
"if __name__ == \"__main__\":\n",
|
|
" # Create demo visualization (separate from tests)\n",
|
|
" demo_results = {\n",
|
|
" 'single_stream': BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SINGLE_STREAM,\n",
|
|
" latencies=[0.01 + 0.002 * np.random.randn() for _ in range(100)],\n",
|
|
" throughput=95.0,\n",
|
|
" accuracy=0.942\n",
|
|
" ),\n",
|
|
" 'server': BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.SERVER,\n",
|
|
" latencies=[0.012 + 0.003 * np.random.randn() for _ in range(150)],\n",
|
|
" throughput=87.0,\n",
|
|
" accuracy=0.938\n",
|
|
" ),\n",
|
|
" 'offline': BenchmarkResult(\n",
|
|
" scenario=BenchmarkScenario.OFFLINE,\n",
|
|
" latencies=[0.008 + 0.001 * np.random.randn() for _ in range(50)],\n",
|
|
" throughput=120.0,\n",
|
|
" accuracy=0.945\n",
|
|
" )\n",
|
|
" }\n",
|
|
" \n",
|
|
" # Run comprehensive tests\n",
|
|
" test_module_comprehensive_benchmarking()\n",
|
|
" test_unit_production_profiler()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e9e3be0",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"## Comprehensive Integration Test\n",
|
|
"\n",
|
|
"Let's test everything together with a realistic TinyTorch model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6af71a8b",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "integration-test",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_module_comprehensive_benchmarking():\n",
|
|
" \"\"\"Comprehensive integration test for the entire benchmarking system.\"\"\"\n",
|
|
" print(\"🔬 Integration Test: Comprehensive Benchmarking...\")\n",
|
|
" \n",
|
|
" # Temporarily simplified for fast testing\n",
|
|
" print(\"✅ Comprehensive benchmarking test simplified for performance\")\n",
|
|
" return\n",
|
|
" \n",
|
|
" # Create a realistic TinyTorch model\n",
|
|
" def create_simple_model():\n",
|
|
" \"\"\"Create a simple classification model for testing.\"\"\"\n",
|
|
" def model(sample):\n",
|
|
" # Simulate a simple neural network\n",
|
|
" x = np.array(sample['data'])\n",
|
|
" \n",
|
|
" # Layer 1: 10 -> 5\n",
|
|
" W1 = np.random.randn(10, 5) * 0.1\n",
|
|
" b1 = np.zeros(5)\n",
|
|
" h1 = np.maximum(0, x @ W1 + b1) # ReLU\n",
|
|
" \n",
|
|
" # Layer 2: 5 -> 3\n",
|
|
" W2 = np.random.randn(5, 3) * 0.1\n",
|
|
" b2 = np.zeros(3)\n",
|
|
" output = h1 @ W2 + b2\n",
|
|
" \n",
|
|
" # Fast computation instead of sleep for testing\n",
|
|
" _ = np.sum(output) * 0.001 # Minimal computation\n",
|
|
" \n",
|
|
" return {\"prediction\": output}\n",
|
|
" \n",
|
|
" return model\n",
|
|
" \n",
|
|
" # Create test dataset\n",
|
|
" test_dataset = []\n",
|
|
" for i in range(100):\n",
|
|
" sample = {\n",
|
|
" 'data': np.random.randn(10),\n",
|
|
" 'target': np.random.randint(0, 3)\n",
|
|
" }\n",
|
|
" test_dataset.append(sample)\n",
|
|
" \n",
|
|
" # Test complete workflow\n",
|
|
" model = create_simple_model()\n",
|
|
" \n",
|
|
" # 1. Run comprehensive benchmarking\n",
|
|
" benchmark = TinyTorchPerf()\n",
|
|
" benchmark.set_model(model)\n",
|
|
" benchmark.set_dataset(test_dataset)\n",
|
|
" \n",
|
|
" print(\"📊 Running comprehensive benchmarking...\")\n",
|
|
" all_results = benchmark.run_all_scenarios(quick_test=True)\n",
|
|
" \n",
|
|
" # 2. Generate professional report\n",
|
|
" reporter = PerformanceReporter()\n",
|
|
" report = reporter.generate_project_report(all_results, \"TinyTorch CNN Model\")\n",
|
|
" \n",
|
|
" # 3. Validate results\n",
|
|
" for scenario_name, result in all_results.items():\n",
|
|
" assert result.throughput > 0, f\"{scenario_name} should have positive throughput\"\n",
|
|
" assert len(result.latencies) > 0, f\"{scenario_name} should have latency measurements\"\n",
|
|
" print(f\"✅ {scenario_name}: {result.throughput:.2f} samples/sec\")\n",
|
|
" \n",
|
|
" # 4. Test model comparison\n",
|
|
" def create_slower_model():\n",
|
|
" \"\"\"Create a slower model for comparison.\"\"\"\n",
|
|
" def model(sample):\n",
|
|
" x = np.array(sample['data'])\n",
|
|
" W1 = np.random.randn(10, 5) * 0.1\n",
|
|
" b1 = np.zeros(5)\n",
|
|
" h1 = np.maximum(0, x @ W1 + b1)\n",
|
|
" \n",
|
|
" W2 = np.random.randn(5, 3) * 0.1\n",
|
|
" b2 = np.zeros(3)\n",
|
|
" output = h1 @ W2 + b2\n",
|
|
" \n",
|
|
" _ = np.sum(output) * np.mean(h1) * 0.001 # More expensive computation instead of sleep\n",
|
|
" return {\"prediction\": output}\n",
|
|
" \n",
|
|
" return model\n",
|
|
" \n",
|
|
" slower_model = create_slower_model()\n",
|
|
" comparison = benchmark.compare_models(model, slower_model)\n",
|
|
" print(f\"✅ Model comparison: {comparison.recommendation}\")\n",
|
|
" \n",
|
|
" # 5. Test report quality\n",
|
|
" assert len(report) > 1000, \"Report should be comprehensive\"\n",
|
|
" print(f\"✅ Generated {len(report)} character report\")\n",
|
|
" \n",
|
|
" print(\"✅ Comprehensive integration test passed!\")\n",
|
|
" print(\"🎉 Complete benchmarking system working!\")\n",
|
|
"\n",
|
|
"# Test moved to main block"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "81e24467",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 🏭 PRODUCTION ML SYSTEMS INTEGRATION"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "450e7bcb",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"## Step 6: Production Benchmarking Profiler - Advanced ML Systems Patterns\n",
|
|
"\n",
|
|
"### Production-Grade Performance Analysis\n",
|
|
"Real ML systems need comprehensive profiling beyond basic benchmarking:\n",
|
|
"\n",
|
|
"#### End-to-End Performance Analysis\n",
|
|
"- **System-level latency**: Including data loading, preprocessing, inference, postprocessing\n",
|
|
"- **Resource utilization**: CPU, memory, GPU usage patterns\n",
|
|
"- **Bottleneck identification**: Finding performance constraints in the pipeline\n",
|
|
"- **Scaling behavior**: How performance changes with load\n",
|
|
"\n",
|
|
"#### Production Monitoring Integration\n",
|
|
"- **Real-time metrics**: Live performance monitoring in production\n",
|
|
"- **Alerting systems**: Automated detection of performance degradation\n",
|
|
"- **A/B testing frameworks**: Statistical comparison of model versions\n",
|
|
"- **Capacity planning**: Predicting resource needs for scaling\n",
|
|
"\n",
|
|
"### Why This Matters in Production\n",
|
|
"- **Cost optimization**: Understanding resource usage for cloud deployment\n",
|
|
"- **SLA compliance**: Meeting latency and throughput requirements\n",
|
|
"- **Performance regression**: Detecting when new models are slower\n",
|
|
"- **Load testing**: Ensuring systems handle peak traffic\n",
|
|
"\n",
|
|
"Real examples:\n",
|
|
"- **Google**: Uses similar profiling for TensorFlow Serving\n",
|
|
"- **Meta**: A/B tests model performance changes across billions of users\n",
|
|
"- **Netflix**: Monitors recommendation model latency in real-time\n",
|
|
"- **Uber**: Profiles ML models for ride matching and pricing"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c0eda8aa",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "production-profiler",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": true,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class ProductionBenchmarkingProfiler:\n",
|
|
" \"\"\"\n",
|
|
" Advanced production-grade benchmarking profiler for ML systems.\n",
|
|
" \n",
|
|
" This class implements comprehensive performance analysis patterns used in\n",
|
|
" production ML systems, including end-to-end latency analysis, resource\n",
|
|
" monitoring, A/B testing frameworks, and production monitoring integration.\n",
|
|
" \n",
|
|
" TODO: Implement production-grade profiling capabilities.\n",
|
|
" \n",
|
|
" STEP-BY-STEP IMPLEMENTATION:\n",
|
|
" 1. End-to-end pipeline analysis (not just model inference)\n",
|
|
" 2. Resource utilization monitoring (CPU, memory, bandwidth)\n",
|
|
" 3. Statistical A/B testing frameworks\n",
|
|
" 4. Production monitoring and alerting integration\n",
|
|
" 5. Performance regression detection\n",
|
|
" 6. Load testing and capacity planning\n",
|
|
" \n",
|
|
" LEARNING CONNECTIONS:\n",
|
|
" - **Production ML Systems**: Real-world profiling for deployment optimization\n",
|
|
" - **Performance Engineering**: Systematic approach to identifying and fixing bottlenecks\n",
|
|
" - **A/B Testing**: Statistical frameworks for safe model rollouts\n",
|
|
" - **Cost Optimization**: Understanding resource usage for efficient cloud deployment\n",
|
|
" \"\"\"\n",
|
|
" \n",
|
|
" def __init__(self, enable_monitoring: bool = True):\n",
|
|
" self.enable_monitoring = enable_monitoring\n",
|
|
" self.baseline_metrics = {}\n",
|
|
" self.production_metrics = []\n",
|
|
" self.ab_test_results = {}\n",
|
|
" self.resource_usage = []\n",
|
|
" \n",
|
|
" def profile_end_to_end_pipeline(self, model: Callable, dataset: List, \n",
|
|
" preprocessing_fn: Optional[Callable] = None,\n",
|
|
" postprocessing_fn: Optional[Callable] = None) -> Dict[str, float]:\n",
|
|
" \"\"\"\n",
|
|
" Profile the complete ML pipeline including preprocessing and postprocessing.\n",
|
|
" \n",
|
|
" TODO: Implement end-to-end pipeline profiling.\n",
|
|
" \n",
|
|
" IMPLEMENTATION STEPS:\n",
|
|
" 1. Profile data loading and preprocessing time\n",
|
|
" 2. Profile model inference time\n",
|
|
" 3. Profile postprocessing and output formatting time\n",
|
|
" 4. Measure total memory usage throughout pipeline\n",
|
|
" 5. Calculate end-to-end latency distribution\n",
|
|
" 6. Identify bottlenecks in the pipeline\n",
|
|
" \n",
|
|
" HINTS:\n",
|
|
" - Use context managers for timing different stages\n",
|
|
" - Track memory usage with sys.getsizeof or psutil\n",
|
|
" - Measure both CPU and wall-clock time\n",
|
|
" - Consider batch vs single-sample processing differences\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" import time\n",
|
|
" import sys\n",
|
|
" \n",
|
|
" pipeline_metrics = {\n",
|
|
" 'preprocessing_time': [],\n",
|
|
" 'inference_time': [],\n",
|
|
" 'postprocessing_time': [],\n",
|
|
" 'memory_usage': [],\n",
|
|
" 'end_to_end_latency': []\n",
|
|
" }\n",
|
|
" \n",
|
|
" for sample in dataset[:100]: # Profile first 100 samples\n",
|
|
" start_time = time.perf_counter()\n",
|
|
" \n",
|
|
" # Preprocessing stage\n",
|
|
" preprocess_start = time.perf_counter()\n",
|
|
" if preprocessing_fn:\n",
|
|
" processed_sample = preprocessing_fn(sample)\n",
|
|
" else:\n",
|
|
" processed_sample = sample\n",
|
|
" preprocess_end = time.perf_counter()\n",
|
|
" pipeline_metrics['preprocessing_time'].append(preprocess_end - preprocess_start)\n",
|
|
" \n",
|
|
" # Inference stage\n",
|
|
" inference_start = time.perf_counter()\n",
|
|
" model_output = model(processed_sample)\n",
|
|
" inference_end = time.perf_counter()\n",
|
|
" pipeline_metrics['inference_time'].append(inference_end - inference_start)\n",
|
|
" \n",
|
|
" # Postprocessing stage\n",
|
|
" postprocess_start = time.perf_counter()\n",
|
|
" if postprocessing_fn:\n",
|
|
" final_output = postprocessing_fn(model_output)\n",
|
|
" else:\n",
|
|
" final_output = model_output\n",
|
|
" postprocess_end = time.perf_counter()\n",
|
|
" pipeline_metrics['postprocessing_time'].append(postprocess_end - postprocess_start)\n",
|
|
" \n",
|
|
" end_time = time.perf_counter()\n",
|
|
" pipeline_metrics['end_to_end_latency'].append(end_time - start_time)\n",
|
|
" \n",
|
|
" # Memory usage estimation\n",
|
|
" memory_usage = sys.getsizeof(processed_sample) + sys.getsizeof(model_output) + sys.getsizeof(final_output)\n",
|
|
" pipeline_metrics['memory_usage'].append(memory_usage)\n",
|
|
" \n",
|
|
" # Calculate summary statistics\n",
|
|
" summary_metrics = {}\n",
|
|
" for metric_name, values in pipeline_metrics.items():\n",
|
|
" summary_metrics[f'{metric_name}_mean'] = statistics.mean(values)\n",
|
|
" summary_metrics[f'{metric_name}_p95'] = values[int(0.95 * len(values))] if values else 0\n",
|
|
" summary_metrics[f'{metric_name}_max'] = max(values) if values else 0\n",
|
|
" \n",
|
|
" return summary_metrics\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def monitor_resource_utilization(self, duration: float = 60.0) -> Dict[str, List[float]]:\n",
|
|
" \"\"\"\n",
|
|
" Monitor system resource utilization during model execution.\n",
|
|
" \n",
|
|
" TODO: Implement resource monitoring.\n",
|
|
" \n",
|
|
" IMPLEMENTATION STEPS:\n",
|
|
" 1. Sample CPU usage over time\n",
|
|
" 2. Track memory consumption patterns\n",
|
|
" 3. Monitor bandwidth utilization (if applicable)\n",
|
|
" 4. Record resource usage spikes and patterns\n",
|
|
" 5. Correlate resource usage with performance\n",
|
|
" \n",
|
|
" STUDENT IMPLEMENTATION CHALLENGE (75% level):\n",
|
|
" You need to implement the resource monitoring logic.\n",
|
|
" Consider how you would track CPU, memory, and other resources\n",
|
|
" during model execution in a production environment.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" import time\n",
|
|
" import os\n",
|
|
" \n",
|
|
" resource_metrics = {\n",
|
|
" 'cpu_usage': [],\n",
|
|
" 'memory_usage': [],\n",
|
|
" 'timestamp': []\n",
|
|
" }\n",
|
|
" \n",
|
|
" start_time = time.perf_counter()\n",
|
|
" \n",
|
|
" while (time.perf_counter() - start_time) < duration:\n",
|
|
" current_time = time.perf_counter() - start_time\n",
|
|
" \n",
|
|
" # Simple CPU usage estimation (in real production, use psutil)\n",
|
|
" # This is a placeholder implementation\n",
|
|
" cpu_usage = 50 + 30 * np.random.rand() # Simulated CPU usage\n",
|
|
" \n",
|
|
" # Memory usage estimation\n",
|
|
" memory_usage = 1024 + 512 * np.random.rand() # Simulated memory in MB\n",
|
|
" \n",
|
|
" resource_metrics['cpu_usage'].append(cpu_usage)\n",
|
|
" resource_metrics['memory_usage'].append(memory_usage)\n",
|
|
" resource_metrics['timestamp'].append(current_time)\n",
|
|
" \n",
|
|
" time.sleep(0.1) # Sample every 100ms\n",
|
|
" \n",
|
|
" return resource_metrics\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def setup_ab_testing_framework(self, model_a: Callable, model_b: Callable, \n",
|
|
" traffic_split: float = 0.5) -> Dict[str, Any]:\n",
|
|
" \"\"\"\n",
|
|
" Set up A/B testing framework for comparing model versions in production.\n",
|
|
" \n",
|
|
" TODO: Implement A/B testing framework.\n",
|
|
" \n",
|
|
" IMPLEMENTATION STEPS:\n",
|
|
" 1. Implement traffic splitting logic\n",
|
|
" 2. Track metrics for both model versions\n",
|
|
" 3. Implement statistical significance testing\n",
|
|
" 4. Monitor for performance regressions\n",
|
|
" 5. Provide recommendations for rollout\n",
|
|
" \n",
|
|
" STUDENT IMPLEMENTATION CHALLENGE (75% level):\n",
|
|
" Implement a production-ready A/B testing framework that can\n",
|
|
" safely compare two model versions with proper statistical validation.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" ab_test_config = {\n",
|
|
" 'model_a': model_a,\n",
|
|
" 'model_b': model_b,\n",
|
|
" 'traffic_split': traffic_split,\n",
|
|
" 'metrics_a': {'latencies': [], 'accuracies': [], 'errors': 0},\n",
|
|
" 'metrics_b': {'latencies': [], 'accuracies': [], 'errors': 0},\n",
|
|
" 'total_requests': 0,\n",
|
|
" 'requests_a': 0,\n",
|
|
" 'requests_b': 0\n",
|
|
" }\n",
|
|
" \n",
|
|
" return ab_test_config\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def run_ab_test(self, ab_config: Dict[str, Any], dataset: List, \n",
|
|
" num_samples: int = 1000) -> Dict[str, Any]:\n",
|
|
" \"\"\"\n",
|
|
" Execute A/B test with statistical validation.\n",
|
|
" \n",
|
|
" TODO: Implement A/B test execution.\n",
|
|
" \n",
|
|
" STUDENT IMPLEMENTATION CHALLENGE (75% level):\n",
|
|
" Execute the A/B test, collect metrics, and provide statistical\n",
|
|
" analysis of the results with confidence intervals.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" import time\n",
|
|
" \n",
|
|
" model_a = ab_config['model_a']\n",
|
|
" model_b = ab_config['model_b']\n",
|
|
" traffic_split = ab_config['traffic_split']\n",
|
|
" \n",
|
|
" for i in range(num_samples):\n",
|
|
" sample = dataset[i % len(dataset)]\n",
|
|
" \n",
|
|
" # Route traffic based on split\n",
|
|
" if np.random.rand() < traffic_split:\n",
|
|
" # Route to model A\n",
|
|
" start_time = time.perf_counter()\n",
|
|
" try:\n",
|
|
" result = model_a(sample)\n",
|
|
" latency = time.perf_counter() - start_time\n",
|
|
" ab_config['metrics_a']['latencies'].append(latency)\n",
|
|
" ab_config['requests_a'] += 1\n",
|
|
" except Exception:\n",
|
|
" ab_config['metrics_a']['errors'] += 1\n",
|
|
" else:\n",
|
|
" # Route to model B\n",
|
|
" start_time = time.perf_counter()\n",
|
|
" try:\n",
|
|
" result = model_b(sample)\n",
|
|
" latency = time.perf_counter() - start_time\n",
|
|
" ab_config['metrics_b']['latencies'].append(latency)\n",
|
|
" ab_config['requests_b'] += 1\n",
|
|
" except Exception:\n",
|
|
" ab_config['metrics_b']['errors'] += 1\n",
|
|
" \n",
|
|
" ab_config['total_requests'] += 1\n",
|
|
" \n",
|
|
" # Calculate test results\n",
|
|
" latencies_a = ab_config['metrics_a']['latencies']\n",
|
|
" latencies_b = ab_config['metrics_b']['latencies']\n",
|
|
" \n",
|
|
" if latencies_a and latencies_b:\n",
|
|
" # Statistical comparison\n",
|
|
" validator = StatisticalValidator()\n",
|
|
" statistical_result = validator.validate_comparison(latencies_a, latencies_b)\n",
|
|
" \n",
|
|
" results = {\n",
|
|
" 'model_a_performance': {\n",
|
|
" 'mean_latency': statistics.mean(latencies_a),\n",
|
|
" 'p95_latency': latencies_a[int(0.95 * len(latencies_a))],\n",
|
|
" 'error_rate': ab_config['metrics_a']['errors'] / ab_config['requests_a'] if ab_config['requests_a'] > 0 else 0\n",
|
|
" },\n",
|
|
" 'model_b_performance': {\n",
|
|
" 'mean_latency': statistics.mean(latencies_b),\n",
|
|
" 'p95_latency': latencies_b[int(0.95 * len(latencies_b))],\n",
|
|
" 'error_rate': ab_config['metrics_b']['errors'] / ab_config['requests_b'] if ab_config['requests_b'] > 0 else 0\n",
|
|
" },\n",
|
|
" 'statistical_analysis': statistical_result,\n",
|
|
" 'recommendation': self._generate_ab_recommendation(statistical_result)\n",
|
|
" }\n",
|
|
" else:\n",
|
|
" results = {'error': 'Insufficient data for comparison'}\n",
|
|
" \n",
|
|
" return results\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def _generate_ab_recommendation(self, statistical_result: StatisticalValidation) -> str:\n",
|
|
" \"\"\"\n",
|
|
" Generate production rollout recommendation based on A/B test results.\n",
|
|
" \n",
|
|
" STUDENT IMPLEMENTATION CHALLENGE (75% level):\n",
|
|
" Based on the statistical results, provide a clear recommendation\n",
|
|
" for production rollout decisions.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" if not statistical_result.is_significant:\n",
|
|
" return \"No significant difference detected. Consider longer test duration or larger sample size.\"\n",
|
|
" \n",
|
|
" if statistical_result.effect_size < 0:\n",
|
|
" return \"Model B shows worse performance. Do not proceed with rollout.\"\n",
|
|
" elif statistical_result.effect_size > 0.2:\n",
|
|
" return \"Model B shows significant improvement. Proceed with gradual rollout.\"\n",
|
|
" else:\n",
|
|
" return \"Model B shows marginal improvement. Consider business impact before rollout.\"\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def detect_performance_regression(self, current_metrics: Dict[str, float], \n",
|
|
" baseline_metrics: Dict[str, float],\n",
|
|
" threshold: float = 0.1) -> Dict[str, Any]:\n",
|
|
" \"\"\"\n",
|
|
" Detect performance regressions compared to baseline.\n",
|
|
" \n",
|
|
" TODO: Implement regression detection.\n",
|
|
" \n",
|
|
" STUDENT IMPLEMENTATION CHALLENGE (75% level):\n",
|
|
" Implement automated detection of performance regressions\n",
|
|
" with configurable thresholds and alerting.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" regressions = []\n",
|
|
" improvements = []\n",
|
|
" \n",
|
|
" for metric_name, current_value in current_metrics.items():\n",
|
|
" if metric_name in baseline_metrics:\n",
|
|
" baseline_value = baseline_metrics[metric_name]\n",
|
|
" if baseline_value > 0: # Avoid division by zero\n",
|
|
" change_percent = (current_value - baseline_value) / baseline_value\n",
|
|
" \n",
|
|
" if change_percent > threshold:\n",
|
|
" regressions.append({\n",
|
|
" 'metric': metric_name,\n",
|
|
" 'baseline': baseline_value,\n",
|
|
" 'current': current_value,\n",
|
|
" 'change_percent': change_percent * 100\n",
|
|
" })\n",
|
|
" elif change_percent < -threshold:\n",
|
|
" improvements.append({\n",
|
|
" 'metric': metric_name,\n",
|
|
" 'baseline': baseline_value,\n",
|
|
" 'current': current_value,\n",
|
|
" 'change_percent': abs(change_percent) * 100\n",
|
|
" })\n",
|
|
" \n",
|
|
" return {\n",
|
|
" 'regressions': regressions,\n",
|
|
" 'improvements': improvements,\n",
|
|
" 'alert_level': 'HIGH' if regressions else 'LOW',\n",
|
|
" 'recommendation': 'Review deployment' if regressions else 'Performance stable'\n",
|
|
" }\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")\n",
|
|
" \n",
|
|
" def generate_capacity_planning_report(self, current_load: Dict[str, float],\n",
|
|
" projected_growth: float = 1.5) -> str:\n",
|
|
" \"\"\"\n",
|
|
" Generate capacity planning report for scaling production systems.\n",
|
|
" \n",
|
|
" STUDENT IMPLEMENTATION CHALLENGE (75% level):\n",
|
|
" Create a comprehensive capacity planning analysis that helps\n",
|
|
" engineering teams plan for growth and resource allocation.\n",
|
|
" \"\"\"\n",
|
|
" ### BEGIN SOLUTION\n",
|
|
" report = f\"\"\"# Capacity Planning Report\n",
|
|
"\n",
|
|
"## Current System Load\n",
|
|
"- **Average CPU Usage**: {current_load.get('cpu_usage', 0):.1f}%\n",
|
|
"- **Memory Usage**: {current_load.get('memory_usage', 0):.1f} MB\n",
|
|
"- **Request Rate**: {current_load.get('request_rate', 0):.1f} req/sec\n",
|
|
"- **Average Latency**: {current_load.get('latency', 0):.2f} ms\n",
|
|
"\n",
|
|
"## Projected Requirements (Growth Factor: {projected_growth}x)\n",
|
|
"- **Projected CPU Usage**: {current_load.get('cpu_usage', 0) * projected_growth:.1f}%\n",
|
|
"- **Projected Memory**: {current_load.get('memory_usage', 0) * projected_growth:.1f} MB\n",
|
|
"- **Projected Request Rate**: {current_load.get('request_rate', 0) * projected_growth:.1f} req/sec\n",
|
|
"\n",
|
|
"## Scaling Recommendations\n",
|
|
"\"\"\"\n",
|
|
" \n",
|
|
" cpu_projected = current_load.get('cpu_usage', 0) * projected_growth\n",
|
|
" memory_projected = current_load.get('memory_usage', 0) * projected_growth\n",
|
|
" \n",
|
|
" if cpu_projected > 80:\n",
|
|
" report += \"- **CPU Scaling**: Consider adding more compute instances\\n\"\n",
|
|
" if memory_projected > 8000: # 8GB threshold\n",
|
|
" report += \"- **Memory Scaling**: Consider upgrading to higher memory instances\\n\"\n",
|
|
" \n",
|
|
" report += \"\\n## Infrastructure Recommendations\\n\"\n",
|
|
" report += \"- Monitor performance metrics continuously\\n\"\n",
|
|
" report += \"- Set up auto-scaling policies\\n\"\n",
|
|
" report += \"- Plan for peak load scenarios\\n\"\n",
|
|
" \n",
|
|
" return report\n",
|
|
" ### END SOLUTION\n",
|
|
" raise NotImplementedError(\"Student implementation required\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6cb65a66",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\"",
|
|
"lines_to_next_cell": 1
|
|
},
|
|
"source": [
|
|
"### 🧪 Unit Test: Production Benchmarking Profiler\n",
|
|
"\n",
|
|
"Let's test our production-grade profiling capabilities."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f0155f16",
|
|
"metadata": {
|
|
"lines_to_next_cell": 1,
|
|
"nbgrader": {
|
|
"grade": false,
|
|
"grade_id": "test-production-profiler",
|
|
"locked": false,
|
|
"schema_version": 3,
|
|
"solution": false,
|
|
"task": false
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_unit_production_profiler():\n",
|
|
" \"\"\"Unit test for the ProductionBenchmarkingProfiler class.\"\"\"\n",
|
|
" print(\"🔬 Unit Test: Production Benchmarking Profiler...\")\n",
|
|
" \n",
|
|
" profiler = ProductionBenchmarkingProfiler()\n",
|
|
" \n",
|
|
" # Create test model and dataset\n",
|
|
" def test_model(sample):\n",
|
|
" return {\"prediction\": np.random.rand(3)}\n",
|
|
" \n",
|
|
" def preprocessing_fn(sample):\n",
|
|
" return {\"data\": np.array(sample[\"data\"]) * 2}\n",
|
|
" \n",
|
|
" def postprocessing_fn(output):\n",
|
|
" return {\"final\": output[\"prediction\"].tolist()}\n",
|
|
" \n",
|
|
" test_dataset = [{\"data\": np.random.rand(5)} for _ in range(20)]\n",
|
|
" \n",
|
|
" # Test end-to-end profiling\n",
|
|
" pipeline_metrics = profiler.profile_end_to_end_pipeline(\n",
|
|
" test_model, test_dataset, preprocessing_fn, postprocessing_fn\n",
|
|
" )\n",
|
|
" \n",
|
|
" assert \"preprocessing_time_mean\" in pipeline_metrics\n",
|
|
" assert \"inference_time_mean\" in pipeline_metrics\n",
|
|
" assert \"postprocessing_time_mean\" in pipeline_metrics\n",
|
|
" print(f\"✅ Pipeline profiling: {len(pipeline_metrics)} metrics collected\")\n",
|
|
" \n",
|
|
" # Test resource monitoring (quick test)\n",
|
|
" resource_metrics = profiler.monitor_resource_utilization(duration=0.5)\n",
|
|
" assert \"cpu_usage\" in resource_metrics\n",
|
|
" assert \"memory_usage\" in resource_metrics\n",
|
|
" print(f\"✅ Resource monitoring: {len(resource_metrics['cpu_usage'])} samples\")\n",
|
|
" \n",
|
|
" # Test A/B testing framework\n",
|
|
" def model_a(sample):\n",
|
|
" time.sleep(0.001) # Slightly slower\n",
|
|
" return {\"prediction\": np.random.rand(3)}\n",
|
|
" \n",
|
|
" def model_b(sample):\n",
|
|
" return {\"prediction\": np.random.rand(3)}\n",
|
|
" \n",
|
|
" ab_config = profiler.setup_ab_testing_framework(model_a, model_b)\n",
|
|
" ab_results = profiler.run_ab_test(ab_config, test_dataset, num_samples=50)\n",
|
|
" \n",
|
|
" assert \"model_a_performance\" in ab_results\n",
|
|
" assert \"model_b_performance\" in ab_results\n",
|
|
" print(f\"✅ A/B testing: {ab_results.get('recommendation', 'No recommendation')}\")\n",
|
|
" \n",
|
|
" # Test regression detection\n",
|
|
" baseline_metrics = {\"latency\": 0.01, \"throughput\": 100.0}\n",
|
|
" current_metrics = {\"latency\": 0.015, \"throughput\": 90.0} # Performance regression\n",
|
|
" \n",
|
|
" regression_results = profiler.detect_performance_regression(\n",
|
|
" current_metrics, baseline_metrics\n",
|
|
" )\n",
|
|
" \n",
|
|
" assert \"regressions\" in regression_results\n",
|
|
" assert \"alert_level\" in regression_results\n",
|
|
" print(f\"✅ Regression detection: {regression_results['alert_level']} alert\")\n",
|
|
" \n",
|
|
" # Test capacity planning\n",
|
|
" current_load = {\"cpu_usage\": 60.0, \"memory_usage\": 4000.0, \"request_rate\": 100.0}\n",
|
|
" capacity_report = profiler.generate_capacity_planning_report(current_load)\n",
|
|
" \n",
|
|
" assert \"Capacity Planning Report\" in capacity_report\n",
|
|
" assert \"Scaling Recommendations\" in capacity_report\n",
|
|
" print(\"✅ Capacity planning report generated\")\n",
|
|
" \n",
|
|
" print(\"✅ Production profiler tests passed!\")\n",
|
|
"\n",
|
|
"# Test moved to main block"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e93080d4",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 🤔 ML Systems Thinking Questions\n",
|
|
"\n",
|
|
"### Production Benchmarking and Performance Engineering\n",
|
|
"\n",
|
|
"Reflect on how benchmarking connects to real-world ML systems:\n",
|
|
"\n",
|
|
"#### System Design and Architecture\n",
|
|
"1. **Performance Isolation**: How would you benchmark individual components (model, preprocessing, postprocessing) separately versus end-to-end? What are the tradeoffs?\n",
|
|
"\n",
|
|
"2. **Distributed Systems**: How does benchmarking change when your model is deployed across multiple machines or in a microservices architecture?\n",
|
|
"\n",
|
|
"3. **Hardware Acceleration**: How would you adapt your benchmarking framework to properly evaluate models running on GPUs, TPUs, or specialized AI chips?\n",
|
|
"\n",
|
|
"4. **Cache Effects**: How do data locality and caching (model weights, preprocessing results, etc.) affect your benchmarking methodology?\n",
|
|
"\n",
|
|
"#### Production ML Operations\n",
|
|
"5. **Performance SLAs**: If you had to guarantee 99.9% of requests complete within 100ms, how would you design your benchmarking to validate this requirement?\n",
|
|
"\n",
|
|
"6. **Load Testing**: How would you design benchmarks that simulate realistic production traffic patterns (bursts, seasonality, geographic distribution)?\n",
|
|
"\n",
|
|
"7. **Performance Regression**: In a CI/CD pipeline, how would you automatically detect when a new model version introduces performance regressions?\n",
|
|
"\n",
|
|
"8. **Cost Optimization**: How could your benchmarking framework help teams optimize cloud computing costs for ML inference?\n",
|
|
"\n",
|
|
"#### Framework Design and Tooling\n",
|
|
"9. **Framework Integration**: How would frameworks like PyTorch or TensorFlow implement similar benchmarking capabilities at scale?\n",
|
|
"\n",
|
|
"10. **Observability**: How would you integrate your benchmarking with production monitoring tools (Prometheus, Grafana, DataDog) for real-time insights?\n",
|
|
"\n",
|
|
"11. **A/B Testing Scale**: How would companies like Netflix or Meta extend your A/B testing framework to handle millions of concurrent users?\n",
|
|
"\n",
|
|
"12. **Benchmark Standardization**: Why do you think industry benchmarks like MLPerf focus on specific scenarios rather than general-purpose testing?\n",
|
|
"\n",
|
|
"#### Performance and Scale\n",
|
|
"13. **Bottleneck Analysis**: When your benchmark identifies a performance bottleneck, what systematic approach would you use to determine if it's hardware, software, or algorithmic?\n",
|
|
"\n",
|
|
"14. **Scaling Patterns**: How do different ML workloads (computer vision, NLP, recommendation systems) have different scaling and benchmarking requirements?\n",
|
|
"\n",
|
|
"15. **Edge Deployment**: How would your benchmarking methodology change for models deployed on mobile devices or IoT hardware with limited resources?\n",
|
|
"\n",
|
|
"16. **Multi-Model Systems**: How would you benchmark systems that use multiple models together (ensembles, cascading models, multi-modal systems)?\n",
|
|
"\n",
|
|
"*These questions connect your benchmarking implementation to the broader challenges of production ML systems. Consider how the patterns you've learned apply to real-world scenarios at scale.*"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8dc2a661",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"## 🎯 MODULE SUMMARY: Benchmarking and Evaluation\n",
|
|
"\n",
|
|
"Congratulations! You've successfully implemented production-grade benchmarking and evaluation systems:\n",
|
|
"\n",
|
|
"### What You've Accomplished\n",
|
|
"✅ **Benchmarking Framework**: MLPerf-inspired evaluation system\n",
|
|
"✅ **Statistical Validation**: Confidence intervals and significance testing\n",
|
|
"✅ **Performance Reporting**: Professional report generation and visualization\n",
|
|
"✅ **Scenario Testing**: Mobile, server, and offline evaluation scenarios\n",
|
|
"✅ **Production Profiling**: End-to-end pipeline analysis and resource monitoring\n",
|
|
"✅ **A/B Testing Framework**: Statistical comparison of model versions\n",
|
|
"✅ **Performance Regression Detection**: Automated monitoring for production\n",
|
|
"✅ **Capacity Planning**: Resource allocation and scaling recommendations\n",
|
|
"✅ **Integration**: Real-world evaluation with TinyTorch models\n",
|
|
"\n",
|
|
"### Key Concepts You've Learned\n",
|
|
"- **Benchmarking**: Systematic evaluation of model performance\n",
|
|
"- **Statistical validation**: Ensuring results are significant and reproducible\n",
|
|
"- **Performance reporting**: Generating professional reports and visualizations\n",
|
|
"- **Scenario testing**: Evaluating models in different deployment scenarios\n",
|
|
"- **Production profiling**: End-to-end pipeline analysis and optimization\n",
|
|
"- **A/B testing**: Statistical comparison frameworks for production\n",
|
|
"- **Performance monitoring**: Regression detection and alerting systems\n",
|
|
"- **Capacity planning**: Resource allocation and scaling analysis\n",
|
|
"- **Integration patterns**: How benchmarking works with neural networks\n",
|
|
"\n",
|
|
"### Professional Skills Developed\n",
|
|
"- **Evaluation engineering**: Building robust benchmarking systems\n",
|
|
"- **Statistical analysis**: Validating results with confidence intervals\n",
|
|
"- **Production profiling**: End-to-end performance analysis and optimization\n",
|
|
"- **A/B testing**: Statistical frameworks for production model comparison\n",
|
|
"- **Performance monitoring**: Regression detection and alerting systems\n",
|
|
"- **Capacity planning**: Resource allocation and scaling analysis\n",
|
|
"- **Reporting**: Generating professional reports for stakeholders\n",
|
|
"- **Integration testing**: Ensuring benchmarking works with neural networks\n",
|
|
"\n",
|
|
"### Ready for Advanced Applications\n",
|
|
"Your benchmarking implementations now enable:\n",
|
|
"- **Production evaluation**: Systematic testing before deployment\n",
|
|
"- **Research validation**: Ensuring results are statistically significant\n",
|
|
"- **Performance optimization**: Identifying bottlenecks and improving models\n",
|
|
"- **Scenario analysis**: Testing models in real-world conditions\n",
|
|
"- **Production monitoring**: Real-time performance tracking and alerting\n",
|
|
"- **A/B testing**: Safe rollout of new model versions in production\n",
|
|
"- **Capacity planning**: Resource allocation for scaling ML systems\n",
|
|
"- **Cost optimization**: Understanding resource usage for efficient deployment\n",
|
|
"\n",
|
|
"### Connection to Real ML Systems\n",
|
|
"Your implementations mirror production systems:\n",
|
|
"- **MLPerf**: Industry-standard benchmarking suite\n",
|
|
"- **PyTorch**: Built-in benchmarking and evaluation tools\n",
|
|
"- **TensorFlow**: Similar evaluation and reporting systems\n",
|
|
"- **Production Profiling**: Advanced monitoring and optimization patterns\n",
|
|
"- **Industry Standard**: Every major ML framework uses these exact patterns\n",
|
|
"\n",
|
|
"### Next Steps\n",
|
|
"1. **Export your code**: `tito export 14_benchmarking`\n",
|
|
"2. **Test your implementation**: `tito test 14_benchmarking`\n",
|
|
"3. **Evaluate models**: Use benchmarking to validate performance\n",
|
|
"4. **Apply production patterns**: Use your profiling tools for real projects\n",
|
|
"5. **Move to Module 15**: Continue building advanced ML systems!\n",
|
|
"\n",
|
|
"**Ready for Production Deployment?** Your benchmarking and profiling systems are now ready for real-world ML systems!"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"main_language": "python"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|