github-starred/TinyTorch
2
0
Fork 0
mirror of https://github.com/MLSysBook/TinyTorch.git synced 2026-05-03 01:07:50 -05:00
Code Issues 7 Packages Projects Releases Wiki Activity
Files
95ba293dd7d06fe9d194eedefdee8f32d4291a2a
TinyTorch/modules/15_acceleration/acceleration_dev.ipynb
Vijay Janapa Reddi 45a9cef548 Major reorganization: Remove setup module, renumber all modules, add tito setup command and numeric shortcuts 
- Removed 01_setup module (archived to archive/setup_module)
- Renumbered all modules: tensor is now 01, activations is 02, etc.
- Added tito setup command for environment setup and package installation
- Added numeric shortcuts: tito 01, tito 02, etc. for quick module access
- Fixed view command to find dev files correctly
- Updated module dependencies and references
- Improved user experience: immediate ML learning instead of boring setup
2025-09-28 07:02:08 -04:00

794 lines
33 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"cells": [
{
"cell_type": "markdown",
"id": "bb43e942",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"# Module 16: Hardware Acceleration - The Free Speedup!\n",
"\n",
"## Learning Objectives\n",
"By the end of this module, you will be able to:\n",
"\n",
"1. **Understand Why Loops Are Slow**: See why your Module 2/4 loops have poor performance\n",
"2. **Implement Cache-Friendly Blocking**: Build blocked matrix multiplication that leverages CPU cache hierarchy\n",
"3. **Visualize Memory Access Patterns**: Understand how cache misses destroy performance\n",
"4. **Build Transparent Backend Systems**: Create automatic switching between implementations\n",
"5. **Apply to Real Models**: Use these principles in MLPs, CNNs, and Transformers\n",
"\n",
"## The Free Speedup Journey\n",
"\n",
"**Key Message**: This is the EASIEST optimization - just use better backends! No accuracy trade-offs, no complex math - just 10-100x faster code.\n",
"\n",
"**The Journey:**\n",
"1. **Baseline**: Your loops from Module 2/4 (educational, 1000x slower)\n",
"2. **Blocking**: Cache-friendly version (educational, 10x faster than loops)\n",
"3. **NumPy**: Production version (optimal, another 10x faster)\n",
"4. **Backend**: Smart switching system (transparent optimization)\n",
"\n",
"**Why This Works**: Same math, better implementation. Free performance with zero downsides!"
]
},
{
"cell_type": "markdown",
"id": "b3809c9d",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Part 1: Baseline Implementation - Your Loops from Module 2/4\n",
"\n",
"Let's start with the educational triple-nested loops you implemented earlier. These were perfect for learning but terrible for performance."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a8e2f798",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| default_exp optimization.acceleration\n",
"\n",
"import time\n",
"import numpy as np\n",
"\n",
"def matmul_naive(a: np.ndarray, b: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Educational matrix multiplication using triple nested loops.\n",
" \n",
" This is the same implementation from Module 2/4 - perfect for learning\n",
" the algorithm, but very slow due to poor cache performance.\n",
" \"\"\"\n",
" m, k = a.shape\n",
" k2, n = b.shape\n",
" assert k == k2, f\"Incompatible shapes: {a.shape} @ {b.shape}\"\n",
" \n",
" # Initialize result matrix\n",
" c = np.zeros((m, n), dtype=np.float32)\n",
" \n",
" # Triple nested loop - the educational implementation\n",
" for i in range(m):\n",
" for j in range(n):\n",
" for l in range(k):\n",
" c[i, j] += a[i, l] * b[l, j]\n",
" \n",
" return c"
]
},
{
"cell_type": "markdown",
"id": "c85ddf51",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"### Test Educational Implementation\n",
"\n",
"Let's test our educational loops and see why they're slow."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "68fb5eed",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"def test_naive_baseline():\n",
" \"\"\"Test naive implementation and measure its performance\"\"\"\n",
" print(\"Testing Naive Implementation...\")\n",
" \n",
" # Test correctness with small matrices\n",
" a = np.array([[1, 2], [3, 4]], dtype=np.float32)\n",
" b = np.array([[5, 6], [7, 8]], dtype=np.float32)\n",
" \n",
" result_naive = matmul_naive(a, b)\n",
" result_numpy = a @ b\n",
" assert np.allclose(result_naive, result_numpy), \"Naive matmul incorrect\"\n",
" print(\"✅ Naive implementation produces correct results\")\n",
" \n",
" # Performance comparison (small sizes only - educational is VERY slow)\n",
" print(\"\\nPerformance comparison:\")\n",
" small_a = np.random.randn(100, 100).astype(np.float32)\n",
" small_b = np.random.randn(100, 100).astype(np.float32)\n",
" \n",
" # Time naive implementation\n",
" start = time.perf_counter()\n",
" _ = matmul_naive(small_a, small_b)\n",
" naive_time = time.perf_counter() - start\n",
" \n",
" # Time NumPy implementation\n",
" start = time.perf_counter()\n",
" _ = small_a @ small_b\n",
" numpy_time = time.perf_counter() - start\n",
" \n",
" speedup = naive_time / numpy_time\n",
" print(f\"Naive loops: {naive_time*1000:.1f} ms\")\n",
" print(f\"NumPy optimized: {numpy_time*1000:.1f} ms\")\n",
" print(f\"NumPy is {speedup:.1f}x faster\")\n",
" \n",
" print(\"✅ Naive baseline established\")\n",
" return naive_time, numpy_time, speedup"
]
},
{
"cell_type": "markdown",
"id": "fd8cdf2e",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Part 2: Understanding Cache Hierarchy - Why Memory Matters More Than Computation\n",
"\n",
"**The Big Insight**: Modern CPUs are FAST at computation but SLOW at memory access. Cache hierarchy makes the difference between fast and slow code.\n",
"\n",
"### CPU Cache Hierarchy Visualization\n",
"```\n",
"Registers: 4 bytes - 1 cycle (instant)\n",
"L1 Cache: 32KB - 3-4 cycles (lightning fast)\n",
"L2 Cache: 256KB - 10-20 cycles (fast)\n",
"L3 Cache: 8MB - 50-100 cycles (slow)\n",
"Main RAM: 16GB - 200+ cycles (VERY slow)\n",
"```\n",
"\n",
"**Key Principle**: Keep your working set in L1/L2 cache for 100x better performance!\n",
"\n",
"### Memory Access Pattern Analysis\n",
"\n",
"Your naive loops access memory like this:\n",
"```python\n",
"for i in range(m):\n",
" for j in range(n):\n",
" for l in range(k):\n",
" c[i,j] += a[i,l] * b[l,j] # b[l,j] jumps around randomly!\n",
"```\n",
"\n",
"**The Problem**: `b[l,j]` creates terrible access patterns:\n",
"- Each `j` increment jumps to a new column (cache miss)\n",
"- Each `l` increment jumps to a new row (another cache miss)\n",
"- For 1000x1000 matrix: 1 billion cache misses!\n",
"\n",
"**The Solution**: Process in blocks that fit in cache."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc2f1d0a",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"def matmul_blocked(a: np.ndarray, b: np.ndarray, block_size: int = 64) -> np.ndarray:\n",
" \"\"\"\n",
" Cache-friendly blocked matrix multiplication.\n",
" \n",
" This version processes data in blocks that fit in CPU cache.\n",
" \n",
" **Memory Analysis**:\n",
" - 64x64 block = 4KB floats = 16KB memory (fits in 32KB L1 cache)\n",
" - 3 blocks (A, B, C) = 48KB total (fits in 256KB L2 cache)\n",
" - Reuses each data element 64 times before evicting from cache\n",
" \n",
" **Why This Works**:\n",
" - Naive: 1 cache miss per operation (terrible)\n",
" - Blocked: 1 cache miss per 64 operations (64x better!)\n",
" \n",
" Args:\n",
" a: Left matrix (m × k)\n",
" b: Right matrix (k × n) \n",
" block_size: Cache-friendly block size (32-128, default 64)\n",
" \"\"\"\n",
" m, k = a.shape\n",
" k2, n = b.shape\n",
" assert k == k2, f\"Incompatible shapes: {a.shape} @ {b.shape}\"\n",
" \n",
" # Initialize result\n",
" c = np.zeros((m, n), dtype=np.float32)\n",
" \n",
" # Process in blocks to maximize cache utilization\n",
" for i in range(0, m, block_size):\n",
" for j in range(0, n, block_size):\n",
" for l in range(0, k, block_size):\n",
" # Define block boundaries\n",
" i_end = min(i + block_size, m)\n",
" j_end = min(j + block_size, n)\n",
" l_end = min(l + block_size, k)\n",
" \n",
" # Extract blocks (these stay in cache)\n",
" a_block = a[i:i_end, l:l_end]\n",
" b_block = b[l:l_end, j:j_end]\n",
" \n",
" # Multiply blocks using NumPy (optimized BLAS)\n",
" c[i:i_end, j:j_end] += a_block @ b_block\n",
" \n",
" return c"
]
},
{
"cell_type": "markdown",
"id": "74d05383",
"metadata": {
"lines_to_next_cell": 1
},
"source": [
"\"\"\"\n",
"## Test Blocked Implementation\n",
"\n",
"Let's see how much faster cache-friendly blocking is compared to educational loops.\n",
"\"\"\"\n",
"\n",
"def test_blocked_optimization():\n",
" \"\"\"Test blocked matrix multiplication performance\"\"\"\n",
" print(\"Testing Blocked Matrix Multiplication...\")\n",
" \n",
" # Test correctness\n",
" a = np.random.randn(200, 200).astype(np.float32)\n",
" b = np.random.randn(200, 200).astype(np.float32)\n",
" \n",
" result_blocked = matmul_blocked(a, b, block_size=64)\n",
" result_numpy = a @ b\n",
" \n",
" assert np.allclose(result_blocked, result_numpy, atol=1e-3), \"Blocked matmul incorrect\"\n",
" print(\"✅ Blocked implementation produces correct results\")\n",
" \n",
" # Performance comparison\n",
" print(\"\\nPerformance comparison:\")\n",
" \n",
" # Educational vs Blocked vs NumPy\n",
" size = 200\n",
" test_a = np.random.randn(size, size).astype(np.float32)\n",
" test_b = np.random.randn(size, size).astype(np.float32)\n",
" \n",
" # Time educational (smaller subset to avoid waiting forever)\n",
" start = time.perf_counter()\n",
" _ = matmul_naive(test_a[:50, :50], test_b[:50, :50])\n",
" naive_time = time.perf_counter() - start\n",
" naive_time_scaled = naive_time * (size/50)**3 # Scale up for comparison\n",
" \n",
" # Time blocked\n",
" start = time.perf_counter()\n",
" _ = matmul_blocked(test_a, test_b, block_size=64)\n",
" blocked_time = time.perf_counter() - start\n",
" \n",
" # Time NumPy\n",
" start = time.perf_counter()\n",
" _ = test_a @ test_b\n",
" numpy_time = time.perf_counter() - start\n",
" \n",
" print(f\"Naive (estimated): {naive_time_scaled*1000:.1f} ms\")\n",
" print(f\"Blocked: {blocked_time*1000:.1f} ms\")\n",
" print(f\"NumPy: {numpy_time*1000:.1f} ms\")\n",
" \n",
" speedup_blocked = naive_time_scaled / blocked_time\n",
" speedup_numpy = naive_time_scaled / numpy_time\n",
" \n",
" print(f\"\\n🚀 SPEEDUP RESULTS:\")\n",
" print(f\"Blocked is {speedup_blocked:.1f}x faster than naive loops!\")\n",
" print(f\"NumPy is {speedup_numpy:.1f}x faster than naive loops!\")\n",
" print(f\"\\n💡 Why blocking works: Better cache utilization!\")\n",
" print(f\" • Naive: 1 cache miss per operation\")\n",
" print(f\" • Blocked: 1 cache miss per 64 operations\")\n",
" print(f\" • NumPy: Professional optimizations + vectorization\")\n",
" \n",
" print(\"✅ Blocked optimization tested successfully\")\n",
" return blocked_time, numpy_time"
]
},
{
"cell_type": "markdown",
"id": "5dd1eddc",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Part 3: NumPy Optimization - Production Performance\n",
"\n",
"Now we'll switch to NumPy for production use. The key insight: NumPy already has these optimizations (and more) built-in."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "510040fa",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"def matmul_numpy(a: np.ndarray, b: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Production matrix multiplication using NumPy.\n",
" \n",
" This is what you should actually use in practice.\n",
" NumPy already has blocking, vectorization, and BLAS optimizations built-in.\n",
" \"\"\"\n",
" return a @ b"
]
},
{
"cell_type": "markdown",
"id": "6dc5cef7",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"### Test Production Implementation\n",
"\n",
"Let's verify that NumPy is indeed the best choice for production."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5450d83e",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"def test_production_performance():\n",
" \"\"\"Test that NumPy is indeed optimal for production use\"\"\"\n",
" print(\"Testing Production Performance...\")\n",
" \n",
" # Test different sizes\n",
" sizes = [200, 500, 800]\n",
" \n",
" print(\"\\nPerformance comparison across the optimization spectrum:\")\n",
" \n",
" for size in sizes:\n",
" print(f\"\\nMatrix size: {size}x{size}\")\n",
" a = np.random.randn(size, size).astype(np.float32)\n",
" b = np.random.randn(size, size).astype(np.float32)\n",
" \n",
" # Time blocked implementation\n",
" start = time.perf_counter()\n",
" _ = matmul_blocked(a, b, block_size=64)\n",
" blocked_time = time.perf_counter() - start\n",
" \n",
" # Time NumPy implementation\n",
" start = time.perf_counter()\n",
" _ = matmul_numpy(a, b)\n",
" numpy_time = time.perf_counter() - start\n",
" \n",
" speedup = blocked_time / numpy_time\n",
" print(f\"Blocked: {blocked_time*1000:6.1f} ms\")\n",
" print(f\"NumPy: {numpy_time*1000:6.1f} ms\")\n",
" print(f\"NumPy is {speedup:.1f}x faster than blocked\")\n",
" \n",
" print(\"\\n💡 Key Insight: NumPy already has these optimizations built-in!\")\n",
" print(\" • Blocking algorithms\")\n",
" print(\" • Vectorization\")\n",
" print(\" • Hardware-specific BLAS libraries\")\n",
" print(\" • Assembly-level optimizations\")\n",
" \n",
" print(\"\\n✅ Production performance verified\")\n",
" return True"
]
},
{
"cell_type": "markdown",
"id": "34430270",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Part 4: Smart Backend System - Transparent Optimization\n",
"\n",
"Now let's build a system that automatically chooses the right implementation. This is how real ML frameworks work!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bb6e536f",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"class OptimizedBackend:\n",
" \"\"\"\n",
" Smart backend that automatically dispatches to optimal implementations.\n",
" \n",
" This demonstrates how real ML frameworks (PyTorch, TensorFlow) work:\n",
" - Single API for users\n",
" - Automatic dispatch to fastest implementation\n",
" - Transparent optimization without code changes\n",
" \"\"\"\n",
" \n",
" def dispatch(self, op: str, *args, **kwargs):\n",
" \"\"\"Dispatch operations to optimal implementations\"\"\"\n",
" if op == \"matmul\":\n",
" return self.matmul(*args, **kwargs)\n",
" else:\n",
" raise NotImplementedError(f\"Operation {op} not implemented\")\n",
" \n",
" def matmul(self, a: np.ndarray, b: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Matrix multiplication with automatic optimization selection.\n",
" \n",
" For production: Always use NumPy (has all optimizations built-in)\n",
" For education: Could switch based on size, but NumPy is always best\n",
" \"\"\"\n",
" # In a real system, you might choose based on:\n",
" # - Matrix size (small vs large)\n",
" # - Hardware available (CPU vs GPU)\n",
" # - Memory constraints\n",
" # \n",
" # But NumPy is almost always the right choice for CPU\n",
" return matmul_numpy(a, b)\n",
"\n",
"# Global backend instance\n",
"_backend = OptimizedBackend()\n",
"\n",
"def matmul(a: np.ndarray, b: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Matrix multiplication using optimal backend.\n",
" \n",
" This is the API students should use - it automatically\n",
" selects the best implementation available.\n",
" \"\"\"\n",
" return _backend.dispatch(\"matmul\", a, b)"
]
},
{
"cell_type": "markdown",
"id": "3bf96063",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"### Test Backend System\n",
"\n",
"Let's verify our backend system works correctly and uses optimal implementations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "daaad52d",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"def test_backend_system():\n",
" \"\"\"Test the backend system\"\"\"\n",
" print(\"Testing Backend System...\")\n",
" \n",
" # Test matrices\n",
" a = np.random.randn(100, 100).astype(np.float32)\n",
" b = np.random.randn(100, 100).astype(np.float32)\n",
" \n",
" # Test that our backend works\n",
" result = matmul(a, b)\n",
" expected = a @ b\n",
" \n",
" assert np.allclose(result, expected), \"Backend matmul incorrect\"\n",
" print(\"✅ Backend produces correct results\")\n",
" \n",
" # Compare performance\n",
" start = time.perf_counter()\n",
" _ = matmul(a, b)\n",
" backend_time = time.perf_counter() - start\n",
" \n",
" start = time.perf_counter()\n",
" _ = a @ b\n",
" numpy_time = time.perf_counter() - start\n",
" \n",
" print(f\"\\nPerformance comparison:\")\n",
" print(f\"Backend: {backend_time*1000:.1f} ms\")\n",
" print(f\"NumPy: {numpy_time*1000:.1f} ms\")\n",
" print(f\"Backend uses optimal NumPy implementation\")\n",
" \n",
" print(\"\\n✅ Backend system works correctly\")\n",
" return True"
]
},
{
"cell_type": "markdown",
"id": "d3ae2f46",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Part 5: Real-World Application Testing\n",
"\n",
"Let's test our optimizations on actual ML model operations: MLP layers, CNN convolutions, and Transformer attention."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a4858d70",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"def test_ml_model_acceleration():\n",
" \"\"\"Test acceleration on real ML model operations\"\"\"\n",
" print(\"Testing Acceleration on Real ML Models...\")\n",
" \n",
" # Test 1: MLP Forward Pass (common in Module 4)\n",
" print(\"\\n1. MLP Forward Pass (256 → 128 → 64):\")\n",
" batch_size, input_dim, hidden_dim, output_dim = 32, 256, 128, 64\n",
" \n",
" # Simulated MLP layers\n",
" x = np.random.randn(batch_size, input_dim).astype(np.float32)\n",
" W1 = np.random.randn(input_dim, hidden_dim).astype(np.float32)\n",
" W2 = np.random.randn(hidden_dim, output_dim).astype(np.float32)\n",
" \n",
" # Time naive implementation (small version)\n",
" start = time.perf_counter()\n",
" h1_naive = matmul_naive(x[:8, :64], W1[:64, :32]) # Scaled down\n",
" h2_naive = matmul_naive(h1_naive, W2[:32, :16]) # Scaled down\n",
" naive_time = time.perf_counter() - start\n",
" \n",
" # Time optimized implementation\n",
" start = time.perf_counter()\n",
" h1_opt = matmul(x, W1)\n",
" h2_opt = matmul(h1_opt, W2)\n",
" opt_time = time.perf_counter() - start\n",
" \n",
" # Scale naive time for comparison\n",
" naive_scaled = naive_time * (32/8) * (256/64) * (128/32)\n",
" speedup = naive_scaled / opt_time\n",
" \n",
" print(f\" Naive (estimated): {naive_scaled*1000:.1f} ms\")\n",
" print(f\" Optimized: {opt_time*1000:.1f} ms\")\n",
" print(f\" Speedup: {speedup:.1f}x faster!\")\n",
" \n",
" # Test 2: CNN-like Convolution (flattened as matrix multiply)\n",
" print(\"\\n2. CNN Convolution (as matrix multiply):\")\n",
" # Simulate im2col operation for 3x3 convolution\n",
" img_patches = np.random.randn(1024, 27).astype(np.float32) # 32x32 image, 3x3 patches\n",
" conv_filters = np.random.randn(27, 64).astype(np.float32) # 64 filters\n",
" \n",
" start = time.perf_counter()\n",
" conv_output = matmul(img_patches, conv_filters)\n",
" conv_time = time.perf_counter() - start\n",
" print(f\" Convolution output: {conv_time*1000:.1f} ms\")\n",
" print(f\" Shape: {conv_output.shape} (1024 locations × 64 filters)\")\n",
" \n",
" # Test 3: Transformer-like Attention (scaled down)\n",
" print(\"\\n3. Transformer Attention (Q·K^T):\")\n",
" seq_len, d_model = 128, 256\n",
" Q = np.random.randn(seq_len, d_model).astype(np.float32)\n",
" K = np.random.randn(seq_len, d_model).astype(np.float32)\n",
" \n",
" start = time.perf_counter()\n",
" attention_scores = matmul(Q, K.T) # Shape: (seq_len, seq_len)\n",
" attn_time = time.perf_counter() - start\n",
" print(f\" Attention computation: {attn_time*1000:.1f} ms\")\n",
" print(f\" Shape: {attention_scores.shape} (128×128 attention matrix)\")\n",
" \n",
" print(f\"\\n✅ All ML model operations accelerated successfully!\")\n",
" print(f\"💡 Key insight: Matrix multiplication is EVERYWHERE in ML!\")\n",
" return True\n",
"\n",
"def run_complete_acceleration_demo():\n",
" \"\"\"Run the complete acceleration demonstration\"\"\"\n",
" print(\"🚀 Complete Hardware Acceleration Demo\")\n",
" print(\"=\" * 55)\n",
" print(\"THE FREE SPEEDUP: From Naive Loops to Optimized Backends\")\n",
" \n",
" # 1. Test naive baseline\n",
" print(\"\\n1. Naive Baseline (your Module 2/4 loops):\")\n",
" naive_results = test_naive_baseline()\n",
" \n",
" # 2. Test blocked optimization\n",
" print(\"\\n2. Cache-Friendly Blocking:\")\n",
" test_blocked_optimization()\n",
" \n",
" # 3. Test production performance\n",
" print(\"\\n3. Production Performance (NumPy):\")\n",
" test_production_performance()\n",
" \n",
" # 4. Test ML model acceleration\n",
" print(\"\\n4. Real ML Model Acceleration:\")\n",
" test_ml_model_acceleration()\n",
" \n",
" # 5. Test backend system\n",
" print(\"\\n5. Smart Backend System:\")\n",
" test_backend_system()\n",
" \n",
" print(\"\\n\" + \"=\" * 55)\n",
" print(\"🎯 HARDWARE ACCELERATION MASTERED\")\n",
" print(\"=\" * 55)\n",
" \n",
" print(\"\\n📚 What You Mastered:\")\n",
" print(\"✅ Why your Module 2/4 loops were slow (cache hierarchy matters!)\")\n",
" print(\"✅ How cache-friendly blocking works (process data in chunks)\")\n",
" print(\"✅ Why NumPy dominates (professional optimizations built-in)\")\n",
" print(\"✅ How to build smart backend systems (automatic optimization)\")\n",
" print(\"✅ Real ML applications (MLPs, CNNs, Transformers all use matmul!)\")\n",
" \n",
" print(\"\\n🎯 The Free Speedup Philosophy:\")\n",
" print(\"• 🚀 Same math, better implementation = 100x speedup\")\n",
" print(\"• 🧠 Educational loops teach algorithms\")\n",
" print(\"• ⚡ Blocked algorithms teach cache optimization\")\n",
" print(\"• 🏭 NumPy provides production performance\")\n",
" print(\"• 🎯 Smart backends make optimization transparent\")\n",
" print(\"• 💡 Understanding the spectrum makes you a better engineer!\")\n",
" \n",
" return naive_results"
]
},
{
"cell_type": "markdown",
"id": "6fa92758",
"metadata": {},
"source": [
"\"\"\"\n",
"# Systems Analysis Summary\n",
"\n",
"This module demonstrates the fundamental principles of hardware acceleration in ML systems:\n",
"\n",
"## 🏗️ **Architecture Principles**\n",
"- **Cache Hierarchy**: Understanding L1/L2/L3 cache and memory access costs\n",
"- **Vectorization**: Leveraging SIMD instructions for parallel computation\n",
"- **Memory Layout**: Contiguous access patterns for optimal performance\n",
"- **Backend Abstraction**: Transparent dispatch between naive and optimized implementations\n",
"\n",
"## ⚡ **Optimization Techniques**\n",
"- **Blocked Algorithms**: Process data in cache-friendly blocks\n",
"- **Vectorized Operations**: Avoid Python loops, use NumPy's optimized routines\n",
"- **In-place Operations**: Minimize memory allocation overhead\n",
"- **Automatic Dispatch**: Choose optimal implementation based on problem size\n",
"\n",
"## 📊 **Performance Understanding**\n",
"- **Measurement First**: Profile real bottlenecks before optimizing\n",
"- **Algorithmic Impact**: O(N³) → O(N²) matters more than 2x constant factors\n",
"- **Hardware Awareness**: CPU cache misses cost 100x more than cache hits\n",
"- **Library Utilization**: Optimized BLAS libraries beat custom implementations\n",
"\n",
"## 🎯 **Real-World Applications**\n",
"- **ML Frameworks**: How PyTorch/TensorFlow apply these same principles\n",
"- **Production Systems**: Where optimization efforts provide real value\n",
"- **Development Practice**: When to optimize vs when to use existing solutions\n",
"\n",
"## 💡 **Key Insights**\n",
"- Cache-friendly algorithms provide 2-5x speedups from memory access patterns alone\n",
"- Vectorization eliminates Python overhead for 10-100x improvements\n",
"- Most NumPy operations are already optimized - focus on system-level improvements\n",
"- Competition frameworks make optimization learning engaging and quantifiable\n",
"- Real ML systems face memory and communication bottlenecks, not pure computation limits\n",
"\n",
"This approach teaches students to think like systems engineers: understand the hardware, measure scientifically, optimize systematically, and focus efforts where they matter most.\n",
"\"\"\"\n",
"\n",
"if __name__ == \"__main__\":\n",
" print(\"Module 16: Hardware Acceleration - The Free Speedup!\")\n",
" print(\"=\" * 60)\n",
" print(\"🚀 THE EASIEST OPTIMIZATION: Better Backends, Zero Trade-offs\")\n",
" \n",
" # Run complete demonstration\n",
" results = run_complete_acceleration_demo()\n",
" \n",
" print(f\"\\n🎉 Module 16: Hardware Acceleration COMPLETE!\")\n",
" print(f\"⚡ Mastered: 10-100x speedups with no accuracy loss\")\n",
" print(f\"🧠 Learned: Cache hierarchy, blocking, vectorization\")\n",
" print(f\"🏭 Applied: MLPs, CNNs, Transformers all benefit\")\n",
" print(f\"🎯 Ready: To build high-performance ML systems!\")"
]
},
{
"cell_type": "markdown",
"id": "4967dd03",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## 🤔 ML Systems Thinking: Interactive Questions\n",
"\n",
"1. **Memory Access Pattern Analysis**: Your educational loops access `b[l, j]` in the innermost loop, creating terrible cache performance. Draw a diagram showing how this access pattern jumps around in memory, calculate the number of cache misses for a 1000×1000 matrix multiply, and explain why this creates exponentially worse performance as matrices get larger.\n",
"\n",
"2. **Cache Hierarchy Optimization**: Your blocked implementation uses 64×64 blocks. Calculate: (a) Total memory footprint of three 64×64 float32 blocks, (b) Why this fits in L1/L2 cache, (c) Cache utilization ratio (reuses per cache miss), and (d) What happens with 256×256 blocks instead (hint: L3 cache limit).\n",
"\n",
"3. **Production Library Justification**: You implemented blocking for education, but NumPy beats it by another 10x. Identify three specific optimizations NumPy has (vectorization, BLAS libraries, assembly kernels) and calculate the development cost vs. performance benefit of implementing these yourself. Why is this a losing proposition for ML engineers?\n",
"\n",
"4. **ML Model Acceleration Strategy**: You tested MLP, CNN, and Transformer operations. For each model type, identify: (a) The dominant matrix operations, (b) Which operations benefit most from acceleration, (c) Memory vs. compute bottlenecks, and (d) Why understanding the optimization spectrum makes you a better ML systems engineer."
]
},
{
"cell_type": "markdown",
"id": "a582121a",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 2
},
"source": [
"## 🎯 MODULE SUMMARY: Hardware Acceleration - The Free Speedup\n",
"\n",
"This module demonstrates the easiest optimization in ML systems: using better backends for free speedups with zero accuracy trade-offs. You learned why understanding the optimization spectrum makes you a better engineer.\n",
"\n",
"### 🛤️ **The Free Speedup Journey**\n",
"- **Educational Foundation**: Your Module 2/4 loops taught you the algorithm (perfect for learning)\n",
"- **Performance Understanding**: Module 15 showed you WHY loops are slow (profiling first)\n",
"- **Optimization Mastery**: Now you achieve 100x speedups by choosing better implementations\n",
"- **Systems Thinking**: Understanding the spectrum from educational to production code\n",
"\n",
"### 🛠️ **What We Built and Tested**\n",
"- **Educational Baseline**: Your triple-nested loops from Module 2/4 (algorithm understanding)\n",
"- **Cache-Friendly Blocking**: 64×64 blocks fitting in L1/L2 cache (10x+ speedup)\n",
"- **NumPy Production**: Leveraging professional BLAS optimizations (another 10x speedup)\n",
"- **Smart Backend System**: Automatic dispatch to optimal implementations\n",
"- **Real ML Applications**: MLP, CNN, Transformer operations using matrix multiplication\n",
"\n",
"### 🧠 **Key Learning Outcomes**\n",
"- **Why loops are slow**: Memory access patterns and cache hierarchy matter most\n",
"- **How blocking helps**: Processing data in cache-friendly chunks improves performance\n",
"- **When to use NumPy**: It already has these optimizations (and more) built-in\n",
"- **Systems thinking**: Understanding enables better decisions about when to optimize\n",
"\n",
"### ⚡ **Performance Spectrum Mastered**\n",
"- **Educational loops**: Algorithm understanding (1000x slower, perfect for learning)\n",
"- **Cache-friendly blocking**: Systems understanding (100x slower, teaches optimization)\n",
"- **NumPy production**: Professional performance (optimal speed, built-in optimizations)\n",
"- **Smart backends**: Engineering understanding (transparent optimization selection)\n",
"\n",
"### 🏆 **Practical Skills Developed**\n",
"- Analyze why educational implementations have poor performance\n",
"- Implement cache-friendly algorithms to understand optimization principles\n",
"- Choose NumPy for production while understanding what it's doing internally\n",
"- Build systems that balance educational value with performance requirements\n",
"\n",
"### 📊 **Systems Insights Gained**\n",
"- **Educational code serves a purpose**: Understanding algorithms enables optimization intuition\n",
"- **Cache hierarchy dominates performance**: Memory access patterns matter more than computation\n",
"- **Libraries beat custom optimization**: NumPy already has expert-level optimizations\n",
"- **Understanding enables better tools**: You can build smarter systems when you know the principles\n",
"\n",
"### 💡 **The Free Speedup Philosophy**\n",
"This is the EASIEST optimization in ML systems: same math, better implementation, massive speedups, zero downsides. You implemented loops to understand algorithms. You implemented blocking to understand cache optimization. Now you use NumPy because it has all optimizations built-in. Understanding this spectrum - from educational to production - makes you a superior ML systems engineer who can make informed optimization decisions."
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "-all",
"main_language": "python",
"notebook_metadata_filter": "-all"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink