github-starred/TinyTorch
2
0
Fork 0
mirror of https://github.com/MLSysBook/TinyTorch.git synced 2026-05-24 12:55:50 -05:00
Code Issues 7 Packages Projects Releases Wiki Activity
Files
gh-pages
TinyTorch/modules/16_compression_ABOUT.html
github-actions[bot] 3c159d9fd8 Deploy live site from main branch - 7bc4f6f835 7bc4f6f835
2025-12-05 00:52:38 +00:00

1000 lines
72 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en" data-content_root="../" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>16. Compression - Pruning and Model Compression &#8212; Tiny🔥Torch</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!-- Loaded before other Sphinx assets -->
<link href="../_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="../_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="../_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="../_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="../_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="../_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="../_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="../_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="../_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="../_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css" />
<link rel="stylesheet" type="text/css" href="../_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="../_static/sphinx-design.min.css?v=95c83b7e" />
<link rel="stylesheet" type="text/css" href="../_static/custom.css?v=009d37f4" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="../_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
<link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
<script src="../_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="../_static/documentation_options.js?v=9eb32ce0"></script>
<script src="../_static/doctools.js?v=9a2dae69"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/clipboard.min.js?v=a7894cd8"></script>
<script src="../_static/copybutton.js?v=f281be69"></script>
<script src="../_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="../_static/togglebutton.js?v=4a39c7ea"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script src="../_static/design-tabs.js?v=f930bc37"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script async="async" src="../_static/sphinx-thebe.js?v=c100c467"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script>DOCUMENTATION_OPTIONS.pagename = 'modules/16_compression_ABOUT';</script>
<script src="../_static/ml-timeline.js?v=76e9b3e3"></script>
<script src="../_static/wip-banner.js?v=04a7e74d"></script>
<script src="../_static/marimo-badges.js?v=e6289128"></script>
<script src="../_static/sidebar-link.js?v=404b701b"></script>
<script src="../_static/hero-carousel.js?v=10341d2a"></script>
<script src="../_static/subscribe-modal.js?v=42919b64"></script>
<link rel="icon" href="../_static/favicon.svg"/>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="17. Memoization - Computational Reuse for Inference" href="17_memoization_ABOUT.html" />
<link rel="prev" title="15. Quantization - Reduced Precision for Efficiency" href="15_quantization_ABOUT.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<input type="checkbox"
class="sidebar-toggle"
id="pst-primary-sidebar-checkbox"/>
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
<input type="checkbox"
class="sidebar-toggle"
id="pst-secondary-sidebar-checkbox"/>
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
<div class="search-button__wrapper">
<div class="search-button__overlay"></div>
<div class="search-button__search-container">
<form class="bd-search d-flex align-items-center"
action="../search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
id="search-input"
placeholder="Search..."
aria-label="Search..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form></div>
</div>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<div class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<a class="navbar-brand logo" href="../intro.html">
<img src="../_static/logo-tinytorch.png" class="logo__image only-light" alt="Tiny🔥Torch - Home"/>
<script>document.write(`<img src="../_static/logo-tinytorch.png" class="logo__image only-dark" alt="Tiny🔥Torch - Home"/>`);</script>
</a></div>
<div class="sidebar-primary-item">
<script>
document.write(`
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
`);
</script></div>
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
<div class="bd-toc-item navbar-nav active">
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🚀 Getting Started</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../getting-started.html">Complete Guide</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🏗 Foundation Tier (01-07)</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../tiers/foundation.html">📖 Tier Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="01_tensor_ABOUT.html">01. Tensor</a></li>
<li class="toctree-l1"><a class="reference internal" href="02_activations_ABOUT.html">02. Activations</a></li>
<li class="toctree-l1"><a class="reference internal" href="03_layers_ABOUT.html">03. Layers</a></li>
<li class="toctree-l1"><a class="reference internal" href="04_losses_ABOUT.html">04. Losses</a></li>
<li class="toctree-l1"><a class="reference internal" href="05_autograd_ABOUT.html">05. Autograd</a></li>
<li class="toctree-l1"><a class="reference internal" href="06_optimizers_ABOUT.html">06. Optimizers</a></li>
<li class="toctree-l1"><a class="reference internal" href="07_training_ABOUT.html">07. Training</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🏛️ Architecture Tier (08-13)</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../tiers/architecture.html">📖 Tier Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="08_dataloader_ABOUT.html">08. DataLoader</a></li>
<li class="toctree-l1"><a class="reference internal" href="09_spatial_ABOUT.html">09. Convolutions</a></li>
<li class="toctree-l1"><a class="reference internal" href="10_tokenization_ABOUT.html">10. Tokenization</a></li>
<li class="toctree-l1"><a class="reference internal" href="11_embeddings_ABOUT.html">11. Embeddings</a></li>
<li class="toctree-l1"><a class="reference internal" href="12_attention_ABOUT.html">12. Attention</a></li>
<li class="toctree-l1"><a class="reference internal" href="13_transformers_ABOUT.html">13. Transformers</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">⏱️ Optimization Tier (14-19)</span></p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../tiers/optimization.html">📖 Tier Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="14_profiling_ABOUT.html">14. Profiling</a></li>
<li class="toctree-l1"><a class="reference internal" href="15_quantization_ABOUT.html">15. Quantization</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">16. Compression</a></li>
<li class="toctree-l1"><a class="reference internal" href="17_memoization_ABOUT.html">17. Memoization</a></li>
<li class="toctree-l1"><a class="reference internal" href="18_acceleration_ABOUT.html">18. Acceleration</a></li>
<li class="toctree-l1"><a class="reference internal" href="19_benchmarking_ABOUT.html">19. Benchmarking</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🏅 Capstone Competition</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../tiers/olympics.html">📖 Competition Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="20_capstone_ABOUT.html">20. Torch Olympics</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🧭 Course Orientation</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../chapters/00-introduction.html">Course Structure</a></li>
<li class="toctree-l1"><a class="reference internal" href="../prerequisites.html">Prerequisites &amp; Resources</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/learning-journey.html">Learning Journey</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapters/milestones.html">Historical Milestones</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🛠️ TITO CLI Reference</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../tito/overview.html">Command Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tito/modules.html">Module Workflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tito/milestones.html">Milestone System</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tito/data.html">Progress &amp; Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tito/troubleshooting.html">Troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../datasets.html">Datasets Guide</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">🤝 Community</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../community.html">Ecosystem</a></li>
<li class="toctree-l1"><a class="reference internal" href="../resources.html">Learning Resources</a></li>
<li class="toctree-l1"><a class="reference internal" href="../credits.html">Credits &amp; Acknowledgments</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
</div>
<div id="rtd-footer-container"></div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="sbt-scroll-pixel-helper"></div>
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-bars"></span>
</button></div>
</div>
<div class="header-article-items__end">
<div class="header-article-item">
<div class="article-header-buttons">
<div class="dropdown dropdown-download-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
<i class="fas fa-download"></i>
</button>
<ul class="dropdown-menu">
<li><a href="../_sources/modules/16_compression_ABOUT.md" target="_blank"
class="btn btn-sm btn-download-source-button dropdown-item"
title="Download source file"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file"></i>
</span>
<span class="btn__text-container">.md</span>
</a>
</li>
<li>
<button onclick="window.print()"
class="btn btn-sm btn-download-pdf-button dropdown-item"
title="Print to PDF"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file-pdf"></i>
</span>
<span class="btn__text-container">.pdf</span>
</button>
</li>
</ul>
</div>
<button onclick="toggleFullScreen()"
class="btn btn-sm btn-fullscreen-button"
title="Fullscreen mode"
data-bs-placement="bottom" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-expand"></i>
</span>
</button>
<script>
document.write(`
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
</button>
`);
</script>
<script>
document.write(`
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
</button>
`);
</script>
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-list"></span>
</button>
</div></div>
</div>
</div>
</div>
<div id="jb-print-docs-body" class="onlyprint">
<h1>16. Compression - Pruning and Model Compression</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#overview">Overview</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-objectives">Learning Objectives</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#build-use-reflect">Build → Use → Reflect</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#implementation-guide">Implementation Guide</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#sparsity-measurement">Sparsity Measurement</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#magnitude-based-pruning-unstructured">Magnitude-Based Pruning (Unstructured)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#structured-pruning-hardware-friendly">Structured Pruning (Hardware-Friendly)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#knowledge-distillation">Knowledge Distillation</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#low-rank-approximation">Low-Rank Approximation</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#complete-compression-pipeline">Complete Compression Pipeline</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#getting-started">Getting Started</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#prerequisites">Prerequisites</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#development-workflow">Development Workflow</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing">Testing</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#comprehensive-test-suite">Comprehensive Test Suite</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#test-coverage-areas">Test Coverage Areas</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#inline-testing-validation">Inline Testing &amp; Validation</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#manual-testing-examples">Manual Testing Examples</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#systems-thinking-questions">Systems Thinking Questions</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#real-world-applications">Real-World Applications</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#compression-theory-foundations">Compression Theory Foundations</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#performance-characteristics-and-trade-offs">Performance Characteristics and Trade-offs</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ready-to-build">Ready to Build?</a></li>
</ul>
</nav>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<section id="compression-pruning-and-model-compression">
<h1>16. Compression - Pruning and Model Compression<a class="headerlink" href="#compression-pruning-and-model-compression" title="Link to this heading">#</a></h1>
<p><strong>OPTIMIZATION TIER</strong> | Difficulty: ⭐⭐⭐ (3/4) | Time: 5-6 hours</p>
<section id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Link to this heading">#</a></h2>
<p>Modern neural networks are massively overparameterized. BERT has 110M parameters but can compress to 40% size with 97% accuracy retention (DistilBERT). GPT-2 can be pruned 90% and retrained to similar performance (Lottery Ticket Hypothesis). Model compression techniques remove unnecessary parameters to enable practical deployment on resource-constrained devices.</p>
<p>This module implements core compression strategies: magnitude-based pruning (removing smallest weights), structured pruning (removing entire channels for hardware efficiency), knowledge distillation (training smaller models from larger teachers), and low-rank approximation (matrix factorization). You’ll understand the critical trade-offs between compression ratio, inference speedup, and accuracy retention.</p>
<p><strong>Important reality check</strong>: The implementations in this module demonstrate compression algorithms using NumPy, focusing on educational understanding of the techniques. Achieving actual inference speedup from sparse models requires specialized hardware support (NVIDIA’s 2:4 sparsity, specialized sparse CUDA kernels) or optimized libraries (torch.sparse, cuSPARSE) beyond this module’s scope. You’ll learn when compression helps versus when it creates overhead without benefits.</p>
</section>
<section id="learning-objectives">
<h2>Learning Objectives<a class="headerlink" href="#learning-objectives" title="Link to this heading">#</a></h2>
<p>By the end of this module, you will be able to:</p>
<ul class="simple">
<li><p><strong>Understand compression fundamentals</strong>: Differentiate between unstructured sparsity (scattered zeros), structured sparsity (removed channels), and architectural compression (distillation)</p></li>
<li><p><strong>Implement magnitude pruning</strong>: Remove weights below importance thresholds to achieve 50-95% sparsity with minimal accuracy loss</p></li>
<li><p><strong>Design structured pruning</strong>: Remove entire computational units (channels, neurons) using importance metrics like L2 norm</p></li>
<li><p><strong>Apply knowledge distillation</strong>: Train student models to match teacher performance using temperature-scaled soft targets</p></li>
<li><p><strong>Analyze compression trade-offs</strong>: Measure when pruning reduces model size without delivering proportional speedup, and understand hardware constraints</p></li>
</ul>
</section>
<section id="build-use-reflect">
<h2>Build → Use → Reflect<a class="headerlink" href="#build-use-reflect" title="Link to this heading">#</a></h2>
<p>This module follows TinyTorch’s <strong>Build → Use → Reflect</strong> framework:</p>
<ol class="arabic simple">
<li><p><strong>Build</strong>: Implement magnitude pruning, structured pruning, knowledge distillation, and low-rank approximation algorithms</p></li>
<li><p><strong>Use</strong>: Apply compression techniques to realistic neural networks and measure sparsity, parameter reduction, and memory savings</p></li>
<li><p><strong>Reflect</strong>: Understand why 90% unstructured sparsity rarely accelerates inference, when structured pruning delivers real speedups, and how compression strategies must adapt to hardware constraints</p></li>
</ol>
</section>
<section id="implementation-guide">
<h2>Implementation Guide<a class="headerlink" href="#implementation-guide" title="Link to this heading">#</a></h2>
<section id="sparsity-measurement">
<h3>Sparsity Measurement<a class="headerlink" href="#sparsity-measurement" title="Link to this heading">#</a></h3>
<p>Before compression, you need to quantify model density:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">measure_sparsity</span><span class="p">(</span><span class="n">model</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Calculate percentage of zero weights in model.&quot;&quot;&quot;</span>
<span class="n">total_params</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">zero_params</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">():</span>
<span class="n">total_params</span> <span class="o">+=</span> <span class="n">param</span><span class="o">.</span><span class="n">size</span>
<span class="n">zero_params</span> <span class="o">+=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">data</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="p">(</span><span class="n">zero_params</span> <span class="o">/</span> <span class="n">total_params</span><span class="p">)</span> <span class="o">*</span> <span class="mf">100.0</span>
</pre></div>
</div>
<p><strong>Why this matters</strong>: Sparsity measurement reveals how much redundancy exists. A 90% sparse model has only 10% active weights, but achieving speedup from this sparsity requires specialized hardware or storage formats.</p>
</section>
<section id="magnitude-based-pruning-unstructured">
<h3>Magnitude-Based Pruning (Unstructured)<a class="headerlink" href="#magnitude-based-pruning-unstructured" title="Link to this heading">#</a></h3>
<p>Remove individual weights with smallest absolute values:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">magnitude_prune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">sparsity</span><span class="o">=</span><span class="mf">0.9</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Remove smallest weights to achieve target sparsity.&quot;&quot;&quot;</span>
<span class="c1"># Collect all weights (excluding biases)</span>
<span class="n">all_weights</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">weight_params</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">():</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span> <span class="c1"># Skip 1D biases</span>
<span class="n">all_weights</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">flatten</span><span class="p">())</span>
<span class="n">weight_params</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">param</span><span class="p">)</span>
<span class="c1"># Find threshold at desired percentile</span>
<span class="n">magnitudes</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">all_weights</span><span class="p">)</span>
<span class="n">threshold</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">percentile</span><span class="p">(</span><span class="n">magnitudes</span><span class="p">,</span> <span class="n">sparsity</span> <span class="o">*</span> <span class="mi">100</span><span class="p">)</span>
<span class="c1"># Zero out weights below threshold</span>
<span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">weight_params</span><span class="p">:</span>
<span class="n">mask</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="n">threshold</span>
<span class="n">param</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">param</span><span class="o">.</span><span class="n">data</span> <span class="o">*</span> <span class="n">mask</span>
</pre></div>
</div>
<p><strong>Characteristics</strong>:</p>
<ul class="simple">
<li><p><strong>Compression</strong>: Can achieve 90%+ sparsity with minimal accuracy loss</p></li>
<li><p><strong>Speed reality</strong>: Creates scattered zeros that don’t accelerate dense matrix operations</p></li>
<li><p><strong>Storage benefit</strong>: Sparse formats (CSR, COO) reduce memory when combined with specialized storage</p></li>
<li><p><strong>Hardware requirement</strong>: Needs sparse tensor support for any speedup (torch.sparse, cuSPARSE)</p></li>
</ul>
<p><strong>Critical insight</strong>: High sparsity ratios don’t equal speedup. Dense matrix operations (GEMM) are highly optimized; sparse operations require irregular memory access and specialized kernels. Without hardware acceleration, 90% sparse models run at similar speeds to dense models.</p>
</section>
<section id="structured-pruning-hardware-friendly">
<h3>Structured Pruning (Hardware-Friendly)<a class="headerlink" href="#structured-pruning-hardware-friendly" title="Link to this heading">#</a></h3>
<p>Remove entire channels or neurons for actual hardware benefits:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">structured_prune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">prune_ratio</span><span class="o">=</span><span class="mf">0.5</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Remove entire channels based on L2 norm importance.&quot;&quot;&quot;</span>
<span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="n">model</span><span class="o">.</span><span class="n">layers</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">layer</span><span class="p">,</span> <span class="n">Linear</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">layer</span><span class="p">,</span> <span class="s1">&#39;weight&#39;</span><span class="p">):</span>
<span class="n">weight</span> <span class="o">=</span> <span class="n">layer</span><span class="o">.</span><span class="n">weight</span><span class="o">.</span><span class="n">data</span>
<span class="c1"># Calculate L2 norm for each output channel</span>
<span class="n">channel_norms</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">norm</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="c1"># Identify channels to remove (lowest importance)</span>
<span class="n">num_channels</span> <span class="o">=</span> <span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">num_to_prune</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">num_channels</span> <span class="o">*</span> <span class="n">prune_ratio</span><span class="p">)</span>
<span class="k">if</span> <span class="n">num_to_prune</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Get indices of weakest channels</span>
<span class="n">prune_indices</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">argpartition</span><span class="p">(</span>
<span class="n">channel_norms</span><span class="p">,</span> <span class="n">num_to_prune</span>
<span class="p">)[:</span><span class="n">num_to_prune</span><span class="p">]</span>
<span class="c1"># Zero entire channels</span>
<span class="n">weight</span><span class="p">[:,</span> <span class="n">prune_indices</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">if</span> <span class="n">layer</span><span class="o">.</span><span class="n">bias</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">layer</span><span class="o">.</span><span class="n">bias</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="n">prune_indices</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span>
</pre></div>
</div>
<p><strong>Characteristics</strong>:</p>
<ul class="simple">
<li><p><strong>Compression</strong>: 30-70% typical (coarser granularity than magnitude pruning)</p></li>
<li><p><strong>Speed benefit</strong>: Smaller dense matrices enable faster computation when architecturally reduced</p></li>
<li><p><strong>Accuracy trade-off</strong>: Loses more accuracy than unstructured pruning at same sparsity level</p></li>
<li><p><strong>Hardware friendly</strong>: Regular memory access patterns work well with standard dense operations</p></li>
</ul>
<p><strong>Critical insight</strong>: Structured pruning achieves lower compression ratios but enables real speedup when combined with architectural changes. Simply zeroing channels doesn’t help—you need to physically remove them from the model architecture to see benefits.</p>
</section>
<section id="knowledge-distillation">
<h3>Knowledge Distillation<a class="headerlink" href="#knowledge-distillation" title="Link to this heading">#</a></h3>
<p>Transfer knowledge from large teacher models to smaller students:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">class</span><span class="w"> </span><span class="nc">KnowledgeDistillation</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Compress models through teacher-student training.&quot;&quot;&quot;</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">teacher_model</span><span class="p">,</span> <span class="n">student_model</span><span class="p">,</span>
<span class="n">temperature</span><span class="o">=</span><span class="mf">3.0</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.7</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">teacher_model</span> <span class="o">=</span> <span class="n">teacher_model</span>
<span class="bp">self</span><span class="o">.</span><span class="n">student_model</span> <span class="o">=</span> <span class="n">student_model</span>
<span class="bp">self</span><span class="o">.</span><span class="n">temperature</span> <span class="o">=</span> <span class="n">temperature</span> <span class="c1"># Soften distributions</span>
<span class="bp">self</span><span class="o">.</span><span class="n">alpha</span> <span class="o">=</span> <span class="n">alpha</span> <span class="c1"># Balance soft vs hard targets</span>
<span class="k">def</span><span class="w"> </span><span class="nf">distillation_loss</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">student_logits</span><span class="p">,</span>
<span class="n">teacher_logits</span><span class="p">,</span> <span class="n">true_labels</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Combined loss: soft targets + hard labels.&quot;&quot;&quot;</span>
<span class="c1"># Temperature-scaled softmax for soft targets</span>
<span class="n">student_soft</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">(</span><span class="n">student_logits</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">temperature</span><span class="p">)</span>
<span class="n">teacher_soft</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">(</span><span class="n">teacher_logits</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">temperature</span><span class="p">)</span>
<span class="c1"># Soft loss: learn from teacher&#39;s knowledge</span>
<span class="n">soft_loss</span> <span class="o">=</span> <span class="n">kl_divergence</span><span class="p">(</span><span class="n">student_soft</span><span class="p">,</span> <span class="n">teacher_soft</span><span class="p">)</span>
<span class="c1"># Hard loss: learn correct answers</span>
<span class="n">student_hard</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">(</span><span class="n">student_logits</span><span class="p">)</span>
<span class="n">hard_loss</span> <span class="o">=</span> <span class="n">cross_entropy</span><span class="p">(</span><span class="n">student_hard</span><span class="p">,</span> <span class="n">true_labels</span><span class="p">)</span>
<span class="c1"># Weighted combination</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">alpha</span> <span class="o">*</span> <span class="n">soft_loss</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">alpha</span><span class="p">)</span> <span class="o">*</span> <span class="n">hard_loss</span>
</pre></div>
</div>
<p><strong>Why distillation works</strong>:</p>
<ul class="simple">
<li><p><strong>Soft targets</strong>: Teacher’s probability distributions reveal uncertainty and class relationships</p></li>
<li><p><strong>Temperature scaling</strong>: Higher temperatures (T=3-5) soften sharp predictions, providing richer training signal</p></li>
<li><p><strong>Architectural freedom</strong>: Student can have completely different architecture, not just pruned weights</p></li>
<li><p><strong>Accuracy preservation</strong>: Students often match 95-99% of teacher performance with 5-10× fewer parameters</p></li>
</ul>
<p><strong>Production example</strong>: DistilBERT uses distillation to compress BERT from 110M to 66M parameters (40% reduction) while retaining 97% accuracy on GLUE benchmarks.</p>
</section>
<section id="low-rank-approximation">
<h3>Low-Rank Approximation<a class="headerlink" href="#low-rank-approximation" title="Link to this heading">#</a></h3>
<p>Compress weight matrices through SVD factorization:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">low_rank_approximate</span><span class="p">(</span><span class="n">weight_matrix</span><span class="p">,</span> <span class="n">rank_ratio</span><span class="o">=</span><span class="mf">0.5</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Factorize matrix using truncated SVD.&quot;&quot;&quot;</span>
<span class="n">m</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="n">weight_matrix</span><span class="o">.</span><span class="n">shape</span>
<span class="c1"># Perform singular value decomposition</span>
<span class="n">U</span><span class="p">,</span> <span class="n">S</span><span class="p">,</span> <span class="n">V</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">svd</span><span class="p">(</span><span class="n">weight_matrix</span><span class="p">,</span> <span class="n">full_matrices</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="c1"># Truncate to target rank</span>
<span class="n">max_rank</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="n">target_rank</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">rank_ratio</span> <span class="o">*</span> <span class="n">max_rank</span><span class="p">))</span>
<span class="n">U_truncated</span> <span class="o">=</span> <span class="n">U</span><span class="p">[:,</span> <span class="p">:</span><span class="n">target_rank</span><span class="p">]</span>
<span class="n">S_truncated</span> <span class="o">=</span> <span class="n">S</span><span class="p">[:</span><span class="n">target_rank</span><span class="p">]</span>
<span class="n">V_truncated</span> <span class="o">=</span> <span class="n">V</span><span class="p">[:</span><span class="n">target_rank</span><span class="p">,</span> <span class="p">:]</span>
<span class="c1"># Reconstruct: W ≈ U @ diag(S) @ V</span>
<span class="k">return</span> <span class="n">U_truncated</span><span class="p">,</span> <span class="n">S_truncated</span><span class="p">,</span> <span class="n">V_truncated</span>
</pre></div>
</div>
<p><strong>Compression math</strong>:</p>
<ul class="simple">
<li><p>Original matrix: m × n parameters</p></li>
<li><p>Factorized: (m × k) + k + (k × n) = k(m + n + 1) parameters</p></li>
<li><p>Compression achieved when: k &lt; mn/(m+n+1)</p></li>
<li><p>Example: (1000×1000) = 1M params → (1000×100 + 100×1000) = 200K params (80% reduction)</p></li>
</ul>
<p><strong>When low-rank works</strong>: Large matrices with redundancy (common in fully-connected layers). <strong>When it fails</strong>: Small matrices or convolutions with less redundancy.</p>
</section>
<section id="complete-compression-pipeline">
<h3>Complete Compression Pipeline<a class="headerlink" href="#complete-compression-pipeline" title="Link to this heading">#</a></h3>
<p>Combine multiple techniques for maximum compression:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">compress_model</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">compression_config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Apply comprehensive compression strategy.&quot;&quot;&quot;</span>
<span class="n">stats</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">&#39;original_params&#39;</span><span class="p">:</span> <span class="nb">sum</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">size</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">model</span><span class="o">.</span><span class="n">parameters</span><span class="p">()),</span>
<span class="s1">&#39;original_sparsity&#39;</span><span class="p">:</span> <span class="n">measure_sparsity</span><span class="p">(</span><span class="n">model</span><span class="p">),</span>
<span class="s1">&#39;applied_techniques&#39;</span><span class="p">:</span> <span class="p">[]</span>
<span class="p">}</span>
<span class="c1"># Apply magnitude pruning</span>
<span class="k">if</span> <span class="s1">&#39;magnitude_prune&#39;</span> <span class="ow">in</span> <span class="n">compression_config</span><span class="p">:</span>
<span class="n">sparsity</span> <span class="o">=</span> <span class="n">compression_config</span><span class="p">[</span><span class="s1">&#39;magnitude_prune&#39;</span><span class="p">]</span>
<span class="n">magnitude_prune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">sparsity</span><span class="o">=</span><span class="n">sparsity</span><span class="p">)</span>
<span class="n">stats</span><span class="p">[</span><span class="s1">&#39;applied_techniques&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;magnitude_</span><span class="si">{</span><span class="n">sparsity</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="c1"># Apply structured pruning</span>
<span class="k">if</span> <span class="s1">&#39;structured_prune&#39;</span> <span class="ow">in</span> <span class="n">compression_config</span><span class="p">:</span>
<span class="n">ratio</span> <span class="o">=</span> <span class="n">compression_config</span><span class="p">[</span><span class="s1">&#39;structured_prune&#39;</span><span class="p">]</span>
<span class="n">structured_prune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">prune_ratio</span><span class="o">=</span><span class="n">ratio</span><span class="p">)</span>
<span class="n">stats</span><span class="p">[</span><span class="s1">&#39;applied_techniques&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;structured_</span><span class="si">{</span><span class="n">ratio</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="n">stats</span><span class="p">[</span><span class="s1">&#39;final_sparsity&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">measure_sparsity</span><span class="p">(</span><span class="n">model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">stats</span>
<span class="c1"># Example usage</span>
<span class="n">config</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">&#39;magnitude_prune&#39;</span><span class="p">:</span> <span class="mf">0.8</span><span class="p">,</span> <span class="c1"># 80% sparsity</span>
<span class="s1">&#39;structured_prune&#39;</span><span class="p">:</span> <span class="mf">0.3</span> <span class="c1"># Remove 30% of channels</span>
<span class="p">}</span>
<span class="n">stats</span> <span class="o">=</span> <span class="n">compress_model</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">config</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Achieved </span><span class="si">{</span><span class="n">stats</span><span class="p">[</span><span class="s1">&#39;final_sparsity&#39;</span><span class="p">]</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2">% sparsity&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p><strong>Multi-stage strategy</strong>: Different techniques target different redundancy types. Magnitude pruning removes unimportant individual weights; structured pruning removes redundant channels; distillation creates fundamentally smaller architectures.</p>
</section>
</section>
<section id="getting-started">
<h2>Getting Started<a class="headerlink" href="#getting-started" title="Link to this heading">#</a></h2>
<section id="prerequisites">
<h3>Prerequisites<a class="headerlink" href="#prerequisites" title="Link to this heading">#</a></h3>
<p>Ensure you understand compression foundations:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Activate TinyTorch environment</span>
<span class="nb">source</span><span class="w"> </span>scripts/activate-tinytorch
<span class="c1"># Verify prerequisite modules</span>
tito<span class="w"> </span><span class="nb">test</span><span class="w"> </span>quantization
</pre></div>
</div>
<p><strong>Required knowledge</strong>:</p>
<ul class="simple">
<li><p>Neural network training and fine-tuning (pruned models need retraining)</p></li>
<li><p>Gradient-based optimization (fine-tuning after compression)</p></li>
<li><p>Quantization techniques (often combined with pruning for multiplicative gains)</p></li>
</ul>
<p><strong>From previous modules</strong>:</p>
<ul class="simple">
<li><p><strong>Tensor operations</strong>: Weight manipulation and masking</p></li>
<li><p><strong>Optimizers</strong>: Fine-tuning compressed models</p></li>
<li><p><strong>Quantization</strong>: Combining compression techniques (10× pruning + 4× quantization = 40× total)</p></li>
</ul>
</section>
<section id="development-workflow">
<h3>Development Workflow<a class="headerlink" href="#development-workflow" title="Link to this heading">#</a></h3>
<ol class="arabic simple">
<li><p><strong>Open the development file</strong>: <code class="docutils literal notranslate"><span class="pre">modules/16_compression/compression_dev.ipynb</span></code></p></li>
<li><p><strong>Implement sparsity measurement</strong>: Calculate percentage of zero weights across model</p></li>
<li><p><strong>Build magnitude pruning</strong>: Remove smallest weights using percentile thresholds</p></li>
<li><p><strong>Create structured pruning</strong>: Remove entire channels based on L2 norm importance</p></li>
<li><p><strong>Implement knowledge distillation</strong>: Build teacher-student training with temperature scaling</p></li>
<li><p><strong>Add low-rank approximation</strong>: Factor large matrices using truncated SVD</p></li>
<li><p><strong>Build compression pipeline</strong>: Combine techniques sequentially</p></li>
<li><p><strong>Export and verify</strong>: <code class="docutils literal notranslate"><span class="pre">tito</span> <span class="pre">module</span> <span class="pre">complete</span> <span class="pre">16</span> <span class="pre">&amp;&amp;</span> <span class="pre">tito</span> <span class="pre">test</span> <span class="pre">compression</span></code></p></li>
</ol>
</section>
</section>
<section id="testing">
<h2>Testing<a class="headerlink" href="#testing" title="Link to this heading">#</a></h2>
<section id="comprehensive-test-suite">
<h3>Comprehensive Test Suite<a class="headerlink" href="#comprehensive-test-suite" title="Link to this heading">#</a></h3>
<p>Run the full test suite to verify compression functionality:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># TinyTorch CLI (recommended)</span>
tito<span class="w"> </span><span class="nb">test</span><span class="w"> </span>compression
<span class="c1"># Direct pytest execution</span>
python<span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span>tests/<span class="w"> </span>-k<span class="w"> </span>compression<span class="w"> </span>-v
</pre></div>
</div>
</section>
<section id="test-coverage-areas">
<h3>Test Coverage Areas<a class="headerlink" href="#test-coverage-areas" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p>✅ <strong>Sparsity measurement</strong>: Correctly counts zero vs total parameters</p></li>
<li><p>✅ <strong>Magnitude pruning</strong>: Achieves target sparsity with appropriate threshold selection</p></li>
<li><p>✅ <strong>Structured pruning</strong>: Removes entire channels, creates block sparsity patterns</p></li>
<li><p>✅ <strong>Knowledge distillation</strong>: Combines soft and hard losses with temperature scaling</p></li>
<li><p>✅ <strong>Low-rank approximation</strong>: Reduces parameters through SVD factorization</p></li>
<li><p>✅ <strong>Compression pipeline</strong>: Sequential application preserves functionality</p></li>
</ul>
</section>
<section id="inline-testing-validation">
<h3>Inline Testing &amp; Validation<a class="headerlink" href="#inline-testing-validation" title="Link to this heading">#</a></h3>
<p>The module includes comprehensive validation:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="err">🔬</span> <span class="n">Unit</span> <span class="n">Test</span><span class="p">:</span> <span class="n">Measure</span> <span class="n">Sparsity</span><span class="o">...</span>
<span class="err">✅</span> <span class="n">measure_sparsity</span> <span class="n">works</span> <span class="n">correctly</span><span class="err">!</span>
<span class="err">🔬</span> <span class="n">Unit</span> <span class="n">Test</span><span class="p">:</span> <span class="n">Magnitude</span> <span class="n">Prune</span><span class="o">...</span>
<span class="err">✅</span> <span class="n">magnitude_prune</span> <span class="n">works</span> <span class="n">correctly</span><span class="err">!</span>
<span class="err">🔬</span> <span class="n">Unit</span> <span class="n">Test</span><span class="p">:</span> <span class="n">Structured</span> <span class="n">Prune</span><span class="o">...</span>
<span class="err">✅</span> <span class="n">structured_prune</span> <span class="n">works</span> <span class="n">correctly</span><span class="err">!</span>
<span class="err">🔬</span> <span class="n">Integration</span> <span class="n">Test</span><span class="p">:</span> <span class="n">Complete</span> <span class="n">compression</span> <span class="n">pipeline</span><span class="o">...</span>
<span class="err">✅</span> <span class="n">Achieved</span> <span class="mf">82.5</span><span class="o">%</span> <span class="n">sparsity</span> <span class="k">with</span> <span class="mi">2</span> <span class="n">techniques</span>
<span class="err">📊</span> <span class="n">Progress</span><span class="p">:</span> <span class="n">Compression</span> <span class="n">module</span> <span class="err">✓</span>
</pre></div>
</div>
</section>
<section id="manual-testing-examples">
<h3>Manual Testing Examples<a class="headerlink" href="#manual-testing-examples" title="Link to this heading">#</a></h3>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">compression_dev</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">magnitude_prune</span><span class="p">,</span> <span class="n">structured_prune</span><span class="p">,</span>
<span class="n">measure_sparsity</span><span class="p">,</span> <span class="n">KnowledgeDistillation</span>
<span class="p">)</span>
<span class="c1"># Test magnitude pruning</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">(</span><span class="n">Linear</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span> <span class="mi">50</span><span class="p">),</span> <span class="n">Linear</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Initial sparsity: </span><span class="si">{</span><span class="n">measure_sparsity</span><span class="p">(</span><span class="n">model</span><span class="p">)</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span>
<span class="n">magnitude_prune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">sparsity</span><span class="o">=</span><span class="mf">0.9</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;After pruning: </span><span class="si">{</span><span class="n">measure_sparsity</span><span class="p">(</span><span class="n">model</span><span class="p">)</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span>
<span class="c1"># Test structured pruning</span>
<span class="n">structured_prune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">prune_ratio</span><span class="o">=</span><span class="mf">0.3</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;After structured: </span><span class="si">{</span><span class="n">measure_sparsity</span><span class="p">(</span><span class="n">model</span><span class="p">)</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2">%&quot;</span><span class="p">)</span>
<span class="c1"># Test knowledge distillation</span>
<span class="n">teacher</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">(</span><span class="n">Linear</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span> <span class="mi">200</span><span class="p">),</span> <span class="n">Linear</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="mi">50</span><span class="p">))</span>
<span class="n">student</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">(</span><span class="n">Linear</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span> <span class="mi">50</span><span class="p">))</span> <span class="c1"># 3× smaller</span>
<span class="n">kd</span> <span class="o">=</span> <span class="n">KnowledgeDistillation</span><span class="p">(</span><span class="n">teacher</span><span class="p">,</span> <span class="n">student</span><span class="p">)</span>
</pre></div>
</div>
</section>
</section>
<section id="systems-thinking-questions">
<h2>Systems Thinking Questions<a class="headerlink" href="#systems-thinking-questions" title="Link to this heading">#</a></h2>
<section id="real-world-applications">
<h3>Real-World Applications<a class="headerlink" href="#real-world-applications" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>Mobile deployment</strong>: DistilBERT achieves 40% size reduction with 97% accuracy retention, enabling BERT on mobile devices</p></li>
<li><p><strong>Edge inference</strong>: MobileNetV2/V3 combine structured pruning with depthwise convolutions for &lt;10MB models running real-time on phones</p></li>
<li><p><strong>Production acceleration</strong>: NVIDIA TensorRT applies automatic pruning + quantization for 3-10× speedup on inference workloads</p></li>
<li><p><strong>Model democratization</strong>: GPT distillation (DistilGPT-2) creates 40% smaller models approaching full performance on consumer hardware</p></li>
</ul>
</section>
<section id="compression-theory-foundations">
<h3>Compression Theory Foundations<a class="headerlink" href="#compression-theory-foundations" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>Lottery Ticket Hypothesis</strong>: Pruned networks can retrain to full accuracy from initial weights, suggesting networks contain sparse “winning ticket” subnetworks</p></li>
<li><p><strong>Overparameterization insights</strong>: Modern networks have excess capacity for easier optimization, not representation—most parameters help training, not inference</p></li>
<li><p><strong>Information bottleneck</strong>: Compression forces models to distill essential knowledge, sometimes improving generalization by removing noise</p></li>
<li><p><strong>Hardware-algorithm co-design</strong>: Effective compression requires algorithms designed for hardware constraints (memory bandwidth, cache locality, SIMD width)</p></li>
</ul>
</section>
<section id="performance-characteristics-and-trade-offs">
<h3>Performance Characteristics and Trade-offs<a class="headerlink" href="#performance-characteristics-and-trade-offs" title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>Unstructured sparsity limitations</strong>: 90% sparse models rarely accelerate without specialized hardware—dense GEMM operations are too optimized</p></li>
<li><p><strong>Structured sparsity benefits</strong>: Removing entire channels enables speedup when architecturally implemented (smaller dense matrices, not just zeros)</p></li>
<li><p><strong>Compression-accuracy curves</strong>: Accuracy degrades gradually until critical sparsity threshold, then collapses—find the “knee” of the curve</p></li>
<li><p><strong>Iterative pruning advantage</strong>: Gradual compression with fine-tuning (10 steps × 10% sparsity increase) achieves higher compression with better accuracy than one-shot pruning</p></li>
<li><p><strong>Multiplicative compression</strong>: Combining techniques multiplies gains—90% pruning (10× reduction) + INT8 quantization (4× reduction) = 40× total compression</p></li>
</ul>
</section>
</section>
<section id="ready-to-build">
<h2>Ready to Build?<a class="headerlink" href="#ready-to-build" title="Link to this heading">#</a></h2>
<p>You’re about to implement compression techniques that transform research models into deployable systems. These optimizations bridge the gap between what’s possible in the lab and what’s practical in production on resource-constrained devices.</p>
<p>Understanding compression from first principles—implementing pruning algorithms yourself rather than using torch.nn.utils.prune—gives you deep insight into the trade-offs between model size, inference speed, and accuracy. You’ll discover why most sparsity doesn’t accelerate inference, when structured pruning actually helps, and how to design compression strategies for different deployment scenarios (mobile apps need aggressive compression; cloud services need balanced approaches).</p>
<p>This module emphasizes honest engineering: you’ll see that achieving 90% sparsity is straightforward but getting speedup from that sparsity requires specialized hardware or libraries beyond these NumPy implementations. Production compression combines multiple techniques sequentially, carefully measuring accuracy after each stage and stopping when degradation exceeds acceptable thresholds.</p>
<p>Take your time with this module. Compression is where theory meets deployment constraints, where algorithmic elegance confronts hardware reality. The techniques you implement here enable real-world ML deployment at scale!</p>
<p>Choose your preferred way to engage with this module:</p>
<div class="sd-container-fluid sd-sphinx-override sd-mb-4 docutils">
<div class="sd-row sd-row-cols-1 sd-row-cols-xs-1 sd-row-cols-sm-2 sd-row-cols-md-3 sd-row-cols-lg-3 docutils">
<div class="sd-col sd-d-flex-row docutils">
<div class="sd-card sd-sphinx-override sd-w-100 sd-shadow-sm sd-card-hover docutils">
<div class="sd-card-body docutils">
<div class="sd-card-title sd-font-weight-bold docutils">
🚀 Launch Binder</div>
<p class="sd-card-text">Run this module interactively in your browser. No installation required!</p>
</div>
<a class="sd-stretched-link sd-hide-link-text reference external" href="https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/16_compression/compression_dev.ipynb"><span>https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/16_compression/compression_dev.ipynb</span></a></div>
</div>
<div class="sd-col sd-d-flex-row docutils">
<div class="sd-card sd-sphinx-override sd-w-100 sd-shadow-sm sd-card-hover docutils">
<div class="sd-card-body docutils">
<div class="sd-card-title sd-font-weight-bold docutils">
⚡ Open in Colab</div>
<p class="sd-card-text">Use Google Colab for GPU access and cloud compute power.</p>
</div>
<a class="sd-stretched-link sd-hide-link-text reference external" href="https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/16_compression/compression_dev.ipynb"><span>https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/16_compression/compression_dev.ipynb</span></a></div>
</div>
<div class="sd-col sd-d-flex-row docutils">
<div class="sd-card sd-sphinx-override sd-w-100 sd-shadow-sm sd-card-hover docutils">
<div class="sd-card-body docutils">
<div class="sd-card-title sd-font-weight-bold docutils">
📖 View Source</div>
<p class="sd-card-text">Browse the Python source code and understand the implementation.</p>
</div>
<a class="sd-stretched-link sd-hide-link-text reference external" href="https://github.com/mlsysbook/TinyTorch/blob/main/modules/16_compression/compression_dev.py"><span>https://github.com/mlsysbook/TinyTorch/blob/main/modules/16_compression/compression_dev.py</span></a></div>
</div>
</div>
</div>
<div class="tip admonition">
<p class="admonition-title">💾 Save Your Progress</p>
<p><strong>Binder sessions are temporary!</strong> Download your completed notebook when done, or switch to local development for persistent work.</p>
</div>
<hr class="docutils" />
<div class="prev-next-area">
<a class="left-prev" href="../modules/15_quantization/ABOUT.html" title="previous page">← Previous Module</a>
<a class="right-next" href="../modules/17_memoization/ABOUT.html" title="next page">Next Module →</a>
</div>
</section>
</section>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
name: "python3",
path: "./modules"
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="left-prev"
href="15_quantization_ABOUT.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">15. Quantization - Reduced Precision for Efficiency</p>
</div>
</a>
<a class="right-next"
href="17_memoization_ABOUT.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">17. Memoization - Computational Reuse for Inference</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> Contents
</div>
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#overview">Overview</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-objectives">Learning Objectives</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#build-use-reflect">Build → Use → Reflect</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#implementation-guide">Implementation Guide</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#sparsity-measurement">Sparsity Measurement</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#magnitude-based-pruning-unstructured">Magnitude-Based Pruning (Unstructured)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#structured-pruning-hardware-friendly">Structured Pruning (Hardware-Friendly)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#knowledge-distillation">Knowledge Distillation</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#low-rank-approximation">Low-Rank Approximation</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#complete-compression-pipeline">Complete Compression Pipeline</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#getting-started">Getting Started</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#prerequisites">Prerequisites</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#development-workflow">Development Workflow</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing">Testing</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#comprehensive-test-suite">Comprehensive Test Suite</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#test-coverage-areas">Test Coverage Areas</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#inline-testing-validation">Inline Testing &amp; Validation</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#manual-testing-examples">Manual Testing Examples</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#systems-thinking-questions">Systems Thinking Questions</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#real-world-applications">Real-World Applications</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#compression-theory-foundations">Compression Theory Foundations</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#performance-characteristics-and-trade-offs">Performance Characteristics and Trade-offs</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ready-to-build">Ready to Build?</a></li>
</ul>
</nav></div>
</div></div>
</div>
<footer class="bd-footer-content">
<div class="bd-footer-content__inner container">
<div class="footer-item">
<p class="component-author">
By Prof. Vijay Janapa Reddi (Harvard University)
</p>
</div>
<div class="footer-item">
<p class="copyright">
© Copyright 2025.
<br/>
</p>
</div>
<div class="footer-item">
</div>
<div class="footer-item">
</div>
</div>
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="../_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="../_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
<footer class="bd-footer">
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink