mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-11-09 10:39:41 -06:00
Vijay Janapa Reddi
58ed2e8b2c
Plain words and formal posture across the landing site, labs, slides, TinyTorch guide, MLS-im docs and Hardware Kits: removed hype and intensifiers, contractions and em-dashes in expository prose, announcers and wordy verbs. Wording only; no code, links, numbers or claims changed.
108 lines
5.1 KiB
Plaintext
108 lines
5.1 KiB
Plaintext
---
|
||
title: "Module 16: Compression"
|
||
module: 16_compression
|
||
notebook: compression
|
||
---
|
||
|
||
Compression is the negotiation between model footprint and model quality. Every technique here (pruning, distillation, low-rank approximation) buys memory and bandwidth savings at some cost to accuracy, and hardware only cashes the savings when the sparsity pattern matches what the silicon can skip. The ratio is the thing; this module makes it measurable.
|
||
|
||
:::{.callout-note title="Module Info"}
|
||
|
||
**OPTIMIZATION TIER** | Difficulty: ●●●○ | Time: 5-7 hours | Prerequisites: 01-14
|
||
|
||
**Prerequisites: Modules 01-14** means you should have:
|
||
|
||
- Built tensors, layers, and the complete training pipeline (Modules 01-08)
|
||
- Implemented profiling tools to measure model characteristics (Module 14)
|
||
- Comfort with weight distributions, parameter counting, and memory analysis
|
||
|
||
If you can profile a model's parameters and understand weight distributions, you are ready.
|
||
:::
|
||
|
||
{{< module-header >}}
|
||
|
||
## Overview
|
||
|
||
A modern language model takes 100GB of storage. A phone gives you less than 1GB. A microcontroller gives you under 100KB. The model that wins the benchmark and the model that ships are not the same artifact, and compression closes the gap between them.
|
||
|
||
Compression techniques test whether a model can represent the task with fewer active weights or a smaller architecture. You implement magnitude pruning and work with the supplied structured pruning, distillation, and low-rank approximation. TinyTorch pruning zeros weights without shrinking dense arrays; SVD savings require retaining and executing the factors. Measure accuracy after each change.
|
||
|
||
## Commands
|
||
|
||
{{< module-workflow >}}
|
||
|
||
## Learning objectives
|
||
|
||
:::{.callout-tip title="By completing this module, you will:"}
|
||
|
||
- **Implement** magnitude-based pruning to zero 80-90% of small weights, then measure what it cost in accuracy
|
||
- **Master** structured pruning that creates hardware-friendly sparsity patterns by zeroing entire channels
|
||
- **Run** the supplied knowledge distillation code and measure the student’s size and accuracy
|
||
- **Understand** compression trade-offs between sparsity ratio, inference speed, memory footprint, and accuracy preservation
|
||
- **Analyze** when to apply different compression techniques based on deployment constraints and performance requirements
|
||
:::
|
||
|
||
## What you'll build
|
||
|
||
::: {#fig-16_compression-diag-1 fig-env="figure" fig-pos="htb" fig-cap="**Three compression mechanisms.** Magnitude and structured pruning zero selected weights or channels without changing dense shapes. Low-rank approximation returns truncated SVD factors; savings require retaining and executing those factors." fig-alt="Magnitude and structured pruning zero selected weights or channels without changing dense shapes. Low-rank approximation returns truncated SVD factors; savings require retaining and executing those factors."}
|
||
|
||
|
||
|
||
:::
|
||
|
||
**The pattern you will enable:**
|
||
```python
|
||
# Compress a model by removing 80% of smallest weights
|
||
magnitude_prune(model, sparsity=0.8)
|
||
sparsity = measure_sparsity(model) # Returns ~80%
|
||
```
|
||
|
||
### What you're not building yet
|
||
|
||
To keep the module focused, you will **not** implement:
|
||
|
||
- Sparse storage formats like CSR (PyTorch provides `torch.sparse` layouts)
|
||
- Iterative prune-and-fine-tune schedules
|
||
- Dynamic pruning during training via hooks and callbacks
|
||
- Joint quantization + pruning pipelines
|
||
|
||
You are building the **algorithms** that decide which weights to remove and how. Sparse execution or structural compaction requires additional implementation. Module 17 compares dense vectorization and tiling; it does not provide sparse kernels.
|
||
|
||
## What you write
|
||
|
||
{{< what-you-write >}}
|
||
|
||
## How you know it works
|
||
|
||
{{< module-checks >}}
|
||
|
||
## When it fails
|
||
|
||
{{< stub-failure >}}
|
||
|
||
`Large weights should survive`
|
||
: From `test_unit_magnitude_prune`. The largest weights were pruned. Sort magnitudes in ascending order and zero the first `prune_count`.
|
||
|
||
`Expected ~50% sparsity, got 33.33333333333333%`
|
||
: From `test_unit_magnitude_prune`. Biases were counted as prunable. Prune only weight matrices, the parameters with `ndim > 1`.
|
||
|
||
## Finished? Read why
|
||
|
||
{{< book-chapter >}}
|
||
|
||
## What's next
|
||
|
||
You have measured sparsity and explored smaller representations. Module 17 examines dense execution costs through vectorization, tiling, and grouped expressions. Your pruned weights retain their dense shape and still participate in multiplication.
|
||
|
||
**Next:** [Module 17: Acceleration](17_acceleration.qmd)
|
||
|
||
### How later modules use this one
|
||
|
||
| Module | What It Does | Your Compression In Action |
|
||
|--------|--------------|---------------------------|
|
||
| **17: Acceleration** | Optimize computation kernels | Compare dense implementations; masked zeros still execute |
|
||
| **18: Memoization** | Cache repeated computations | Measure model allocations separately from cache bytes |
|
||
| **19: Benchmarking** | Measure end-to-end performance | Confirm pruned models keep dense throughput until sparse kernels exist |
|
||
|
||
: **How compression feeds into acceleration, memoization, and benchmarking.** {#tbl-16-compression-downstream-usage}
|