cs249r_book/tinytorch/guide/modules/07_optimizers.qmd

---
title: "Module 07: Optimizers"
module: 07_optimizers
notebook: optimizers
---

A gradient is a direction, not a step. The optimizer picks a step size, a history window, and a per-parameter scaling rule. Its state, momentum buffers and running second moments, often costs more memory than the model itself. Choose well and a model converges in an afternoon. Choose poorly and VRAM runs out on epoch one.

:::{.callout-note title="Module Info"}

**FOUNDATION TIER** | Difficulty: ●●○○ | Time: 3-5 hours | Prerequisites: 01-06

**Prerequisites: Modules 01-06** means you need:

- Tensor operations and parameter storage
- DataLoader for efficient batch processing
- Understanding of forward/backward passes (autograd)
- Why gradients point toward higher loss

If you understand how `loss.backward()` computes gradients and why we need to update parameters to minimize loss, you are ready.
:::

{{< module-header >}}

## Overview

You have gradients. Now what? An optimizer is the rule that turns a gradient into a parameter update, and it can make the difference between a model that converges in an afternoon and one that diverges on the first batch. Picture optimization as hiking in fog: you can feel the slope under your feet but cannot see the valley. Each optimizer is a different strategy for choosing your next step.

You will meet three, writing the first two yourself: SGD with momentum (the foundation), Adam with adaptive per-parameter learning rates (the most widely used modern optimizer), and AdamW with decoupled weight decay (the default for transformers). They differ in memory cost, convergence speed, and how forgiving they are when you guess the learning rate wrong, which matters more than most practitioners admit.

## Commands

{{< module-workflow >}}

## Learning objectives

:::{.callout-tip title="By completing this module, you will:"}

- **Implement** SGD with momentum to reduce oscillations and accelerate convergence in narrow valleys
- **Master** Adam's adaptive learning rate mechanism with first and second moment estimation
- **Understand** memory trade-offs (plain SGD: 2x, SGD with momentum: 3x, Adam/AdamW: 4x the parameter bytes, counting weights, gradients, and optimizer state) and computational complexity per step
- **Connect** optimizer state management to checkpointing and distributed training considerations
:::

## What you'll build

::: {#fig-07_optimizers-diag-1 fig-env="figure" fig-pos="htb" fig-cap="**Optimizer state and weight decay.** SGD uses coupled weight decay and optional momentum. Adam forms bias-corrected moments from the gradient plus coupled decay. AdamW forms moments from the gradient and applies weight decay separately." fig-alt="SGD uses coupled weight decay and optional momentum. Adam forms bias-corrected moments from the gradient plus coupled decay. AdamW forms moments from the gradient and applies weight decay separately."}

![](../assets/images/diagrams/07_optimizers-diag-1.svg)

:::

**The pattern you will enable:**
```python
# Training loop with optimizer
optimizer = Adam(model.parameters(), lr=0.001)
loss.backward()  # Compute gradients (Module 06)
optimizer.step()  # Update parameters using gradients
optimizer.zero_grad()  # Clear gradients for next iteration
```

### What you're not building yet

To keep this module focused, you will **not** implement:

- Learning rate schedules (that is Module 08: Training)
- Gradient clipping (that is Module 08: Training)
- Second-order optimizers like L-BFGS (rarely used in deep learning due to memory cost)
- Distributed optimizer sharding (production frameworks use techniques like ZeRO)

**You are building the core optimization algorithms.** Advanced training techniques come in Module 08.

## What you write

{{< what-you-write >}}

## How you know it works

{{< module-checks >}}

## When it fails

{{< stub-failure >}}

`assert np.allclose(param2.data, expected_second, rtol=1e-5)`
:   From `test_unit_sgd_optimizer`. The second SGD step is wrong because the momentum buffer did not accumulate. Update it as `momentum * buffer + grad`, then step with the buffer.

`m_hat should equal grad at step 1, got [0.01 0.02]`
:   From `test_unit_adam_update_moments`. No bias correction. Divide `m` by `1 - beta1**t` and `v` by `1 - beta2**t`.

`m_hat should equal grad at step 1, got [inf inf]`
:   From `test_unit_adam_update_moments`. The correction ran before the update count was incremented, so `t = 0` and `1 - beta1**0` is zero. Increment the count first.

## Finished? Read why

{{< book-chapter >}}

## What's next

:::{.callout-note title="Up next: Module 08, Training"}

You have an optimizer that takes one step. Module 08 wraps it in the loop that takes a million: epochs, validation, checkpointing, gradient clipping, and learning rate schedules. The question it answers is the one this chapter raised but did not solve: *how do you actually drive a model to convergence without babysitting it?*
:::

**Next:** [Module 08: Training](08_training.qmd)

### How later modules use this one

| Module | What It Does | Your Optimizers In Action |
|--------|--------------|---------------------------|
| **08: Training** | Complete training loops | `for epoch in range(10): loss.backward(); optimizer.step()` |
| **09: Convolutions** | Convolutional networks | `AdamW` optimizes millions of CNN parameters efficiently |
| **13: Transformers** | Attention mechanisms | Large models require careful optimizer selection |

: **How optimizers feed into subsequent training modules.** {#tbl-07-optimizers-downstream-usage}