_calculate_improvements() guards latency/memory/energy division against
opt_metrics[metric] <= 0 with a documented "fallback to 1.0", but
accuracy_retention = opt_metrics['accuracy'] / base_metrics['accuracy']
had no such guard on the denominator. A baseline model with 0.0 accuracy
(a broken/failing baseline -- exactly the kind of input a benchmarking
suite should handle gracefully) raises ZeroDivisionError and crashes the
whole comparison, while the sibling metrics computed two lines above
degrade gracefully.
Added the same guard pattern already used for the other three metrics.
The function's own docstring already promised "Handle division by zero
with fallback to 1.0" as a general rule -- this fix makes the accuracy
branch actually honor it.
Verified: the old code raises ZeroDivisionError for
base_metrics={'accuracy': 0.0}; the fixed code returns
accuracy_retention=1.0 instead, and the normal (non-zero) case is
unaffected (0.6/0.5 = 1.2, unchanged).
benchmark_model()'s throughput_samples_per_sec (float(1000 / avg_latency))
and generate_submission()'s speedup (float(baseline_latency /
optimized_latency)) both divide by a latency measured with time.time()
around a single tiny forward pass, with no zero-guard. time.time() has
coarse resolution on Windows (commonly ~15.6ms); for a small model, a
fast forward pass can legitimately measure 0.0 elapsed time across all
runs, making avg_latency (and therefore optimized_latency downstream)
exactly 0.0 and raising ZeroDivisionError.
An existing test (line ~1347) asserts throughput_samples_per_sec > 0, and
another (line ~1564) asserts speedup > 0, so the fix floors each
denominator at a small epsilon (1e-6) rather than returning 0.0 or None --
both of which would fail those assertions and would also misrepresent an
immeasurably-fast result as zero throughput.
Verified: reproduced ZeroDivisionError for avg_latency=0.0 and
optimized_latency=0.0 with the old expressions; confirmed the floored
versions return large-but-finite positive numbers instead, and that
normal (non-zero) latency values are unaffected.
_cached_generation_step() writes the new token's K/V into the cache via
update(), then calls cache_obj.get() and uses its result directly as
K_all/V_all for the attention computation. But KVCache.get() is designed
(and tested in test_unit_kvcache) to return only tokens already made
visible by advance() -- exactly NOT the token just written by update(),
since advance() is meant to run once per token after all layers finish.
So the new token's own K/V was never included in its own attention
computation, and on the very first generation step (cache empty, nothing
advanced yet), cache.get() returns a zero-length sequence, crashing the
matmul with "zero-size array to reduction operation maximum which has no
identity" -- cached generation was broken from the first token onward.
Fixed by concatenating the current step's own K_heads/V_heads (already
computed, no extra work needed) onto cache.get()'s history, instead of
using the cache's history alone.
Verified with a stub attention module (identity projections) against the
real KVCache: reproduced the crash with the old logic on the first
token (zero-length K/V from cache.get()), and confirmed the fixed
function produces the correct self-attention output ([1,0,0,0], the
token attending fully to itself) instead of crashing.
Module 06's test_unit_broadcast_gradients ran Tensor(data, requires_grad=True)
before enable_autograd() upgrades the constructor to accept it, so the exported
notebook cell raised TypeError. Match the sibling test_unit_function_classes
convention and set requires_grad as an attribute instead; the constructor stays
minimal by design. Fixes#1893.
The PR converted the SGD markdown cell to a raw string (r""") but kept \\
in the redrawn ASCII art. In a raw string \\ is two literal backslashes, so
the loss-surface, oscillation, and momentum diagrams rendered with doubled
backslashes in the generated notebook (correct-looking in the .py source).
Normalize the art to single backslashes, matching the foundations cell.
AvgPool2d returned a tensor with no _grad_fn, so average pooling had no
gradient path (MaxPool2d and Conv2d both wire theirs). Add AvgPool2dBackward,
which distributes each output gradient equally (1/kernel_area) across its
window and accumulates overlaps, padding the gradient buffer with zeros to
match the forward pass. Wire it into AvgPool2d.forward.
From #1885, taking the AvgPool backward feature only: singular name to match
the *Backward convention, zero (not -inf) padding, and verified by numeric
gradient check. The PR diagram edits are not included.
The naive 1/(1+exp(-x)) overflows exp for large negative x and the unit
test hid the resulting RuntimeWarning by suppressing the whole category.
Use the piecewise-stable form (each branch keeps its exponent <= 0) under
np.errstate, since np.where evaluates both branches and the discarded one
can still overflow harmlessly. Drop the test-wide warning suppressor.
Based on #1870; uses errstate so the piecewise form is actually silent.
quantize_int8 set scale=1.0 for a constant tensor and clamped zero_point
to the INT8 range, so for any constant |c| > 127 the zero_point saturated
and dequantization silently returned a wrong value (c=500 recovered as 128).
For |c| > 127 use zero_point = +-1 and scale = |c| so
(0 - zero_point) * scale = c holds exactly. Adds a test exercising that branch.
Reapplies the fix from #1874 on dev with a corrected scale (the PR used
scale = -c/(-zero_point), which flips the sign and recovers -c).
The 3-D mask reshape used seq_len for both the query and key dimensions,
raising ValueError for cross-attention (seq_q != seq_k) and only working
when seq_q == seq_k. Read both dimensions and reshape to
(batch, 1, seq_q, seq_k) so the mask broadcasts across heads correctly.
Extracted from #1875 (branch was based on main; reapplied on dev).
The module declares rng = np.random.default_rng(7) at the top so that all
stochastic operations are reproducible for a given seed. Dropout ignored this
and called np.random.random(shape), which draws from the global unseeded state.
Masks were different on every run even when the module seed was set, making
Dropout unit tests non-reproducible and causing hard-to-debug numerical
differences between training runs.
Changed to rng.random(shape) so Dropout participates in the same reproducible
stream as every other random operation in the module.
All three classes are declared with no base class, so super().__init__()
resolves to object.__init__(), a silent no-op. The calls made these look like
they inherited from Layer (as Linear does in 03_layers) when they do not.
Removing them makes the code honest: these are standalone modules that
implement parameters() directly. The right long-term fix is to export Layer
from tinytorch.core and wire these classes into that hierarchy, but that
requires a separate refactor across multiple modules.
The old test called transpose(0, 2) on a (2, 2, 2) tensor and only checked
that the output shape was (2, 2, 2). Because all three dims are equal, any axis
swap produces the same shape, so the test would have passed even if transpose
was completely broken and returned the input unchanged.
Switched to a (2, 3, 4) tensor where all three dimension sizes are distinct.
Now transpose(0, 2) must produce (4, 3, 2), a shape only achievable by swapping
exactly axis 0 and axis 2. Also added a data-value check verifying that element
[i,j,k] in the original appears at [k,j,i] in the transposed result, catching
implementations that swap the wrong pair of axes.
When the shape contains -1, the inferred size was computed with integer
division (self.size // known_size). If the total size is not evenly divisible
by the known dimensions, floor division silently produces a truncated value
and the element-count check catches it only after the wrong number has already
been placed in new_shape, producing a confusing error.
For example, Tensor([1..7]).reshape(-1, 3) should fail immediately because 7
is not divisible by 3, but the old code inferred 2 (7//3), then raised a
generic element-count mismatch that did not explain the actual problem.
Added a divisibility check before the division so the error fires at the right
place with a clear message, matching PyTorch behavior.
tracked_mul() wraps a scalar other into other_tensor = Tensor(other) and
calls _ensure_grad_attrs() on it correctly, but then passed the original raw
scalar to MulBackward instead of other_tensor. MulBackward.apply() pulls both
operands out of saved_tensors and calls .data on them, so any scalar multiply
in a tracked computation graph crashed on the backward pass with AttributeError
because int and float have no .data attribute.
Fixed by passing other_tensor to MulBackward so both operands in saved_tensors
are always Tensor instances. The forward computation still uses the original
scalar unchanged.
When bias=None, only x and weight are saved in saved_tensors (2 entries),
but apply() always returned a 3-tuple of grad_input, grad_weight, grad_bias.
The backward walk zips saved_tensors with the return tuple so the extra
grad_bias (None) was silently dropped by zip truncation. This made bias-free
Conv2d appear to train correctly while the tuple length mismatch left the
code one refactor away from corrupting gradients silently.
Fixed by returning only (grad_input, grad_weight) when bias is None so
the return tuple always has the same length as saved_tensors.
GPT._create_causal_mask was returning an additive -inf mask
(upper-triangular with -inf, 0 for allowed positions) but
_apply_mask in module 12 expects a binary mask (1=allow, 0=block)
and computes adder = (1 - mask) * MASK_VALUE.
With the old mask, every allowed position (value 0) got masked by
(1-0)*(-1e9) = -1e9, blocking ALL attention -- both past and future
tokens. Only an infinite value in the mask produced the correct
-inf addend for blocked positions.
Fix: use np.tril(np.ones(...)) which is the binary lower-triangular
convention that _apply_mask expects.
When B is a 1D vector, numpy's .T is a no-op (returns the same 1D
array), so the previous code called np.matmul(grad_output, b.data)
which either failed on mismatched shapes or silently computed a dot
product instead of the outer product needed for grad_A.
For A(m,k) @ b(k,) -> out(m,):
grad_A = np.outer(grad_output, b) (m,k) -- was broken
grad_b = a.T @ grad_output (k,) -- was broken for 1D a
Fixes the symmetric case (1D a) with grad_B = np.outer(a, grad_output).
The PR's inline ← arrow form crams annotations into the line ends and
clutters the diagram box. The original vertical layout keeps the box
clean and places the explanations below where they're visually separate.
Pandoc subscript requires no spaces between paired tildes. The two tildes
on this line are separated by sentence text with spaces, so the bare ~
renders as literal. The backslash escape was defensive but unnecessary.
#1617 unseeded the module-level rng in tinytorch/src/03_layers/03_layers.py
to make the perceptron milestone produce different weights on every run.
But Linear.__init__ reads weight init from that rng, so the change made
Linear init nondeterministic across the entire codebase. The integration
test tests/integration/test_training_flow.py::test_deep_network_gradient_chain
specifically depends on Linear having stable init; empirical sweep over
5000 random draws shows the test fails 27.5% of the time under unseeded
init, which is why the Windows CI run failed in #1617.
Restore the seed=7 default in 03_layers.py and instead rebind layers.rng
to an unseeded RNG locally inside the perceptron milestone, mirroring
the pattern already used by the XOR milestone (#1618). This keeps the
perceptron's "different weights every run" promise without breaking
unrelated tests.
The module-level `rng = np.random.default_rng(7)` at the top of
`src/03_layers/03_layers.py` was reused for every `Linear(...)` weight
init, so every run produced identical weights. Milestone 1's UI text
("No random seed - each run will be different!") contradicted this, and
the issue reporter rightly expected weights to vary.
Drop the seed in two places:
- `src/03_layers/03_layers.py`: the module-level `rng` is now unseeded.
Local seeded RNGs inside test/demo blocks (around lines 829, 848, 981)
are unchanged - those want determinism for reproducible self-tests.
- `milestones/01_1958_perceptron/01_rosenblatt_forward.py`: the data-gen
RNG is also unseeded so the cluster points vary too, matching the
on-screen claim.
Verified by running the milestone twice: cluster positions, predictions,
and accuracy now differ between runs (e.g., 100% "got lucky" vs 0%
"random guessing"). `Linear(...)` weights also differ across fresh
constructions within a single run.
Relates to #1611
gamma and beta were created as Tensor(np.ones/zeros(n)) with no
requires_grad flag, defaulting to False after enable_autograd() patches
Tensor.__init__. The _LayerNormBackward.apply() guards on
beta.requires_grad and gamma.requires_grad, so gradients were silently
never computed for either parameter -- LayerNorm could not learn its
scale and shift during training.
Fix: pass requires_grad=True at construction so the backward pass
computes grad_gamma and grad_beta correctly.
Also remove the manual param.requires_grad = True workaround from
test_layernorm_gradient_flow() which was masking the bug.
quantize_int8() had a special case for tensors where all elements are the
same value (max == min). It set scale=1.0 and zero_point=0 and returned an
all-zeros INT8 tensor. On dequantization: (0 - 0) * 1.0 = 0.0 for every
element, so the reconstructed tensor is all zeros regardless of what the
original constant was. A bias layer initialised to a non-zero constant, or
any weight tensor that happens to be uniform, is silently zeroed out after
a quantize/dequantize roundtrip.
Root cause: the contributor correctly handled the zero-range edge case to
avoid division by zero when computing scale, but forgot that zero_point must
encode the constant so that dequantization can recover it.
Fix: compute zero_point = round(-min_val) (clamped to [-128, 127]) so that
the invariant (0 - zero_point) * scale = min_val holds for constant tensors.
Also tightens the unit test: the old test only asserted scale_const == 1.0
and never checked that the value survived the roundtrip, which is why this
went undetected. New test explicitly dequantizes and asserts recovery of both
positive and negative constants.
Audit (parent commit): every tinytorch/src/NN_xxx/ABOUT.md is a stale
duplicate of tinytorch/quarto/modules/NN_xxx.qmd, with the QMD strictly
newer/richer (each QMD has additional sections like Your TinyTorch,
PyTorch comparison, MLPerf section that ABOUT.md is missing).
Drift confirms ABOUT.md is unmaintained: e.g., 12_attention/ABOUT.md
still says "GPT-3 training (4x inference)" while the QMD was corrected
to 5x. Single-source-of-truth in quarto/modules/ stops the recurrence.
The retired Jupyter Book site (tinytorch/site-legacy/) was the only
historical consumer; it is deleted in the next commit.
Also deleted:
- tools/dev/fix_about_titles.py -- one-shot ABOUT.md title fixer
- tools/dev/fix_mermaid_diagrams.py -- one-shot ABOUT.md mermaid tweaker
Both operated on the now-deleted ABOUT.md files and have no other use.
Updated tinytorch/README.md so the documented repo tree no longer
shows ABOUT.md under each module folder, and repointed a stale
"setup guide" link from site/getting-started.md to quarto/getting-started.qmd.
The condition guarding the accuracy branch was len(outputs.data.shape) > 1,
intended to detect multi-class classification. A single-output regression
model (shape (N,1)) also satisfies this, so it entered the argmax path where
argmax of a single column is always 0. For any regression task evaluate()
silently returned meaningless accuracy instead of 0.0, with no error raised.
The root assumption -- 2D outputs implies classification -- breaks for the
common (N,1) regression case. Fix: add outputs.data.shape[-1] > 1 so only
outputs with more than one logit enter the classification branch.
Also tightens the unit test: the existing test used Linear(2,1)+MSELoss but
only checked isinstance(accuracy, float), which passed even when the bug
fired. New test asserts regression accuracy == 0.0 and adds a separate
classification sub-case (Linear(2,3)) to verify the argmax path still works.
Two root causes for the 20 failing gradient correctness tests:
1. GELU forward/backward approximation mismatch: forward used
sigmoid(1.702*x) but backward computed derivative of the tanh
approximation. Now both use the sigmoid form consistently.
2. Finite-difference tests used float64 arithmetic but Tensor stores
float32, causing precision loss in numerical gradients. Tests now
use float32 throughout with appropriate tolerances.
Also fixed BCE and composed gradient tests that created disconnected
Tensors from .data copies, breaking the autograd graph.
view() and masked_fill() were present in the exported core/tensor.py
(landed via #1391) but missing from the source file, causing the
TestTensorPyTorchCompat tests to fail CI with AttributeError.
Syncs src/01_tensor/01_tensor.py with the exported package:
- Tensor([t1, t2]) stacking via np.stack in __init__
- view(*shape) as a reshape alias
- masked_fill(mask, value) for transformer attention patterns
- contiguous() upgraded to np.ascontiguousarray for C-order memory layout
Add PyTorch-compatible utility methods so students can follow PyTorch
tutorials without hitting AttributeError. All three are thin wrappers
over existing attributes — zero risk to downstream modules.
Tanh was the only activation not patched by enable_autograd(). calling Tanh()(x).backward() silently left x.grad as None because the forward never attached a computation graph.
changes mirror the existing tracked_sigmoid_forward pattern:
- new TanhBackward(Function) class using saved output, since d/dx tanh(x) = 1 - tanh(x)^2 can be computed from the forward result
- new tracked_tanh_forward in enable_autograd() that attaches TanhBackward when input requires_grad
- Tanh added to the activations import and patch install lines
- regression test in test_06_autograd_progressive.py asserts tanh grad values at known points
closes#1341
The requires_grad loop now checks isinstance(param, Tensor) before setting
the attribute, so test stubs or models that return raw numpy arrays from
parameters() do not crash with AttributeError.
Linear layers create Tensors with requires_grad=False by default. If the
optimizer hasn't been constructed yet (or Module 07 is incomplete), the
tracked_mse_forward gate fails silently: loss gets no _grad_fn, backward()
returns immediately, and param.grad stays None — causing optimizer.step()
to skip all updates and the test assertion "Parameters should change after
optimizer update" to fail.
Fix: explicitly iterate model.parameters() in trainer_init and set
requires_grad=True, making the Trainer the authoritative owner of gradient
tracking regardless of optimizer setup order.
Fixes#1334
Replace np.random.randn/rand/seed with np.random.default_rng(7) across
all 93 source modules, tests, and milestones for reproducible, isolated
random state.
- Fix pool_size=(2,2) → kernel_size=2, stride=2 in 02_lecun_cifar10.py
- Normalize all Conv2D/MaxPool2D/BatchNorm2D references to lowercase d
(Conv2d, MaxPool2d, BatchNorm2d) across comments, strings, and ASCII art
- Fix Conv2d error string in 09_convolutions.py source module
- Delete milestones/extras/ directory (drifted copies, 2 placeholders,
1 broken script — maintenance liability that caused the duplicate bug)
- Add test_milestones_smoke.py: 19 lightweight tests that import every
milestone script and construct every model class in 0.07s, catching
API drift without data downloads
- Revert ASCII diagrams to literal '</w>' and '<UNK>' values for
pedagogical clarity (diagrams should be concrete, not symbolic)
- Fix incomplete docstring migration where expected outputs still
used raw string literals while inputs used Tokenizer.TOK_EOW
- Standardize assertion error message interpolation style
Implement two systems engineering enhancements to Module 06:
- no_grad() context manager with global _GRAD_TRACKING_ENABLED flag,
checked by all tracked operations to skip graph construction during
inference. Supports safe nesting and matches PyTorch's API.
- retain_graph=False parameter on backward() that releases _grad_fn
references after gradient propagation, fixing memory growth across
training iterations. Matches PyTorch's default behavior.
847/847 tests passing, zero regressions.
- Add missing markdown cell before test_unit_reduce_broadcast_grad
(every other unit test in the module has one)
- Restructure _reduce_broadcast_grad docstring from Args/Returns/Algorithm
to TODO/APPROACH/EXAMPLE/HINT to match the module's scaffolding pattern
M05 DataLoader:
- Standardize channel-format heuristic across RandomHorizontalFlip,
RandomCrop, and _pad_image to all use shape[0] for channels-first
detection (was inconsistent: flip used shape[-1], crop used shape[0])
M12 Attention:
- _apply_mask now uses Tensor API (ones_like - mask) instead of raw
mask.data access, preserving autograd gradient flow
- MultiHeadAttention.forward uses mask.reshape() instead of
mask.data.reshape() for same reason
M18 Memoization:
- Fix prefill: process prompt tokens one-at-a-time instead of as a
batch, so KV cache is populated through the normal generation path.
Previously cache slots remained zeros after prefill.
M03 Layers:
- Sequential.forward/.__call__ now accept training= parameter and
pass it through to Dropout layers (eval mode was broken)
M12 Attention:
- Fix memory multiplier: was 7x (double-counted), now 4x incremental
(1x forward + 1x grad + 2x optimizer). Aligned ABOUT.md to match.
- Fix mask docstring: True/False -> 1/0 to match implementation
M14 Profiling:
- Add __enter__/__exit__ to Profiler (6 progressive tests used
context manager but class had no protocol support)
- Fix FLOP prose formula: remove batch_size to match implementation
M15 Quantization:
- Fix "symmetric" -> "asymmetric (min-max)" terminology throughout
- Fix Magic Formula prose to match code: q = x/s + zp (not (x-zp)/s)
- Add parentheses to zero-point hint formula for clarity
