Plain words and formal posture across the landing site, labs, slides, TinyTorch
guide, MLS-im docs and Hardware Kits: removed hype and intensifiers, contractions
and em-dashes in expository prose, announcers and wordy verbs. Wording only; no
code, links, numbers or claims changed.
Memory figures now divide by powers of ten everywhere (1 KB = 10^3 B,
1 MB = 10^6 B, 1 GB = 10^9 B), matching the book's prose convention.
Removes MiB/GiB labels and 1024-based divisors from src modules, tito
system info, milestones, tests, book chapters, guide pages, and the
regenerated code listings. Binary units remain only where a module
explicitly teaches the SI-vs-IEC distinction.
Release-hardening and transformer-era release of TinyTorch, audited as one
tree. This commit brings tinytorch/ and the TinyTorch CI workflows to the
state of the audit/tinytorch-release branch, where the individual changes
are recorded commit by commit.
- Milestones: transformer-era parts (TinyGPT, sequence routing, TinyCopilot,
conversational Q&A), Milestone 06 "MLPerf to Generative Serving (2018)",
Milestone 07 kernels; per-part recording; NumPy-referenced losses and
causality, attention-content and position probes; try-it prompts after a
pass.
- Framework: dropout eval mode, target gradients, float32 stays float32,
layer validation, LoRA adapters.
- tito: loopback login callback with a state nonce, consented progress sync,
TLS-verified update check, Ctrl-C after a pass keeps the pass.
- Grading: graded tests compare values against NumPy references; tests that
could not fail now check something.
- CI: the publish preflight runs its tests and ships the commit it tested;
workflow inputs pass through env; third-party actions pinned.
- Guide and book: read line by line against the code; every demo is a real
recording.
See tinytorch/CHANGELOG.md, section 0.2.0.
The module page drops 'not building col2im', shows the trainable call, and
lists the two failure messages a student will hit. Chapter 17 derives the
three gradients, quotes col2im and Im2colConv2dFunction.backward, reports
the measured training-step speedup, and replaces the write-col2im exercise
with one on recomputing the patch matrix instead of saving it.
The module page lists the new objective, the pattern, the prerequisite on
Conv2d, and the two failure messages a student will actually hit. Chapter
17 quotes im2col and im2col_conv2d, contrasts the offset loop with chapter
9's per-position model, traces the 3x3 example, reports the measured
speedups, and adds a col2im exercise. Chapter 9 points forward to it.
Callout titles were written five ways (Coming Up: X — Y, Coming Up — X: Y,
hyphens, and none on 01/15/16); each page now has an 'Up next' callout
where it had one, a linked Next: line, and a real 'How later modules use
this one' heading instead of assorted bold pseudo-headings. The
boilerplate '@tbl-... traces how this module is reused' sentence is gone.
Module Info prerequisite leads use one phrasing.
- Code blocks go back to wrapping: with scroll, trailing `# comments`
were cut off at the right edge with no cue, worst on phones. The
module command blocks already put comments on their own lines.
- Home page on phones: comparison cards stack instead of widening the
page to 531px, and the timeline is one left-aligned column.
- Dark mode: the home lead paragraph was hard-coded dark gray; the
shared navbar stays light, so its title is kept dark; the sidebar logo
gets a light plate so "TORCH" stays visible.
- Tables use smaller type on phones; definition-list entries
(Troubleshooting) get space between them.
- Credits printed a Markdown link as raw text inside an HTML block;
"1998." and "2018." paragraphs on Milestones 04 and 06 rendered as
ordered lists; "** Foundation Tier**" showed literal asterisks; the
Optimization page's nested bullets were flattened and it named its
cycle three ways; Datasets field lines ran together.
- Names: Quick Start is titled Quick Start; "TinyTorch Olympics" is
"Torch Olympics"; Community's educators section points to For
Instructors; milestone pages carry their title in front matter so
their contents list is not nested under it; the Glossary's A-Z
headings appear in its contents list.
- tools/diagrams/STYLE.md, an internal style guide, is no longer
published into the site and its search index.
- Module 01 links Module 02 from What's next.
- Glossary: opens with a "Tools and course terms" section (terminal,
virtual environment, activate, tito, Jupyter Lab, notebook, cell, the
two meanings of kernel, module, export, unit and integration tests,
milestone, Binder), adds weight, bias, and parameter, gives
broadcasting a two-line example, drops the Lab Guide wording, moves
into Getting Started in the sidebar, and is linked from Quick Start.
The newcomer reviewer found none of the tool words it stumbled on.
- Module 01's lead ("saturate the ALUs or stall on cache misses") told
the first page a beginner reads that they were underqualified; it now
says what a tensor is and what they will write.
- Figures on 02, 05, 06, 07, 08 carried a one-line caption inside the
image above the real caption (two of them overclaiming: "correctly
accumulating", "guarantees mathematical equivalence"); removed, canvas
trimmed, renders checked. 05's "drop_last drops remainder" labeled an
option TinyTorch's DataLoader does not have.
- Top of page: the tall audio and Binder cards become one compact row
(the Overview heading moves up about 150px); the Binder button says
the session is discarded when you leave.
- Commands: each command's comment sits on its own line so the block
survives a phone screen; the four steps `complete` runs move into
"How you know it works".
- How you know it works: unit tests and integration files are lists
(two columns on wide screens), and test names are escaped, since two
`__init__` names on Module 19 were rendering as bold "init".
- What you write: a stacked definition list instead of a two-column
table that clipped on phones.
- Headings in sentence case on all 20 pages; one "What you're not
building yet" label; H3/H4 a clearer step down from H2.
- Code blocks scroll instead of soft-wrapping, and the copy button no
longer covers the first line.
- Slides viewer: on a load failure the spinner stops and a plain
download link remains.
- Module 20: one name (Torch Olympics), Capstone in its info line, and
the Olympics section moved ahead of the troubleshooting sections.
- 13 and 20 say "module" for site pages; 15's lead sentence reworded.
- TITO CLI: six pages become two. "Workflow & Commands" replaces the
overview, Module Workflow, Milestone System, and Progress & Data
pages (their URLs redirect), with every command taken from the
current parser and every behavior checked in tito's code. It corrects
two claims the old pages made: module reset keeps no backup (it
regenerates the notebook), and backups exist only for
`system update` and `package reset --backup`. Troubleshooting drops
from 1,075 lines to the problems students hit, quoting tito's real
messages; its old "rm -rf .venv; tito setup" fix could not work,
since tito lives in .venv.
- For Instructors: a new page covering what students write (47
regions), the three staging tiers, the nbgrader workflow and its
exchange caveat, and the fact a course must plan around: the book
prints the reference solutions.
- About the Course: Welcome and Big Picture merge into one page
(big-picture.html redirects); the Lab Guide language is gone.
- Torch Olympics folds into Module 20, Team into Team & Credits
(both old URLs redirect).
- Sidebar: milestones close the tier that unlocks them, a Teaching
section holds the instructor and developer pages, and the navbar
matches. Remaining `cd milestones; python ...` blocks on tier hubs
and the datasets page use tito.
Every module page now has the same shape: Overview, Start and finish,
Learning Objectives, What You'll Build, What you write, How you know
it works, When it fails, Finished? Read why, What's Next. Pages drop
from 530-760 source lines to about 110.
Four sections are generated from the repository at render time:
- module-checks: the unit tests the notebook runs (by their headings
in src/), the integration files `tito module complete` selects
(same rule as tito), and the milestones this module unlocks (parsed
from tito/commands/milestone.py).
- book-chapter: the companion chapter by its real title, with the PDF
link and a note that the book prints reference solutions.
- stub-failure: what a bare NotImplementedError means.
- (from earlier) module-workflow and what-you-write.
"When it fails" lists at most three messages per module, each one
reproduced by injecting the named bug into src/ and running that
module's unit tests; none is paraphrased.
Removed from every page: Core Concepts, API Reference, Production
Context with its PyTorch tabs, Check Your Understanding / Self-Check,
Key Takeaways, Further Reading (now in the book), Get Started (it
repeated the Binder link and linked the full solution), and the
implementation-roadmap tables, which listed pre-solved scaffold as
student work. The three optimization-tier narrative sections
(14, 17, 18) are kept aside for the tier hub.
Two shortcodes read the module's source file at render time, so the
page cannot drift from the notebook students receive:
- what-you-write lists every solution region the student tier strips
(plain and role="core"; scaffold regions ship solved), with the
owning class resolved through nesting, @method_of, or a later
Class.name = fn assignment, and the docstring's first line.
- module-workflow gives the start/resume/complete commands, the
notebook path, the export target from #| default_exp, and the four
checks complete runs, in the order it runs them.
Module 01 uses both. Its five-part roadmap listed scaffold methods as
student work (reshape, sum, ...), and its Get Started box repeated the
Binder link and linked the full solution source; both are gone.
Each of the 20 module pages carried an identical 298-line HTML block
(audio, Binder, source link, slide viewer) differing only in slug,
notebook, and slide name. Move it into a guide/_extensions/tinytorch
shortcode driven by page front matter (module, notebook, slides), and
move the H1 into a front-matter title.
Rendered DOM is identical for modules 02-20. Module 01 now gets the
same title block as the others (its H1 had not been promoted), and
every page's browser title reads 'Module NN: Name' instead of the bare
file slug.
The ::: {.panel-tabset} opener was the last line of the Production
Context section cut in 33352a7cb1, so its closing fence was orphaned:
the page rendered a literal ':::', and Your TinyTorch / PyTorch showed as
two separate H2s in the table of contents instead of two tabs.
Also retargets the Check Your Understanding line that sent students to
'revisit Core Concepts', a section that no longer exists on the page, at
the book chapter.
Caught by an external review of the trimmed page.
First page done under the two-artifact split. The website page carries
what you need to start; the book carries why it works.
Migrated into book/01_tensors.qmd first, so nothing is lost:
- Reductions and axis semantics. Sum, Mean and Max are Part 5 of module
01 and the chapter did not mention them at all: what axis collapses,
why keepdims matters when a loss gradient has to broadcast back, and
why collapsing the last axis of a row-major array walks contiguous
memory while collapsing the first walks with a stride.
- The three seminal references (Harris 2020, Lawson 1979, Baydin 2018).
The book cited no literature; the guide's Further Reading was the only
place they appeared.
Then removed from guide/modules/01_tensor.qmd: Core Concepts,
Architecture, Production Context, Key Takeaways, Further Reading, and
the "Why Tensors Matter at Scale" subsection nested under PyTorch.
953 -> 632 lines, and 408 -> 15 lines of Python. What remains is the
trailhead: Overview, Learning Objectives, What You'll Build, API
Reference, Common Errors, the TinyTorch/PyTorch contrast, a pointer to
the chapter, Check Your Understanding, and Get Started.
Two things for the remaining nineteen:
- The cut cannot be driven from H2 boundaries alone. "Why Tensors Matter
at Scale" is an H3 nested under ## PyTorch, not under ## Production
Context where it belongs logically, so a section-level pass leaves it
behind. Each page needs the orphan check by eye.
- The pointer uses the published URL, not a relative path. The Book PDF
is injected into assets/downloads/ at deploy time and is not in the
repo, so a relative link fails the internal-link check, which is how
this one was caught.
guide/modules/01_tensor.qmd showed
def __add__(self, other):
return Tensor(self.data + other.data)
src/01_tensor:448 is `return Add.apply(self, _as_tensor(other))`. The
listing predates the Function.apply refactor. Wrapping the result in a
fresh Tensor creates a new leaf, so a student who copies it gets the
right numbers and no graph node, and Module 06's autograd then produces
no gradient with no error. The book's listing is correct, because the
book pulls through tools/listings.py while the guide hand-copies.
Also on that page: "A reshape is an O(1) metadata update, the underlying
memory is completely untouched." True of NumPy, false here. TinyTorch's
reshape dispatches through Function.apply, whose Tensor constructor
copies, which the page's own checklist already said ("TinyTorch copies
on wrap"). Now says both.
And guide/modules/08_training.qmd:581 said the Trainer applies the
schedule "after each epoch" while its own listing twelve lines earlier
comments "at the start of each epoch so LR is set before training
begins". src applies get_lr(self.epoch) before any batch.
An audit comparing each guide chapter against its book chapter found
that where the two disagree about the implementation, the book is right
every time. The books were written against src/; the guides were not
re-verified after it settled. Four confirmed against src:
- Sinusoidal positional encoding "has no length limit" and "can
extrapolate to any length". src/11_embeddings raises
ValueError("Sequence runs past the sinusoidal position table"). A
student following the guide hits an exception. The book says the table
is still allocated and evaluating past it needs a bigger allocation.
Also "Sinusoidal PE: 0 bytes" -- zero *parameters*, but src
materialises the full table.
- BPE training is "O(n^2)", stated three different ways across the page
and used to derive a 10 s / 16.7 min extrapolation. train() runs one
_count_byte_pairs and one _merge_pair per merge, so it is O(n x m).
The book says O(NM).
- The GPT parameter total. An earlier commit corrected the text to say
the output projection is a separate Linear, but left the arithmetic:
25.6 + 1.0 + 37.8 = 64.4M silently omits the 25.6M head it had just
described. Corrected to ~90M. The Q2 walkthrough still assumed tying
("usually tied to embeddings (0 additional)", ~125M); untied it is
~163M, and its punchline inverts: blocks 85M against 78M of embedding
and head is nearly even, not a rout.
Left alone deliberately: the O(n^2) at 10_tokenization:816 is attention
scaling with sequence length, which is correct.
Caught by looking at a rendered page rather than the diff. The
01_tensor figure caption read "...Module 06 adds graph recording and
backward propagation. ... Module 06 extends this operation path with
graph recording and backward propagation."
When the in-image footers moved into the Quarto captions, the overlap
check compared only the first 35 characters, so a footer that restated
the caption in different words was appended anyway. Six of the eight
captions repeated themselves:
- 01_tensor-diag-1: Module 06's role, twice
- 01_tensor-diag-2: the whole sentence, already carried by the caption's
title and the clause before it
- 03_layers-diag-1: Sequential composing layers, twice
- 09_convolutions-diag-1: stride 1, no padding, and the 54 output were
already in the caption; only the zero bias and the backward behaviour
were new, so only those are kept
- 10_tokenization-diag-1: merges applied in order within each word,
twice
- 11_embeddings-diag-1: repeated IDs accumulating into one row, twice
Each is trimmed to the part the caption did not already say. 12_attention
and 04_losses were fine as they were.
Per-page review against src/ found four snippets that do not run and two
that run but do the wrong thing.
Do not run:
- 11_embeddings: `Tokenizer(vocab_size=50000)`. Tokenizer is the
interface base class with no __init__, so this raises TypeError.
BPETokenizer is the one that takes vocab_size.
- 06_autograd documented `set_grad_enabled(mode: bool)`. That name
appears nowhere in the repo. The row was also a duplicate: the real
API, is_grad_enabled(), was already documented on the line above.
- 19_benchmarking documented `compare_submissions`. Zero occurrences in
src/19_benchmarking; the only occurrence in the repo was that table
row.
- 09_convolutions promised module 10 builds a WordPiece tokenizer, in
two places. It builds CharTokenizer and BPETokenizer; WordPiece
appears only as an aside about BERT.
Run, but wrong:
- 04_losses showed `return LogSoftmax.apply(x)`, dropping the dim
argument the wrapper accepts (src passes `dim=dim`). A student gets a
log_softmax whose dim= is silently ignored, i.e. the wrong axis with
no error.
- 16_compression's structured_prune pruned every Linear. src collects
them and prunes `linears[:-1]`, deliberately excluding the classifier
because "its output channels are the classes, so zeroing them removes
classes, not neurons". Copying the listing silently deletes output
classes. Also switched to src's stable argsort.
Two reasoning errors:
- 19_benchmarking's Pareto example declared model C (12ms, 91%)
dominated by A (10ms, 90%). A is faster but less accurate, so neither
dominates and C was Pareto-optimal. The page's own Q4 analyses the
identical structure correctly. C is now 89% so the example teaches
what it claims, with a note on the near-miss case.
- 13_transformers assumed the output projection is weight-tied ("0
extra"). src builds `Linear(embed_dim, vocab_size, bias=False)` and
counts it, and the page's own figure alt text calls it a separate
head. Also: the causal mask was described as upper triangular filled
with -inf, while the listing directly beneath it is np.tril of 1.0/0.0.
And the measurement overclaims, all against milestones/06_2018_mlperf,
which states compression and speedup "are outcomes to measure, not
benefits the script assumes" and that a small workload "may become
slower": "compress 8x and speed up generation 10x" (the repo ceiling is
4x), INT8 "speeds up inference 2x", pruning "halves memory" (capstone's
own measure_memory sums param.data.nbytes, "zeros still occupy
storage"), "sparsity creates memory savings", pruning "while preserving
accuracy", profiling "routinely uncovers 10x speedups", and vectorised
NumPy "10-100x faster". 15_quantization also pointed at module 16 for
quantization-aware training, which module 16 does not implement and
which 15's own "What You're NOT Building" list already excluded.
Three of the 109 current-generation diagrams carry a caption baked into
the image; fourteen of the twenty-five older ones do. Quarto already
prints a caption underneath, so the reader got the same voice twice in
two typefaces, and the canvas carried dead space to hold it.
Eight of those footers were safe to identify: a single #D1D5DB rule in
the trailing stretch of the file, with only text after it. Their text is
appended to the page's fig-cap before removal, so nothing is lost --
several carried qualifications the caption did not have, such as the
stride/padding/bias of the convolution example and that the
cross-entropy decimals are rounded.
The other six are not touched. In those, #D1D5DB is an internal grid
line rather than a footer rule: in 16_neuron-removal it appears on line
5 of 102. Treating it as a separator would have deleted the diagram's
own labels. They are also book-only files with no guide page to hold a
caption.
Canvases are re-sized to the content. The first attempt measured rect y
without adding rect height and clipped the bottom row of
01_tensor-diag-1; caught by rendering, which is why every one of the
eight was re-rendered and inspected after the fix.
The website diagrams in guide/assets/images/svg/ predate the current
visual standard in guide/tools/diagrams/STYLE.md. They used flat
#111827 boxes at rx=3, L-shaped connectors the spec forbids, captions
baked into the image that duplicate the Quarto fig-cap, and canvases
sized well past their content.
Rebuilt all 11 to the spec: 680 viewBox sized to content, greyscale
palette with at most one #ff8246 accent, rx=2, uniform 1pt strokes,
straight H/V connectors only. Three carried real defects beyond style:
- tito-progress-tracking had a box no arrow reached, and otherwise
duplicated tito-module-workflow. Redrawn around .tito/progress.json
and the two commands that read it, which is what its page is about.
- tito-test-hierarchy claimed increasing scope and drew six identical
boxes. Now a widening stack.
- The optimization fan-ins bent four arrows into a bus. The target is
widened to span its inputs so every arrow drops straight down.
Removing the in-image captions exposed that fig-gpt-architecture never
had a Quarto caption, so it rendered a bare "Figure 1"; its prose also
described the stack bottom-up while the figure reads top-down. Both
fixed. The module-flow caption named blue/green/orange tiers the figure
no longer uses.
Every SVG paints an opaque white canvas, which on the dark theme landed
as a hard-edged white slab. Referenced through <img src> the SVG is an
isolated document, so site CSS cannot reach inside it and an internal
prefers-color-scheme block would track the OS rather than the site
toggle. dark-mode.scss now frames them as deliberate plates instead,
which covers all 145 diagrams rather than only these 11.
Also repoints the .gitignore downloads whitelist at guide/, left stale
by the quarto -> guide rename.
Verified by building the site and screenshotting every figure at 1440px
light, 1440px dark, and 430px; audited all 62 pages for overflow and
broken images.
Vijay Janapa Reddi