Adds a Playwright-based smoke check that runs after the Quarto build
and before the dev preview deploy. Catches the four classes of
regression we just fixed in commit 6fdf81dd4:
• <link rel=stylesheet> 404s (broken site_libs/ deploy)
• JS console errors during page load
• Blank-page renders (homepage body shorter than 1.5× viewport)
• Navbar collapse breakpoint drift across the shared chrome (asserts
expanded at 1200/1400, collapsed at 992/1199 in light + dark)
Layout follows the publish-guard pattern already in use:
.github/scripts/visual_smoke.py — test logic, runnable locally
against any Quarto _build/ tree
.github/workflows/infra-visual-smoke.yml
— workflow_call reusable that
runs the script on an uploaded
build artifact
tinytorch-preview-dev.yml is split into three jobs:
1. build-site — builds Quarto, injects PDFs, uploads _build/ artifact
2. smoke — `uses:` the reusable; deploy is gated on this passing
3. deploy — downloads the same artifact (so we ship the exact
bytes smoke validated) and SSH-pushes to the dev repo
Verified locally:
• script passes 8/8 (4 viewports × 2 schemes) on the fixed dev tip
• script catches the regression: with collapse-below reverted to "lg"
and re-rendered, smoke fails 4/8 with [NAVBAR_COLLAPSE_AT_XL] at
exactly 992 + 1199 px in light + dark — the same widths a human
would notice the wrap
Other sites' preview-dev workflows are intentionally untouched. Once
this proof-of-concept settles, copying the build-site/smoke/deploy
split to labs/kits/mlsysim/site/etc. is mechanical (the reusable
workflow doesn't change). A follow-up infra-visual-cross-site.yml
will run nightly across every site in a matrix and add the cross-site
consistency assertions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Default RUN /bin/sh doubles 82414 inside double quotes to the shell PID, so
command substitution with Docker escaped dollar-paren was mis-parsed
(Phase 10: syntax error on open paren). Use POSIX backticks for command
substitution, brace variables for echo, and a short file-header note.
The project's source convention (navbar, footer, announcements,
instructor course-map, etc.) already treats mlsysbook.ai/vol1/ and
mlsysbook.ai/vol2/ as the canonical URLs. The /book/vol1/ nesting was
an artifact of the legacy single-volume textbook still occupying /book/
on live, kept alive by a special-case mapping in the dev URL rewriter.
This refactor aligns the actual deploy paths with the source convention
before Monday's release locks in citation-grade URLs.
Public URL change:
was: https://mlsysbook.ai/book/vol1/ + .../book/vol2/
is: https://mlsysbook.ai/vol1/ + .../vol2/
Variable change (set on harvard-edge/cs249r_book):
VOL1_DEPLOY_PATH=vol1 (new)
VOL2_DEPLOY_PATH=vol2 (new)
BOOK_DEPLOY_PATH=book (will be deleted post-merge)
Workflow changes:
book-publish-live.yml
- Reads VOL1_DEPLOY_PATH + VOL2_DEPLOY_PATH (fail-fast on empty).
- Each volume deploys to its own top-level path on gh-pages.
- Skip-list regex now includes both top-level paths plus 'book' so
the legacy single-volume textbook at /book/ stays untouched
(the user-facing front door remains the OLD textbook until an
explicit cutover via the Cloudflare redirect).
- Root index.html redirect now targets /$VOL1_PATH/ instead of
/$BOOK_PATH/vol1/. The CF redirect is what users see; this is
a fallback only.
- Validation, commit message, and step summary lines log both
volume paths separately rather than a single book parent.
book-preview-dev.yml
- Same VOL1/VOL2 read with fail-fast guard.
- Cleans /vol1/, /vol2/, AND legacy /book/ on the dev preview repo
to avoid zombie content from the previous nested deploy.
- Copies preview-site/vol1 → /vol1/ and preview-site/vol2 → /vol2/
separately (no longer wraps both under /book/).
- Drops the /book/ chooser page on dev (the dev landing page
already has volume cards). The chooser file stays in the repo
for the eventual live cutover.
publish-all-live.yml
- Step summary now lists Volume I and Volume II as separate links.
rewrite-dev-urls.sh
- vol1 → vol1, vol2 → vol2 (identity mapping; matches every other
subsite). The PREFIX depth math stays generic for any future
nested subsite.
site/index.qmd
- Volume cards link to vol1/ and vol2/ (top-level relative). On
live this resolves to mlsysbook.ai/vol1/ and mlsysbook.ai/vol2/.
Result: zero asymmetry between the volume URL convention used in source
and the actual deploy paths. The dev URL rewriter no longer needs a
special case. Citations made against Monday's release URLs will be
permanent and aligned with the project's everyday URL vocabulary.
* fix(dev-mirror): compute prefix from dev-side depth in rewrite-dev-urls.sh
The previous implementation hard-coded PREFIX="../" for any non-root
subsite, which silently mis-rewrote every absolute mlsysbook.ai link on
the dev preview for nested subsites (vol1, vol2 — they live at
/book/vol1/ and /book/vol2/ on dev). The most visible symptom was the
navbar title-href landing one level too shallow: clicking the navbar
title from inside Vol I went to /book/ instead of the unified landing
page at the dev root.
Fix: derive PREFIX from the number of path segments in the calling
subsite's dev-side path (book/vol1 → 2 hops → '../../') and use the
mlsysbook.ai key (not the dev-path) for self-link detection. Add an
explicit error if the caller passes a subsite name that is not in the
SUBSITES map, instead of silently producing wrong rewrites.
Sample rewrites with the fix:
vol1 page https://mlsysbook.ai/ → ../../
vol1 page https://mlsysbook.ai/vol2/ → ../../book/vol2/
vol1 page https://mlsysbook.ai/kits/ → ../../kits/
kits page https://mlsysbook.ai/ → ../
kits page https://mlsysbook.ai/vol1/ → ../book/vol1/
Live builds are unaffected — they use the original absolute URLs.
* feat(book): per-volume announcement bars (Crimson / ETH-Blue)
Split the shared book announcement bar into two volume-scoped files so
each volume gets audience-appropriate copy AND inherits the right brand
tint. Vol I keeps the Harvard-Crimson tint (its theme accent) and the
Foundations-flavored content; Vol II picks up the ETH-Blue tint (its
theme accent) and Scale-flavored content that leads with the new
volume launch and the cross-ecosystem build path.
Files:
- announcement-vol1.yml — new, Vol I copy, no hard-coded color (uses
`type: primary` so .announcement / .alert-primary get $accent =
$brand-crimson via theme-harvard.scss)
- announcement-vol2.yml — new, Vol II copy, same pattern but theme
feeds $accent = $brand-eth-blue via theme-eth.scss
- announcement.yml — emptied to a no-op with a deprecation note;
keep for one release cycle to avoid breaking any external metadata
reference, then delete
The CSS that translates `type: primary` into the per-theme tint already
lived in book/quarto/assets/styles/_base-styles.scss (`.announcement {
background: linear-gradient(... lighten($accent, 52%) ...) }`). No
SCSS changes needed — the previous behavior of a single shared bar
just hid that the tint was already theme-driven.
Resolves the "Vol II announcement should be ETH-themed" QA note.
* feat(theme): cross-site dark-mode persistence + FOUC guard
Make dark-mode preference flow seamlessly across every subsite under
mlsysbook.ai (Quarto-built and Next.js alike) and eliminate the
theme-flash that dark-mode readers see on first paint.
Quarto subsites (book / labs / kits / slides / instructors / mlsysim /
tinytorch / unified site):
- shared/config/site-head.html now inlines a tiny pre-paint script
that reads `quarto-color-scheme` from localStorage (or falls back
to OS preference) and applies `data-bs-theme`,
`data-quarto-color-scheme`, and `style.color-scheme` on <html>
BEFORE any other script runs. Eliminates the visible flash that
was happening because Quarto's own toggle script runs late.
- Listens for `storage` events so a toggle in tab A propagates to
tab B without a refresh.
- Inlined deliberately: the script is tiny, must be synchronous in
<head> to avoid the flash, and inlining sidesteps per-subsite
asset path differences. Canonical externalized source kept at
shared/scripts/theme-persist.js for documentation/testability —
if you change one, mirror to the other.
StaffML (Next.js):
- public/theme-bootstrap.js now reads the Quarto-side key as a
fallback when StaffML has no local preference, so a user toggling
dark mode on the book lands here in dark mode on first visit.
- components/ThemeProvider.tsx mirrors writes back to
`quarto-color-scheme`, so navigating onward to any Quarto subsite
inherits StaffML's choice. Both subsystems retain their own keys
as primary so each app's behavior is unchanged in isolation.
The `quarto-color-scheme` key is the bridge contract — keep it stable
across all theme code paths.
* test(audit): Playwright site-audit script (sidebar / darkmode / assets)
Single Playwright-driven QA script that the release-prep plan needs in
three flavors. Implemented as one CLI with three subcommands so the
shared boilerplate (browser launch, URL list, output dirs, screenshot
naming) lives in one place and the per-site source-of-truth list does
too.
Subcommands:
sidebar Assert every Quarto subsite exposes a populated, visible
#quarto-sidebar / .sidebar-navigation. Skips sites that
intentionally have no sidebar (landing, slides, StaffML).
Catches the regression where Vol I/II builds dropped the
sidebar after a config refactor.
darkmode Force dark-mode via localStorage + data attributes, scroll
top→bottom in 800px chunks (so lazy content renders), and
screenshot full-page into _audit/darkmode/<site>.png for
eyeball review. Surfaces "half-themed" widgets that CSS
linters can't find (announcement bar, footer tiles, code
blocks, etc.).
assets Listen for failed network requests + 4xx/5xx responses on
every site URL. Catches the broken <img> embeds reported
during dev-mirror review (TinyTorch big-picture PDF
viewer, Vol II cover) before they hit production.
Targets dev / live / local with --target. Use --only <substring> to
narrow scope. JSON report written to _audit/<cmd>.json for CI ingest.
Exits non-zero on issues so it can become a blocking CI check once the
baseline is clean.
Requires `npm i -D playwright && npx playwright install chromium`.
adds checks for two bug classes that each produced silent, long-lived failures:
1. workflow fork-safety (from #1344). any pull_request-triggered workflow that references ${{ vars.* }} or non-GITHUB_TOKEN ${{ secrets.* }} breaks silently on fork PRs because repo vars/secrets are not exposed in that context. a week of broken fork CI on #1306, #1331, #1339 before anyone noticed.
2. marimo widget-in-gated-cell (exposed by lab_01's broken state, fixed in #1339). when an @app.cell has mo.stop() AND defines mo.ui.* widgets that appear in its return tuple, those widgets don't exist until the gate unblocks, cascading "undefined dependency" failures through every cell that depends on them.
## changes
- `.github/scripts/check_workflow_fork_safety.py`: standalone python + pyyaml, parses each workflow, identifies those triggered by pull_request, flags unsafe vars/secrets references with file:line:token pointers and a fix hint. exempts secrets.GITHUB_TOKEN which is always available.
- `.github/workflows/ci-sanity.yml`: new workflow triggered on .github/ changes that runs the fork-safety check. catches contributors who don't have pre-commit installed.
- `.pre-commit-config.yaml`: wires the fork-safety script as a local pre-commit hook under a new "SECTION 3.5: CI SANITY" so it runs on workflow edits.
- `labs/tests/test_static.py`: new TestMarimoDataflow class with test_no_widget_defined_in_gated_cell. ast-based, flags cells that are both gated and define returned widgets. marked xfail for now because 32 of 33 labs currently have the pattern; the systematic refactor is separate scope. once labs are converted to the proper pattern (widget in own cell, gate cell is pure mo.stop, see vol2/lab_05_dist_train), remove the xfail.
- `labs-validate-dev.yml`, `kits-validate-dev.yml`, `mlsysim-validate-dev.yml`: the "validate build output" step now echoes the resolved env var and fails loudly with an explanation if the var is empty, rather than silently checking a wrong path. would have diagnosed #1344 in 30 seconds instead of a week.
## verification
- fork-safety check: ok, 47 workflows scanned, 8 pull_request-exposed, 0 violations
- marimo check: ok across all 33 labs
- pytest labs/tests/test_static.py: 656 passed, 4 skipped, 33 xfailed, 1 xpassed
- vol2/lab_05_dist_train is xpassed (the canonical reference lab with the proper pattern)
The three GitHub Actions workflows that synced newsletter posts from
Buttondown previously called .github/scripts/sync_newsletter.py with a
bare `pip install requests`. They now install the news CLI's
requirements and call `news pull`.
- sync-newsletter.yml (daily scheduled sync)
- site-publish-live.yml (pre-build sync for live site)
- site-preview-dev.yml (pre-build sync for dev preview)
Ports the production-grade logic from the old script into
cli/commands/pull.py:
- HTML body handling (wrap in ```{=html}``` block for Quarto)
- Auto-categorization based on subject keywords
- Guest author detection ("Written by X" pattern)
- Skip Buttondown placeholder images (image-generator.buttondown.email)
- Incremental sync with early termination after 3 consecutive unchanged
- _stats.yml generation (issue count + subscriber count)
- --no-stats flag for runs that should not update the counter
Delete .github/scripts/sync_newsletter.py — the CLI is now the single
source of truth. No other callers.
Architecture:
- Merge landing, about, community, newsletter into one site/ project
- Move navbar-common.yml to shared/config/ (used by 12 configs)
- Create shared/config/footer-site.yml for centralized footer
- Create shared/scripts/subscribe-modal.js as canonical copy
- Single _quarto.yml replaces 4 independent configs
- One site_libs/ copy replaces four
Features gained:
- Google Analytics on ALL hub pages (was only on book volumes)
- Subscribe modal on landing page (was missing)
- Centralized footer with consistent links
Workflows updated:
- site-preview-dev.yml: matrix strategy → single build job
- site-publish-live.yml: loop over subsites → single build + deploy
- sync-newsletter.yml: builds from unified site project
- publish-all-live.yml: removed stale subsite input
- rewrite-dev-urls.sh: added --shallow flag for unified builds
All 12 navbar-common.yml references updated:
book vol1/vol2, site (unified), slides, instructors, interviews,
kits, labs, mlsysim
Move about/, community/, landing/, newsletter/ into site/ to
declutter the repo root. Instructors stays top-level as curriculum
content alongside book/, kits/, labs/, etc.
- Create unified site-preview-dev.yml (replaces 3 individual workflows)
- Create site-publish-live.yml for live deployment of all site subsites
- Add deploy_site and deploy_instructors to publish-all-live.yml
- Update all workflow paths, _quarto.yml relative paths, and scripts
- Fix landing page skip-list to include about/community/newsletter
- Update .gitignore newsletter paths
Replace the generate-navbar.py script approach with direct metadata-files
references to navbar-common.yml. Quarto natively merges navbar arrays,
so site-local dropdowns defined inline in each _quarto.yml get appended
to the shared navbar automatically.
- Delete generate-navbar.py script (no longer needed)
- Delete all _navbar-generated.yml and _navbar-local.yml files
- Each subsite now references ../book/quarto/config/shared/html/navbar-common.yml
directly via metadata-files
- Sites with local dropdowns (slides, instructors, mlsysim) define them inline
in their _quarto.yml navbar.left — Quarto merges with the shared left items
- Zero tooling, zero generated files, zero maintenance scripts
Single source of truth for the navigation bar in navbar-common.yml.
New 5-dropdown structure: Read | Build | Teach | Community | About.
- Add generate-navbar.py script to merge shared + site-local dropdowns
- Add _navbar-local.yml for slides, instructors, mlsysim (site-specific menus)
- Generate _navbar-generated.yml for all 6 subsites (committed, no CI changes needed)
- Strip hardcoded navbars from all _quarto.yml configs (landing, vol1, vol2, slides,
instructors, mlsysim, kits, labs, newsletter)
- Add community/ and about/ paths to rewrite-dev-urls.sh for dev preview
To update navbar: edit navbar-common.yml, run generate-navbar.py --all, commit.
Extract duplicated sed-based mlsysbook.ai URL rewriting from three
dev preview workflows into .github/scripts/rewrite-dev-urls.sh.
Each workflow now calls the script with its subsite name and build
directory. The site map lives in one place — adding a new subsite
means updating one array instead of every workflow.
Remove 19 unused scripts that were not referenced in any workflows or configuration files:
- 13 validation scripts in .github/tinytorch-scripts/ (never integrated into CI/CD)
- TINYTORCH_RELEASE_PROCESS.md documentation
- Duplicate gs_compress_pdf.py script
- Unused book scripts (footnotes, reorganize_scripts)
- Unused check_no_emojis.py script
This commit introduces a script that automatically generates changelog entries by analyzing git commit history and summarizing changes using OpenAI.
The script fetches changes from the `dev` branch since the last gh-pages publish, summarizes the changes using OpenAI, and formats them into a changelog entry. It then prepends the new entry to the existing changelog file.
The script uses git commands to get the commit history and OpenAI to summarize the changes. It also includes logic to sort the changes by chapter order and to categorize them as major or minor updates.
This enables a more automated and insightful changelog generation process.
