Vol II now has section IDs on every numbered header (prior commit), so
both `ids` and `case` scopes pass on both volumes. Replace the two
per-scope hooks with a single `book-check-headers` running
`./binder check headers` (no flags).
The `labels` split (orphans vol1-only, duplicates both-vols) stays in
place — Vol II still references not-yet-authored chapters, which the
orphans scope correctly flags. Collapse that one once those chapters
land.
Hook count: 47 -> 46.
Vol II was being skipped by the headers and labels hooks because both
hooks carried a `--vol1` filter. The filter was needed for ONE scope in
each group (headers/ids, labels/orphans), not the other (headers/case,
labels/duplicates which both pass cleanly on vol2 today).
Split each group into per-scope hooks so the universally-clean scope
covers both volumes:
book-check-headers-ids vol1 only (--scope ids --vol1)
book-check-headers-case both vols (--scope case)
book-check-labels-orphans vol1 only (--scope orphans --vol1)
book-check-labels-duplicates both vols (--scope duplicates)
Net effect: vol2 now gets headline-case enforcement + figure/table/listing
duplicate-label detection on every commit. No regression to vol1 coverage
(both vol1-only scopes still run on vol1; the both-vols scopes also see
vol1).
When vol2 catches up — every section ID'd, every forward-ref resolved
to a real chapter — collapse each pair back to a single
`book-check-<group>` running `./binder check <group>`. The TODO is
inline in `.pre-commit-config.yaml`.
Hook count: 45 -> 47.
Replace the dated 60-row "Check Groups & Scopes" table (which still
mentioned a `rendering` group split out months ago and was missing the
migrated index/lego-dead-code scopes) with a description of the
discovery contract:
./binder check # full catalogue
./binder check <group> help # per-group rows incl. default flag
Adding 60+ rows of stale truth-duplication to a Markdown table is the
opposite of maintainable; the binder help output now is the source of
truth.
Document the "default=True vs default=False" contract that pre-commit
and CI consume, and rewrite the Pre-commit Integration + Adding a new
check sections to walk a future maintainer through the actual flow:
write _run_<scope>, add a Scope() entry, done — no YAML edit needed.
Closes the loop on `./binder build epub`. The build pipeline now ends
with a smoke + epubcheck pass against the just-built EPUB, matching
what CI runs, so a local success means "you built it AND it validates
against the same baseline CI enforces." Before this commit, the build
simply exited after the post-process sanitizer — leaving the user to
remember `./binder check epub` as a separate step. Most users didn't.
Sequence for `./binder build epub --vol1` is now:
1. hygiene preflight (~100ms; fast-fail on source issues)
2. Quarto render (~2 min)
3. epub_postprocess.py hook (sanitizes renderer-emitted issues)
4. smoke + epubcheck (~7s; new; validates the final EPUB)
Post-flight runs on all four EPUB build paths (build_full,
build_volume, build_chapters, build_chapters_with_volume) for
symmetry with the existing --skip-hygiene behaviour.
Failure modes
-------------
- smoke finds a reader-compat issue (CSS custom properties, external
resource refs): build exits nonzero, file still written. Release
scripts gated on exit code fail fast rather than shipping a broken
artifact.
- epubcheck reports counts within baseline: build succeeds with a
yellow summary line showing the counts ("all within baseline — no
regression"). No regression → no failure.
- epubcheck counts exceed the baseline recorded in
book/tools/audit/epubcheck-baseline.json: build exits nonzero with
per-volume deltas and a pointer to `--update-baseline` for
accepted increases.
- Java / epubcheck missing locally: smoke still runs; epubcheck
skipped with a single yellow hint line ("install via `pip install
epubcheck` or `./binder doctor`"). Build does NOT fail — CI will
catch anything this local pass would have.
--skip-validate
---------------
Matches --skip-hygiene in shape. Bypasses the post-flight step for:
* iterating on a known-broken build
* local environments without Java / epubcheck
* emergency artifact generation
Prints a loud yellow warning so the bypass is never silent.
Also done in this commit
------------------------
- Inlined "EPUB — How This Integrates with the Book Workflow" into
book/cli/README.md so the author / release / CI flows are
documented in one place new contributors will actually find.
- Added --skip-validate to the `./binder build --help` output.
Validation
----------
./binder build --help → lists --skip-validate
BuildCommand class signatures → skip_validate parameter on
all four build_* methods
baseline matches recorded counts → post-flight would pass silently
Replaces the flat MAX_FATAL / MAX_ERRORS gating in CI with a
per-volume baseline file. The check fails only when the current
epubcheck counts exceed what is recorded in
`book/tools/audit/epubcheck-baseline.json`, so:
* Pre-existing issues do not block PRs that don't touch them.
* Any new FATAL / ERROR / WARNING does block the PR.
* A cleanup PR that lowers the counts must also include an
`--update-baseline` commit so the ratchet tightens.
This is strictly better than `--max-errors N`: with a flat threshold
of N, a cleanup that brings the count from N to N-10 leaves the
ceiling at N forever and allows silent regrowth. With the ratchet,
every cleanup explicitly lowers the ceiling.
CLI surface
-----------
./binder check epub --scope epubcheck \
--baseline book/tools/audit/epubcheck-baseline.json
./binder check epub --scope epubcheck \
--baseline book/tools/audit/epubcheck-baseline.json \
--update-baseline # rewrite to current counts
When --baseline is supplied, --max-fatal / --max-errors are ignored.
The baseline file is JSON with a `volumes` map of per-volume
{FATAL, ERROR, WARNING} counts.
Under-baseline behaviour
------------------------
When current counts are *below* baseline, the run passes and prints
a hint that `--update-baseline` is available. It deliberately does
NOT auto-update: silently lowering the bar without a commit would
mean regression detection depends on build order, not on a tracked
artifact.
Baseline file (initial)
-----------------------
vol1: 0 FATAL, 0 ERROR, 0 WARNING
vol2: 0 FATAL, 0 ERROR, 0 WARNING
Matches the current clean state after the fix/epub-issues work.
Any new issue in either volume blocks the PR.
CI wiring
---------
.github/workflows/book-validate-dev.yml now passes --baseline
instead of --max-fatal / --max-errors. The inline env block and
step-level comment document the update-baseline workflow so a
cleanup PR knows to refresh the file.
Validation
----------
./binder check epub --scope epubcheck \
--baseline book/tools/audit/epubcheck-baseline.json → exit 0
(clean tree matches baseline)
YAML parse of CI workflow → OK
The legacy `book/tools/scripts/utilities/validate_epub.py` duplicated
most of what epubcheck covers (mimetype, container.xml, OPF structure,
XHTML well-formedness, unclosed tags, unescaped ampersands) and had a
pre-existing broken exit code that made it unusable in the default
`./binder check epub` run.
Two of its checks were genuinely unique — CSS custom properties and
external resource references. Those patterns produce real user-
reported breakage on older readers (ClearView / Tolino firmware pre-
2023) that epubcheck does not flag because they are valid EPUB 3
spec. Those unique checks now live in _epub_checks.py as
`run_smoke_checks_on()`, exposed via the new scope:
./binder check epub --scope smoke
Smoke runs in <1s, does not require Java, and fails on detection
(error severity) because the patterns caused real load failures.
Default `./binder check epub` now runs three scopes in order:
hygiene (source, pre-build)
smoke (built EPUB, reader compat, no Java)
epubcheck (built EPUB, W3C validation, needs Java)
Together they form a three-layer defense: source, built-EPUB without
Java, built-EPUB with Java. A regression has to escape all three to
reach a reader.
Removed
-------
book/tools/scripts/utilities/validate_epub.py — duplicate of epubcheck
plus two unique checks now owned by _epub_checks.py.
Validation
----------
./binder check epub → 0 issues, 3 scopes run, ~7s
./binder check epub --scope smoke → 0 issues, ~200ms
Folds both EPUB checks into the binder CLI so pre-commit, CI, and
interactive `./binder check epub` all exercise the same Python code
(no subprocess-to-script). Adds epubcheck to book/tools/dependencies
so pip install gives every contributor the validator, and simplifies
the CI job from ~60 lines of shell+Java setup to three binder calls.
New CLI surface
---------------
./binder check epub # both scopes (default)
./binder check epub --scope hygiene # source-level invariants
./binder check epub --scope epubcheck # W3C validator on built EPUBs
./binder check epub --scope epubcheck \
--max-fatal 0 --max-errors 0 # tighten thresholds
./binder check epub --json # machine-readable output
Implementation
--------------
New file book/cli/commands/_epub_checks.py holds the primitives:
find_hygiene_issues() — walks SVG + BIB source, returns
EpubIssue records for the four invariant
classes (svg-c0, svg-dupe-marker,
bib-url-escape-{underscore,percent},
bib-url-raw-angle).
run_epubcheck_on() — invokes the epubcheck binary or Python
wrapper, parses JSON messages, returns
one EpubIssue per location with severity
counts for threshold checks.
emit_github_annotations() — under GITHUB_ACTIONS=true, emits
`::error file=...` so findings appear
inline on the PR diff.
_discover_built_epubs() — finds the most recent EPUB per volume
under book/quarto/_build/epub-vol*/.
validate.py gains:
- GROUPS['epub'] with 'hygiene' and 'epubcheck' scopes (the old
'structure' scope delegated to validate_epub.py and had a
pre-existing broken exit code; retired from default runs).
- New --max-fatal and --max-errors flags.
- Native _run_epub_hygiene and _run_epubcheck methods that call
_epub_checks.py directly and wrap results into ValidationIssue.
No subprocess-to-my-own-script.
Pre-commit wiring
-----------------
book-epub-hygiene now runs `./book/binder check epub --scope hygiene`.
Trigger files extended to include validate.py and _epub_checks.py so
the hook re-runs when the check itself changes.
CI wiring
---------
epub-validate job drops ~60 lines of shell + java jar download and
calls `./binder check epub --scope epubcheck` twice: once with --json
to capture a machine-readable artifact, once without for the
threshold-gated human summary. Java is still installed via
actions/setup-java because epubcheck is a jar.
Requirements
------------
`epubcheck>=5.1.0` added to book/tools/dependencies/requirements.txt
so every contributor gets the validator alongside the rest of the
binder runtime. Wrapper bundles the jar; JRE 8+ is still a host
requirement (brew install temurin / apt install default-jre). When
neither the Python wrapper nor the system binary is available, the
binder check emits an `epubcheck-missing` issue with install
instructions rather than silently passing.
Removed
-------
book/tools/audit/checks/epub_hygiene.py — its logic now lives in
book/cli/commands/_epub_checks.py.
Documentation
-------------
book/cli/README.md gains a "EPUB Checks — Two Layers, One CLI
Surface" section with an error-code table, fix-at-source guidance
per code, and the defense-in-depth rationale (hygiene →
epub_postprocess → epubcheck) so future contributors understand
where each check fits in the pipeline.
Validation
----------
./binder check epub → 0 issues, ~7s (both scopes)
./binder check epub --scope hygiene → 0 issues, ~100ms
pre-commit run book-epub-hygiene → passes
YAML parse of pre-commit + CI → both OK
Adds space before $\times$ when it appears between two inline Python
values (product pattern), preventing LaTeX from treating the entire
sequence as one unbreakable token that overflows callout boxes and
page margins. Multiplier patterns (e.g. "3× faster") left unchanged.
Also centralizes pre-commit checks:
- Migrate inline bash div-fence checks into binder (div-fences scope)
- Add times-product-spacing check to catch future regressions
- Update binder CLI README with full command/scope reference
- Remove redundant BINDER_NATIVE_AUDIT.md
- Completed full Volume 1 refactor to Safe Class Namespace pattern.
- Fixed render errors and verified all 16 chapters.
- Updated 'binder' CLI with native validation and maintenance namespaces.
- Enhanced VS Code extension with Chapter Navigator and Run History.
- Integrated 'binder validate' into pre-commit workflows.
* Restructure: Move book content to book/ subdirectory
- Move quarto/ → book/quarto/
- Move cli/ → book/cli/
- Move docker/ → book/docker/
- Move socratiQ/ → book/socratiQ/
- Move tools/ → book/tools/
- Move scripts/ → book/scripts/
- Move config/ → book/config/
- Move docs/ → book/docs/
- Move binder → book/binder
Git history fully preserved for all moved files.
Part of repository restructuring to support MLSysBook + TinyTorch.
Pre-commit hooks bypassed for this commit as paths need updating.
* Update pre-commit hooks for book/ subdirectory
- Update all quarto/ paths to book/quarto/
- Update all tools/ paths to book/tools/
- Update config/linting to book/config/linting
- Update project structure checks
Pre-commit hooks will now work with new directory structure.
* Update .gitignore for book/ subdirectory structure
- Update quarto/ paths to book/quarto/
- Update assets/ paths to book/quarto/assets/
- Maintain all existing ignore patterns
* Update GitHub workflows for book/ subdirectory
- Update all quarto/ paths to book/quarto/
- Update cli/ paths to book/cli/
- Update tools/ paths to book/tools/
- Update docker/ paths to book/docker/
- Update config/ paths to book/config/
- Maintain all workflow functionality
* Update CLI config to support book/ subdirectory
- Check for book/quarto/ path first
- Fall back to quarto/ for backward compatibility
- Maintain full CLI functionality
* Create new root and book READMEs for dual structure
- Add comprehensive root README explaining both projects
- Create book-specific README with quick start guide
- Document repository structure and navigation
- Prepare for TinyTorch integration
