Replace pointers to the private project rules/docs tree (relative .claude/rules
and .claude/docs paths) in code comments and docstrings with neutral phrasing
("the project prose style guide", etc.). Load-bearing runtime paths that the
tooling reads or writes are left intact.
- labs-validate-dev.yml: the Pyodide/WASM export smoke test imported the
pre-taxonomy-refactor path `from mlsysim.core.engine import Engine`. The
refactor moved Engine to mlsysim/engine/engine.py (exported at the package
top level). Use the canonical public import `from mlsysim import Engine`.
Fixes the "ModuleNotFoundError: No module named 'mlsysim.core.engine'" failure.
- LIVE_INTERVIEWER_PLAN.md: "pre-selects" -> "preselects" (codespell dictionary).
First-pass topic_chapter_map.yaml (87 topics → book chapters, primary +
also_see + rationale) plus the design analysis for connecting each StaffML
question back to recommended textbook reading. Seeds the future work
tracked in harvard-edge/cs249r_book#1822. Not yet wired into the build.
StaffML is an assessment instrument, not a textbook: recall is a first-class
skill and L1/L2 questions are warm-up/screening (paper sec:levels). A flat
'must reason' bar wrongly rejects valid recall items. Grade the floor by Bloom
level instead — L1/L2 pass at tp>=1, L3 at tp>=2, L4/L5/L6+ at tp>=3 — so the
gate fails only a level/depth MISMATCH (recall masquerading as L5), not recall
doing its job. Re-applied to the backfill: 2,700 below-floor (28%, concentrated
at L4-L6+), down from a misleading flat 43%.
The pipeline validated correctness only (schema/format/level-fit/coherence/math)
— nothing checked whether a question TEACHES, so ~22% of published questions are
correct-but-vacuous (recall/vocab + decorative math). Add gate_teaching_power:
passes iff tp>=3 and not vacuous, calibrated against gold-standard exemplars (not
same-cell peers, the level_fit blind spot that lets vacuity propagate). Validated
on 146 labeled questions: 90% within 1pt of the independent scorer, 9/9 trusted
golds pass. Documented the bar in AUTHORING.md.
The flagged titles were machine-generated slug stubs ("Cloud Gpu Virtualization
L3 0") leaking track prefix, level/index, and wrong acronym case. Regenerated
descriptive, scenario-grounded titles (e.g. "Partitioning an A100 MIG for 7B and
1B Models") via content-aware pass; spot-checked for accuracy against scenarios.
Asserts the actual rendered output (jsdom + testing-library), not just types:
MCQ marks correct-green/wrong-red on reveal; $...$ renders as KaTeX; bare-$
currency renders verbatim (regression guard); bold survives the math-split pass.
The MATH_SPLIT regex matched any $...$ span on a line, so cost questions
using bare-$ currency ("Maintenance ($3M) ... ($50)") rendered the text
between two dollar signs as garbled KaTeX. ~5% of published vault questions
(544/9,525) hit this. Gate math on a LaTeX-signal in the body (backslash,
caret, underscore, brace); currency has none, real math always does.
Verified: 3 currency samples render as text, 3 real-math strings as math.
A1 — MCQ self-check: new MCQOptions component renders selectable choices when a
question carries details.options + correct_index. Pre-reveal the learner picks
an option; on reveal the correct option is marked green and a wrong pick red,
keeping all distractors. Deliberately a self-check — NOT wired into the
SR/scoring pipeline; the open self-rating (Skip/Wrong/Partial/Nailed) stays
authoritative. Rendered in both practice and gauntlet flows. Non-MCQ
open-response path is unchanged.
A4 — scenario/question/answer text now routes through MarkdownText everywhere
(practice scenario was raw GlossaryText; gauntlet scenario, common_mistake,
realistic_solution, and the results-phase review were raw text), so **bold**
and $math$ render instead of showing literal markers.
Grading wiring — the live reveal path now gates numeric grading on
isNumericQuestion(current) (previously ungated) and calls gradeNapkinAnswer,
preserving the setNapkinResult shape and extending the display to show units
and an "→ equivalent" marker for unit-equivalent answers.
MarkdownText is a custom regex renderer (no react-markdown/remark stack), so
KaTeX is integrated by splitting math segments out FIRST — before the bold /
code / number-highlight / glossary passes — so LaTeX bodies are never mangled.
New MathText component calls katex.renderToString (SSR-safe, throwOnError:false)
and is used for both inline ($...$) and display ($$...$$) math. Math now renders
wherever MarkdownText is used: scenario, napkin_math, realistic_solution, and
common_mistake.
Add extractFinalQuantity and gradeNapkinAnswer to corpus.ts, layered on top
of the existing unit-blind extractFinalNumber (whose last-token behavior stays
pinned by tests). js-quantities — the JS equivalent of Pint, already used
server-side in mlsysim — validates unit tokens, handles SI prefixes (mW vs MW),
dimensional analysis, and base conversion.
- extractFinalQuantity scans number+token candidates, parses each with Qty(),
and returns the last one that parses to a real unit (honoring =>/answer:/final:
markers first). Non-unit tokens like "40 layers"/"40 epochs" fail to parse and
are skipped, fixing the last-token bug for unit-bearing answers.
- gradeNapkinAnswer converts compatible quantities to a common base and grades
via checkNapkinMath (logic unchanged); incompatible dimensions return a
way_off "Wrong quantity (a vs b)" result; otherwise it falls back to the
legacy bare-number path. Display values + units are returned for the UI.
Adds js-quantities + @types/js-quantities. (KaTeX, added in the same manifest
bump, is consumed by the math-rendering change that follows.) Ten new tests
cover ms/µs equivalence, GB/s vs GB incompatibility, mW vs MW prefix safety,
"1.31 MB across 40 layers", marker lines, and bare-number fallback.
- commOverheadPct now divides the overlap-discounted communication time (the
part that actually lands in iterTimeMs) by iterTimeMs, instead of dividing the
raw allreduce time by an iteration time that only counted 70% of it.
- Fix the inline comment that claimed 14 bytes/param for Adam optimizer state;
the code uses 12 (fp32 master + momentum + variance), the standard breakdown.
- Replace the redundant Math.min(gpusPerNode, ...) with the equivalent ternary
it always collapsed to (no numeric change), with a clarifying comment.
- parseJsonRequest enforces the per-endpoint size cap against the actual decoded
body length, not just the client-supplied Content-Length header (which can be
omitted or chunked away). Shared across /ask, /interview, and /waitlist.
- checkRateLimit fails CLOSED (limiter_unavailable -> 503) when RATE_LIMIT_KV is
missing, and /interview now always rate-limits. Previously /interview skipped
limiting entirely when the binding was absent (unbounded LLM spend on the
highest-token path) while /ask threw and 500'd — an inconsistent split.
The effect keyed on summary?.id but closed over summary, so a same-id record
swapped for a richer one merged against a stale value (and eslint-react would
flag the deps). Read summary through a ref kept current each render: re-fetch
only on id change, but always merge against the latest summary.
- Route question hydration and worker search through the shared transport and
centralized config; corpus-provider's /manifest probe + SW handshake now use
navigator.serviceWorker.ready so the SET_VAULT_API_ORIGIN message isn't
dropped while the worker is still installing.
- Track hydrated ids explicitly instead of inferring from a truthy
realistic_solution, so recall/MCQ questions with an empty solution stop
re-fetching from the worker on every access.
- QUESTION_COUNT_DISPLAY now derives from the authoritative manifest count with
magnitude-aware rounding, so a partial bundle can no longer render '0+'.
- checkNapkinMath divides by |modelAnswer| (with a zero guard) instead of
max(modelAnswer, 1), so sub-unit answers are graded on true relative error.
- extractFinalNumber requires a leading digit, so a stray comma no longer
parses to 0 and becomes a bogus 'final answer'.
Covered by napkin-grading.test.ts and question-count-display.test.ts.
vault-api.ts (VaultApiClient: retry + circuit breaker, built 'to close
Soumith R3-F-2') was imported by nothing — every real fetch used a bare
fetch() with no retry, breaker, or release header, and its typed
getQuestion() assumed a nested-details response the deployed worker does
not return. Delete it and add:
- vault-config.ts: single source of truth for data mode + worker base URL,
so the React provider and the non-React fetchers agree (production, with
the env unset, now resolves to the worker default instead of 'no worker').
- vault-fetch.ts: one shape-agnostic transport with per-attempt timeout,
bounded retry on transient failures, the X-Vault-Release header, and a
per-origin circuit breaker with correct single-probe half-open semantics
(the orphaned client admitted every concurrent caller as a probe).
Covered by vault-fetch.test.ts (7 tests).
Question hydration (getQuestionFullDetail) now catches errors so the
interview proceeds even when the vault worker is unavailable. Chain
navigation calls also wrapped with tryHydrate helper.
Active phase layout: reduced padding, h-screen overflow-hidden on main
to prevent double scrollbars with the ecosystem nav bars.
Welcome page now leads with "Start a Study Path" hero card pointing to
/plans, followed by a 3-card explainer (Vault/Practice/Mock Interview)
that explains what each section does. "Try one random question" and
"Skip to Vault" moved to compact fallback links at the bottom.
Nav promotes Study Plans from secondary Tools dropdown to primary nav
between Practice and Mock Interview.
Extends the Study Plans system with user-built paths. Users pick a
track, optionally focus on one competency area, choose a starting level
(default L1), and optionally set an interview target date. Questions
are ordered level-by-level with topic-alphabetical sorting within each
level. Resume-where-you-left-off works via the existing PlanProgress
system. Interview prep mode shows daily quota, days remaining, and
on-track/behind status via PrepDashboard. Custom paths are included
in progress export/import/clear.
Adds getTopicProgressMap() to progress.ts computing per-topic completion
with L-level breakdown across all 87 topics and 13 competency areas.
Progress page now shows collapsible area sections with individual topic
bars showing correct/total, highest mastered level, and color-coded
progress. Legacy heatmap moved behind a collapsible toggle. Header
counters show topics touched and total corpus size.
Maps 13 competency areas to MLSysBook chapter slugs. After scoring a
question in Practice or Study Plans, users see "Read more" links to
relevant textbook chapters on mlsysbook.ai. Chapter-level links only
for now (section anchors can be added when URLs stabilize).
Step 6: /interview route with setup phase. Track/level/duration selectors,
optional competency area focus, session creation via createSession().
Nav link added (Mic2 icon) between Mock Interview and Progress.
Step 7: Active phase with chat transcript, composer (Cmd+Enter to send),
timer (count-up with duration target), area coverage chips, collapsible
NapkinCalc + HardwareRef tools. Chain navigation on AI response: advance,
retreat, switch area, or conclude based on conductor metadata.
Step 8: Feedback phase with overall score (0-100), per-area rating bars,
strengths/weaknesses lists, practice recommendations with question links,
expandable full transcript, New Interview / Progress actions.
Step 10: FirstRunExplainer interview mode with 4 bullets.
Step 11: Session resume handling with localStorage persistence and
resume/start-new modal on page reload during active interview.
Results saved to localStorage (staffml_interviews key, capped at 100).
Analytics events fired at started/completed transitions.
Five items scoped: onboarding flow, chapter linking, sequential
L-level paths, NeetCode-style progress tracking, interview prep mode.
Three-phase build order with dependencies mapped. Data model additions
and open questions documented for review.
Step 1: Four chain selection functions in corpus.ts for the interview
conductor: getChainsByArea, getChainsByTopic, getChainsForInterview,
getChainEntryPoint. New ChainSummary type enriches ChainInfo with topic,
area, and zone per member. Built from existing _chainIndex at module load.
Step 2: interview-types.ts defines the full type contract between client
and worker: InterviewConfig, InterviewSession, TranscriptEntry,
ConductorMeta (intent + nextAction), ConductorResponse, AreaAssessment,
InterviewReport, PracticeRecommendation, InterviewRequestPayload.
5 new vitest tests verify chain selection returns valid results for
cloud/L3-L5, primary tier sorts first, area filtering works, and entry
point selection finds the target level.
Updated the live interviewer plan with:
- Full inventory of existing assets to leverage: 843 chains (2,853
questions in curated difficulty progressions), gauntlet phase machine,
worker LLM routing (6 providers), spaced repetition system, 14 reusable
components, napkin math checking, glossary (831 terms)
- Chain-first question selection: the AI navigates chains for depth
(follow-ups are real vault questions with canonical solutions) and
switches chains for breadth (coverage-driven area transitions)
- Chain-aware system prompt: zone progressions guide cognitive moves
(fluency -> diagnosis -> evaluation), level progressions enable
zoom in/out based on candidate performance
- Updated Phase 1 scope to reuse gauntlet phase machine, existing
components, and SR integration
Comprehensive design document covering goals, data model, architecture,
system prompt design, UI wireframes, and phased implementation plan.
Based on feedback from Peter (EAIF/NMC WG) about how real staff-level
embedded ML interviews work: conversational, unstructured, zoom in/out,
napkin math together, diagram descriptions, acronym-friendly.
Key design decisions: stateless worker, client-side question pool
selection (30 hydrated questions per session), single-prompt conductor
(evaluate + select + generate per turn), intent-based dialogue grammar.
Three fixes addressing user feedback from Peter (EAIF/NMC WG):
Visual questions: SVGs were never mirrored from vault/visuals/ to
public/question-visuals/, causing 404s on all platforms (not Chrome-specific).
Added mirror-visuals.sh standalone script and wired it as a fallback in
build-local-corpus.mjs so npm run dev always populates the 236 SVGs.
Glossary tooltips: Parsed 831 terms from MLSysBook vol1/vol2 glossaries
into glossary.json. New GlossaryText component annotates acronyms (92
matchable patterns) with hover tooltips showing expansion and definition.
Wired into scenario text and MarkdownText so questions, solutions, and
common-mistake text all get inline acronym hover. Uses existing MetaTooltip
component (accessible, pure CSS, keyboard-focusable). First occurrence
only per render to avoid clutter.
Rubric quality: Replaced naive first-3-sentences extraction with a scoring
system that prioritizes sentences containing technical terms, causal
reasoning, and quantitative claims. Better filler removal and smarter
common-mistake sentence selection.
The mobile hamburger menu rendered inside a position:sticky parent
with overflow-y:visible, so when its content exceeded the viewport
the overflow items sat below the fold and a swipe scrolled the page
instead of the menu. Items only became reachable once the sticky
bar released at the body bottom.
Cap the expanded menu at calc(100dvh - 60px) with overflow-y:auto
and overscroll-behavior:contain so the menu owns its own scroll and
doesn't chain to the page.
Vijay Janapa Reddi
github-actions[bot]
dependabot[bot]andGitHub
farhan
