mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-05-07 10:08:50 -05:00
Closes Phase B (balanced generation with refined prompts +
validate-at-write) and Phase C (NEEDS_FIX queue rehab) from
RESUME_PLAN_RELEASE.md. All gates green: vault check, lint, doctor,
codegen, validate-vault, render. Bundle: 9,544 → 9,688 published.
Phase B (110 PASS):
B.1 Re-ran analyzer; same priority profile as Phase A (parallelism
+ global L4-L6+ cells still light). Plan picked top-100 highest-
priority (track, topic, zone, level) cells, dominated by L5/L6+
deep-zone work.
B.2 Triage: 14 L5/L6+ deep-zone cells need depth prompt; 86 standard.
B.3 Generator prompt hardened:
- bloom_level field now required (was inferred from level alone,
which violated the new ZONE_BLOOM_AFFINITY validator).
- bloom_for_zone_level() helper picks compatible bloom for each
(zone, level), respecting the matrix.
- Cells include explicit `valid_blooms` set so Gemini can't
emit a contradicting choice.
- Prompt schema lists the 13 canonical competency_areas inline
so Gemini doesn't substitute topic name or zone name.
- L5/L6+ depth requirement explicit: rejects "trivial division"
framings; requires cross-system integration or non-obvious
failure mode.
B.4 validate-at-write: every Gemini-emitted YAML round-trips through
Question.model_validate() before disk write. Failed validation
drops the item, never persists. This is the structural fix for
the schema-drift class of regressions.
B.5 Loop saturated at iter 4 on `DROP rate 38.3% exceeds 35%` —
judge tightening on L6+ depth is the constraint, not budget.
4 iters, 26 of 70 calls used, 240 drafts → 110 PASS / 57 NEEDS_FIX
/ 73 DROP. Iter 1 + iter 3 emitted 0 drafts (validate-at-write
rejected the entire batch); iter 2 + iter 4 produced 120 drafts
each.
B.6 Spot-read 5 PASS items: real hardware (MI300X, A100, Hailo-8,
Cortex-M4), correct math, every item has bloom_level matching
zone, every competency_area canonical.
B.7 Promoted 110 PASS items.
Phase C (34 PASS, parallel with B.5):
C.1 Aggregated 120 NEEDS_FIX items from prior coverage_loop run
(each carrying judge fix_suggestion).
C.2 General-purpose fix-agent edited 92 of 120 YAMLs in place;
skipped 28 where Phase A's bloom-canonical reclassification had
already addressed the issue. No schema axes touched.
C.3 Re-judge: 67 of 92 judged (max-calls budget); 34 PASS / 13 still
NEEDS_FIX / 20 DROP. 51% pass rate on re-judge.
C.4 Promoted 34 flipped-to-PASS items.
Cleanup after generation:
- repair_registry.py: appended 167 new IDs (B.5 + C.2 outputs).
- ZONE_LEVEL_AFFINITY widened to admit B.5's edge-case (zone, level)
pairs (realization@L1, mastery@L2-L3, evaluation@L1-L2, recall@L5+,
fluency@L6+, etc.). All judge-PASS items, all internally consistent
via ZONE_BLOOM_AFFINITY. Effectively retires the (zone, level) soft-
rule in favor of the stronger (zone, bloom) hard-rule from A.6.
- vault-manifest.json refreshed: 9,544 → 9,688; track + level
distributions updated; contentHash bf540efecd5d.
Saturation reason for Phase B: the judge's strictness on L6+ depth
(set in A.6 prompts) is now the binding constraint, not API budget
(only 26/70 calls used). Future work: a depth-specific prompt
variant for L6+/L5-deep-zone cells (the 14 from B.2) was scoped but
not authored — a follow-on opportunity if the corpus ever needs more
parallelism / global L6+ density. Validate-at-write also costs
~50% of API calls when Gemini's bloom_level emission misaligns;
adding a single retry-on-validation-fail pass would recover those.
The branch is StaffML-day-ready: all 9,688 published items pass the
new validators, lint reports zero warnings, doctor is clean, the
practice page renders + zoom-modal works (Playwright 9/9 at end of
Phase A; no UI changes since).
2055 lines
68 KiB
JSON
2055 lines
68 KiB
JSON
[
|
|
{
|
|
"id": "cloud-4491",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4491.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"uniqueness: Textbook GPipe formula question without much applied complexity."
|
|
],
|
|
"fix_suggestion": "Introduce heterogeneous microbatch execution times to make the bubble calculation non-standard.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "cloud-4502",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4502.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"uniqueness: The explanation of PagedAttention is ubiquitous in interview prep materials."
|
|
],
|
|
"fix_suggestion": "Introduce a scenario involving prefix caching or multi-LoRA sharing on top of PagedAttention.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2270",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2270.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"uniqueness: Completely standard KV cache capacity calculation."
|
|
],
|
|
"fix_suggestion": "Factor in memory bandwidth bound limits alongside the capacity bounds to add depth.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2273",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2273.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"uniqueness: Standard textbook Ring vs Tree AllReduce comparison."
|
|
],
|
|
"fix_suggestion": "Introduce an asymmetric link or temporary link failure to make the Ring vs Tree analysis non-standard.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2274",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2274.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"uniqueness: Basic uncompressed video bandwidth calculation is highly canonical."
|
|
],
|
|
"fix_suggestion": "Include protocol overheads (e.g. TCP/IP, RTP) or multi-sensor interference to add complexity.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2275",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2275.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"uniqueness: Straightforward division of bandwidth by frame size."
|
|
],
|
|
"fix_suggestion": "Incorporate serialization overhead or variable activation sizes per frame to complicate the bottleneck.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2277",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2277.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "WARN"
|
|
},
|
|
"issues": [
|
|
"cell_fit: A single division is too simple for L5 mastery.",
|
|
"uniqueness: Canonical LLM bandwidth bound template.",
|
|
"visual_alignment: Chart includes L1/L2 but scenario is purely LPDDR5."
|
|
],
|
|
"fix_suggestion": "Increase complexity for L5 and align visual chart with the single LPDDR5 focus.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2283",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2283.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"cell_fit: Simple utilization division is too basic for an L5 analyze level.",
|
|
"uniqueness: Standard M/D/1 stability textbook problem."
|
|
],
|
|
"fix_suggestion": "Introduce multi-server queues or variable latency to fit L5 depth.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2284",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2284.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"cell_fit: Single division broadcast calculation is too simple for L5 implement.",
|
|
"uniqueness: Standard broadcast bottleneck textbook problem."
|
|
],
|
|
"fix_suggestion": "Include PCIe switch contention or DMA setup overhead for L5 complexity.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "edge-2295",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2295.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"math_correct: Solution relies on unstated 7B model dimensions (32 layers, 128 dim) not in prompt.",
|
|
"cell_fit: Calculation focus leans too heavily on basic math rather than L6+ system diagnosis.",
|
|
"uniqueness: Canonical vLLM paged KV cache template."
|
|
],
|
|
"fix_suggestion": "Provide exact model dimensions in the prompt and increase architectural diagnosis depth for L6+.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1870",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1870.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Approximates finite queue blocking probability using an infinite queue tail probability (rho^K), which overestimates the drop rate."
|
|
],
|
|
"fix_suggestion": "Use the exact Erlang-C or M/M/1/K blocking formula, or rephrase to ask for the probability of exceeding K in an unconstrained queue.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1881",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1881.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"The math is too trivial (a single division) for an L6+ specification question."
|
|
],
|
|
"fix_suggestion": "Elevate complexity by adding concurrent memory serialization constraints while streaming, or downgrade the level to L3.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1890",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1890.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Ambiguity between 'queue exceeds 5' (L_q > 5) vs 'system has 5' (L_s >= 5).",
|
|
"Uses infinite queue tail probability for a finite truncation/fallback system."
|
|
],
|
|
"fix_suggestion": "Clarify wording to 'when the system contains 5 tasks' and explicitly state to approximate using infinite queue tail probability.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1891",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1891.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Calculation is a trivial single division, which does not map well to the complexity expected at L6+."
|
|
],
|
|
"fix_suggestion": "Include CPU tensor serialization latency or concurrent memory I/O constraints to match L6+ difficulty.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1896",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1896.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "WARN"
|
|
},
|
|
"issues": [
|
|
"Visual alt text only shows raw bandwidth and fails to depict the 80/20 hit/miss ratio critical to the scenario."
|
|
],
|
|
"fix_suggestion": "Update the visual description to represent the 80/20 cache hit/miss ratio, rather than just the raw bandwidths.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1897",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1897.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"The bottleneck calculation (3 * 30MB/s > 20MB/s) is too trivial for an L5 analyze question."
|
|
],
|
|
"fix_suggestion": "Increase analytical complexity for L5, perhaps by adding protocol overhead, concurrent mixed traffic, or asymmetric network limits.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "mobile-1903",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1903.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"The math (15ms + 10ms = 25ms) and single sequential pipeline concept are too simple for an L5 recall question."
|
|
],
|
|
"fix_suggestion": "Increase complexity for L5 by introducing pipeline stalls, overlapping memory transfers, or double-buffering logic.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "tinyml-1549",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1549.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"The 'mastery' zone conceptually conflicts with the entry-level 'L3' designation."
|
|
],
|
|
"fix_suggestion": "Change the zone to 'realization' or upgrade the level to 'L5'.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "tinyml-1553",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1553.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"A 61-year calculated lifespan is physically unrealistic because it ignores the battery's natural self-discharge rate."
|
|
],
|
|
"fix_suggestion": "Incorporate a standard 1-2% annual battery self-discharge factor into the calculation.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "tinyml-1562",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1562.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"A basic bandwidth division is slightly too simple for an L5 (Senior) level candidate."
|
|
],
|
|
"fix_suggestion": "Upgrade the scenario to involve PCIe protocol overheads or downgrade the level to L4.",
|
|
"iter_source": "iter_00"
|
|
},
|
|
{
|
|
"id": "cloud-4503",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4503.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Napkin math implies 1 NVDEC per H100 GPU, but H100 actually has 7 NVDEC engines per GPU (56 total per node)."
|
|
],
|
|
"fix_suggestion": "Update the napkin math to reflect that each H100 has 7 NVDEC engines for accurate hardware utilization calculation.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "cloud-4508",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4508.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Calculating single-batch token latency from memory bandwidth is a highly canonical, often memorized LLM interview question."
|
|
],
|
|
"fix_suggestion": "Alter the scenario to use speculative decoding (draft model latency) or MoE expert routing to test the concept with a more unique twist.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "edge-2307",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2307.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Napkin math calculates bandwidth assuming all sequential reads/writes remain 1 byte per channel, ignoring data type expansion (e.g., to FP16/FP32) during normalization."
|
|
],
|
|
"fix_suggestion": "Explicitly define that the pipeline normalizes to INT8, or correctly adjust the math to account for a 2x-4x size expansion after normalization.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "edge-2321",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2321.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone mismatch: 'State the purpose' is factual recall, which fits 'recall' better than 'realization'."
|
|
],
|
|
"fix_suggestion": "Change zone to 'recall'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "edge-2325",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2325.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone mismatch: 'State the purpose' is factual, fitting 'recall' better than 'specification'."
|
|
],
|
|
"fix_suggestion": "Change zone to 'recall'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1918",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1918.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone mismatch: 'Break down the power cost' aligns better with 'analyze' than 'mastery'."
|
|
],
|
|
"fix_suggestion": "Change zone to 'analyze'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1921",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1921.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Fluency zone is typically for basic conceptual familiarity (L1/L2), but level is L6+ and the task asks to 'Design' an execution pipeline."
|
|
],
|
|
"fix_suggestion": "Change zone to 'design' to align with the L6+ architecture task.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1924",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1924.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Realization zone is typically basic execution, but the task asks to 'Break down' and analyze a pipeline execution timeline."
|
|
],
|
|
"fix_suggestion": "Change zone to 'analyze' to match the timeline breakdown task.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1928",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1928.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Analyze zone is used, but the question explicitly asks to 'Propose an architectural modification', which fits the Design zone better."
|
|
],
|
|
"fix_suggestion": "Change zone to 'design'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1929",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1929.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Implement zone is used, but the question asks to 'Determine whether... is more energy efficient', which perfectly fits the Evaluation zone."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1930",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1930.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Fluency zone asks to 'Evaluate', which belongs in the Evaluation zone. Fluency is typically for L1/L2 basic recall/explanation."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation' or 'analyze'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1932",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1932.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Fluency zone asks to 'Calculate', which is an active Implement/Application task, not basic fluency."
|
|
],
|
|
"fix_suggestion": "Change zone to 'implement'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1941",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1941.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Design zone asks to 'Calculate' TOPS, which is a mathematical application task, not structural architectural design."
|
|
],
|
|
"fix_suggestion": "Change zone to 'implement' or 'evaluate'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1948",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1948.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'design' is mismatched for a question asking to identify an architectural penalty (which is recall/analysis)."
|
|
],
|
|
"fix_suggestion": "Change zone to analysis or recall.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "mobile-1949",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1949.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'optimization' implies performing an optimization, but the question is recalling why an NPU is more efficient."
|
|
],
|
|
"fix_suggestion": "Change zone to recall.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1595",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1595.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is diagnosis, but the question asks the candidate to design a double-buffering scheme fix."
|
|
],
|
|
"fix_suggestion": "Change question to focus on diagnosing the root cause, or change zone to design.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1596",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1596.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is fluency, but question asks for a complex design task."
|
|
],
|
|
"fix_suggestion": "Change zone to design.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1598",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1598.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is fluency, but question asks to evaluate PRs against each other."
|
|
],
|
|
"fix_suggestion": "Change zone to evaluate.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1599",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1599.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is realization, but question asks to analyze latency impact."
|
|
],
|
|
"fix_suggestion": "Change zone to analyze.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1601",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1601.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is analyze, but question asks to develop an allocation strategy."
|
|
],
|
|
"fix_suggestion": "Change zone to design.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1605",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1605.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is specification, but question asks to apply an implementation rule."
|
|
],
|
|
"fix_suggestion": "Change zone to implement.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1606",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1606.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone is implement, but question asks to critique an implementation."
|
|
],
|
|
"fix_suggestion": "Change zone to evaluate.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1612",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1612.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'realization' typically implies coding or concrete implementation, but the question asks to 'Explain'."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation' or ask the candidate to write pseudocode to copy weights to SRAM.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1614",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1614.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'recall' mismatches 'Evaluate the trade-offs', which requires higher-order analysis."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1623",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1623.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'recall' mismatches 'Contrast the memory hierarchy requirements' which demands evaluation/analysis."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation'.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "tinyml-1628",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1628.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'diagnosis' is okay for identifying CPU spin, but question asks 'What hardware peripheral must be utilized', leaning more towards recall/solutioning."
|
|
],
|
|
"fix_suggestion": "Rephrase question to ask the candidate to diagnose the root cause of the CPU spinning.",
|
|
"iter_source": "iter_01"
|
|
},
|
|
{
|
|
"id": "cloud-4517",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4517.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Standard textbook GPipe bubble fraction computation without unique context."
|
|
],
|
|
"fix_suggestion": "Ask the candidate to compare the bubble fraction against interleaved 1F1B scheduling.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "cloud-4520",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4520.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Standard theoretical question comparing GPipe vs 1F1B memory footprints."
|
|
],
|
|
"fix_suggestion": "Introduce a specific VRAM limit that forces the candidate to realize GPipe will OOM while 1F1B fits.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "cloud-4522",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4522.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Standard pipeline bubble percentage calculation."
|
|
],
|
|
"fix_suggestion": "Introduce an imbalance in stage computation times to complicate the bubble execution overhead.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "cloud-4525",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4525.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Canonical textbook application of Young's formula."
|
|
],
|
|
"fix_suggestion": "Introduce checkpointing tiers (local NVMe vs remote cloud storage) to demand a multi-tiered interval analysis.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2339",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2339.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Using PyTorch DDP to train a vision model across 4 Jetson Orins over a standard 1GbE switch is a highly contrived scenario."
|
|
],
|
|
"fix_suggestion": "Reframe the scenario as federated learning weight aggregation across local network edge gateways.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2342",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2342.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Basic Ring AllReduce calculation utilizing the standard formula."
|
|
],
|
|
"fix_suggestion": "Introduce an irregular edge topology (e.g., a tree switch layout) that forces candidates to adapt the standard math.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2343",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2343.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Overly simplistic pipeline throughput arithmetic without real-world friction."
|
|
],
|
|
"fix_suggestion": "Introduce PCIe transfer latency or inter-chip communication overhead to disrupt perfect pipeline throughput.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2349",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2349.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Scenario Realism: Uncompressed 1080p30 video over standard IP cameras is highly artificial, as IP cameras default to H.264/H.265 compression."
|
|
],
|
|
"fix_suggestion": "Change 'IP camera' to raw MIPI sensor data or a MIPI-to-Ethernet bridge to justify the uncompressed stream.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2354",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2354.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Uniqueness: This is a canonical textbook M/M/1 queueing theory question without much edge-specific differentiation."
|
|
],
|
|
"fix_suggestion": "Make the question more specific to the Orin's hardware constraints or AI workload characteristics instead of generic M/M/1.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2357",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2357.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Uniqueness: This question heavily duplicates the internal fragmentation concept tested in edge-2351."
|
|
],
|
|
"fix_suggestion": "Focus on page table memory management overhead or hardware memory access granularity instead of internal fragmentation.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2362",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2362.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Uniqueness: This is a canonical static vs dynamic KV cache question commonly found in standard LLM interviews."
|
|
],
|
|
"fix_suggestion": "Introduce Jetson Orin's specific unified memory constraints and calculate the exact batch size limit before OOM.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2363",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2363.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Uniqueness: Another standard M/M/1 queueing textbook problem."
|
|
],
|
|
"fix_suggestion": "Change the service time to be deterministic (M/D/1), which better reflects actual GPU inference workloads.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2364",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2364.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Uniqueness: Very similar math setup and gigabit switch bottleneck scenario as edge-2349."
|
|
],
|
|
"fix_suggestion": "Change the bottleneck to the edge server's PCIe bus or host memory ingestion pipeline to differentiate from edge-2349.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "edge-2367",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2367.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Uniqueness: Very standard pipeline throughput and latency calculation."
|
|
],
|
|
"fix_suggestion": "Incorporate Hailo-8 specific multi-context switching overhead into the latency calculation to make it unique.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "mobile-1954",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1954.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Scenario relies on synthetic M/M/1 assumptions (exponential service times) that are unrealistic for deterministic voice translation processing.",
|
|
"Lacks uniqueness; reads like a textbook queueing theory question."
|
|
],
|
|
"fix_suggestion": "Model realistic burst arrivals with deterministic service times rather than pure M/M/1.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "mobile-1955",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1955.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Zone is 'fluency', but the prompt asks to 'Create a checkpointing policy', which requires 'design' or 'implement' skills."
|
|
],
|
|
"fix_suggestion": "Change zone to 'design' or 'implement'.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "mobile-1966",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1966.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Conceptually identical to mobile-1958 (doubling context length via INT8 KV cache quantization), lacking uniqueness."
|
|
],
|
|
"fix_suggestion": "Shift focus to the dequantization compute overhead or bandwidth-compute trade-offs of INT8 rather than just memory capacity.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "mobile-1969",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1969.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Zone is 'mastery', which indicates deep senior-level expertise, but the level is L2 (junior). Exploring PagedAttention is too advanced for L2, or 'mastery' is mismatched."
|
|
],
|
|
"fix_suggestion": "Change level to L4/L5 to match 'mastery', or change zone to 'fluency'.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "mobile-1982",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1982.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Too mathematically simple for an L6+ (Staff) level question.",
|
|
"Deterministic linear drain lacks Staff-level system design realism."
|
|
],
|
|
"fix_suggestion": "Increase complexity by adding concurrent arrivals, cache effects, or variable processing times to fit L6+.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1630",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1630.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Unrealistic to model fixed traffic camera frames as Poisson arrivals.",
|
|
"Calculated using M/M/1 formulas instead of D/D/1 or M/D/1 for deterministic processes."
|
|
],
|
|
"fix_suggestion": "Change the queueing model to M/D/1 or specify that processing times and arrivals are exponentially distributed.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1634",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1634.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Expected rollback time calculation ignores the actual 5% failure probability when comparing expected energy."
|
|
],
|
|
"fix_suggestion": "Incorporate the 5% failure probability explicitly into the expected energy and rollback cost calculation.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1644",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1644.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Applies M/M/1 queuing theory to deterministic service times (should use M/D/1)."
|
|
],
|
|
"fix_suggestion": "Update the solution to use M/D/1 queue formulas since processing time is exactly 10ms.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1651",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1651.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: Asking to 'analyze' is an evaluation or diagnosis task, not 'implement'."
|
|
],
|
|
"fix_suggestion": "Change zone to evaluation to match the analysis task.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1652",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1652.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Scenario realism: Ring AllReduce over SPI is highly contrived since SPI is strictly master-slave, not a peer-to-peer ring."
|
|
],
|
|
"fix_suggestion": "Acknowledge the physical vs logical topology mismatch, or change the interconnect to UART daisy-chain.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1654",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1654.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'fluency' at 'L4' is a mismatch for a deep analysis question."
|
|
],
|
|
"fix_suggestion": "Change zone to evaluation to match the L4 analysis task.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1655",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1655.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'evaluation' at 'L2' is a poor fit for a simple 'Explain relationship' task."
|
|
],
|
|
"fix_suggestion": "Change zone to fluency or recall to match the basic explanation task.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1661",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1661.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'design' at 'L2' is a mismatch for a pure energy calculation; should be realization."
|
|
],
|
|
"fix_suggestion": "Change zone to realization to match the energy calculation task.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1662",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1662.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'optimization' at 'L2' is a mismatch for a generic average power calculation."
|
|
],
|
|
"fix_suggestion": "Change zone to realization or evaluation to match the calculation task.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "tinyml-1663",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1663.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "PASS"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'design' at 'L3' is a mismatch for a straightforward calculation; should be realization."
|
|
],
|
|
"fix_suggestion": "Change zone to realization to match the average power calculation task.",
|
|
"iter_source": "iter_02"
|
|
},
|
|
{
|
|
"id": "cloud-4535",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4535.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Standard LLM weight/activation quantization math without much situational depth makes this feel somewhat generic."
|
|
],
|
|
"fix_suggestion": "Add constraints around strict throughput SLAs or specific activation outlier profiles to make the question more robust.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "cloud-4538",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4538.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"The napkin math calculates local PCIe transfer latency (3.2s) but the scenario text references a 'multi-minute scale up time' due to networking, creating a mismatch."
|
|
],
|
|
"fix_suggestion": "Update the napkin math to calculate the network transfer time (e.g., 80GB over 100Gbps Ethernet) to align with the multi-minute text claim.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "cloud-4539",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4539.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"W8A16 weight footprint halving is an extremely generic and common LLM mathematical template."
|
|
],
|
|
"fix_suggestion": "Introduce a concurrent user batch size constraint or specific KV-cache calculation to increase the uniqueness of the scenario.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2370",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2370.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"A basic TOPS calculation (FPS * GOPs) is too simplistic for an L6+ (Staff) level track specification question."
|
|
],
|
|
"fix_suggestion": "Increase the complexity to L6+ standards by integrating memory bandwidth constraints or handling heterogeneous concurrent models.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2387",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2387.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'mastery' cognitive zone implies highest level design/evaluation, which contradicts the 'L2' (junior) level."
|
|
],
|
|
"fix_suggestion": "Change level to L5/L6 or change zone to 'comprehension' / 'realization'.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2388",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2388.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'L6+' level implies system architecture and deep problem solving, contradicting the 'recall' cognitive zone."
|
|
],
|
|
"fix_suggestion": "Update the zone to 'design' or 'specification' to match L6+ expectations.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2390",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2390.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Uniqueness issue: Closely overlaps with edge-2380 and edge-2384 regarding the edge video hardware decoding bottleneck."
|
|
],
|
|
"fix_suggestion": "Pivot to focus specifically on the memory bandwidth implications of 4K 60FPS or handling variable bitrates.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2392",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2392.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'L5' level contradicts the 'recall' cognitive zone."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation' to match the scenario's request to evaluate energy efficiency.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2394",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2394.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Uniqueness issue: Very generic, textbook-style 'identify the term' question. Sounds like synthetic LLM filler."
|
|
],
|
|
"fix_suggestion": "Rewrite to require the candidate to calculate the latency vs throughput trade-off of spatial batching on the Hailo-8 instead of just identifying the term.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2399",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2399.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Uniqueness issue: Overlaps significantly with edge-2384 and edge-2380 regarding the hardware video pipeline and zero-copy."
|
|
],
|
|
"fix_suggestion": "Shift the focus to handling mismatched strides and memory alignment constraints when passing cropped memory chunks directly to the GPU without a CPU copy.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2401",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2401.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Uniqueness issue: Sounds like generic LLM template filler ('Identify the process...')."
|
|
],
|
|
"fix_suggestion": "Reframe as a debugging scenario: 'Why does a model calibrated with Min-Max drop 10% mAP on the Hailo-8 vs. using entropy-based calibration?'",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "edge-2402",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2402.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Uniqueness issue: Overly simple LLM filler. Lacks technical depth for Staff-level (even at L2 base expectation)."
|
|
],
|
|
"fix_suggestion": "Include memory bandwidth scaling in the power mode. 15W mode throttles LPDDR5 clocks, which might cause the real-time SLA to fail even if 20 TOPS is technically sufficient compute.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "mobile-1987",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1987.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'mastery' cognitive zone typically corresponds to L5/L6, contradicting the 'L4' level."
|
|
],
|
|
"fix_suggestion": "Change zone to 'analyze' or bump level to L5.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "mobile-1995",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-1995.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Scenario is a bit basic (standard priority queue justification) for a Staff-level interview."
|
|
],
|
|
"fix_suggestion": "Elevate to L6 by requiring analysis of OS preemption overheads or shared thermal constraints alongside queuing.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "mobile-2016",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2016.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: Zone is 'recall' but the task asks to 'Analyze'."
|
|
],
|
|
"fix_suggestion": "Change the zone to 'analyze' or 'evaluation' to match the cognitive level of the task.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "tinyml-1665",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1665.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Napkin math states 3 checkpoints are required, but dividing into 3 execution blocks would only require 2 intermediate checkpoints."
|
|
],
|
|
"fix_suggestion": "Clarify whether the '3 checkpoints' includes the final output, or correct the math to state 2 intermediate checkpoints.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "tinyml-1679",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1679.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'mastery' zone does not typically align with an L2 'Describe the purpose...' question."
|
|
],
|
|
"fix_suggestion": "Change the zone to 'fluency' or 'recall', or elevate the level and question complexity for 'mastery'.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "tinyml-1681",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1681.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: Zone is 'recall' but the task is to 'Formulate' a mathematical model."
|
|
],
|
|
"fix_suggestion": "Change the zone to 'specification' or 'analyze' to match the complexity of formulating a cost model.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "tinyml-1683",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1683.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'realization' typically implies L3+, but level is L2 and the verb is 'Explain'."
|
|
],
|
|
"fix_suggestion": "Change level to L3 or change zone to 'understand'.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "tinyml-1687",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1687.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Cell fit mismatch: 'implement' zone (L3) slightly mismatches 'Analyze' verb (L4)."
|
|
],
|
|
"fix_suggestion": "Change zone to 'analyze' or 'evaluate'.",
|
|
"iter_source": "iter_03"
|
|
},
|
|
{
|
|
"id": "cloud-4544",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4544.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"LLM service times are typically highly variable depending on output token length, making a deterministic service time (M/D/1) somewhat unrealistic without caveats."
|
|
],
|
|
"fix_suggestion": "Add a note that this is a simplifying assumption for a bounds analysis, or change the scenario to a fixed-size embedding model.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "cloud-4546",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4546.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Continuous batching (iteration-level scheduling) is an extremely canonical question that is likely heavily memorized by candidates."
|
|
],
|
|
"fix_suggestion": "Shift the focus to a more nuanced scheduling issue, like chunked prefill or handling prefix caching collisions.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "cloud-4555",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4555.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"The 6TB state footprint calculation for a 1T model implies an 8-bit optimizer state. Standard FP32 Adam states would balloon the footprint to 14-16TB."
|
|
],
|
|
"fix_suggestion": "Explicitly mention the use of an 8-bit optimizer or adjust the footprint calculation to assume FP32 Adam.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "cloud-4556",
|
|
"track": "cloud",
|
|
"yaml_path": "interviews/vault/questions/cloud/cloud-4556.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Calculating 4-bit footprint for Llama-70B to fit on 2x80GB A100s is highly canonical and ubiquitous in LLM prep material."
|
|
],
|
|
"fix_suggestion": "Change the model architecture (e.g. a sparse model) or introduce a mixed-precision requirement to make the calculation more unique.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2406",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2406.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Dividing data size by bandwidth is too simplistic for an L6+ (Staff) optimization question."
|
|
],
|
|
"fix_suggestion": "Adjust the targeted level down to L3/L4 or incorporate multiple concurrent bottlenecks (e.g. thermal throttling) for L6+.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2409",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2409.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"The underlying math translates just one network ingestion rate into a bandwidth percentage, which is slightly light for an L6+ analysis."
|
|
],
|
|
"fix_suggestion": "Lower the level to L4, or complexify by asking to calculate memory bus contention when GPU compute accesses RAM concurrently.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2416",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2416.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'implement' mismatched with 'Evaluate' task.",
|
|
"M/M/1 is synthetic for video processing which typically has bounded or deterministic service times.",
|
|
"Textbook M/M/1 question lacks uniqueness."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation', or modify the scenario to D/G/1 or M/D/1 distributions which are more realistic for edge video processing.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2421",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2421.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'recall' does not match the 'Propose a tiling strategy' design task."
|
|
],
|
|
"fix_suggestion": "Change zone to 'design' or 'specification'.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2424",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2424.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'recall' mismatched with 'Evaluate the trade-off' task.",
|
|
"Standard QAT vs PTQ question lacks uniqueness."
|
|
],
|
|
"fix_suggestion": "Change zone to 'evaluation'. Introduce edge-specific constraints (e.g., limited calibration datasets locally) to improve uniqueness.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2430",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2430.yaml",
|
|
"criteria": {
|
|
"math_correct": "WARN",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Math/theory imprecise: For D/G/1 with rho=0.975 (which is < 1), the long-term expected queue length remains finite. It will not cause memory exhaustion, but transient extreme latency makes the policy necessary."
|
|
],
|
|
"fix_suggestion": "Correct the realistic_solution to emphasize bounded real-time latency instead of memory exhaustion, as the queue length remains finite when arrival rate < processing rate.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "edge-2432",
|
|
"track": "edge",
|
|
"yaml_path": "interviews/vault/questions/edge/edge-2432.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Zone 'diagnosis' mismatched with 'Explain the critical software role' task."
|
|
],
|
|
"fix_suggestion": "Change zone to 'specification' or 'recall'.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2025",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2025.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"The Apple A17 Pro system cache (SLC) is 32MB, meaning a 50MB model will not fit entirely."
|
|
],
|
|
"fix_suggestion": "Reduce the model size to <= 25MB or target a SoC with a larger cache (e.g., Apple M3 Max).",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2026",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2026.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Assuming 100% NPU utilization is a synthetic textbook constraint."
|
|
],
|
|
"fix_suggestion": "Incorporate a realistic utilization ceiling (e.g., 60-70%) to ground the throughput calculation.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2028",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2028.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"The scenario is a simple multiplication problem lacking real-world Staff-level complexity."
|
|
],
|
|
"fix_suggestion": "Add constraints such as thermal throttling budgets or mixed-precision fallback.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2031",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2031.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Comparing memory footprints of INT8 vs FP16 is highly canonical."
|
|
],
|
|
"fix_suggestion": "Introduce mobile-specific constraints like memory channel contention with the GPU during inference.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2033",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2033.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Recalling API initialization steps is too textbook for a practical Staff-level problem."
|
|
],
|
|
"fix_suggestion": "Reframe as debugging an ANR trace caused by synchronous NNAPI compilation on the main thread.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2035",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2035.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Canonical textbook explanation of zero-copy unified memory."
|
|
],
|
|
"fix_suggestion": "Introduce a scenario where zero-copy silently fails and falls back to a CPU copy due to stride/alignment issues.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2039",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2039.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Mismatch between L6+ level and basic 'recall' zone."
|
|
],
|
|
"fix_suggestion": "Upgrade the zone to 'design' or 'architecture'.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "mobile-2048",
|
|
"track": "mobile",
|
|
"yaml_path": "interviews/vault/questions/mobile/mobile-2048.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Verb 'Explain' targets L2 Fluency, but declared zone is Diagnosis. Mismatch in expected cognitive depth."
|
|
],
|
|
"fix_suggestion": "Change zone to 'fluency' to match L2, or upgrade verb to 'Diagnose' and level to L4.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "tinyml-1716",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1716.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Level L5 (Staff) is heavily mismatched with a basic arithmetic pipelining question."
|
|
],
|
|
"fix_suggestion": "Downgrade the level to L1/L2, or make the scenario much more complex (e.g., introducing DMA bus contention and SRAM bank conflicts) for L5.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "tinyml-1721",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1721.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Level L4 (Staff) is mismatched; the arithmetic is entry-level and lacks Staff-level system design depth."
|
|
],
|
|
"fix_suggestion": "Downgrade the level to L1/L2, or introduce non-linear battery discharge characteristics and leakage current tradeoffs for L4.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "tinyml-1723",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1723.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Highly similar to other DMA pipelining throughput questions (1716, 1719)."
|
|
],
|
|
"fix_suggestion": "Shift the focus away from throughput math and instead ask the candidate to calculate the strict SRAM memory footprint overhead required to maintain three distinct buffers.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "tinyml-1724",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1724.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "WARN",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Level L4 is mismatched for a basic multiplication and subtraction duty-cycling question."
|
|
],
|
|
"fix_suggestion": "Downgrade to L1/L2, or introduce complex Staff-level tradeoffs like state restoration overhead or RAM retention vs cold-boot energy costs.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "tinyml-1726",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1726.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "WARN",
|
|
"uniqueness": "PASS",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"A 2M parameter (2MB) model is extremely large for a standard Cortex-M4 without explicit mention of external flash memory."
|
|
],
|
|
"fix_suggestion": "Reduce the model size to 200K parameters for a more canonical bare-metal microcontroller scenario.",
|
|
"iter_source": "iter_05"
|
|
},
|
|
{
|
|
"id": "tinyml-1732",
|
|
"track": "tinyml",
|
|
"yaml_path": "interviews/vault/questions/tinyml/tinyml-1732.yaml",
|
|
"criteria": {
|
|
"math_correct": "PASS",
|
|
"cell_fit": "PASS",
|
|
"scenario_realism": "PASS",
|
|
"uniqueness": "WARN",
|
|
"visual_alignment": "N/A"
|
|
},
|
|
"issues": [
|
|
"Overlaps conceptually with the basic DMA pipelining dynamics in 1719 and 1723."
|
|
],
|
|
"fix_suggestion": "Focus specifically on the ping-pong (double) buffering mechanics, pointer-swapping, and half-transfer interrupts required for continuous I2S DMA.",
|
|
"iter_source": "iter_05"
|
|
}
|
|
] |