mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-03-11 19:03:34 -05:00
Clean up temporary review and analysis files
Remove temporary files from repository: - COMPREHENSIVE_MODULE_REVIEW.md - Module review (temporary) - paper/CITATIONS_TO_ADD.md - Citation recommendations (temporary) - paper/CLAIM_EVIDENCE_MATRIX.md - Evidence validation (temporary) - paper/EVIDENCE_INVENTORY.md - Evidence tracking (temporary) - paper/LITERATURE_REVIEW_ASSESSMENT.md - Literature review (temporary) - paper/PYTHON_DEVELOPER_TECHNICAL_REVIEW.md - Code review (temporary) - paper/NEW_CITATIONS.bib - Temporary citations (content in references.bib) - paper/proposed_figures.tex - Figure proposals (temporary) Update .gitignore to prevent tracking these file types: - Add patterns for *_REVIEW*, *_MATRIX, *_INVENTORY, *_ASSESSMENT - Add NEW_CITATIONS.bib and proposed_figures.tex patterns - These files are AI-generated temporary analysis artifacts Update paper.pdf with latest compilation including caption styling. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Vijay Janapa Reddi
@@ -1,327 +0,0 @@
|
||||
# TinyTorch: Citations to Add - Quick Reference
|
||||
|
||||
## CRITICAL FIXES (Must do before submission)
|
||||
|
||||
### 1. Fix Corrupted Bib Entries
|
||||
|
||||
**bruner1960process** - Currently wrong paper about "Narrative Approach and Mentalization"
|
||||
|
||||
Replace with:
|
||||
```bibtex
|
||||
@book{bruner1960process,
|
||||
author = {Bruner, Jerome S.},
|
||||
title = {The Process of Education},
|
||||
year = {1960},
|
||||
publisher = {Harvard University Press},
|
||||
address = {Cambridge, MA}
|
||||
}
|
||||
```
|
||||
|
||||
OR use the original scaffolding paper:
|
||||
```bibtex
|
||||
@article{wood1976role,
|
||||
author = {Wood, David and Bruner, Jerome S. and Ross, Gail},
|
||||
title = {The role of tutoring in problem solving},
|
||||
journal = {Journal of Child Psychology and Psychiatry},
|
||||
volume = {17},
|
||||
number = {2},
|
||||
pages = {89--100},
|
||||
year = {1976},
|
||||
doi = {10.1111/j.1469-7610.1976.tb00381.x}
|
||||
}
|
||||
```
|
||||
|
||||
**perkins1992transfer** - Currently wrong paper about "infant mortality"
|
||||
|
||||
Replace with:
|
||||
```bibtex
|
||||
@incollection{perkins1992transfer,
|
||||
author = {Perkins, David N. and Salomon, Gavriel},
|
||||
title = {Transfer of Learning},
|
||||
booktitle = {International Encyclopedia of Education},
|
||||
edition = {2nd},
|
||||
year = {1992},
|
||||
publisher = {Pergamon Press},
|
||||
address = {Oxford, England}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL ADDITIONS (Strongly recommended)
|
||||
|
||||
### 2. Systems Thinking Foundation
|
||||
|
||||
```bibtex
|
||||
@book{meadows2008thinking,
|
||||
author = {Meadows, Donella H.},
|
||||
title = {Thinking in Systems: A Primer},
|
||||
year = {2008},
|
||||
publisher = {Chelsea Green Publishing},
|
||||
address = {White River Junction, VT}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to cite:** When using "systems thinking" terminology - add to Related Work Section 3.2
|
||||
|
||||
**Suggested text addition:**
|
||||
> "TinyTorch's incremental system construction develops systems thinking~\citep{meadows2008thinking}—the ability to understand how components interact within complex systems—through direct implementation rather than abstract instruction."
|
||||
|
||||
### 3. Compiler Course Model
|
||||
|
||||
```bibtex
|
||||
@book{aho2006compilers,
|
||||
author = {Aho, Alfred V. and Lam, Monica S. and Sethi, Ravi and Ullman, Jeffrey D.},
|
||||
title = {Compilers: Principles, Techniques, and Tools},
|
||||
edition = {2nd},
|
||||
year = {2006},
|
||||
publisher = {Addison-Wesley},
|
||||
address = {Boston, MA}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to cite:** When comparing to "compiler course model" - Introduction and Related Work
|
||||
|
||||
**Suggested text addition:**
|
||||
> "The curriculum follows compiler course pedagogy~\citep{aho2006compilers}: students build a complete system module-by-module, experiencing how components integrate through direct implementation."
|
||||
|
||||
### 4. Operating Systems Pedagogy
|
||||
|
||||
```bibtex
|
||||
@book{arpaci2018operating,
|
||||
author = {Arpaci-Dusseau, Remzi H. and Arpaci-Dusseau, Andrea C.},
|
||||
title = {Operating Systems: Three Easy Pieces},
|
||||
year = {2018},
|
||||
publisher = {Arpaci-Dusseau Books},
|
||||
url = {http://pages.cs.wisc.edu/~remzi/OSTEP/}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to cite:** When discussing "build the whole stack" approach
|
||||
|
||||
**Suggested text addition in Related Work:**
|
||||
> "TinyTorch's incremental system construction draws pedagogical inspiration from compiler courses~\citep{aho2006compilers} and operating systems courses~\citep{arpaci2018operating}, where students build complete systems to develop systems thinking~\citep{meadows2008thinking} through component integration."
|
||||
|
||||
---
|
||||
|
||||
## HIGH-VALUE ADDITIONS
|
||||
|
||||
### 5. Active Learning Foundation
|
||||
|
||||
```bibtex
|
||||
@article{freeman2014active,
|
||||
author = {Freeman, Scott and Eddy, Sarah L. and McDonough, Miles and Smith, Michelle K. and Okoroafor, Nnadozie and Jordt, Hannah and Wenderoth, Mary Pat},
|
||||
title = {Active learning increases student performance in science, engineering, and mathematics},
|
||||
journal = {Proceedings of the National Academy of Sciences},
|
||||
volume = {111},
|
||||
number = {23},
|
||||
pages = {8410--8415},
|
||||
year = {2014},
|
||||
doi = {10.1073/pnas.1319030111}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to cite:** When justifying hands-on, build-from-scratch approach
|
||||
|
||||
**Suggested text addition:**
|
||||
> "This constructionist, hands-on pedagogy aligns with meta-analytic evidence that active learning significantly improves student performance in STEM education compared to traditional lecture-based instruction~\citep{freeman2014active}."
|
||||
|
||||
### 6. Academic Workforce Citations (Replace Robert Half/Keller)
|
||||
|
||||
```bibtex
|
||||
@article{brynjolfsson2017machine,
|
||||
author = {Brynjolfsson, Erik and Mitchell, Tom},
|
||||
title = {What can machine learning do? Workforce implications},
|
||||
journal = {Science},
|
||||
volume = {358},
|
||||
number = {6370},
|
||||
pages = {1530--1534},
|
||||
year = {2017},
|
||||
doi = {10.1126/science.aap8062}
|
||||
}
|
||||
|
||||
@article{ransbotham2020expanding,
|
||||
author = {Ransbotham, Sam and Kiron, David and Gerbert, Philipp and Reeves, Martin},
|
||||
title = {Expanding AI's Impact with Organizational Learning},
|
||||
journal = {MIT Sloan Management Review},
|
||||
volume = {61},
|
||||
number = {4},
|
||||
year = {2020}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to use:** Introduction, replace or supplement Robert Half/Keller citations
|
||||
|
||||
**Suggested reframe:**
|
||||
> "Machine learning systems engineering requires tacit knowledge that resists automation~\citep{brynjolfsson2017machine}: understanding why Adam requires 2× optimizer state memory, when attention's O(N²) scaling becomes prohibitive, how to navigate accuracy-latency-memory tradeoffs. While workforce demand for ML systems skills has grown substantially~\citep{ransbotham2020expanding}, traditional ML education separates algorithms from systems..."
|
||||
|
||||
### 7. Progressive Disclosure HCI/Learning
|
||||
|
||||
```bibtex
|
||||
@article{mayer2009multimedia,
|
||||
author = {Mayer, Richard E. and Moreno, Roxana},
|
||||
title = {Nine ways to reduce cognitive load in multimedia learning},
|
||||
journal = {Educational Psychologist},
|
||||
volume = {38},
|
||||
number = {1},
|
||||
pages = {43--52},
|
||||
year = {2003},
|
||||
doi = {10.1207/S15326985EP3801_6}
|
||||
}
|
||||
```
|
||||
|
||||
OR
|
||||
|
||||
```bibtex
|
||||
@book{nielsen1993usability,
|
||||
author = {Nielsen, Jakob},
|
||||
title = {Usability Engineering},
|
||||
year = {1993},
|
||||
publisher = {Academic Press},
|
||||
address = {Boston, MA}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to cite:** When discussing progressive disclosure pattern as contribution
|
||||
|
||||
**Suggested text addition:**
|
||||
> "Progressive disclosure—revealing complexity incrementally to manage cognitive load~\citep{mayer2009multimedia}—addresses this through runtime feature activation..."
|
||||
|
||||
---
|
||||
|
||||
## CITE WHAT YOU ALREADY HAVE
|
||||
|
||||
### 8. Bloom's Taxonomy for Assessment
|
||||
|
||||
**Already in bib:** `thompson2008bloom`
|
||||
|
||||
**Where to cite:** Section 4.4 (NBGrader Assessment) or when discussing assessment levels
|
||||
|
||||
**Suggested text addition:**
|
||||
> "NBGrader tests progress from lower-order skills (recall tensor operations) to higher-order skills (analyze gradient flow, evaluate optimization tradeoffs), following Bloom's taxonomy for computing education~\citep{thompson2008bloom}."
|
||||
|
||||
### 9. Vygotsky - Scaffolding/ZPD
|
||||
|
||||
**Already in bib:** `vygotsky1978mind`
|
||||
|
||||
**Where to cite:** Related Work section when discussing scaffolding, or when discussing progressive difficulty
|
||||
|
||||
**Suggested text addition:**
|
||||
> "Module prerequisites create scaffolding~\citep{vygotsky1978mind}, ensuring students work within their zone of proximal development: challenging enough to require new learning, familiar enough to build on mastered concepts."
|
||||
|
||||
---
|
||||
|
||||
## RECOMMENDED SEARCH QUERIES
|
||||
|
||||
### For Recent ML Education Work (2023-2024)
|
||||
|
||||
**Google Scholar:**
|
||||
- `"machine learning education" 2023 2024 SIGCSE`
|
||||
- `"teaching machine learning" 2023 2024 ICER`
|
||||
- `"ML systems" education 2023 2024`
|
||||
|
||||
**ACM Digital Library:**
|
||||
- Search SIGCSE 2023, 2024 proceedings for: "machine learning" OR "deep learning"
|
||||
- Search ICER 2023, 2024 proceedings for: "ML" OR "framework"
|
||||
|
||||
**arXiv:**
|
||||
- `cat:cs.CY "machine learning education" 2023`
|
||||
|
||||
**Target:** Find 2-3 recent papers showing awareness of current ML education research
|
||||
|
||||
---
|
||||
|
||||
## OPTIONAL BUT VALUABLE
|
||||
|
||||
### 10. Notional Machines
|
||||
|
||||
```bibtex
|
||||
@phdthesis{sorva2013notional,
|
||||
author = {Sorva, Juha},
|
||||
title = {Notional Machines and Introductory Programming Education},
|
||||
school = {Aalto University},
|
||||
year = {2013},
|
||||
type = {Doctoral dissertation}
|
||||
}
|
||||
```
|
||||
|
||||
**Where to cite:** When discussing "mental models of framework internals"
|
||||
|
||||
**Suggested addition:**
|
||||
> "Students develop notional machines~\citep{sorva2013notional}—mental models of framework internals—through implementation: how tensors store gradients, how autograd builds computational graphs, how optimizers manage state."
|
||||
|
||||
### 11. JAX Discussion in Related Work
|
||||
|
||||
**Already have bradbury2018jax in bib but not discussed**
|
||||
|
||||
**Suggested addition to Related Work:**
|
||||
> "\textbf{JAX}~\citep{bradbury2018jax} offers an alternative functional paradigm through composable transformations (\texttt{jax.grad}, \texttt{vmap}). While pedagogically valuable for understanding functional programming applied to ML, JAX assumes framework usage rather than framework construction, positioning it complementary to TinyTorch's build-from-scratch approach."
|
||||
|
||||
---
|
||||
|
||||
## PRIORITY ORDER
|
||||
|
||||
### Must Do (1-2 hours):
|
||||
1. Fix bruner1960process (corrupted)
|
||||
2. Fix perkins1992transfer (corrupted)
|
||||
3. Add meadows2008thinking (systems thinking)
|
||||
4. Add aho2006compilers (compiler course model)
|
||||
5. Cite vygotsky1978mind (already in bib)
|
||||
6. Cite thompson2008bloom (already in bib)
|
||||
|
||||
### Should Do (2-3 hours):
|
||||
7. Search SIGCSE/ICER 2023-2024 for recent work
|
||||
8. Add brynjolfsson2017machine + ransbotham2020expanding (replace industry citations)
|
||||
9. Add freeman2014active (active learning)
|
||||
10. Add arpaci2018operating (OS course pedagogy)
|
||||
|
||||
### Could Do (1-2 hours):
|
||||
11. Add mayer2009multimedia or nielsen1993usability (progressive disclosure)
|
||||
12. Add sorva2013notional (notional machines)
|
||||
13. Add JAX discussion in related work
|
||||
|
||||
---
|
||||
|
||||
## BIBTEX FILE CLEANUP
|
||||
|
||||
**Remove these if not cited in final paper:**
|
||||
- Any uncited technical papers (baydin2018automatic, etc.)
|
||||
- Duplicate or superseded entries
|
||||
|
||||
**Keep these even if lightly cited:**
|
||||
- Historical ML papers (rosenblatt1958perceptron, rumelhart1986learning, lecun1998gradient) - important for milestone validation
|
||||
- Technical framework papers (pytorch04release, tensorflow20) - important for historical context
|
||||
|
||||
---
|
||||
|
||||
## WHERE TO ADD IN PAPER
|
||||
|
||||
### Introduction (Lines 174-180):
|
||||
- Replace/supplement Robert Half/Keller with Brynjolfsson, Ransbotham
|
||||
- Add compiler/OS course references when comparing to "compiler course model"
|
||||
|
||||
### Related Work - Educational Frameworks (Lines 377-388):
|
||||
- Add JAX discussion
|
||||
- Cite recent ML education work if found
|
||||
|
||||
### Related Work - Learning Theory (Lines 389-400):
|
||||
- Cite Vygotsky when discussing scaffolding
|
||||
- Add Mayer/Nielsen for progressive disclosure grounding
|
||||
- Add Freeman for active learning
|
||||
|
||||
### Related Work - New Subsection (INSERT):
|
||||
Add new subsection "Systems Pedagogy Foundations":
|
||||
```latex
|
||||
\subsection{Systems Pedagogy Foundations}
|
||||
|
||||
TinyTorch's incremental system construction draws pedagogical inspiration from compiler courses~\citep{aho2006compilers} and operating systems courses~\citep{arpaci2018operating}, where students build complete systems (compilers from lexer to code generator, OS kernels from process management to file systems) to develop systems thinking~\citep{meadows2008thinking} through component integration. This "build the whole stack" approach has proven effective for teaching complex systems concepts in CS education.
|
||||
```
|
||||
|
||||
### Section 4.4 NBGrader (Around line 820):
|
||||
- Cite thompson2008bloom when discussing assessment levels
|
||||
|
||||
---
|
||||
|
||||
**Total estimated time for all Must Do + Should Do items: 3-5 hours**
|
||||
|
||||
**Expected impact on citation quality: 6.5/10 → 8.5/10**
|
||||
@@ -1,250 +0,0 @@
|
||||
# TinyTorch Paper: Claim-Evidence Matrix
|
||||
|
||||
Quick reference: Every major claim mapped to evidence strength and required action.
|
||||
|
||||
---
|
||||
|
||||
## How to Read This Matrix
|
||||
|
||||
- **Status Icons:**
|
||||
- ✅ **STRONG** - Can defend to reviewers, keep as-is
|
||||
- ⚠️ **MEDIUM** - Evidence exists but needs hedging/clarification
|
||||
- ❌ **WEAK** - No evidence, remove or significantly hedge
|
||||
|
||||
- **Action Codes:**
|
||||
- **KEEP** - Claim is well-supported
|
||||
- **HEDGE** - Add qualifiers (hypothesized, estimated, designed to)
|
||||
- **VERIFY** - Check source and update
|
||||
- **REMOVE** - No supporting evidence, delete claim
|
||||
|
||||
---
|
||||
|
||||
## QUANTITATIVE CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| 3:1 supply/demand ratio | Line 176 | Industry report (keller2025ai) | ❌ WEAK | VERIFY or REMOVE |
|
||||
| 150,000 practitioners | Line 176 | Industry report (roberthalf2024talent) | ❌ WEAK | REMOVE |
|
||||
| 78% job posting growth | Line 176 | Industry report | ❌ WEAK | REMOVE |
|
||||
| 40-50% executives cite shortage | Line 176 | Industry report (keller2025ai) | ❌ WEAK | VERIFY or REMOVE |
|
||||
| Adam 2× optimizer state | Lines 176, 359, 481 | Mathematical derivation | ✅ STRONG | KEEP |
|
||||
| Adam 4× total training memory | Lines 176, 359, 740 | Mathematical derivation | ✅ STRONG | KEEP |
|
||||
| Conv2d 109× parameter efficiency | Lines 354, 359, 537 | Calculated: 896 vs 98,336 | ✅ STRONG | KEEP |
|
||||
| MNIST 180 MB | Line 734 | Calculation: 60k×784×4 = 188 MB | ✅ STRONG | KEEP |
|
||||
| ImageNet 670 GB | Line 734 | Calculation: 1.2M×224²×3×4 = 722 GB | ✅ STRONG | KEEP |
|
||||
| GPT-3 training 2.6 TB | Line 740 | Calculation: 175B×4×4 = 2.8 TB | ✅ STRONG | KEEP |
|
||||
| CIFAR conv 241M ops | Line 775 | Calculation: 128×32×28²×3×5² | ✅ STRONG | KEEP |
|
||||
| 20 modules | Lines 169, 180, 354 | Codebase: 20 directories exist | ✅ STRONG | KEEP |
|
||||
| 60-80 hours curriculum | Lines 169, 352 | No empirical data | ❌ WEAK | ADD "estimated" |
|
||||
| 2-3 weeks bootcamp | Line 352 | Contradicts 60-80hrs (implies 80-120) | ❌ WEAK | REMOVE or RECONCILE |
|
||||
| 283 NBGrader cells | Implicit | Verified in codebase | ✅ STRONG | KEEP (if mentioned) |
|
||||
| 167 test files | Not mentioned | Only 1 found in codebase | ❌ WEAK | REMOVE claim |
|
||||
|
||||
---
|
||||
|
||||
## TECHNICAL CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| Progressive disclosure reduces cognitive load | Lines 361, 717 | Grounded in Sweller (1988), already hedged | ⚠️ MEDIUM | KEEP (hedged as hypothesis) |
|
||||
| Systems-first improves learning | Lines 359, 730 | Design exists, no comparison data | ⚠️ MEDIUM | HEDGE as "enables" not "demonstrates" |
|
||||
| Historical milestones validate correctness | Lines 167, 270, 354 | Templates exist, no student completion data | ❌ WEAK | CHANGE to "provides validation targets" |
|
||||
| Monkey-patching enables progressive disclosure | Lines 354, 598-642 | Implementation exists in codebase | ✅ STRONG | KEEP |
|
||||
| Module 01 has dormant gradient features | Lines 361, 603-609 | Code exists in tensor_dev.py | ✅ STRONG | KEEP |
|
||||
| PyTorch 0.4 Variable/Tensor merger parallel | Lines 395, 723 | Cited pytorch04release | ✅ STRONG | KEEP |
|
||||
| 10-100× speedup from vectorization | Lines 501, 542, 775 | Plausible given Table 1 data | ⚠️ MEDIUM | KEEP (comparative claim) |
|
||||
| TinyTorch 90-424× slower than PyTorch | Table 1, Line 777 | If measured, strong; if estimated, hedge | ⚠️ MEDIUM | ADD measurement methodology footnote |
|
||||
|
||||
---
|
||||
|
||||
## PEDAGOGICAL CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| Constructionism supports build-from-scratch | Line 393 | Cited Papert (1980) | ✅ STRONG | KEEP |
|
||||
| Cognitive apprenticeship via modeling | Line 395 | Cited Collins et al. (1989) | ✅ STRONG | KEEP |
|
||||
| Productive failure pedagogy | Lines 397, 777 | Cited Kapur (2008) | ✅ STRONG | KEEP |
|
||||
| Threshold concepts (autograd, memory) | Line 399 | Cited Meyer & Land (2003) | ✅ STRONG | KEEP |
|
||||
| Situated cognition via building | Line 730 | Cited Lave & Wenger (1991) | ✅ STRONG | KEEP |
|
||||
| Compiler course model analogy | Line 272 | No citation for compiler pedagogy | ⚠️ MEDIUM | ADD citation or soften |
|
||||
| Students transition users → engineers | Lines 180, 352 | Design goal, no assessment | ❌ WEAK | HEDGE as "aims to" |
|
||||
| Makes tacit knowledge explicit | Lines 180, 359 | Design goal, no knowledge tests | ❌ WEAK | HEDGE as "designed to" |
|
||||
| Validates via milestone recreation | Lines 167, 270, 569 | Milestones are templates, not proven | ❌ WEAK | CHANGE to "provides targets" |
|
||||
| Three integration models work | Lines 352, 817-826 | Design described, deployment unproven | ⚠️ MEDIUM | HEDGE as "supports" not "validated" |
|
||||
|
||||
---
|
||||
|
||||
## ARTIFACT CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| 20 modules implemented | Throughout | All 20 directories exist in codebase | ✅ STRONG | KEEP |
|
||||
| NBGrader infrastructure complete | Lines 363, 841-882 | 283 solution cells, metadata exists | ✅ STRONG | KEEP |
|
||||
| Infrastructure unvalidated at scale | Line 882 | Acknowledged in paper | ✅ STRONG | KEEP (honest scoping) |
|
||||
| PyTorch-inspired package architecture | Lines 363, 885-906 | nbdev exports, import structure exists | ✅ STRONG | KEEP |
|
||||
| TinyDigits dataset exists | Line 535 | Directory exists in datasets/ | ✅ STRONG | KEEP |
|
||||
| TinyTalks dataset exists | Line 535 | Directory exists in datasets/ | ✅ STRONG | KEEP |
|
||||
| Datasets <50MB combined | Line 535 | Needs verification | ⚠️ MEDIUM | VERIFY size with du -sh |
|
||||
| Datasets offline-first | Line 535 | Design intent, verifiable | ✅ STRONG | KEEP |
|
||||
| 6 historical milestones (1958-2024) | Lines 571-586 | 6 milestone directories exist | ✅ STRONG | KEEP |
|
||||
| Students achieve 95% MNIST | Line 532 | Target accuracy, not proven student result | ⚠️ MEDIUM | CLARIFY as "target" |
|
||||
| Students achieve 75% CIFAR-10 | Line 537 | Target accuracy, not proven student result | ⚠️ MEDIUM | CLARIFY as "target" |
|
||||
| 70 years of ML history | Lines 167, 270, 354 | 1958-2024 = 66 years, close enough | ✅ STRONG | KEEP (acceptable rounding) |
|
||||
| Milestones use only student code | Lines 167, 569 | Design intent, unverified | ⚠️ MEDIUM | HEDGE as "designed to use" |
|
||||
|
||||
---
|
||||
|
||||
## LEARNING OUTCOME CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| Students learn systems thinking | Lines 270, 359 | Curriculum structure, no assessment | ❌ WEAK | HEDGE as "designed to teach" |
|
||||
| Improves production readiness | Line 359 | No job placement or skill transfer data | ❌ WEAK | HEDGE or REMOVE |
|
||||
| Better than algorithm-only approach | Implied | No controlled comparison | ❌ WEAK | REMOVE or explicitly state "requires comparison" |
|
||||
| Cognitive load reduction | Lines 361, 717 | Already hedged as hypothesis | ⚠️ MEDIUM | KEEP (properly hedged) |
|
||||
| Transfer to PyTorch/TensorFlow | Lines 352, 1023 | Design intent, no transfer tasks | ❌ WEAK | HEDGE as "should transfer" |
|
||||
| Students complete curriculum | Implied | No completion tracking | ❌ WEAK | ACKNOWLEDGE as unknown |
|
||||
| Memory reasoning becomes automatic | Line 532 | Pedagogical goal, no assessment | ❌ WEAK | HEDGE as "aim" |
|
||||
|
||||
---
|
||||
|
||||
## DEPLOYMENT CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| Three integration models | Lines 817-826 | Design documented, deployment unproven | ⚠️ MEDIUM | KEEP with "supports" not "validated" |
|
||||
| Self-paced learning (primary) | Line 821 | Design described, usage unknown | ⚠️ MEDIUM | KEEP as design, note validation needed |
|
||||
| Institutional integration | Line 823 | Model described, no adoptions known | ⚠️ MEDIUM | KEEP as option, not proven |
|
||||
| Team onboarding | Line 825 | Model described, no company usage | ⚠️ MEDIUM | KEEP as potential use |
|
||||
| CPU-only accessibility | Lines 832-835 | Design verified in codebase | ✅ STRONG | KEEP |
|
||||
| 4GB RAM requirement | Line 835 | Design intent, should verify | ⚠️ MEDIUM | VERIFY via measurement |
|
||||
| Works on Chromebooks | Line 835 | Plausible given Python-only | ⚠️ MEDIUM | KEEP with "should work" |
|
||||
| NBGrader scalability projections | Lines 867-868 | Projections, not measured | ⚠️ MEDIUM | KEEP as "projected" |
|
||||
| 30 students: 10 min grading | Line 867 | Projection, not empirical | ⚠️ MEDIUM | Already hedged as "projected" |
|
||||
| 1000+ students: 2hr turnaround | Line 867 | Projection, not empirical | ⚠️ MEDIUM | Already hedged as "projected" |
|
||||
|
||||
---
|
||||
|
||||
## COMPARISON CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| vs. micrograd: Complete scope | Lines 379, 438 | micrograd is ~200 lines, TinyTorch is 20 modules | ✅ STRONG | KEEP |
|
||||
| vs. MiniTorch: Systems emphasis | Lines 381, 439 | Different pedagogical focus documented | ✅ STRONG | KEEP |
|
||||
| vs. tinygrad: Scaffolded pedagogy | Lines 383, 440 | Design differences clear | ✅ STRONG | KEEP |
|
||||
| vs. CS231n: Cumulative framework | Lines 385, 441 | CS231n has isolated assignments | ✅ STRONG | KEEP |
|
||||
| vs. d2l.ai: Framework construction | Lines 387, 441 | d2l.ai uses frameworks, doesn't build | ✅ STRONG | KEEP |
|
||||
| vs. fast.ai: Bottom-up approach | Lines 387, 442 | fast.ai is top-down, documented | ✅ STRONG | KEEP |
|
||||
| Table 1: Framework comparison | Lines 407-433 | All frameworks exist and verifiable | ✅ STRONG | KEEP |
|
||||
|
||||
---
|
||||
|
||||
## FUTURE WORK CLAIMS
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| Fall 2025 empirical validation planned | Lines 366, 1004 | Stated intent | ⚠️ MEDIUM | KEEP as plan, not commitment |
|
||||
| Controlled studies planned | Line 1004 | Research plan described | ⚠️ MEDIUM | KEEP as plan |
|
||||
| Cognitive load measurement planned | Line 717 | Methodology mentioned | ⚠️ MEDIUM | KEEP as plan |
|
||||
| Transfer task assessment planned | Line 1004 | Future work described | ⚠️ MEDIUM | KEEP as plan |
|
||||
| Maintenance commitment through 2027 | Line 929 | Author commitment | ✅ STRONG | KEEP (personal commitment) |
|
||||
|
||||
---
|
||||
|
||||
## SCOPE LIMITATIONS (Already Well-Documented)
|
||||
|
||||
| Claim | Location | Evidence | Status | Action |
|
||||
|-------|----------|----------|--------|--------|
|
||||
| No GPU programming covered | Lines 967-969 | Explicitly stated | ✅ STRONG | KEEP |
|
||||
| No distributed training | Lines 969 | Explicitly stated | ✅ STRONG | KEEP |
|
||||
| No production deployment | Lines 969-970 | Explicitly stated | ✅ STRONG | KEEP |
|
||||
| CPU-only pedagogical choice | Lines 973-974 | Justified | ✅ STRONG | KEEP |
|
||||
| 100-1000× slower than PyTorch | Lines 777, 979 | Acknowledged trade-off | ✅ STRONG | KEEP |
|
||||
| NBGrader unvalidated at scale | Line 882 | Honestly acknowledged | ✅ STRONG | KEEP |
|
||||
| Learning outcomes unproven | Line 977 | Should be more prominent | ⚠️ MEDIUM | MOVE to Introduction |
|
||||
| Materials in English only | Line 983 | Acknowledged limitation | ✅ STRONG | KEEP |
|
||||
|
||||
---
|
||||
|
||||
## SUMMARY STATISTICS
|
||||
|
||||
### By Evidence Strength:
|
||||
- ✅ **STRONG (Keep as-is):** 45 claims (62%)
|
||||
- ⚠️ **MEDIUM (Hedge/verify):** 20 claims (28%)
|
||||
- ❌ **WEAK (Remove/rewrite):** 7 claims (10%)
|
||||
|
||||
### By Required Action:
|
||||
- **KEEP:** 45 claims (62%)
|
||||
- **HEDGE:** 13 claims (18%)
|
||||
- **VERIFY:** 4 claims (6%)
|
||||
- **REMOVE:** 3 claims (4%)
|
||||
- **CLARIFY:** 7 claims (10%)
|
||||
|
||||
### Critical Action Items (Must Fix):
|
||||
1. Workforce statistics (4 claims) - VERIFY or REMOVE
|
||||
2. Milestone validation language (3 claims) - CHANGE to "provides targets"
|
||||
3. Time estimates (2 claims) - ADD "estimated"
|
||||
4. Systems-first effectiveness (1 claim) - HEDGE as "enables"
|
||||
|
||||
### High Priority (Should Fix):
|
||||
1. Learning outcome claims (6 claims) - HEDGE as design goals
|
||||
2. Deployment model effectiveness (3 claims) - CLARIFY as options
|
||||
3. Performance measurements (1 claim) - ADD methodology
|
||||
|
||||
---
|
||||
|
||||
## HOW TO USE THIS MATRIX
|
||||
|
||||
### Before Submission:
|
||||
1. Review all ❌ **WEAK** claims - these MUST be fixed
|
||||
2. Check all ⚠️ **MEDIUM** claims - hedge appropriately
|
||||
3. Verify ✅ **STRONG** claims haven't been overstated
|
||||
|
||||
### During Revision:
|
||||
1. Use "Action" column to know what to do
|
||||
2. Cross-reference "Location" to find exact lines
|
||||
3. Check "Evidence" column to understand why change is needed
|
||||
|
||||
### For Reviewer Response:
|
||||
1. Point to ✅ **STRONG** claims when defending contributions
|
||||
2. Acknowledge ⚠️ **MEDIUM** items as future work
|
||||
3. Show you've removed ❌ **WEAK** unsupported claims
|
||||
|
||||
---
|
||||
|
||||
## QUICK DECISION GUIDE
|
||||
|
||||
**When you see this claim type → Do this:**
|
||||
|
||||
| Claim Type | Has Evidence? | Action |
|
||||
|------------|---------------|--------|
|
||||
| Mathematical calculation | Yes (formula) | KEEP - cite calculation |
|
||||
| Code implementation | Yes (in codebase) | KEEP - cite GitHub |
|
||||
| Learning theory | Yes (peer-reviewed citation) | KEEP - cite paper |
|
||||
| Design feature | Yes (implemented) | KEEP - describe design |
|
||||
| Pedagogical effectiveness | No (not measured) | HEDGE - "designed to," "aims to" |
|
||||
| Learning outcome | No (no assessment) | HEDGE - "hypothesized," "should" |
|
||||
| Student performance | No (no data) | REMOVE or mark as "target" |
|
||||
| Industry statistic | Maybe (verify source) | VERIFY source or REMOVE |
|
||||
| Time estimate | No (no tracking) | ADD "estimated" |
|
||||
| Future plan | N/A (it's a plan) | KEEP with "planned" language |
|
||||
|
||||
---
|
||||
|
||||
## FINAL CHECKLIST
|
||||
|
||||
Before submission, verify:
|
||||
- [ ] All ❌ claims fixed (removed or hedged)
|
||||
- [ ] All workforce statistics verified or removed
|
||||
- [ ] "60-80 hours" includes "estimated"
|
||||
- [ ] Milestone "validation" → "validation targets"
|
||||
- [ ] Systems-first "demonstrates" → "enables"
|
||||
- [ ] Learning outcomes hedged as design goals
|
||||
- [ ] No unverified performance numbers
|
||||
- [ ] Limitations visible in Introduction
|
||||
|
||||
**When all boxes checked:** Ready to submit ✅
|
||||
|
||||
---
|
||||
|
||||
This matrix provides the evidence foundation for all revision documents. Use it as the source of truth for claim verification.
|
||||
@@ -1,314 +0,0 @@
|
||||
# TinyTorch Paper: Evidence Inventory
|
||||
|
||||
**What We Can Prove vs. What We're Claiming**
|
||||
|
||||
---
|
||||
|
||||
## ✅ STRONG EVIDENCE (Can Defend to Reviewers)
|
||||
|
||||
### Technical Calculations (All Verified)
|
||||
|
||||
| Claim | Evidence | Status |
|
||||
|-------|----------|--------|
|
||||
| Adam 2× optimizer state | momentum + variance = 2× model params | ✅ Mathematically verified |
|
||||
| Adam 4× total training memory | weights + grads + momentum + variance | ✅ Mathematically verified |
|
||||
| Conv2d 109× parameter efficiency | 896 params vs 98,336 params | ✅ Calculated and verified |
|
||||
| MNIST: ~180 MB | 60,000 × 784 × 4 = 188 MB | ✅ Within rounding error |
|
||||
| ImageNet: ~670 GB | 1.2M × 224×224×3 × 4 = 722.5 GB | ✅ Within rounding error |
|
||||
| GPT-3 training: ~2.6 TB | 175B × 4 × 4 = 2.8 TB | ✅ Within rounding error |
|
||||
| CIFAR conv: 241M ops | 128×32×28×28×3×5×5 = 241,228,800 | ✅ Exact |
|
||||
|
||||
**Reviewer Defense:** "All memory and complexity calculations are mathematically derived and verified against standard formulas."
|
||||
|
||||
---
|
||||
|
||||
### Implementation Artifacts (All Exist)
|
||||
|
||||
| Claim | Evidence | Verification Command | Status |
|
||||
|-------|----------|---------------------|--------|
|
||||
| 20 modules implemented | 20 directories in modules/ | `ls -1 modules/\|grep ^[0-9]\|wc -l` | ✅ 20 found |
|
||||
| NBGrader infrastructure | 283 solution cells | `grep -r "BEGIN SOLUTION" modules/\|wc -l` | ✅ 283 found |
|
||||
| Progressive disclosure code | Dormant features in Module 01 | `modules/01_tensor/tensor_dev.py:606-609` | ✅ Implemented |
|
||||
| PyTorch-inspired package | nbdev export directives | `grep "default_exp" modules/*/\*.py` | ✅ Found |
|
||||
| TinyDigits dataset | Dataset directory exists | `ls datasets/tinydigits/` | ✅ Exists |
|
||||
| TinyTalks dataset | Dataset directory exists | `ls datasets/tinytalks/` | ✅ Exists |
|
||||
| Milestone templates | 6 milestone directories | `ls milestones/0*/` | ✅ 6 found |
|
||||
|
||||
**Reviewer Defense:** "All claimed infrastructure is publicly available and documented at github.com/harvard-edge/TinyTorch"
|
||||
|
||||
---
|
||||
|
||||
### Learning Theory Grounding (Well-Cited)
|
||||
|
||||
| Claim | Evidence | Citation | Status |
|
||||
|-------|----------|----------|--------|
|
||||
| Cognitive load theory | Cited Sweller (1988) | Line 717, references.bib:51 | ✅ Peer-reviewed |
|
||||
| Constructionism | Cited Papert (1980) | Line 393, references.bib:366 | ✅ Peer-reviewed |
|
||||
| Cognitive apprenticeship | Cited Collins et al. (1989) | Line 395, references.bib:104 | ✅ Peer-reviewed |
|
||||
| Productive failure | Cited Kapur (2008) | Line 397, references.bib:382 | ✅ Peer-reviewed |
|
||||
| Threshold concepts | Cited Meyer & Land (2003) | Line 399, references.bib:397 | ✅ Peer-reviewed |
|
||||
| Situated learning | Cited Lave & Wenger (1991) | Line 730, references.bib:92 | ✅ Peer-reviewed |
|
||||
|
||||
**Reviewer Defense:** "Pedagogical design grounded in established CS education research with peer-reviewed citations."
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ WEAK EVIDENCE (Needs Hedging or Removal)
|
||||
|
||||
### Workforce Statistics (Cannot Verify)
|
||||
|
||||
| Claim | Citation | Problem | Status |
|
||||
|-------|----------|---------|--------|
|
||||
| 3:1 supply/demand ratio | keller2025ai | Industry report, not peer-reviewed | ❌ Unverifiable |
|
||||
| 150,000 practitioners worldwide | roberthalf2024talent | Specific number without source quote | ❌ Unverifiable |
|
||||
| 78% job posting growth | roberthalf2024talent | No page number or quote provided | ❌ Unverifiable |
|
||||
| 40-50% executives cite shortage | keller2025ai | Range suggests uncertainty | ❌ Unverifiable |
|
||||
|
||||
**Reviewer Challenge:** "These are industry marketing materials, not research. Can you cite peer-reviewed workforce studies?"
|
||||
|
||||
**Recommendation:** Remove specific numbers, keep general statement:
|
||||
```latex
|
||||
Industry surveys identify demand-supply imbalances for ML systems
|
||||
engineers~\citep{roberthalf2024talent,keller2025ai}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Time Estimates (No Empirical Data)
|
||||
|
||||
| Claim | Evidence | Problem | Status |
|
||||
|-------|----------|---------|--------|
|
||||
| 60-80 hours curriculum | NONE | No student tracking data | ❌ Unsupported |
|
||||
| 2-3 weeks bootcamp | NONE | Contradicts 60-80 hours (implies 80-120hrs) | ❌ Inconsistent |
|
||||
|
||||
**Reviewer Challenge:** "What data supports these time estimates? How many students completed the curriculum?"
|
||||
|
||||
**Recommendation:** Add "estimated based on pilot testing"
|
||||
|
||||
---
|
||||
|
||||
### Learning Outcomes (Design Goals, Not Proven Results)
|
||||
|
||||
| Claim | Evidence | Problem | Status |
|
||||
|-------|----------|---------|--------|
|
||||
| "Students transition from users to engineers" | Curriculum design | No pre/post assessment | ❌ Unproven outcome |
|
||||
| "Makes tacit knowledge explicit" | Module structure | No knowledge transfer tests | ❌ Design goal |
|
||||
| "Validates correctness through milestones" | Milestone templates exist | No student completion data | ❌ Overstated |
|
||||
| "Reduces cognitive load" | Already hedged as hypothesis | Properly scoped | ✅ Acceptable hedging |
|
||||
|
||||
**Reviewer Challenge:** "How do you know students learn better with this approach? Where's the comparison data?"
|
||||
|
||||
**Recommendation:** Change to design goals rather than proven outcomes:
|
||||
- "aims to transition students"
|
||||
- "designed to make tacit knowledge explicit"
|
||||
- "provides validation targets through milestones"
|
||||
|
||||
---
|
||||
|
||||
## 🔍 MISSING EVIDENCE (Should Collect for Future Paper)
|
||||
|
||||
### Student Usage Data
|
||||
- ❌ Number of students who completed curriculum
|
||||
- ❌ Completion rate per module
|
||||
- ❌ Drop-off points (which modules students abandon)
|
||||
- ❌ Time per module (actual measurements)
|
||||
- ❌ Background characteristics (ML experience, programming proficiency)
|
||||
|
||||
### Milestone Achievement Data
|
||||
- ❌ Percentage achieving target accuracies (95% MNIST, 75% CIFAR)
|
||||
- ❌ Common implementation bugs (qualitative failure analysis)
|
||||
- ❌ Debugging time per milestone
|
||||
- ❌ Success rate: students who attempt vs. complete milestones
|
||||
|
||||
### Learning Outcome Assessments
|
||||
- ❌ Pre/post knowledge tests
|
||||
- ❌ Transfer tasks (debugging PyTorch code with TinyTorch knowledge)
|
||||
- ❌ Comparison with control group (traditional ML course students)
|
||||
- ❌ Cognitive load measurements (dual-task, self-report scales)
|
||||
- ❌ Six-month retention follow-up
|
||||
|
||||
### Deployment Evidence
|
||||
- ❌ Number of institutions using curriculum
|
||||
- ❌ Student enrollment numbers
|
||||
- ❌ TA/instructor feedback
|
||||
- ❌ Integration model effectiveness (self-paced vs. institutional)
|
||||
|
||||
**Timeline:** Fall 2025 deployment can collect this data
|
||||
|
||||
---
|
||||
|
||||
## 📊 EVIDENCE STRENGTH BY CLAIM TYPE
|
||||
|
||||
### Mathematical/Technical Claims: 95% Strong
|
||||
- All calculations verified
|
||||
- Code implementations exist
|
||||
- Can reproduce all numbers
|
||||
- **Action:** None needed, these are solid
|
||||
|
||||
### Infrastructure Claims: 90% Strong
|
||||
- Modules, datasets, NBGrader all exist
|
||||
- Publicly available and verifiable
|
||||
- Package structure documented
|
||||
- **Action:** Verify dataset sizes, clarify test count
|
||||
|
||||
### Learning Theory Claims: 85% Strong
|
||||
- Well-cited peer-reviewed sources
|
||||
- Design grounded in established research
|
||||
- Properly hedged (progressive disclosure as "hypothesized")
|
||||
- **Action:** Ensure consistent hedging throughout
|
||||
|
||||
### Pedagogical Effectiveness Claims: 30% Strong
|
||||
- Design exists and is well-documented
|
||||
- No empirical validation of learning outcomes
|
||||
- Time estimates unsubstantiated
|
||||
- Milestone "validation" overstated
|
||||
- **Action:** Hedge as design goals, not proven results
|
||||
|
||||
### Workforce Motivation Claims: 20% Strong
|
||||
- Based on industry reports, not research
|
||||
- Cannot verify specific statistics
|
||||
- May not be appropriate for academic paper
|
||||
- **Action:** Remove specifics or verify sources
|
||||
|
||||
---
|
||||
|
||||
## 🎯 WHAT REVIEWERS WILL ACCEPT
|
||||
|
||||
### Acceptable Claims (Evidence Exists)
|
||||
✅ "We implemented a 20-module curriculum"
|
||||
✅ "Progressive disclosure uses monkey-patching for runtime activation"
|
||||
✅ "Adam requires 2× optimizer state (momentum + variance)"
|
||||
✅ "Conv2d achieves 109× parameter efficiency over dense layers"
|
||||
✅ "Design grounded in cognitive load theory~\citep{sweller1988}"
|
||||
✅ "Curriculum provides historical milestone templates"
|
||||
✅ "NBGrader infrastructure enables automated assessment"
|
||||
|
||||
### Questionable Claims (Needs Hedging)
|
||||
⚠️ "Students transition from users to engineers" → "aims to transition"
|
||||
⚠️ "Validates correctness through milestones" → "provides validation targets"
|
||||
⚠️ "60-80 hours completion time" → "estimated 60-80 hours"
|
||||
⚠️ "Makes tacit knowledge explicit" → "designed to make explicit"
|
||||
|
||||
### Unacceptable Claims (Remove or Verify)
|
||||
❌ "3:1 supply/demand ratio" (cannot verify)
|
||||
❌ "150,000 practitioners worldwide" (cannot verify)
|
||||
❌ "78% job posting growth" (cannot verify)
|
||||
❌ "Students recreate 70 years of ML history" (milestones are templates, not proven)
|
||||
|
||||
---
|
||||
|
||||
## 📝 RECOMMENDED EVIDENCE LANGUAGE
|
||||
|
||||
### For Unverified Claims:
|
||||
**DON'T SAY:**
|
||||
- "X demonstrates that..."
|
||||
- "This proves..."
|
||||
- "Evidence shows..."
|
||||
- "Validates that..."
|
||||
|
||||
**DO SAY:**
|
||||
- "X is designed to..."
|
||||
- "We hypothesize that..."
|
||||
- "This approach aims to..."
|
||||
- "Preliminary observations suggest..." (if you have pilot data)
|
||||
|
||||
### For Future Work:
|
||||
**DON'T SAY:**
|
||||
- "Will be tested in Fall 2025"
|
||||
|
||||
**DO SAY:**
|
||||
- "Empirical validation planned for Fall 2025 deployment"
|
||||
- "Requires controlled studies comparing to traditional approaches"
|
||||
- "Future work will measure..."
|
||||
|
||||
---
|
||||
|
||||
## 🔬 EVIDENCE QUALITY TIERS
|
||||
|
||||
### Tier 1: Mathematical/Reproducible Evidence
|
||||
- Anyone can verify these claims
|
||||
- Examples: Conv2d 109×, Adam 4×, memory calculations
|
||||
- **Strength:** Unassailable
|
||||
|
||||
### Tier 2: Implemented Artifacts
|
||||
- Reviewers can inspect code
|
||||
- Examples: 20 modules, NBGrader cells, milestone templates
|
||||
- **Strength:** Strong (publicly verifiable)
|
||||
|
||||
### Tier 3: Cited Learning Theory
|
||||
- Grounded in peer-reviewed research
|
||||
- Examples: Cognitive load theory, constructionism
|
||||
- **Strength:** Acceptable (design justification)
|
||||
|
||||
### Tier 4: Design Claims
|
||||
- Infrastructure exists but effectiveness unproven
|
||||
- Examples: Integration models, progressive disclosure
|
||||
- **Strength:** Acceptable if hedged as design goals
|
||||
|
||||
### Tier 5: Learning Outcome Claims
|
||||
- No empirical validation yet
|
||||
- Examples: "Students learn better," "Reduces cognitive load"
|
||||
- **Strength:** Weak (requires hedging or future work framing)
|
||||
|
||||
### Tier 6: External Statistics
|
||||
- Industry reports, not research
|
||||
- Examples: Workforce numbers
|
||||
- **Strength:** Very weak (verify or remove)
|
||||
|
||||
---
|
||||
|
||||
## 🎓 FINAL GUIDANCE
|
||||
|
||||
### What This Paper CAN Claim:
|
||||
1. "We designed and implemented a complete 20-module ML systems curriculum"
|
||||
2. "The design is grounded in established learning theory (X, Y, Z)"
|
||||
3. "Progressive disclosure is a novel pedagogical pattern for ML education"
|
||||
4. "Systems-first integration differs from traditional algorithm-focused curricula"
|
||||
5. "All infrastructure is open-source and publicly available"
|
||||
6. "The curriculum provides historical milestone templates for validation"
|
||||
|
||||
### What This Paper CANNOT (Yet) Claim:
|
||||
1. "Students learn better with this approach" (no comparison data)
|
||||
2. "Curriculum takes 60-80 hours" (no timing data)
|
||||
3. "Students successfully recreate ML history" (no completion data)
|
||||
4. "Progressive disclosure reduces cognitive load" (no measurements)
|
||||
5. "Specific workforce shortage statistics" (cannot verify sources)
|
||||
|
||||
### Paper Positioning:
|
||||
**This is a design contribution with empirical validation planned, not a learning outcomes study with proven effectiveness.**
|
||||
|
||||
Frame as:
|
||||
- "We present a curriculum design..."
|
||||
- "This approach is hypothesized to..."
|
||||
- "Future work will empirically validate..."
|
||||
|
||||
NOT as:
|
||||
- "We prove that..."
|
||||
- "Results show that..."
|
||||
- "Students demonstrate improved..."
|
||||
|
||||
---
|
||||
|
||||
## 📋 EVIDENCE COLLECTION PRIORITY
|
||||
|
||||
### Before Submission (Critical):
|
||||
1. ✅ Verify or remove workforce statistics
|
||||
2. ✅ Hedge learning outcome claims
|
||||
3. ✅ Clarify milestone templates vs. validation
|
||||
4. ✅ Add "estimated" to time claims
|
||||
|
||||
### For Fall 2025 (High Priority):
|
||||
1. ⏳ Student completion tracking
|
||||
2. ⏳ Time-per-module measurements
|
||||
3. ⏳ Milestone achievement rates
|
||||
4. ⏳ Pre/post knowledge assessments
|
||||
|
||||
### For Future Research (Medium Priority):
|
||||
1. ⏳ Cognitive load experiments
|
||||
2. ⏳ Transfer task assessments
|
||||
3. ⏳ Comparison with control groups
|
||||
4. ⏳ Long-term retention studies
|
||||
|
||||
---
|
||||
|
||||
**Bottom Line:** You have strong evidence for what you built. You have weak evidence for how well it works. Frame accordingly.
|
||||
@@ -1,787 +0,0 @@
|
||||
# TinyTorch Literature Review Assessment
|
||||
**Dr. James Patterson - Research Literature Expert**
|
||||
|
||||
**Date:** 2025-11-18
|
||||
**Paper Analyzed:** `/Users/VJ/GitHub/TinyTorch/paper/paper.tex`
|
||||
**References:** `/Users/VJ/GitHub/TinyTorch/paper/references.bib`
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The TinyTorch paper demonstrates **solid core coverage** of educational ML frameworks and learning theory, but has **critical gaps** in three areas that could hurt in peer review:
|
||||
|
||||
1. **CORRUPTED BIBLIOGRAPHY ENTRIES** - Two citations have completely wrong metadata (bruner1960, perkins1992)
|
||||
2. **Missing Recent Work** - No 2023-2024 ML systems education research cited
|
||||
3. **Weak Industry Evidence** - Workforce gap claims rely solely on non-academic sources
|
||||
|
||||
**Overall Assessment:** 7/10 citation strategy. Strong pedagogical grounding, fair competitive positioning, but needs 5-7 strategic additions and 2 critical fixes before submission.
|
||||
|
||||
---
|
||||
|
||||
## 1. Related Work Coverage
|
||||
|
||||
### 1.1 Educational ML Frameworks - STRONG ✅
|
||||
|
||||
**What's Cited:**
|
||||
- micrograd (Karpathy 2022) - autograd minimalism
|
||||
- MiniTorch (Rush 2020) - Cornell Tech curriculum
|
||||
- tinygrad (Hotz 2023) - inspectable production system
|
||||
- d2l.ai (Zhang 2021) - comprehensive algorithmic foundations
|
||||
- fast.ai (Howard 2020) - top-down layered API approach
|
||||
|
||||
**Assessment:**
|
||||
- All major direct competitors cited ✅
|
||||
- Comparisons are **fair and strategic** - acknowledge strengths while positioning TinyTorch's unique systems-first angle
|
||||
- micrograd comparison is accurate (scalar-only, no systems focus)
|
||||
- MiniTorch comparison is respectful and differentiating (math rigor vs. systems-first)
|
||||
- tinygrad comparison correctly identifies scaffolding gap
|
||||
|
||||
**What's MISSING:**
|
||||
- **JAX educational materials** - Only cited as code (bradbury2018jax) but not discussed in related work despite JAX being increasingly used for educational purposes (functional paradigm, explainability)
|
||||
- **nnfs.io** (Neural Networks from Scratch) - Harrison Kinsley's book/course, similar bottom-up approach but focuses more on algorithms than systems
|
||||
- **PyTorch tutorials evolution** - The official PyTorch tutorials have become quite pedagogical, worth brief mention
|
||||
|
||||
**Recommendation:**
|
||||
- Add brief mention of JAX's functional approach in Related Work:
|
||||
> "JAX~\citep{bradbury2018jax} offers an alternative functional paradigm through composable transformations, teaching automatic differentiation via `jax.grad` and vectorization through `vmap`. While pedagogically valuable for understanding functional programming applied to ML, JAX assumes framework usage rather than framework construction, positioning it complementary to but distinct from TinyTorch's build-from-scratch approach."
|
||||
|
||||
### 1.2 University Courses - GOOD BUT INCOMPLETE ⚠️
|
||||
|
||||
**What's Cited:**
|
||||
- Stanford CS231n (Johnson 2016) - CNNs course with NumPy assignments
|
||||
- CMU DL Systems (Chen 2022) - production ML systems course
|
||||
- Harvard TinyML (Banbury 2021) - embedded deployment focus
|
||||
|
||||
**Assessment:**
|
||||
- CS231n citation is appropriate - isolated exercises vs. cumulative framework
|
||||
- CMU DL Systems positioned correctly as "advanced follow-on" to TinyTorch
|
||||
- **TinyML comparison is EXCELLENT** - clear differentiation (edge deployment vs. framework internals)
|
||||
|
||||
**What's MISSING:**
|
||||
- **Berkeley CS182/282A** (Deep Learning) - Widely adopted course, some from-scratch assignments
|
||||
- **Full Stack Deep Learning** (FSDL) - Production ML course, natural comparison point
|
||||
- **MIT 6.S191** - Introductory deep learning, large MOOC audience
|
||||
- **Other Cornell courses** beyond MiniTorch
|
||||
|
||||
**Verdict:** Adequate coverage of major courses. CS231n and CMU DL Systems are the most important, both cited. Berkeley/MIT would strengthen "landscape coverage" but not critical.
|
||||
|
||||
---
|
||||
|
||||
## 2. Learning Theory Grounding
|
||||
|
||||
### 2.1 Pedagogical Citations - STRONG BUT UNBALANCED ✅⚠️
|
||||
|
||||
**What's Cited:**
|
||||
- **Constructionism** - Papert 1980 ✅
|
||||
- **Cognitive Apprenticeship** - Collins 1989 ✅ (cited 3x, most frequent learning theory)
|
||||
- **Cognitive Load** - Sweller 1988 ✅
|
||||
- **Productive Failure** - Kapur 2008 ✅ (cited 3x)
|
||||
- **Threshold Concepts** - Meyer 2003 ✅
|
||||
- **Situated Learning** - Lave & Wenger 1991 ✅
|
||||
|
||||
**Assessment:**
|
||||
This is **exceptionally strong grounding** in learning theory. The six theories cited cover:
|
||||
- Knowledge construction (Papert, Collins)
|
||||
- Cognitive constraints (Sweller, Kapur)
|
||||
- Conceptual transformation (Meyer)
|
||||
- Contextual learning (Lave & Wenger)
|
||||
|
||||
**Citation frequencies reveal emphasis:**
|
||||
- Cognitive Apprenticeship (3x) - heavily emphasized
|
||||
- Productive Failure (3x) - heavily emphasized
|
||||
- Others (1x each) - mentioned but less central
|
||||
|
||||
**CRITICAL PROBLEMS:**
|
||||
|
||||
1. **CORRUPTED bruner1960process entry:**
|
||||
```bibtex
|
||||
@article{bruner1960process,
|
||||
author = {Frolli, A and Cerciello, F...}, # WRONG AUTHORS
|
||||
title = {Narrative Approach and Mentalization.}, # WRONG TITLE
|
||||
year = {1960}, # But doi is from 2023!
|
||||
```
|
||||
This appears to be searching for Bruner's scaffolding work but got wrong paper. **FIX IMMEDIATELY** - should cite:
|
||||
- Bruner, J. S. (1960). "The Process of Education" - classic scaffolding work
|
||||
- OR Wood, Bruner, Ross (1976). "The role of tutoring in problem solving" - original scaffolding paper
|
||||
|
||||
2. **CORRUPTED perkins1992transfer entry:**
|
||||
```bibtex
|
||||
@article{perkins1992transfer,
|
||||
author = {Burstein, R and Henry, NJ...}, # WRONG - this is about infant mortality
|
||||
title = {Mapping 123 million neonatal, infant and child deaths...}, # COMPLETELY WRONG
|
||||
journal = {Nature},
|
||||
year = {1992}, # But doi says 2019!
|
||||
```
|
||||
This is cited ONCE but appears to be looking for transfer of learning work. Should be:
|
||||
- Perkins, D. N., & Salomon, G. (1992). "Transfer of learning" - International encyclopedia of education
|
||||
|
||||
**What's MISSING:**
|
||||
|
||||
3. **Bloom's Taxonomy** - Paper mentions "systems analysis" and "evaluation" levels but doesn't cite Bloom
|
||||
- You have `thompson2008bloom` in bib but NEVER CITED in paper!
|
||||
- This is about CS assessment using Bloom's - should use it when discussing NBGrader assessment
|
||||
|
||||
4. **Scaffolding** - You discuss scaffolding 5+ times but only cite it via Cognitive Apprenticeship
|
||||
- Missing direct scaffolding citations: Wood, Bruner, Ross (1976) or Vygotsky (1978) ZPD
|
||||
- You HAVE vygotsky1978mind in bib but NEVER CITE IT!
|
||||
|
||||
5. **Active Learning** - You discuss peer instruction, engagement, but no citations
|
||||
- Missing: Freeman et al. (2014) PNAS meta-analysis on active learning effectiveness
|
||||
- Missing: Mazur (1997) Peer Instruction
|
||||
|
||||
6. **Chunking/Progressive Disclosure** - You claim this as innovation but don't cite HCI/progressive disclosure literature
|
||||
- Missing: Nielsen (1993) progressive disclosure in UI design
|
||||
- Missing: Mayer (2009) multimedia learning principles
|
||||
|
||||
**Recommendations:**
|
||||
|
||||
**CRITICAL FIXES (must do before submission):**
|
||||
1. Fix bruner1960process - find correct Bruner scaffolding citation
|
||||
2. Fix perkins1992transfer - find correct transfer learning citation
|
||||
3. Cite thompson2008bloom when discussing NBGrader assessment levels
|
||||
4. Cite vygotsky1978mind when discussing scaffolding/ZPD
|
||||
|
||||
**STRATEGIC ADDITIONS (strengthen theoretical foundation):**
|
||||
5. Add Freeman et al. (2014) active learning meta-analysis to justify hands-on approach
|
||||
6. Add progressive disclosure HCI literature (Nielsen or Mayer) to ground the progressive disclosure pattern claim
|
||||
|
||||
### 2.2 CS Education Research - ADEQUATE ⚠️
|
||||
|
||||
**What's Cited:**
|
||||
- NBGrader (Blank 2019) ✅
|
||||
- Learner-Centered Design (Guzdial 2015) ✅
|
||||
- Peer Instruction (Porter 2013) ✅
|
||||
- Auto-grading review (Ihantola 2010) ✅
|
||||
- Teaching OOP (Kölling 2001) ✅
|
||||
- CS Ed Research (Fincher 2004) ✅
|
||||
|
||||
**Assessment:**
|
||||
- Good breadth of CS education foundations
|
||||
- NBGrader citation is essential - correctly used
|
||||
- Guzdial citation adds learner-centered credibility
|
||||
- Porter peer instruction supports engagement claims
|
||||
|
||||
**What's MISSING:**
|
||||
- **Computing education research methodology** - If you're making pedagogical claims, should cite:
|
||||
- Robins et al. (2003) "Learning and Teaching Programming: A Review and Discussion"
|
||||
- Bennedsen & Caspersen (2007) on failure rates in CS1
|
||||
- **Notional machines** - Your "mental models of framework internals" aligns with notional machine concept:
|
||||
- Sorva (2013) "Notional Machines and Introductory Programming Education"
|
||||
- **SIGCSE/ICER recent work on ML education** - Missing 2023-2024 papers from top CS Ed venues
|
||||
|
||||
**Recommendation:**
|
||||
- Consider adding notional machine citation when discussing "mental models"
|
||||
- Conduct targeted search for 2023-2024 SIGCSE/ICER papers on ML education
|
||||
|
||||
---
|
||||
|
||||
## 3. Systems Education Context
|
||||
|
||||
### 3.1 Systems Thinking - WEAK ❌
|
||||
|
||||
**What's Cited:**
|
||||
- TVM compiler (Chen 2018) ✅
|
||||
- PyTorch autograd (Paszke 2017) ✅
|
||||
- Roofline model (Williams 2009) ✅ (only in future work!)
|
||||
- ASTRA-sim (Chakkaravarthy 2023) ✅ (only in future work!)
|
||||
|
||||
**Assessment:**
|
||||
This is the **weakest area** of the related work. You claim "systems-first" as a major contribution but barely ground it in systems education literature.
|
||||
|
||||
**CRITICAL GAPS:**
|
||||
|
||||
1. **No systems thinking pedagogy citations**
|
||||
- You use "systems thinking" 20+ times but cite NO systems thinking education research
|
||||
- Missing: Meadows (2008) "Thinking in Systems" - foundational
|
||||
- Missing: Senge (1990) "The Fifth Discipline" - organizational learning
|
||||
- Missing: Richmond (1993) "Systems thinking: critical thinking skills for the 1990s and beyond"
|
||||
|
||||
2. **No compiler course pedagogogy**
|
||||
- You compare to "compiler course model" but cite NO compiler education papers
|
||||
- Missing: Aho et al. (2006) "Compilers: Principles, Techniques, and Tools" (Dragon Book)
|
||||
- Missing: Appel (1998) "Modern Compiler Implementation" series
|
||||
- Should cite actual compiler courses that use incremental construction
|
||||
|
||||
3. **No operating systems pedagogy**
|
||||
- TinyTorch's "build the whole system" mirrors OS courses (xv6, Pintos, Nachos)
|
||||
- Missing: Arpaci-Dusseau (2018) "Operating Systems: Three Easy Pieces"
|
||||
- Missing: Ousterhout (1999) "Teaching Operating Systems Using Log-Based Kernels"
|
||||
|
||||
4. **No software engineering education**
|
||||
- Package organization, module dependencies, integration - all SE concepts
|
||||
- Missing: any SE education citations
|
||||
|
||||
**What's PARTIALLY There:**
|
||||
|
||||
5. **ML Systems papers** - You cite:
|
||||
- MLSys book (Reddi 2024) ✅ - good!
|
||||
- FlashAttention (Dao 2022) ✅ - technical paper, not educational
|
||||
- Horovod, DeepSpeed - technical papers, not educational
|
||||
|
||||
**Recommendation - ADD IMMEDIATELY:**
|
||||
|
||||
**Systems Thinking Foundation:**
|
||||
```bibtex
|
||||
@book{meadows2008thinking,
|
||||
author = {Meadows, Donella H.},
|
||||
title = {Thinking in Systems: A Primer},
|
||||
year = {2008},
|
||||
publisher = {Chelsea Green Publishing}
|
||||
}
|
||||
```
|
||||
|
||||
**Compiler Course Model:**
|
||||
```bibtex
|
||||
@book{aho2006compilers,
|
||||
author = {Aho, Alfred V. and Lam, Monica S. and Sethi, Ravi and Ullman, Jeffrey D.},
|
||||
title = {Compilers: Principles, Techniques, and Tools},
|
||||
edition = {2nd},
|
||||
year = {2006},
|
||||
publisher = {Addison-Wesley}
|
||||
}
|
||||
```
|
||||
|
||||
**OS Course Model:**
|
||||
```bibtex
|
||||
@book{arpaci2018operating,
|
||||
author = {Arpaci-Dusseau, Remzi H. and Arpaci-Dusseau, Andrea C.},
|
||||
title = {Operating Systems: Three Easy Pieces},
|
||||
year = {2018},
|
||||
publisher = {Arpaci-Dusseau Books}
|
||||
}
|
||||
```
|
||||
|
||||
Then add paragraph in Related Work:
|
||||
> "TinyTorch's incremental system construction draws pedagogical inspiration from compiler courses~\citep{aho2006compilers} and operating systems courses~\citep{arpaci2018operating}, where students build complete systems (compilers from lexer to code generator, OS kernels from process management to file systems) to develop systems thinking~\citep{meadows2008thinking} through component integration. This ``build the whole stack'' approach has proven effective for teaching complex systems concepts in CS education."
|
||||
|
||||
### 3.2 ML Systems Workforce - PROBLEMATIC ⚠️❌
|
||||
|
||||
**What's Cited:**
|
||||
- Robert Half (2024) - industry report ⚠️
|
||||
- Keller Executive Search (2025) - industry report ⚠️
|
||||
|
||||
**Assessment:**
|
||||
The workforce gap motivates the entire paper, but you rely on **non-peer-reviewed industry sources**. This is risky for academic venues.
|
||||
|
||||
**Problems:**
|
||||
1. These are recruiting firms, not research organizations
|
||||
2. No academic backing for "3:1 demand-supply ratio"
|
||||
3. No academic backing for "only 150,000 skilled practitioners"
|
||||
4. "78% year-over-year growth" might be inflated by industry hype
|
||||
|
||||
**What's MISSING:**
|
||||
|
||||
Academic sources on ML workforce:
|
||||
- **ACM/IEEE computing workforce reports**
|
||||
- **Bureau of Labor Statistics data** on ML engineer demand
|
||||
- **Academic papers on AI skills gap:**
|
||||
- Ransbotham et al. (2020) MIT Sloan on AI adoption barriers
|
||||
- Brynjolfsson & Mitchell (2017) on AI workforce requirements
|
||||
- **Industry-academic collaboration papers**
|
||||
- **Survey papers from CACM, IEEE Computer, or similar**
|
||||
|
||||
**Recommendation - CRITICAL for SIGCSE/Educational Venues:**
|
||||
|
||||
Either:
|
||||
1. **Downplay workforce claims** - Make it secondary motivation, not primary
|
||||
2. **Add academic backing** - Find peer-reviewed sources on AI skills gap
|
||||
3. **Reframe as "tacit knowledge problem"** - This is your strongest argument and doesn't need industry stats
|
||||
|
||||
**Suggested Academic Citations:**
|
||||
|
||||
```bibtex
|
||||
@article{ransbotham2020expanding,
|
||||
author = {Ransbotham, Sam and Kiron, David and Gerbert, Philipp and Reeves, Martin},
|
||||
title = {Expanding AI's Impact with Organizational Learning},
|
||||
journal = {MIT Sloan Management Review},
|
||||
year = {2020}
|
||||
}
|
||||
|
||||
@article{brynjolfsson2017machine,
|
||||
author = {Brynjolfsson, Erik and Mitchell, Tom},
|
||||
title = {What can machine learning do? Workforce implications},
|
||||
journal = {Science},
|
||||
volume = {358},
|
||||
number = {6370},
|
||||
pages = {1530--1534},
|
||||
year = {2017}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Citation Balance
|
||||
|
||||
### 4.1 Citation Distribution Analysis
|
||||
|
||||
**Total unique citations:** ~40 entries in references.bib
|
||||
**Actually cited in paper:** ~35-38 (some in bib never cited)
|
||||
|
||||
**By Category:**
|
||||
- **Educational Frameworks:** 6 citations (micrograd, MiniTorch, tinygrad, d2l.ai, fast.ai, CS231n)
|
||||
- **Learning Theory:** 8 citations (Papert, Collins, Sweller, Kapur, Meyer, Lave, Bruner*, Perkins*)
|
||||
- **CS Education:** 6 citations (NBGrader, Guzdial, Porter, Ihantola, Kölling, Fincher)
|
||||
- **ML Systems:** 4 citations (Reddi MLSys book, Chen DL Systems, Williams Roofline, ASTRA-sim)
|
||||
- **Technical ML:** 10 citations (PyTorch, TensorFlow, autograd papers, optimization papers, FlashAttention, etc.)
|
||||
- **Workforce/Industry:** 2 citations (Robert Half, Keller)
|
||||
- **Other:** 4 citations (MLPerf, CIFAR, historical ML papers)
|
||||
|
||||
**Assessment:**
|
||||
|
||||
**Over-cited Areas:**
|
||||
- **Technical ML papers** - You cite lots of algorithm papers (Vaswani attention, Kingma Adam, etc.) but these are for milestone validation, not related work. This is fine but heavy.
|
||||
|
||||
**Under-cited Areas:**
|
||||
- **ML Systems Education** - Only 2 real educational systems citations (Reddi book, Chen course)
|
||||
- **Systems Thinking Pedagogy** - Zero citations despite 20+ mentions
|
||||
- **Recent Work (2023-2024)** - Only 2-3 recent citations, rest are 2020 or older
|
||||
|
||||
**Citation Age Distribution:**
|
||||
- 2024-2025: 2 citations (Reddi MLSys book, Keller workforce)
|
||||
- 2022-2023: 4 citations (micrograd, tinygrad, Chen DL Systems, ASTRA-sim)
|
||||
- 2018-2021: 10 citations (NBGrader, fast.ai, d2l.ai, JAX, etc.)
|
||||
- 2000-2017: 15 citations (learning theory classics, foundational ML)
|
||||
- Pre-2000: 5 citations (Papert, Sweller, Bruner, etc.)
|
||||
|
||||
**Verdict:**
|
||||
- Good balance between classic theory (pre-2000) and recent work (2020+)
|
||||
- **TOO FEW 2023-2024 citations** for a 2025 paper - looks outdated
|
||||
- Learning theory foundations are appropriately older (Papert 1980, Sweller 1988)
|
||||
|
||||
### 4.2 Uncited Entries in references.bib
|
||||
|
||||
**Entries in bib but NEVER cited in paper:**
|
||||
1. `thompson2008bloom` - Bloom's taxonomy for CS assessment ❗ SHOULD CITE
|
||||
2. `vygotsky1978mind` - Mind in Society, scaffolding/ZPD ❗ SHOULD CITE
|
||||
3. `bradbury2018jax` - JAX framework - mentioned in text but not formally cited
|
||||
4. Several technical papers (Baydin autograd survey, Chen gradient checkpointing, etc.)
|
||||
|
||||
**Recommendation:**
|
||||
- Remove uncited technical papers from bib (clutter)
|
||||
- ADD citations for thompson2008bloom and vygotsky1978mind in appropriate sections
|
||||
|
||||
---
|
||||
|
||||
## 5. Competitive Positioning
|
||||
|
||||
### 5.1 Framework Comparison Table (Table 1) - EXCELLENT ✅
|
||||
|
||||
**Assessment:**
|
||||
This table is **pedagogically brilliant** and **strategically sound**:
|
||||
|
||||
**Strengths:**
|
||||
- Separates educational vs. production frameworks clearly
|
||||
- Five comparison dimensions are well-chosen: Purpose, Scope, Systems Focus, Target Outcome
|
||||
- TinyTorch row is bolded - appropriate emphasis
|
||||
- Fair to competitors - doesn't strawman
|
||||
|
||||
**Strategic Positioning:**
|
||||
- micrograd: acknowledged strength (understand backprop) while showing limitation (no systems)
|
||||
- MiniTorch: respectful ("build from first principles") while differentiating (embedded systems from M01)
|
||||
- tinygrad: acknowledges sophistication ("understand compilation") while showing gap (scaffolding)
|
||||
- PyTorch/TensorFlow: correctly positioned as "what comes after" TinyTorch
|
||||
|
||||
**Minor Suggestions:**
|
||||
- Consider adding "Assessment" column to highlight NBGrader infrastructure advantage
|
||||
- Consider adding "Hardware Requirements" to emphasize CPU-only accessibility
|
||||
|
||||
### 5.2 Detailed Framework Comparisons (Prose) - GOOD ✅
|
||||
|
||||
**micrograd comparison (line 379):**
|
||||
> "pedagogical clarity comes from intentional minimalism... necessarily omits systems concerns"
|
||||
|
||||
**Verdict:** ✅ Fair and strategic. Acknowledges strength (clarity) while identifying gap (no systems).
|
||||
|
||||
**MiniTorch comparison (line 381):**
|
||||
> "core curriculum emphasizes mathematical rigor... TinyTorch differs through systems-first emphasis"
|
||||
|
||||
**Verdict:** ✅ Respectful differentiation. Doesn't claim MiniTorch is bad, just different focus.
|
||||
|
||||
**tinygrad comparison (line 383):**
|
||||
> "pedagogically valuable through its inspectable design, tinygrad assumes significant background"
|
||||
|
||||
**Verdict:** ✅ Acknowledges value while identifying accessibility gap.
|
||||
|
||||
**d2l.ai comparison (line 387):**
|
||||
> "excels at algorithmic understanding through framework usage"
|
||||
|
||||
**Verdict:** ✅ Generous acknowledgment of widespread adoption and algorithmic strength.
|
||||
|
||||
**fast.ai comparison (line 387):**
|
||||
> "distinctive top-down pedagogy starts with practical applications"
|
||||
|
||||
**Verdict:** ✅ Accurately describes complementary approach without criticism.
|
||||
|
||||
**Overall Competitive Positioning:** STRONG. Fair, strategic, differentiated without being aggressive.
|
||||
|
||||
---
|
||||
|
||||
## 6. "What Are Reviewers Likely to Ask 'Why Didn't You Cite X?'"
|
||||
|
||||
### 6.1 High-Risk Omissions
|
||||
|
||||
**SIGCSE/ICER Reviewers Will Ask About:**
|
||||
|
||||
1. **"Why no recent SIGCSE/ICER papers on ML education?"**
|
||||
- Risk: HIGH for SIGCSE submission
|
||||
- You cite nothing from 2023-2024 SIGCSE/ICER
|
||||
- Need: Search SIGCSE 2023, 2024, ICER 2023, 2024 for ML education papers
|
||||
|
||||
2. **"Why no systems thinking education citations despite 'systems-first' claim?"**
|
||||
- Risk: HIGH for any venue
|
||||
- You use "systems thinking" 20+ times but cite zero systems thinking pedagogy
|
||||
- Need: Meadows (2008) or similar foundational systems thinking work
|
||||
|
||||
3. **"Why no Bloom's taxonomy when discussing assessment?"**
|
||||
- Risk: MEDIUM for education venues
|
||||
- You have thompson2008bloom in bib but don't cite it
|
||||
- Need: Cite when discussing NBGrader assessment levels
|
||||
|
||||
4. **"Why no scaffolding citations when you discuss scaffolding 5+ times?"**
|
||||
- Risk: MEDIUM for education venues
|
||||
- You have vygotsky1978mind in bib but don't cite it
|
||||
- Need: Cite Vygotsky or Wood/Bruner/Ross when discussing scaffolding
|
||||
|
||||
**MLSys/Technical Reviewers Will Ask About:**
|
||||
|
||||
5. **"Why no compiler course citations when you compare to 'compiler course model'?"**
|
||||
- Risk: MEDIUM for MLSys
|
||||
- You claim to follow compiler course pedagogy but cite no compiler education
|
||||
- Need: Dragon Book or equivalent compiler course reference
|
||||
|
||||
6. **"Why cite industry reports (Robert Half, Keller) for workforce claims?"**
|
||||
- Risk: MEDIUM for academic venues
|
||||
- These aren't peer-reviewed sources
|
||||
- Need: Academic sources on AI skills gap OR downplay workforce motivation
|
||||
|
||||
**General Academic Reviewers Will Ask About:**
|
||||
|
||||
7. **"Why no recent work? Most citations are 2020 or older."**
|
||||
- Risk: LOW-MEDIUM for 2025 submission
|
||||
- Only 2-3 citations from 2023-2024
|
||||
- Need: More recent ML education / ML systems education work
|
||||
|
||||
### 6.2 Medium-Risk Omissions
|
||||
|
||||
8. **"Why no active learning meta-analysis when you claim hands-on effectiveness?"**
|
||||
- Freeman et al. (2014) PNAS is the canonical citation for active learning superiority
|
||||
- Risk: LOW-MEDIUM
|
||||
|
||||
9. **"Why no progressive disclosure HCI literature when you claim it as innovation?"**
|
||||
- Nielsen or Mayer on progressive disclosure in UI/learning
|
||||
- Risk: LOW-MEDIUM
|
||||
|
||||
10. **"Why no notional machines when discussing mental models?"**
|
||||
- Sorva (2013) on notional machines in programming education
|
||||
- Risk: LOW
|
||||
|
||||
### 6.3 Low-Risk Omissions (Nice to Have)
|
||||
|
||||
11. Berkeley CS182, MIT 6.S191, Full Stack Deep Learning courses
|
||||
12. nnfs.io (Neural Networks from Scratch book/course)
|
||||
13. More JAX educational materials discussion
|
||||
14. Software engineering education (package design, modularity)
|
||||
15. Educational data mining / learning analytics for assessing NBGrader
|
||||
|
||||
---
|
||||
|
||||
## 7. Strategic Recommendations
|
||||
|
||||
### 7.1 MUST FIX Before Submission (Critical Issues)
|
||||
|
||||
**Priority 1: Fix Corrupted Bib Entries** ⚠️⚠️⚠️
|
||||
1. Fix `bruner1960process` - Currently has wrong paper (Narrative/Mentalization)
|
||||
- Find: Bruner "The Process of Education" (1960) OR Wood, Bruner, Ross (1976) scaffolding paper
|
||||
2. Fix `perkins1992transfer` - Currently has infant mortality paper
|
||||
- Find: Perkins & Salomon (1992) "Transfer of learning"
|
||||
|
||||
**Priority 2: Cite What You Already Have**
|
||||
3. Cite `thompson2008bloom` when discussing NBGrader assessment (Section 4.4)
|
||||
4. Cite `vygotsky1978mind` when discussing scaffolding/ZPD (Section 3.2.2 Related Work)
|
||||
|
||||
**Priority 3: Add Systems Thinking Foundation**
|
||||
5. Add Meadows (2008) "Thinking in Systems" - cite when using "systems thinking"
|
||||
6. Add compiler course reference (Dragon Book or equivalent) - cite when comparing to "compiler course model"
|
||||
|
||||
### 7.2 SHOULD ADD for Stronger Positioning (High Value)
|
||||
|
||||
**Priority 4: Recent ML Education Work**
|
||||
7. Search SIGCSE 2023, 2024 proceedings for ML education papers
|
||||
8. Search ICER 2023, 2024 proceedings for ML/programming education papers
|
||||
9. Search MLSys 2023, 2024 for any education-track papers
|
||||
|
||||
**Priority 5: Strengthen Workforce Motivation**
|
||||
10. Replace or supplement Robert Half/Keller with academic sources:
|
||||
- Brynjolfsson & Mitchell (2017) Science paper on ML workforce
|
||||
- Ransbotham et al. (2020) MIT Sloan on AI adoption barriers
|
||||
- OR downplay workforce statistics, emphasize "tacit knowledge problem"
|
||||
|
||||
**Priority 6: Active Learning Foundation**
|
||||
11. Add Freeman et al. (2014) active learning meta-analysis to justify hands-on approach
|
||||
|
||||
### 7.3 COULD ADD for Completeness (Medium Value)
|
||||
|
||||
**Priority 7: Progressive Disclosure Grounding**
|
||||
12. Add Nielsen (1993) or Mayer (2009) on progressive disclosure in learning/UI
|
||||
|
||||
**Priority 8: Notional Machines**
|
||||
13. Add Sorva (2013) on notional machines when discussing "mental models of internals"
|
||||
|
||||
**Priority 9: OS Course Pedagogy**
|
||||
14. Add OS education reference (OSTEP or similar) to strengthen "build the whole system" comparison
|
||||
|
||||
### 7.4 REMOVE or DOWNPLAY (Clutter Reduction)
|
||||
|
||||
**Priority 10: Clean Up Bib**
|
||||
15. Remove uncited technical papers from references.bib (e.g., Baydin autograd survey if not cited)
|
||||
16. Remove or integrate technical milestone papers (Rosenblatt, Rumelhart, LeCun) - currently just used for historical validation, not related work
|
||||
|
||||
---
|
||||
|
||||
## 8. Citation Quality Checklist
|
||||
|
||||
Going through your current citations:
|
||||
|
||||
### Educational Frameworks
|
||||
- ✅ micrograd (Karpathy 2022) - GitHub repo, appropriate
|
||||
- ✅ MiniTorch (Rush 2020) - Cornell Tech, appropriate
|
||||
- ✅ tinygrad (Hotz 2023) - GitHub repo, appropriate
|
||||
- ✅ d2l.ai (Zhang 2021) - arXiv + Cambridge Press, peer-reviewed equivalent
|
||||
- ✅ fast.ai (Howard 2020) - Published in *Information* journal, peer-reviewed
|
||||
- ✅ CS231n (Johnson 2016) - Course webpage, appropriate for course citation
|
||||
|
||||
### Learning Theory
|
||||
- ✅ Papert (1980) - Classic, appropriate
|
||||
- ✅ Collins (1989) - Routledge chapter, peer-reviewed
|
||||
- ✅ Sweller (1988) - Cognitive Science journal, peer-reviewed
|
||||
- ✅ Kapur (2008) - Cognition and Instruction, peer-reviewed
|
||||
- ✅ Meyer (2003) - Book chapter, peer-reviewed
|
||||
- ✅ Lave & Wenger (1991) - Cambridge University Press, peer-reviewed
|
||||
- ❌ Bruner (1960) - **CORRUPTED ENTRY** - fix immediately
|
||||
- ❌ Perkins (1992) - **CORRUPTED ENTRY** - fix immediately
|
||||
- ⚠️ Vygotsky (1978) - **IN BIB BUT NEVER CITED** - should cite
|
||||
|
||||
### CS Education
|
||||
- ✅ NBGrader (Blank 2019) - JOSE, peer-reviewed
|
||||
- ✅ Guzdial (2015) - Morgan & Claypool, peer-reviewed
|
||||
- ✅ Porter (2013) - ACM SIGCSE, peer-reviewed
|
||||
- ✅ Ihantola (2010) - Koli Calling, peer-reviewed
|
||||
- ✅ Kölling (2001) - ITiCSE, peer-reviewed
|
||||
- ✅ Fincher (2004) - Taylor & Francis, peer-reviewed
|
||||
- ⚠️ Thompson (2008) - **IN BIB BUT NEVER CITED** - should cite
|
||||
|
||||
### Workforce/Industry
|
||||
- ⚠️ Robert Half (2024) - **Industry press release, not peer-reviewed**
|
||||
- ⚠️ Keller (2025) - **Industry report, not peer-reviewed**
|
||||
|
||||
### ML Systems
|
||||
- ✅ Reddi MLSys book (2024) - IEEE conference, peer-reviewed
|
||||
- ✅ Chen DL Systems (2022) - CMU course, appropriate
|
||||
- ✅ Williams Roofline (2009) - CACM, peer-reviewed
|
||||
- ✅ ASTRA-sim (2020, 2023) - IEEE ISPASS, peer-reviewed
|
||||
|
||||
### Technical ML
|
||||
- ✅ PyTorch autograd (Paszke 2017) - NIPS workshop, appropriate
|
||||
- ✅ TVM (Chen 2018) - OSDI, peer-reviewed
|
||||
- ✅ FlashAttention (Dao 2022) - NeurIPS, peer-reviewed
|
||||
- ✅ Vaswani attention (2017) - NeurIPS, peer-reviewed (note: doi shows 2025 reprint, original was 2017)
|
||||
- ✅ Adam (Kingma 2014) - arXiv preprint widely cited, appropriate
|
||||
- ✅ Horovod (Sergeev 2018) - arXiv preprint, appropriate
|
||||
- ✅ DeepSpeed (Rasley 2020) - ACM KDD, peer-reviewed
|
||||
- ✅ Historical ML (Rosenblatt 1958, Rumelhart 1986, LeCun 1998) - Classic papers, appropriate
|
||||
|
||||
**Quality Issues:**
|
||||
1. **2 corrupted entries** (bruner1960, perkins1992) - CRITICAL FIX
|
||||
2. **2 industry reports** (Robert Half, Keller) - risky for academic venues
|
||||
3. **2 uncited entries** (thompson2008bloom, vygotsky1978mind) - should cite or remove
|
||||
|
||||
**Overall Citation Quality:** 7/10
|
||||
- Strong peer-reviewed foundation for learning theory and CS education
|
||||
- Good mix of classic and recent work
|
||||
- Technical citations are appropriate
|
||||
- MAJOR issues: 2 corrupted entries, industry sources for key motivation
|
||||
|
||||
---
|
||||
|
||||
## 9. Comparison Table: TinyTorch vs. What Reviewers Expect
|
||||
|
||||
| Aspect | What You Have | What Reviewers Expect | Gap |
|
||||
|--------|---------------|----------------------|-----|
|
||||
| **Educational Frameworks** | 6 major frameworks cited (micrograd, MiniTorch, tinygrad, d2l.ai, fast.ai, CS231n) | All major educational frameworks | ✅ GOOD |
|
||||
| **Learning Theory** | 8 theories cited (constructionism, cognitive apprenticeship, cognitive load, productive failure, threshold concepts, situated learning, + 2 corrupted) | Foundational learning theories with 3-5 key frameworks | ✅ STRONG (once corrupted entries fixed) |
|
||||
| **Systems Thinking** | Zero pedagogy citations despite 20+ mentions | At least 1-2 systems thinking education sources | ❌ CRITICAL GAP |
|
||||
| **Compiler Course Model** | Mentioned but not cited | Citation to compiler education when claiming to follow model | ⚠️ GAP |
|
||||
| **Recent Work (2023-2024)** | 2-3 recent citations | 5-10 recent citations showing awareness of field | ⚠️ GAP |
|
||||
| **Workforce Motivation** | 2 industry reports | Academic sources or downplayed motivation | ⚠️ RISKY |
|
||||
| **Scaffolding** | Discussed but not directly cited | Vygotsky or Wood/Bruner/Ross when discussing scaffolding | ⚠️ GAP |
|
||||
| **Assessment** | NBGrader cited but no Bloom's taxonomy | Bloom's when discussing assessment levels | ⚠️ MINOR GAP |
|
||||
| **Active Learning** | Hands-on approach claimed but not grounded | Freeman et al. meta-analysis or similar | ⚠️ MINOR GAP |
|
||||
| **Competitive Positioning** | Fair, strategic, differentiated | Fair comparison acknowledging strengths | ✅ EXCELLENT |
|
||||
|
||||
---
|
||||
|
||||
## 10. Final Recommendations
|
||||
|
||||
### 10.1 Must-Do Before Submission (1-2 hours)
|
||||
|
||||
1. **Fix corrupted bruner1960process entry** - search for correct Bruner or Wood/Bruner/Ross scaffolding paper
|
||||
2. **Fix corrupted perkins1992transfer entry** - search for correct Perkins & Salomon transfer of learning paper
|
||||
3. **Add systems thinking citation** - Meadows (2008) "Thinking in Systems"
|
||||
4. **Add compiler course citation** - Aho et al. (2006) "Compilers" (Dragon Book)
|
||||
5. **Cite vygotsky1978mind** when discussing scaffolding/ZPD
|
||||
6. **Cite thompson2008bloom** when discussing NBGrader assessment
|
||||
|
||||
### 10.2 Should-Do for Strong Submission (4-6 hours)
|
||||
|
||||
7. **Search SIGCSE 2023-2024** for recent ML education papers - add 2-3 if found
|
||||
8. **Replace/supplement workforce citations** with academic sources:
|
||||
- Brynjolfsson & Mitchell (2017) Science
|
||||
- Ransbotham et al. (2020) MIT Sloan
|
||||
9. **Add Freeman et al. (2014)** active learning meta-analysis
|
||||
10. **Add OS course pedagogy** (Arpaci-Dusseau OSTEP) to strengthen systems course comparison
|
||||
|
||||
### 10.3 Could-Do for Comprehensive Submission (2-4 hours)
|
||||
|
||||
11. **Add progressive disclosure HCI literature** (Nielsen or Mayer)
|
||||
12. **Add notional machines** (Sorva 2013) for mental models
|
||||
13. **Add JAX discussion** in related work (functional paradigm)
|
||||
14. **Search MLSys 2023-2024** for any education-focused papers
|
||||
|
||||
### 10.4 Reframing Recommendations
|
||||
|
||||
**Current Introduction Opening:**
|
||||
> "Machine learning deployment faces a critical workforce bottleneck: demand for ML systems engineers outstrips supply by over 3:1..."
|
||||
|
||||
**Suggested Reframe (de-emphasize industry stats):**
|
||||
> "Machine learning systems engineering requires tacit knowledge that resists formal instruction: understanding why Adam requires 2× optimizer state memory, when attention's O(N²) scaling becomes prohibitive, how to navigate accuracy-latency-memory tradeoffs in production. These engineering judgment calls depend on mental models of framework internals, traditionally acquired through years of debugging PyTorch or TensorFlow rather than formal courses~\citep{reddi2024mlsysbook}. While workforce demand for such skills has grown substantially~\citep{brynjolfsson2017machine}, current ML education..."
|
||||
|
||||
**Why:** Leads with the "tacit knowledge problem" (your strongest argument), supported by academic citation, with workforce as secondary motivation.
|
||||
|
||||
---
|
||||
|
||||
## 11. Success Metrics
|
||||
|
||||
After implementing recommendations, your citation strategy should score:
|
||||
|
||||
| Criterion | Current | Target | Priority |
|
||||
|-----------|---------|--------|----------|
|
||||
| **Related Work Coverage** | 7/10 | 9/10 | HIGH |
|
||||
| **Learning Theory Grounding** | 6/10 (corrupted entries) | 9/10 | CRITICAL |
|
||||
| **Systems Education Context** | 4/10 | 8/10 | CRITICAL |
|
||||
| **Citation Balance** | 7/10 | 8/10 | MEDIUM |
|
||||
| **Competitive Positioning** | 9/10 | 9/10 | ✅ MAINTAIN |
|
||||
| **Citation Quality** | 7/10 (corrupted entries) | 9/10 | CRITICAL |
|
||||
| **Recency** | 6/10 | 8/10 | MEDIUM |
|
||||
| **Academic Rigor** | 7/10 (industry sources) | 9/10 | HIGH |
|
||||
|
||||
**Overall:** 6.5/10 → **8.5/10** with recommended changes
|
||||
|
||||
---
|
||||
|
||||
## 12. Reviewer Likelihood of "Missing Citation" Comments
|
||||
|
||||
### By Venue:
|
||||
|
||||
**SIGCSE/ICER Submission:**
|
||||
- **HIGH RISK (>70% chance):**
|
||||
- "Why no recent SIGCSE/ICER work on ML education?"
|
||||
- "Why no Bloom's taxonomy when discussing assessment?"
|
||||
- "Why no scaffolding citations (Vygotsky, Wood/Bruner/Ross)?"
|
||||
- **MEDIUM RISK (30-70%):**
|
||||
- "Why no systems thinking pedagogy despite systems-first claim?"
|
||||
- "Why no active learning citations?"
|
||||
- "Fix corrupted Bruner/Perkins entries"
|
||||
|
||||
**MLSys/Systems Submission:**
|
||||
- **HIGH RISK (>70%):**
|
||||
- "Why no compiler course citations when comparing to compiler pedagogy?"
|
||||
- "Why no systems thinking education citations?"
|
||||
- **MEDIUM RISK (30-70%):**
|
||||
- "Why cite industry reports for workforce claims?"
|
||||
- "Why no OS course pedagogy for 'build the whole system' claim?"
|
||||
|
||||
**General CS Education Venues (e.g., TOCE, CSE):**
|
||||
- **HIGH RISK:**
|
||||
- Systems thinking pedagogy gap
|
||||
- Recent work gap (2023-2024)
|
||||
- Corrupted bib entries
|
||||
- **MEDIUM RISK:**
|
||||
- Scaffolding citations
|
||||
- Active learning foundations
|
||||
- Progressive disclosure HCI literature
|
||||
|
||||
---
|
||||
|
||||
## 13. Citation Search Queries for Missing Work
|
||||
|
||||
To fill gaps, search these:
|
||||
|
||||
### Google Scholar Queries:
|
||||
|
||||
1. **Recent ML Education:**
|
||||
- `"machine learning education" 2023 2024 SIGCSE`
|
||||
- `"teaching machine learning" 2023 2024 ICER`
|
||||
- `"ML systems education" 2023 2024`
|
||||
- `"deep learning pedagogy" 2023 2024`
|
||||
|
||||
2. **Systems Thinking Education:**
|
||||
- `"systems thinking" education pedagogy`
|
||||
- `"teaching systems thinking" computer science`
|
||||
- `Meadows "Thinking in Systems"`
|
||||
|
||||
3. **Compiler Course Pedagogy:**
|
||||
- `"compiler course" pedagogy education`
|
||||
- `"teaching compilers" incremental construction`
|
||||
- `Aho "Compilers: Principles, Techniques, and Tools"`
|
||||
|
||||
4. **Scaffolding:**
|
||||
- `Wood Bruner Ross 1976 scaffolding`
|
||||
- `Vygotsky "zone of proximal development" education`
|
||||
|
||||
5. **AI Workforce (Academic):**
|
||||
- `Brynjolfsson Mitchell 2017 machine learning workforce`
|
||||
- `"AI skills gap" academic research`
|
||||
- `"machine learning talent" shortage study`
|
||||
|
||||
6. **Active Learning:**
|
||||
- `Freeman 2014 "active learning increases student performance"`
|
||||
|
||||
7. **Progressive Disclosure:**
|
||||
- `Nielsen progressive disclosure`
|
||||
- `Mayer multimedia learning progressive complexity`
|
||||
|
||||
### ACM Digital Library Queries:
|
||||
|
||||
- Search SIGCSE 2023, 2024 proceedings: `"machine learning" OR "deep learning" OR "neural networks"`
|
||||
- Search ICER 2023, 2024 proceedings: `"machine learning" OR "ML" OR "framework"`
|
||||
|
||||
### arXiv Queries:
|
||||
|
||||
- `cat:cs.CY "machine learning education" 2023`
|
||||
- `cat:cs.LG "teaching" OR "education" OR "pedagogy" 2023 2024`
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The TinyTorch paper has **solid core citations** for educational frameworks and learning theory, **excellent competitive positioning**, but **critical gaps** in systems thinking pedagogy, recent work, and two corrupted bib entries.
|
||||
|
||||
**Priority actions:**
|
||||
1. **Fix corrupted entries** (bruner1960, perkins1992) - blocking issue
|
||||
2. **Add systems thinking** (Meadows 2008) - supports main claim
|
||||
3. **Add compiler course reference** (Aho 2006) - supports pedagogical model
|
||||
4. **Cite existing entries** (vygotsky1978mind, thompson2008bloom)
|
||||
5. **Search for 2023-2024 ML education work** - shows awareness of field
|
||||
6. **Replace/supplement workforce citations** with academic sources
|
||||
|
||||
With these changes, the citation strategy moves from 6.5/10 to 8.5/10, significantly reducing reviewer pushback risk.
|
||||
|
||||
---
|
||||
|
||||
**Files Referenced:**
|
||||
- `/Users/VJ/GitHub/TinyTorch/paper/paper.tex` (1035 lines, ~40,000 words)
|
||||
- `/Users/VJ/GitHub/TinyTorch/paper/references.bib` (528 lines, 40 entries)
|
||||
|
||||
**Assessment Completed:** 2025-11-18
|
||||
**Reviewer:** Dr. James Patterson (Literature Review Specialist)
|
||||
@@ -1,291 +0,0 @@
|
||||
% ============================================================================
|
||||
% NEW CITATIONS TO ADD TO references.bib
|
||||
% TinyTorch Literature Review Recommendations
|
||||
% Generated: 2025-11-18
|
||||
% ============================================================================
|
||||
|
||||
% ----------------------------------------------------------------------------
|
||||
% CRITICAL FIXES - Replace corrupted entries
|
||||
% ----------------------------------------------------------------------------
|
||||
|
||||
% REPLACE bruner1960process (currently has wrong paper)
|
||||
% Original scaffolding work - choose ONE:
|
||||
|
||||
% OPTION A: The Process of Education (1960)
|
||||
@book{bruner1960process,
|
||||
author = {Bruner, Jerome S.},
|
||||
title = {The Process of Education},
|
||||
year = {1960},
|
||||
publisher = {Harvard University Press},
|
||||
address = {Cambridge, MA},
|
||||
isbn = {9780674710009}
|
||||
}
|
||||
|
||||
% OPTION B: Original scaffolding paper (1976) - RECOMMENDED for scaffolding discussion
|
||||
@article{wood1976scaffolding,
|
||||
author = {Wood, David and Bruner, Jerome S. and Ross, Gail},
|
||||
title = {The role of tutoring in problem solving},
|
||||
journal = {Journal of Child Psychology and Psychiatry},
|
||||
volume = {17},
|
||||
number = {2},
|
||||
pages = {89--100},
|
||||
year = {1976},
|
||||
doi = {10.1111/j.1469-7610.1976.tb00381.x},
|
||||
publisher = {Wiley}
|
||||
}
|
||||
|
||||
% REPLACE perkins1992transfer (currently has infant mortality paper)
|
||||
@incollection{perkins1992transfer,
|
||||
author = {Perkins, David N. and Salomon, Gavriel},
|
||||
title = {Transfer of Learning},
|
||||
booktitle = {International Encyclopedia of Education},
|
||||
edition = {2nd},
|
||||
editor = {Husen, Torsten and Postlethwaite, T. Neville},
|
||||
year = {1992},
|
||||
publisher = {Pergamon Press},
|
||||
address = {Oxford, England}
|
||||
}
|
||||
|
||||
% ----------------------------------------------------------------------------
|
||||
% CRITICAL ADDITIONS - Systems Thinking and CS Education Foundations
|
||||
% ----------------------------------------------------------------------------
|
||||
|
||||
% Systems thinking foundation - CRITICAL for "systems-first" claim
|
||||
@book{meadows2008thinking,
|
||||
author = {Meadows, Donella H.},
|
||||
title = {Thinking in Systems: A Primer},
|
||||
year = {2008},
|
||||
publisher = {Chelsea Green Publishing},
|
||||
address = {White River Junction, VT},
|
||||
isbn = {9781603580557},
|
||||
editor = {Wright, Diana}
|
||||
}
|
||||
|
||||
% Compiler course pedagogy - CRITICAL for "compiler course model" claim
|
||||
@book{aho2006compilers,
|
||||
author = {Aho, Alfred V. and Lam, Monica S. and Sethi, Ravi and Ullman, Jeffrey D.},
|
||||
title = {Compilers: Principles, Techniques, and Tools},
|
||||
edition = {2nd},
|
||||
year = {2006},
|
||||
publisher = {Addison-Wesley},
|
||||
address = {Boston, MA},
|
||||
isbn = {9780321486814},
|
||||
note = {The Dragon Book}
|
||||
}
|
||||
|
||||
% Operating systems pedagogy - for "build the whole system" comparison
|
||||
@book{arpaci2018operating,
|
||||
author = {Arpaci-Dusseau, Remzi H. and Arpaci-Dusseau, Andrea C.},
|
||||
title = {Operating Systems: Three Easy Pieces},
|
||||
year = {2018},
|
||||
publisher = {Arpaci-Dusseau Books},
|
||||
url = {http://pages.cs.wisc.edu/~remzi/OSTEP/},
|
||||
note = {Version 1.00}
|
||||
}
|
||||
|
||||
% ----------------------------------------------------------------------------
|
||||
% HIGH-VALUE ADDITIONS - Strengthen Theoretical Foundation
|
||||
% ----------------------------------------------------------------------------
|
||||
|
||||
% Active learning meta-analysis - justifies hands-on approach
|
||||
@article{freeman2014active,
|
||||
author = {Freeman, Scott and Eddy, Sarah L. and McDonough, Miles and Smith, Michelle K. and Okoroafor, Nnadozie and Jordt, Hannah and Wenderoth, Mary Pat},
|
||||
title = {Active learning increases student performance in science, engineering, and mathematics},
|
||||
journal = {Proceedings of the National Academy of Sciences},
|
||||
volume = {111},
|
||||
number = {23},
|
||||
pages = {8410--8415},
|
||||
year = {2014},
|
||||
doi = {10.1073/pnas.1319030111},
|
||||
publisher = {National Academy of Sciences}
|
||||
}
|
||||
|
||||
% Academic workforce citation - replace/supplement Robert Half/Keller
|
||||
@article{brynjolfsson2017machine,
|
||||
author = {Brynjolfsson, Erik and Mitchell, Tom},
|
||||
title = {What can machine learning do? Workforce implications},
|
||||
journal = {Science},
|
||||
volume = {358},
|
||||
number = {6370},
|
||||
pages = {1530--1534},
|
||||
year = {2017},
|
||||
doi = {10.1126/science.aap8062},
|
||||
publisher = {American Association for the Advancement of Science}
|
||||
}
|
||||
|
||||
% Academic AI adoption barriers - supports workforce motivation
|
||||
@article{ransbotham2020expanding,
|
||||
author = {Ransbotham, Sam and Kiron, David and Gerbert, Philipp and Reeves, Martin},
|
||||
title = {Expanding AI's Impact with Organizational Learning},
|
||||
journal = {MIT Sloan Management Review},
|
||||
volume = {61},
|
||||
number = {4},
|
||||
pages = {1--6},
|
||||
year = {2020},
|
||||
url = {https://sloanreview.mit.edu/projects/expanding-ais-impact-with-organizational-learning/}
|
||||
}
|
||||
|
||||
% Progressive disclosure / cognitive load management
|
||||
@article{mayer2003cognitive,
|
||||
author = {Mayer, Richard E. and Moreno, Roxana},
|
||||
title = {Nine ways to reduce cognitive load in multimedia learning},
|
||||
journal = {Educational Psychologist},
|
||||
volume = {38},
|
||||
number = {1},
|
||||
pages = {43--52},
|
||||
year = {2003},
|
||||
doi = {10.1207/S15326985EP3801_6},
|
||||
publisher = {Routledge}
|
||||
}
|
||||
|
||||
% Alternative: Nielsen's progressive disclosure (HCI perspective)
|
||||
@book{nielsen1993usability,
|
||||
author = {Nielsen, Jakob},
|
||||
title = {Usability Engineering},
|
||||
year = {1993},
|
||||
publisher = {Academic Press},
|
||||
address = {Boston, MA},
|
||||
isbn = {9780125184069}
|
||||
}
|
||||
|
||||
% ----------------------------------------------------------------------------
|
||||
% OPTIONAL BUT VALUABLE - Strengthen Pedagogical Claims
|
||||
% ----------------------------------------------------------------------------
|
||||
|
||||
% Notional machines - for "mental models" discussion
|
||||
@phdthesis{sorva2013notional,
|
||||
author = {Sorva, Juha},
|
||||
title = {Notional Machines and Introductory Programming Education},
|
||||
school = {Aalto University},
|
||||
year = {2013},
|
||||
type = {Doctoral dissertation},
|
||||
url = {https://aaltodoc.aalto.fi/handle/123456789/11299},
|
||||
address = {Espoo, Finland}
|
||||
}
|
||||
|
||||
% CS1/CS2 failure rates and learning difficulties
|
||||
@inproceedings{bennedsen2007failure,
|
||||
author = {Bennedsen, Jens and Caspersen, Michael E.},
|
||||
title = {Failure rates in introductory programming},
|
||||
booktitle = {ACM SIGCSE Bulletin},
|
||||
volume = {39},
|
||||
number = {2},
|
||||
pages = {32--36},
|
||||
year = {2007},
|
||||
doi = {10.1145/1272848.1272879},
|
||||
publisher = {ACM}
|
||||
}
|
||||
|
||||
% Programming education review
|
||||
@article{robins2003learning,
|
||||
author = {Robins, Anthony and Rountree, Janet and Rountree, Nathan},
|
||||
title = {Learning and Teaching Programming: A Review and Discussion},
|
||||
journal = {Computer Science Education},
|
||||
volume = {13},
|
||||
number = {2},
|
||||
pages = {137--172},
|
||||
year = {2003},
|
||||
doi = {10.1076/csed.13.2.137.14200},
|
||||
publisher = {Taylor \& Francis}
|
||||
}
|
||||
|
||||
% Peer instruction validation (if needed beyond Porter 2013)
|
||||
@article{mazur1997peer,
|
||||
author = {Mazur, Eric},
|
||||
title = {Peer Instruction: A User's Manual},
|
||||
journal = {American Journal of Physics},
|
||||
volume = {67},
|
||||
number = {4},
|
||||
pages = {359--360},
|
||||
year = {1999},
|
||||
doi = {10.1119/1.19265},
|
||||
note = {Book review of Mazur (1997) Prentice Hall}
|
||||
}
|
||||
|
||||
% ----------------------------------------------------------------------------
|
||||
% PLACEHOLDER - Add Recent ML Education Work (2023-2024)
|
||||
% ----------------------------------------------------------------------------
|
||||
|
||||
% TODO: Search SIGCSE 2023-2024, ICER 2023-2024, MLSys 2023-2024
|
||||
% Add 2-3 recent papers on ML education here
|
||||
% Example format:
|
||||
|
||||
% @inproceedings{author2023mleducation,
|
||||
% author = {LastName, FirstName and LastName, FirstName},
|
||||
% title = {Teaching ML Systems in Computer Science Education},
|
||||
% booktitle = {Proceedings of the 54th ACM Technical Symposium on Computer Science Education (SIGCSE)},
|
||||
% year = {2023},
|
||||
% pages = {XXX--XXX},
|
||||
% doi = {10.1145/XXXXXXX.XXXXXXX},
|
||||
% publisher = {ACM}
|
||||
% }
|
||||
|
||||
% ----------------------------------------------------------------------------
|
||||
% NOTES ON EXISTING ENTRIES TO CITE
|
||||
% ----------------------------------------------------------------------------
|
||||
|
||||
% These are ALREADY in references.bib but NOT CITED in paper.tex:
|
||||
% - thompson2008bloom - SHOULD cite when discussing NBGrader assessment levels
|
||||
% - vygotsky1978mind - SHOULD cite when discussing scaffolding/ZPD
|
||||
% - bradbury2018jax - Could discuss JAX in related work (functional paradigm)
|
||||
|
||||
% ============================================================================
|
||||
% USAGE INSTRUCTIONS
|
||||
% ============================================================================
|
||||
|
||||
% CRITICAL FIXES (Must do):
|
||||
% 1. REPLACE bruner1960process entry in references.bib with one above
|
||||
% 2. REPLACE perkins1992transfer entry in references.bib with one above
|
||||
% 3. ADD meadows2008thinking (systems thinking foundation)
|
||||
% 4. ADD aho2006compilers (compiler course model)
|
||||
|
||||
% HIGH-VALUE ADDITIONS (Strongly recommended):
|
||||
% 5. ADD arpaci2018operating (OS course pedagogy)
|
||||
% 6. ADD freeman2014active (active learning justification)
|
||||
% 7. ADD brynjolfsson2017machine + ransbotham2020expanding (replace industry citations)
|
||||
% 8. ADD mayer2003cognitive OR nielsen1993usability (progressive disclosure)
|
||||
|
||||
% OPTIONAL (Nice to have):
|
||||
% 9. ADD sorva2013notional (notional machines)
|
||||
% 10. ADD robins2003learning (programming education foundations)
|
||||
% 11. Search and add 2-3 recent (2023-2024) ML education papers
|
||||
|
||||
% EXISTING ENTRIES TO CITE:
|
||||
% 12. Cite vygotsky1978mind when discussing scaffolding
|
||||
% 13. Cite thompson2008bloom when discussing assessment
|
||||
% 14. Consider citing bradbury2018jax in related work
|
||||
|
||||
% ============================================================================
|
||||
% WHERE TO CITE IN paper.tex
|
||||
% ============================================================================
|
||||
|
||||
% meadows2008thinking:
|
||||
% - Related Work: "systems thinking~\citep{meadows2008thinking}"
|
||||
% - Introduction: when discussing systems-first approach
|
||||
|
||||
% aho2006compilers:
|
||||
% - Introduction: "compiler course pedagogy~\citep{aho2006compilers}"
|
||||
% - Related Work: new subsection on Systems Pedagogy Foundations
|
||||
|
||||
% arpaci2018operating:
|
||||
% - Related Work: "operating systems courses~\citep{arpaci2018operating}"
|
||||
|
||||
% freeman2014active:
|
||||
% - Related Work: "active learning improves student performance~\citep{freeman2014active}"
|
||||
|
||||
% brynjolfsson2017machine + ransbotham2020expanding:
|
||||
% - Introduction: replace/supplement Robert Half/Keller citations
|
||||
% - Reframe workforce motivation as tacit knowledge problem
|
||||
|
||||
% mayer2003cognitive:
|
||||
% - Related Work: "progressive disclosure reduces cognitive load~\citep{mayer2003cognitive}"
|
||||
% - Progressive Disclosure section
|
||||
|
||||
% vygotsky1978mind (already in bib):
|
||||
% - Related Work: "scaffolding~\citep{vygotsky1978mind}"
|
||||
% - When discussing zone of proximal development
|
||||
|
||||
% thompson2008bloom (already in bib):
|
||||
% - NBGrader section: "Bloom's taxonomy for computing~\citep{thompson2008bloom}"
|
||||
|
||||
% ============================================================================
|
||||
@@ -1,700 +0,0 @@
|
||||
# TinyTorch Paper: Senior Python Developer Technical Review
|
||||
|
||||
**Reviewer Perspective**: Senior Python Developer with 10+ years building production ML systems at major tech companies. Extensive experience with PyTorch, TensorFlow, JAX, and production ML infrastructure.
|
||||
|
||||
**Review Date**: 2025-11-18
|
||||
|
||||
**Paper**: TinyTorch: Build Your Own Machine Learning Framework From Tensors to Systems
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Overall Assessment**: This is a pedagogically ambitious and well-structured curriculum with strong educational design, but the paper makes several claims about Python implementation details and production relevance that need significant revision. The paper would benefit from more honest acknowledgment of the educational vs. production trade-offs and correction of several technical inaccuracies.
|
||||
|
||||
**Recommendation**: Major revisions required before publication. The core curriculum idea is sound, but technical claims need careful review by Python/ML systems practitioners.
|
||||
|
||||
**Would I hire someone who completed this?** Yes, with caveats. They'd have excellent mental models of framework internals but would need significant additional training on production systems, modern Python practices, and real-world ML engineering.
|
||||
|
||||
---
|
||||
|
||||
## 1. Code Pedagogy & Python Best Practices
|
||||
|
||||
### 1.1 MAJOR CONCERNS: Python Anti-Patterns Being Taught
|
||||
|
||||
#### Issue #1: Monkey-Patching as Core Pedagogical Pattern (Lines 598-642, Section 4)
|
||||
|
||||
The paper presents monkey-patching as a "pedagogical innovation" for progressive disclosure. **This is concerning from a production Python perspective.**
|
||||
|
||||
**Code Example from Paper (Listing 2.2, lines 619-634):**
|
||||
```python
|
||||
def enable_autograd():
|
||||
"""Monkey-patch Tensor with gradients"""
|
||||
def backward(self, gradient=None):
|
||||
# ... implementation
|
||||
|
||||
# Monkey-patch: replace methods
|
||||
Tensor.backward = backward
|
||||
print("Autograd activated!")
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
|
||||
1. **Teaching bad habits**: Monkey-patching is widely considered an anti-pattern in production Python. It makes code unpredictable, breaks IDEs, confuses type checkers, and violates the principle of least surprise.
|
||||
|
||||
2. **Better alternatives exist**: The stated goal (gradual feature revelation) could be achieved through:
|
||||
- Abstract base classes with concrete implementations
|
||||
- Composition over inheritance patterns
|
||||
- Protocol-based typing (PEP 544)
|
||||
- Feature flags within the class
|
||||
- Separate `SimpleTensor` → `GradientTensor` class hierarchy
|
||||
|
||||
3. **Misleading PyTorch comparison**: The paper claims this "models how frameworks like PyTorch evolved (Variable/Tensor merger)" (line 365). This is **technically incorrect**. PyTorch 0.4's Variable/Tensor merger was a **compile-time refactoring**, not runtime monkey-patching. The C++ codebase was restructured—users didn't see runtime method replacement.
|
||||
|
||||
**Specific Technical Inaccuracy (line 723):**
|
||||
> "Early PyTorch (pre-0.4) separated data (`torch.Tensor`) from gradients (`torch.autograd.Variable`). PyTorch 0.4 (April 2018) consolidated functionality into `Tensor`, matching TinyTorch's pattern."
|
||||
|
||||
This statement conflates two different software engineering approaches:
|
||||
- PyTorch: Static type system changes, compile-time refactoring
|
||||
- TinyTorch: Runtime method replacement via monkey-patching
|
||||
|
||||
Students learning this pattern might think PyTorch actually uses monkey-patching internally, which would horrify the PyTorch core team.
|
||||
|
||||
**Recommendation**:
|
||||
- Remove claims that this mirrors PyTorch's evolution
|
||||
- Acknowledge monkey-patching as "pedagogically expedient but not production practice"
|
||||
- Add a "Production Python Note" box explaining why real frameworks don't do this
|
||||
- Consider alternative implementations for progressive disclosure
|
||||
|
||||
#### Issue #2: Missing Type Hints (Observed in code examples throughout)
|
||||
|
||||
**Code Example (Listing 2.1, lines 515-531):**
|
||||
```python
|
||||
class Tensor:
|
||||
def __init__(self, data):
|
||||
self.data = np.array(data, dtype=np.float32)
|
||||
self.shape = self.data.shape
|
||||
|
||||
def memory_footprint(self):
|
||||
"""Calculate exact memory in bytes"""
|
||||
return self.data.nbytes
|
||||
|
||||
def __matmul__(self, other):
|
||||
if self.shape[-1] != other.shape[0]:
|
||||
raise ValueError(
|
||||
f"Shape mismatch: {self.shape} @ {other.shape}"
|
||||
)
|
||||
return Tensor(self.data @ other.data)
|
||||
```
|
||||
|
||||
**What's Wrong:**
|
||||
- No type hints on any methods
|
||||
- Modern Python (3.5+) strongly encourages type hints for maintainability
|
||||
- PyTorch, TensorFlow, JAX all have extensive type annotations
|
||||
- Missing opportunity to teach static typing, which is critical in production ML
|
||||
|
||||
**What It Should Look Like:**
|
||||
```python
|
||||
from typing import Union, List, Tuple
|
||||
import numpy as np
|
||||
from numpy.typing import NDArray, ArrayLike
|
||||
|
||||
class Tensor:
|
||||
def __init__(self, data: ArrayLike, dtype: np.dtype = np.float32) -> None:
|
||||
self.data: NDArray[np.float32] = np.array(data, dtype=dtype)
|
||||
self.shape: Tuple[int, ...] = self.data.shape
|
||||
|
||||
def memory_footprint(self) -> int:
|
||||
"""Calculate exact memory in bytes"""
|
||||
return self.data.nbytes
|
||||
|
||||
def __matmul__(self, other: 'Tensor') -> 'Tensor':
|
||||
if self.shape[-1] != other.shape[0]:
|
||||
raise ValueError(
|
||||
f"Shape mismatch: {self.shape} @ {other.shape}"
|
||||
)
|
||||
return Tensor(self.data @ other.data)
|
||||
```
|
||||
|
||||
**Impact on Students:**
|
||||
- Students complete 20 modules without learning type hints
|
||||
- When they encounter production PyTorch code with type annotations, they'll be confused
|
||||
- Missing opportunity to teach mypy, which is essential for large Python codebases
|
||||
|
||||
**Recommendation**: Add type hints to all code examples, with a dedicated section explaining static typing in Python.
|
||||
|
||||
#### Issue #3: Adam Optimizer Implementation (Listing 1.2, lines 246-266)
|
||||
|
||||
**Code from Paper:**
|
||||
```python
|
||||
class Adam:
|
||||
def __init__(self, params, lr=0.001):
|
||||
self.params = params
|
||||
self.lr = lr
|
||||
# 2× optimizer state:
|
||||
# momentum + variance
|
||||
self.m = [Tensor.zeros_like(p) for p in params]
|
||||
self.v = [Tensor.zeros_like(p) for p in params]
|
||||
|
||||
def step(self):
|
||||
for p, m, v in zip(self.params, self.m, self.v):
|
||||
m = 0.9*m + 0.1*p.grad
|
||||
v = 0.999*v + 0.001*p.grad**2
|
||||
p.data -= (self.lr * m / (v.sqrt() + 1e-8))
|
||||
```
|
||||
|
||||
**Critical Problems:**
|
||||
|
||||
1. **Bug**: The `m` and `v` updates don't actually modify `self.m` and `self.v`. This creates local variables that get garbage collected. The next `step()` call uses the original (wrong) values.
|
||||
|
||||
**Should be:**
|
||||
```python
|
||||
def step(self):
|
||||
for i, p in enumerate(self.params):
|
||||
self.m[i] = 0.9 * self.m[i] + 0.1 * p.grad
|
||||
self.v[i] = 0.999 * self.v[i] + 0.001 * p.grad**2
|
||||
p.data -= (self.lr * self.m[i] / (self.v[i].sqrt() + 1e-8))
|
||||
```
|
||||
|
||||
2. **Wrong hyperparameters**: The code uses `0.9` for beta1 and `0.999` for beta2, but the update rule is wrong. Adam uses:
|
||||
```
|
||||
m_t = beta1 * m_{t-1} + (1 - beta1) * grad
|
||||
v_t = beta2 * v_{t-1} + (1 - beta2) * grad^2
|
||||
```
|
||||
|
||||
The code has `0.9*m + 0.1*grad` which is correct, but `0.999*v + 0.001*grad**2` is correct. However...
|
||||
|
||||
3. **Missing bias correction**: Adam requires bias correction terms that aren't shown:
|
||||
```python
|
||||
m_hat = m / (1 - beta1**t)
|
||||
v_hat = v / (1 - beta2**t)
|
||||
```
|
||||
Without these, the optimizer will behave incorrectly in early training steps.
|
||||
|
||||
**Impact**: Students implementing this will create a broken optimizer that appears to work but has subtle bugs. When they read PyTorch's Adam implementation, they'll be confused by the additional complexity.
|
||||
|
||||
**Recommendation**: Either fix the implementation or add a prominent note: "Simplified Adam for pedagogy—production implementations require bias correction and careful state management."
|
||||
|
||||
### 1.2 POSITIVE: Good Pedagogical Choices
|
||||
|
||||
Despite the issues above, several code patterns are excellent:
|
||||
|
||||
1. **Explicit loop-based implementations** (Listing 3.1, lines 749-771): The 7-nested-loop convolution is pedagogically brilliant. Students see exactly what's happening.
|
||||
|
||||
2. **Memory profiling integration** (Listing 2.1, line 521-523): Teaching `memory_footprint()` from day one is excellent.
|
||||
|
||||
3. **Progressive complexity**: Starting with explicit loops, then introducing vectorization is the right approach.
|
||||
|
||||
4. **Error messages** (line 527-529): Good use of f-strings for informative errors.
|
||||
|
||||
---
|
||||
|
||||
## 2. Technical Accuracy: Python, NumPy, PyTorch Claims
|
||||
|
||||
### 2.1 CRITICAL: Misleading Performance Claims
|
||||
|
||||
**Table 2.1 (lines 779-793): Runtime Comparison**
|
||||
|
||||
| Operation | TinyTorch | PyTorch | Ratio |
|
||||
|-----------|-----------|---------|-------|
|
||||
| `matmul` (1K×1K) | 1.0 s | 0.9 ms | 1,090× |
|
||||
| `conv2d` (CIFAR batch) | 97 s | 10 ms | 10,017× |
|
||||
| `softmax` (10K elem) | 6 ms | 0.05 ms | 134× |
|
||||
|
||||
**Problems:**
|
||||
|
||||
1. **Unfair comparison**: PyTorch's CPU performance uses MKL/OpenBLAS (highly optimized BLAS libraries), while TinyTorch uses pure Python loops. The comparison should be:
|
||||
- TinyTorch vs. NumPy (fair Python-to-Python)
|
||||
- NumPy vs. PyTorch CPU (library optimizations)
|
||||
- PyTorch CPU vs. PyTorch GPU (hardware acceleration)
|
||||
|
||||
2. **Missing NumPy baseline**: Since TinyTorch "builds on NumPy" (line 772), why not show NumPy's performance?
|
||||
|
||||
**My estimate**: NumPy would be ~10-100× slower than PyTorch CPU, not 1000×, because NumPy also uses MKL.
|
||||
|
||||
3. **Architectural mismatch**: The `matmul` comparison seems wrong. If TinyTorch uses `np.matmul` under the hood (as stated: "minimal NumPy reliance until concepts are established"), it should be much closer to PyTorch CPU. A 1000× difference suggests pure Python nested loops, which contradicts earlier claims about using NumPy.
|
||||
|
||||
**Specific Quote (lines 772-774):**
|
||||
> "This explicit implementation illustrates TinyTorch's pedagogical philosophy: **minimal NumPy reliance until concepts are established**."
|
||||
|
||||
This is contradictory. Earlier (line 184), the abstract says students "implement PyTorch's core components—tensors, autograd, optimizers—to gain framework transparency" **using only NumPy**.
|
||||
|
||||
**Which is it?**
|
||||
- "Only NumPy" (abstract, line 184)
|
||||
- "Minimal NumPy reliance" (section 3.2, line 772)
|
||||
|
||||
These are different claims. If you're using `np.matmul`, the 1000× slowdown claim is misleading.
|
||||
|
||||
**Recommendation**:
|
||||
1. Clarify exactly which operations use NumPy vs. pure Python
|
||||
2. Provide fair comparisons: TinyTorch vs. NumPy vs. PyTorch CPU vs. PyTorch GPU
|
||||
3. Remove misleading performance ratios
|
||||
|
||||
### 2.2 CONCERN: Oversimplified Memory Calculations
|
||||
|
||||
**Example (lines 534, 740):**
|
||||
> "Matrix multiplication A @ B where both are (1000, 1000) FP32 requires 12MB peak memory: 4MB for A, 4MB for B, and 4MB for the output."
|
||||
|
||||
**What's Missing:**
|
||||
|
||||
1. **Intermediate allocations**: In practice, `np.matmul` creates temporary arrays during computation. Peak memory is often higher than input + output.
|
||||
|
||||
2. **Memory alignment**: Modern CPUs require memory alignment for SIMD operations. NumPy may allocate extra bytes.
|
||||
|
||||
3. **Python object overhead**: Each `Tensor` object has Python overhead (dict, refcount, etc.). For small tensors, this dominates.
|
||||
|
||||
4. **Fragmentation**: Memory allocators don't guarantee contiguous allocation. Peak RSS is often higher than theoretical minimum.
|
||||
|
||||
**Real-world example:**
|
||||
```python
|
||||
import numpy as np
|
||||
import psutil
|
||||
import os
|
||||
|
||||
process = psutil.Process(os.getpid())
|
||||
before = process.memory_info().rss / 1024**2
|
||||
|
||||
A = np.random.randn(1000, 1000).astype(np.float32)
|
||||
B = np.random.randn(1000, 1000).astype(np.float32)
|
||||
C = A @ B
|
||||
|
||||
after = process.memory_info().rss / 1024**2
|
||||
print(f"Theoretical: 12 MB")
|
||||
print(f"Actual: {after - before:.1f} MB") # Often 15-20 MB
|
||||
```
|
||||
|
||||
**Impact**: Students will be confused when their memory profiling shows higher usage than calculated.
|
||||
|
||||
**Recommendation**: Add a note about "theoretical minimum vs. practical memory usage" and teach students to measure actual memory with `psutil` or similar tools.
|
||||
|
||||
### 2.3 POSITIVE: Accurate Complexity Analysis
|
||||
|
||||
The paper correctly identifies:
|
||||
- Convolution as O(B × C_out × H_out × W_out × C_in × K_h × K_w) ✓
|
||||
- Attention as O(N²) ✓
|
||||
- Adam's 2× optimizer state overhead ✓
|
||||
|
||||
These are accurate and well-explained.
|
||||
|
||||
---
|
||||
|
||||
## 3. Real-World Relevance & Production Readiness
|
||||
|
||||
### 3.1 MAJOR GAP: What Students WON'T Learn
|
||||
|
||||
The paper acknowledges GPU/distributed training gaps (lines 964-972) but **understates** how critical these are.
|
||||
|
||||
**Missing skills for production ML engineering:**
|
||||
|
||||
1. **GPU fundamentals** (not optional):
|
||||
- Memory hierarchy (global/shared/local/register)
|
||||
- Kernel fusion and graph optimization
|
||||
- Mixed precision training (FP16/BF16)
|
||||
- Gradient accumulation across devices
|
||||
|
||||
**Reality**: 95% of production ML training uses GPUs. CPU-only knowledge is insufficient.
|
||||
|
||||
2. **Distributed training** (essential for modern models):
|
||||
- Data parallelism (DDP, FSDP)
|
||||
- Model parallelism (pipeline, tensor)
|
||||
- Gradient synchronization
|
||||
- Communication bottlenecks
|
||||
|
||||
**Reality**: GPT-3 requires 1024 GPUs. Can't train modern models on single CPU.
|
||||
|
||||
3. **Modern Python tooling**:
|
||||
- Type checking (mypy, pyright)
|
||||
- Linting (ruff, pylint)
|
||||
- Testing frameworks (pytest, hypothesis)
|
||||
- Profiling (py-spy, scalene)
|
||||
- Packaging (poetry, uv)
|
||||
|
||||
4. **Production ML systems**:
|
||||
- Model serving (TorchServe, TensorFlow Serving)
|
||||
- ONNX export and optimization
|
||||
- Quantization beyond int8 (int4, mixed precision)
|
||||
- Latency SLAs and throughput optimization
|
||||
- A/B testing and model versioning
|
||||
|
||||
**Quote from paper (line 356):**
|
||||
> "Students needing immediate GPU/distributed training skills are better served by PyTorch tutorials"
|
||||
|
||||
**Counter-argument**: This dismisses the **primary skill gap** in ML engineering. The industry needs GPU-literate engineers, not CPU-only framework builders.
|
||||
|
||||
**Recommendation**: Reframe positioning from "prepares students for ML systems engineering" to "prepares students to understand framework internals before learning production systems."
|
||||
|
||||
### 3.2 CONCERN: "Systems-First" Framing vs. Reality
|
||||
|
||||
The paper claims "systems-first curriculum" (line 363) but the actual systems content is **introductory at best**.
|
||||
|
||||
**What the paper calls "systems":**
|
||||
- Calculating memory footprints (basic arithmetic)
|
||||
- Counting FLOPs (complexity analysis)
|
||||
- Measuring wall-clock time (basic profiling)
|
||||
|
||||
**What production systems engineering actually involves:**
|
||||
- Memory bandwidth analysis and cache optimization
|
||||
- Kernel fusion and compiler optimizations
|
||||
- Network topology and communication patterns
|
||||
- Batch scheduling and request routing
|
||||
- Cost optimization and resource allocation
|
||||
|
||||
**Example: Module 14 "Profiling" (line 500)**
|
||||
The paper lists this as teaching "bottleneck identification, measurement overhead" but doesn't mention:
|
||||
- Flame graphs
|
||||
- Line profilers (line_profiler)
|
||||
- Memory profilers (memory_profiler, memray)
|
||||
- GPU profilers (nvprof, nsys)
|
||||
- Distributed profilers (torch.profiler with TensorBoard)
|
||||
|
||||
**Recommendation**: Rename "systems-first" to "systems-aware" or "systems-oriented" to set appropriate expectations.
|
||||
|
||||
### 3.3 POSITIVE: Strong Foundation for Further Learning
|
||||
|
||||
**What students WILL gain:**
|
||||
|
||||
1. **Mental models**: Understanding computational graphs, gradient flow, and memory layout
|
||||
2. **Debugging intuition**: Knowing where gradients come from helps debug shape mismatches
|
||||
3. **Architecture understanding**: Why Conv2d has fewer parameters than Dense
|
||||
4. **Trade-off reasoning**: Accuracy vs. speed vs. memory
|
||||
|
||||
**These are valuable**, just not sufficient for production work.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Concerns & Code Quality
|
||||
|
||||
### 4.1 CRITICAL: Autograd Implementation Pattern
|
||||
|
||||
**Code from listing 3.2 (lines 619-634) and Section 4:**
|
||||
|
||||
The monkey-patching approach for autograd has a **fundamental software engineering problem**:
|
||||
|
||||
**Problem: Thread Safety**
|
||||
```python
|
||||
# Module 01-04: Tensor without autograd
|
||||
x = Tensor([1.0, 2.0])
|
||||
y = x * 2 # No gradient tracking
|
||||
|
||||
# Module 05: Enable autograd
|
||||
enable_autograd() # GLOBAL STATE CHANGE
|
||||
|
||||
# Now ALL tensors track gradients
|
||||
z = Tensor([3.0, 4.0]) # Gradients enabled
|
||||
w = x * 3 # Old tensor now has gradients!
|
||||
```
|
||||
|
||||
**Issues:**
|
||||
|
||||
1. **Global mutable state**: `enable_autograd()` modifies the `Tensor` class globally. This is a classic anti-pattern.
|
||||
|
||||
2. **No way to disable**: Once enabled, you can't turn it off without reloading the module.
|
||||
|
||||
3. **Context-dependent behavior**: Same code behaves differently based on whether `enable_autograd()` was called.
|
||||
|
||||
4. **Testing nightmare**: Tests that call `enable_autograd()` affect all subsequent tests.
|
||||
|
||||
**Production approach (PyTorch):**
|
||||
```python
|
||||
# Context manager for gradient control
|
||||
with torch.no_grad():
|
||||
y = x * 2 # No gradients
|
||||
|
||||
y = x * 2 # Gradients tracked
|
||||
|
||||
# Explicit control
|
||||
x = torch.tensor([1.0], requires_grad=False) # No gradients
|
||||
y = torch.tensor([1.0], requires_grad=True) # Track gradients
|
||||
```
|
||||
|
||||
**Recommendation**: Implement autograd as an opt-in feature, not a monkey-patched global state change.
|
||||
|
||||
### 4.2 CONCERN: Error Handling & Edge Cases
|
||||
|
||||
**Throughout code examples**, error handling is minimal:
|
||||
|
||||
**Example problems:**
|
||||
|
||||
1. **Division by zero** (Adam optimizer, line 265):
|
||||
```python
|
||||
p.data -= (self.lr * m / (v.sqrt() + 1e-8))
|
||||
```
|
||||
What if `v` contains NaN or Inf? No check.
|
||||
|
||||
2. **Shape broadcasting** (Tensor operations):
|
||||
No validation that broadcasting rules are satisfied. Silent errors.
|
||||
|
||||
3. **Out of memory**:
|
||||
No guidance on handling OOM errors when creating large tensors.
|
||||
|
||||
4. **Gradient explosion/vanishing**:
|
||||
No clipping, no checks for NaN/Inf in gradients.
|
||||
|
||||
**Production code includes:**
|
||||
- Input validation
|
||||
- Explicit error messages
|
||||
- Gradient clipping
|
||||
- NaN/Inf detection
|
||||
- Resource limits
|
||||
|
||||
**Recommendation**: Add a "Production Hardening" module teaching error handling and edge cases.
|
||||
|
||||
### 4.3 POSITIVE: Clean API Design
|
||||
|
||||
Despite implementation issues, the API design is good:
|
||||
|
||||
1. **PyTorch-compatible imports**: `from tinytorch.nn import Linear` feels familiar
|
||||
2. **Consistent method names**: `.forward()`, `.backward()`, `.step()`
|
||||
3. **Progressive accumulation**: Each module adds capabilities naturally
|
||||
|
||||
---
|
||||
|
||||
## 5. Framework Comparisons
|
||||
|
||||
### 5.1 CONCERN: Oversimplified PyTorch Comparison
|
||||
|
||||
**Table 1.1 (lines 409-435): Framework Comparison**
|
||||
|
||||
The table positions TinyTorch as teaching "systems" while PyTorch has "advanced (implicit)" systems focus.
|
||||
|
||||
**This is misleading.**
|
||||
|
||||
**PyTorch's systems engineering** includes:
|
||||
- Custom CUDA kernels
|
||||
- Distributed communication
|
||||
- Memory allocation strategies
|
||||
- Graph optimization
|
||||
- Operator fusion
|
||||
- Mixed precision training
|
||||
|
||||
These are **explicit and documented**, not "implicit." PyTorch has extensive documentation on:
|
||||
- `torch.cuda` API
|
||||
- Distributed training
|
||||
- TorchScript compilation
|
||||
- ONNX export
|
||||
|
||||
**Recommendation**: Change "Advanced (implicit)" to "Advanced (production-focused)" for PyTorch/TensorFlow.
|
||||
|
||||
### 5.2 POSITIVE: Fair Positioning vs. Educational Frameworks
|
||||
|
||||
The comparisons to micrograd, MiniTorch, and tinygrad are fair and accurate (lines 379-448).
|
||||
|
||||
---
|
||||
|
||||
## 6. Specific Technical Corrections Needed
|
||||
|
||||
### 6.1 Line-by-Line Issues
|
||||
|
||||
**Line 180-181**: "understanding *why* Adam requires 2× optimizer state memory"
|
||||
- **Correction**: Adam requires **3× memory** in practice:
|
||||
- 1× for parameters
|
||||
- 1× for first moment (momentum)
|
||||
- 1× for second moment (variance)
|
||||
- **Plus** gradient memory during backward pass
|
||||
- Total: 4× during training (params + gradients + m + v)
|
||||
|
||||
**Line 264-265**: Adam update rule missing bias correction
|
||||
- **Fix**: Add bias correction terms or note this is simplified
|
||||
|
||||
**Line 534**: "12MB peak memory for 1K×1K matmul"
|
||||
- **Clarification**: Add "theoretical minimum; actual usage may be higher due to allocation overhead"
|
||||
|
||||
**Line 723**: "PyTorch 0.4 consolidated functionality into Tensor, matching TinyTorch's pattern"
|
||||
- **Correction**: Remove "matching TinyTorch's pattern" as PyTorch didn't use monkey-patching
|
||||
|
||||
**Line 773**: "minimal NumPy reliance"
|
||||
- **Clarification**: Specify exactly which operations use NumPy primitives vs. pure Python
|
||||
|
||||
**Line 184**: "using only NumPy"
|
||||
- **Contradiction**: This conflicts with "minimal NumPy" claim on line 773
|
||||
|
||||
### 6.2 Missing Technical Details
|
||||
|
||||
**Should add:**
|
||||
|
||||
1. **Computational graph memory**: How much memory does the graph itself consume?
|
||||
|
||||
2. **Gradient accumulation**: Why is this important for large batch sizes?
|
||||
|
||||
3. **In-place operations**: Why do they matter for memory efficiency?
|
||||
|
||||
4. **View vs. copy**: When does NumPy return a view vs. allocating new memory?
|
||||
|
||||
---
|
||||
|
||||
## 7. Practical Value Assessment
|
||||
|
||||
### 7.1 Would I Hire Someone Who Completed This?
|
||||
|
||||
**Yes, with significant caveats.**
|
||||
|
||||
**What they'd know:**
|
||||
- ✅ How autograd works
|
||||
- ✅ Why attention is O(N²)
|
||||
- ✅ Memory calculation fundamentals
|
||||
- ✅ Basic optimization trade-offs
|
||||
- ✅ Framework architecture patterns
|
||||
|
||||
**What they'd still need:**
|
||||
- ❌ GPU programming (CUDA, cuDNN)
|
||||
- ❌ Distributed training
|
||||
- ❌ Production serving
|
||||
- ❌ Modern Python tooling (mypy, ruff, pytest)
|
||||
- ❌ MLOps (experiment tracking, versioning)
|
||||
- ❌ Real-world debugging (profilers, debuggers)
|
||||
|
||||
**Hiring scenario:**
|
||||
|
||||
- **ML Research Engineer role**: Strong candidate with good fundamentals
|
||||
- **ML Infrastructure role**: Would need 6+ months additional training
|
||||
- **Production ML Engineer role**: Would need significant mentoring on systems/GPU
|
||||
|
||||
**Comparison to other backgrounds:**
|
||||
|
||||
- **TinyTorch grad vs. CS229 grad**: TinyTorch student has better internals knowledge
|
||||
- **TinyTorch grad vs. PyTorch power user**: PyTorch user has better practical skills
|
||||
- **TinyTorch grad vs. Stanford CS231n student**: Roughly comparable, different emphases
|
||||
|
||||
### 7.2 What Would They Still Need to Learn?
|
||||
|
||||
**Immediate gaps (3-6 months):**
|
||||
1. Modern Python development (type hints, testing, tooling)
|
||||
2. GPU fundamentals and CUDA basics
|
||||
3. Production serving and deployment
|
||||
4. Experiment tracking and MLOps
|
||||
|
||||
**Medium-term gaps (6-12 months):**
|
||||
1. Distributed training systems
|
||||
2. Advanced optimization techniques
|
||||
3. Model compression beyond basic quantization
|
||||
4. Cost optimization and resource allocation
|
||||
|
||||
**Long-term expertise (1-2 years):**
|
||||
1. Custom CUDA kernels
|
||||
2. Compiler optimizations
|
||||
3. Hardware-software co-design
|
||||
4. Large-scale system architecture
|
||||
|
||||
---
|
||||
|
||||
## 8. Overall Recommendations
|
||||
|
||||
### 8.1 Critical Changes Needed
|
||||
|
||||
1. **Fix Adam implementation** (Listing 1.2): Add bias correction or explicit disclaimer
|
||||
|
||||
2. **Remove/revise monkey-patching claims**: Don't present this as "how PyTorch works"
|
||||
|
||||
3. **Add type hints**: Teach modern Python practices
|
||||
|
||||
4. **Clarify NumPy usage**: "Only NumPy" vs. "minimal NumPy" contradiction
|
||||
|
||||
5. **Fix memory calculations**: Add overhead discussion
|
||||
|
||||
6. **Tone down "systems-first" claims**: Be honest about scope limitations
|
||||
|
||||
### 8.2 Recommended Additions
|
||||
|
||||
1. **"Production Python" boxes**: Throughout paper, add notes on production practices
|
||||
|
||||
2. **Comparison table**: TinyTorch concepts → PyTorch implementations
|
||||
|
||||
3. **"What's Next" section**: Clear roadmap from TinyTorch to production work
|
||||
|
||||
4. **Error handling module**: Teach production-grade code quality
|
||||
|
||||
5. **Type checking module**: Introduce mypy and static typing
|
||||
|
||||
### 8.3 Positioning Recommendations
|
||||
|
||||
**Current framing**: "TinyTorch prepares students for ML systems engineering"
|
||||
|
||||
**Better framing**: "TinyTorch builds mental models of framework internals, preparing students to learn production ML systems"
|
||||
|
||||
**Key message**: This is **foundational education**, not **production training**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Strengths Worth Emphasizing
|
||||
|
||||
Despite criticisms above, the paper has significant strengths:
|
||||
|
||||
1. **Pedagogical innovation**: Progressive disclosure (despite implementation concerns) is creative
|
||||
|
||||
2. **Historical milestones**: Brilliant motivational device connecting history to implementation
|
||||
|
||||
3. **Integration testing**: Understanding that components must compose is crucial
|
||||
|
||||
4. **Memory-first thinking**: Teaching memory awareness from day one is excellent
|
||||
|
||||
5. **Accessibility**: CPU-only design democratizes access to ML education
|
||||
|
||||
6. **Honest scope**: Section 5.1 (lines 962-972) honestly acknowledges GPU/distributed gaps
|
||||
|
||||
---
|
||||
|
||||
## 10. Final Verdict
|
||||
|
||||
**Technical Accuracy**: 6/10
|
||||
- Several misleading claims about PyTorch, performance, and production systems
|
||||
- Good on complexity analysis, weak on implementation details
|
||||
|
||||
**Code Quality**: 5/10
|
||||
- Monkey-patching anti-pattern is concerning
|
||||
- Missing type hints throughout
|
||||
- Adam implementation has bugs
|
||||
- Good API design
|
||||
|
||||
**Pedagogical Value**: 8/10
|
||||
- Excellent curriculum structure
|
||||
- Creative teaching techniques
|
||||
- Clear learning progression
|
||||
- Good accessibility
|
||||
|
||||
**Production Relevance**: 4/10
|
||||
- Significant gaps in GPU, distributed, and production skills
|
||||
- Overstates "systems" preparation
|
||||
- Good foundation but not sufficient alone
|
||||
|
||||
**Overall**: 6/10 - Strong educational concept with significant technical issues that need addressing.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Suggested Text Revisions
|
||||
|
||||
### Original (Line 180-181):
|
||||
> "understanding *why* Adam requires 2× optimizer state memory"
|
||||
|
||||
### Suggested:
|
||||
> "understanding why Adam requires additional optimizer state memory (momentum and variance buffers, often doubling memory footprint)"
|
||||
|
||||
---
|
||||
|
||||
### Original (Lines 723-725):
|
||||
> "Early PyTorch (pre-0.4) separated data (`torch.Tensor`) from gradients (`torch.autograd.Variable`). PyTorch 0.4 (April 2018) consolidated functionality into `Tensor`, matching TinyTorch's pattern."
|
||||
|
||||
### Suggested:
|
||||
> "Early PyTorch (pre-0.4) separated data (`torch.Tensor`) from gradients (`torch.autograd.Variable`). PyTorch 0.4 (April 2018) consolidated functionality into `Tensor` through a compile-time refactoring of the C++ codebase. While the end result—a unified Tensor class—resembles TinyTorch's design, PyTorch's implementation used static type system changes rather than runtime method enhancement."
|
||||
|
||||
---
|
||||
|
||||
### Original (Line 356):
|
||||
> "Students needing immediate GPU/distributed training skills are better served by PyTorch tutorials"
|
||||
|
||||
### Suggested:
|
||||
> "TinyTorch provides foundations for understanding framework internals; students should follow up with GPU programming (PyTorch tutorials, NVIDIA DLI) and distributed training courses (PyTorch Distributed, DeepSpeed) for production ML engineering roles."
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
This is an ambitious and thoughtful educational project that would significantly benefit from technical review by production Python/ML engineers. The core curriculum idea is sound, but the paper needs revision to:
|
||||
|
||||
1. Fix technical inaccuracies
|
||||
2. Remove misleading comparisons
|
||||
3. Add modern Python practices
|
||||
4. Set realistic expectations about production readiness
|
||||
|
||||
With these revisions, this could be an excellent contribution to ML education. As written, it risks teaching anti-patterns alongside valuable concepts.
|
||||
|
||||
**Bottom line**: I'd enthusiastically recommend this course to someone who wants to understand framework internals, with the explicit caveat that they'll need significant additional training for production ML work.
|
||||
BIN
paper/paper.pdf
BIN
paper/paper.pdf
Binary file not shown.
@@ -1,396 +0,0 @@
|
||||
% PROPOSED PEDAGOGICAL FIGURES FOR TINYTORCH PAPER
|
||||
% Generated: 2025-11-17
|
||||
% Status: Draft - Ready for Review and Integration
|
||||
|
||||
\documentclass{article}
|
||||
\usepackage{tikz}
|
||||
\usetikzlibrary{shapes,arrows,positioning,decorations.pathreplacing,calc}
|
||||
\usepackage{geometry}
|
||||
\geometry{margin=1in}
|
||||
\usepackage{xcolor}
|
||||
|
||||
% Define colors matching paper
|
||||
\definecolor{accentcolor}{RGB}{255,87,34}
|
||||
\definecolor{dormantgray}{RGB}{200,200,200}
|
||||
\definecolor{activeorange}{RGB}{255,152,0}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\section*{Proposed Pedagogical Figures for TinyTorch Paper}
|
||||
|
||||
% ============================================================
|
||||
% FIGURE A: PROGRESSIVE DISCLOSURE TIMELINE
|
||||
% ============================================================
|
||||
\subsection*{Figure A: Progressive Disclosure Timeline (HIGHEST PRIORITY)}
|
||||
\textbf{Location:} Section 3.1 (Progressive Disclosure), after Listing 2\\
|
||||
\textbf{Pedagogical Value:} Visualizes the paper's most novel contribution - how Tensor capabilities evolve across modules while maintaining single mental model.
|
||||
|
||||
\begin{figure}[h]
|
||||
\centering
|
||||
\begin{tikzpicture}[
|
||||
scale=0.9,
|
||||
every node/.style={font=\small},
|
||||
module/.style={rectangle, draw, fill=blue!20, minimum width=1.2cm, minimum height=0.8cm},
|
||||
dormant/.style={rectangle, draw=dormantgray, fill=dormantgray!20, text=gray},
|
||||
active/.style={rectangle, draw=activeorange, fill=activeorange!30, text=black, font=\small\bfseries}
|
||||
]
|
||||
|
||||
% Timeline axis
|
||||
\draw[thick, ->] (0,0) -- (14,0) node[right] {Modules};
|
||||
|
||||
% Module markers
|
||||
\foreach \x/\label in {1/01, 3.5/03, 6/05, 8.5/09, 11/13, 13.5/20} {
|
||||
\draw (\x, 0.1) -- (\x, -0.1);
|
||||
\node[below] at (\x, -0.2) {\texttt{M\label}};
|
||||
}
|
||||
|
||||
% Feature layers - stacked above timeline
|
||||
% Layer 1: Basic Tensor (always present)
|
||||
\node[active] at (1, 1.5) {\texttt{.data}};
|
||||
\node[active] at (2.5, 1.5) {\texttt{.shape}};
|
||||
\draw[activeorange, thick] (0.3, 1.5) -- (13.7, 1.5);
|
||||
\node[left, font=\scriptsize] at (0.2, 1.5) {Core};
|
||||
|
||||
% Layer 2: Dormant until Module 05
|
||||
\node[dormant] at (1, 2.5) {\texttt{.requires\_grad}};
|
||||
\draw[dormantgray, thick, dashed] (0.3, 2.5) -- (5.5, 2.5);
|
||||
\node[active] at (7, 2.5) {\texttt{.requires\_grad}};
|
||||
\draw[activeorange, thick] (6.3, 2.5) -- (13.7, 2.5);
|
||||
\node[left, font=\scriptsize] at (0.2, 2.5) {Gradient};
|
||||
|
||||
\node[dormant] at (2.5, 3.2) {\texttt{.grad}};
|
||||
\draw[dormantgray, thick, dashed] (0.3, 3.2) -- (5.5, 3.2);
|
||||
\node[active] at (7, 3.2) {\texttt{.grad}};
|
||||
\draw[activeorange, thick] (6.3, 3.2) -- (13.7, 3.2);
|
||||
|
||||
\node[dormant] at (1.5, 3.9) {\texttt{.backward()}};
|
||||
\draw[dormantgray, thick, dashed] (0.3, 3.9) -- (5.5, 3.9);
|
||||
\node[active] at (7, 3.9) {\texttt{.backward()}};
|
||||
\draw[activeorange, thick] (6.3, 3.9) -- (13.7, 3.9);
|
||||
|
||||
% Activation marker at Module 05
|
||||
\node[draw, fill=yellow!30, circle, font=\scriptsize\bfseries] at (6, 4.8) {ACTIVATION};
|
||||
\draw[thick, ->] (6, 4.6) -- (6, 4.1);
|
||||
|
||||
% Annotations
|
||||
\node[align=center, font=\scriptsize] at (3, 5.5) {
|
||||
\textbf{Modules 01-04:}\\
|
||||
Features visible but dormant\\
|
||||
\texttt{.backward()} is no-op
|
||||
};
|
||||
|
||||
\node[align=center, font=\scriptsize] at (10, 5.5) {
|
||||
\textbf{Modules 05-20:}\\
|
||||
Autograd fully active\\
|
||||
Gradients flow automatically
|
||||
};
|
||||
|
||||
% Legend
|
||||
\node[dormant, minimum width=1cm] at (2, -1.5) {Dormant};
|
||||
\node[active, minimum width=1cm] at (4.5, -1.5) {Active};
|
||||
|
||||
\end{tikzpicture}
|
||||
\caption{Progressive disclosure of \texttt{Tensor} capabilities across modules. Gradient-related features (\texttt{.requires\_grad}, \texttt{.grad}, \texttt{.backward()}) exist from Module 01 but remain dormant (gray, dashed) until Module 05 activates them via monkey-patching (orange, solid). Students work with a single \texttt{Tensor} interface throughout, but capabilities expand progressively. This manages cognitive load while maintaining conceptual unity.}
|
||||
\label{fig:progressive-timeline}
|
||||
\end{figure}
|
||||
|
||||
\clearpage
|
||||
|
||||
% ============================================================
|
||||
% FIGURE B: MEMORY HIERARCHY BREAKDOWN
|
||||
% ============================================================
|
||||
\subsection*{Figure B: Memory Hierarchy Breakdown (HIGH PRIORITY)}
|
||||
\textbf{Location:} Section 4.1 (Memory Profiling), after Table 1\\
|
||||
\textbf{Pedagogical Value:} Clarifies that "Adam uses 3× parameter memory" refers to optimizer state, while activations typically dominate total memory. Visual makes this concrete.
|
||||
|
||||
\begin{figure}[h]
|
||||
\centering
|
||||
\begin{tikzpicture}[
|
||||
scale=0.85,
|
||||
every node/.style={font=\small}
|
||||
]
|
||||
|
||||
% Define bar widths and positions
|
||||
\def\barwidth{1.5}
|
||||
\def\unitheight{0.3}
|
||||
|
||||
% SGD Memory Breakdown (Left)
|
||||
\node[font=\normalsize\bfseries] at (2, 8) {SGD Optimizer};
|
||||
|
||||
% Parameters (1x)
|
||||
\fill[blue!60] (0.5, 0) rectangle +({\barwidth}, {1*\unitheight});
|
||||
\node[right, font=\scriptsize] at (2.1, {0.5*\unitheight}) {Parameters (1×)};
|
||||
|
||||
% Gradients (1x)
|
||||
\fill[green!60] (0.5, {1*\unitheight}) rectangle +({\barwidth}, {1*\unitheight});
|
||||
\node[right, font=\scriptsize] at (2.1, {1.5*\unitheight}) {Gradients (1×)};
|
||||
|
||||
% Activations (10-100x)
|
||||
\fill[red!40] (0.5, {2*\unitheight}) rectangle +({\barwidth}, {30*\unitheight});
|
||||
\node[right, font=\scriptsize, align=left] at (2.1, {17*\unitheight}) {
|
||||
Activations\\(10-100×)\\
|
||||
\textbf{Dominates!}
|
||||
};
|
||||
|
||||
% Total annotation
|
||||
\draw[thick, <->] (-0.3, 0) -- (-0.3, {32*\unitheight});
|
||||
\node[left, font=\scriptsize, align=right] at (-0.4, {16*\unitheight}) {
|
||||
Total:\\32× params
|
||||
};
|
||||
|
||||
% Adam Memory Breakdown (Right)
|
||||
\node[font=\normalsize\bfseries] at (8, 8) {Adam Optimizer};
|
||||
|
||||
% Parameters (1x)
|
||||
\fill[blue!60] (6.5, 0) rectangle +({\barwidth}, {1*\unitheight});
|
||||
\node[right, font=\scriptsize] at (8.1, {0.5*\unitheight}) {Parameters (1×)};
|
||||
|
||||
% Gradients (1x)
|
||||
\fill[green!60] (6.5, {1*\unitheight}) rectangle +({\barwidth}, {1*\unitheight});
|
||||
\node[right, font=\scriptsize] at (8.1, {1.5*\unitheight}) {Gradients (1×)};
|
||||
|
||||
% Adam states: momentum (1x)
|
||||
\fill[orange!60] (6.5, {2*\unitheight}) rectangle +({\barwidth}, {1*\unitheight});
|
||||
\node[right, font=\scriptsize] at (8.1, {2.5*\unitheight}) {Momentum (1×)};
|
||||
|
||||
% Adam states: variance (1x)
|
||||
\fill[orange!80] (6.5, {3*\unitheight}) rectangle +({\barwidth}, {1*\unitheight});
|
||||
\node[right, font=\scriptsize] at (8.1, {3.5*\unitheight}) {Variance (1×)};
|
||||
|
||||
% Brace for optimizer states
|
||||
\draw[decorate, decoration={brace, amplitude=5pt}]
|
||||
(6.3, {2*\unitheight}) -- (6.3, {4*\unitheight})
|
||||
node[midway, left, xshift=-3pt, font=\scriptsize] {+2× states};
|
||||
|
||||
% Activations (10-100x) - same as SGD
|
||||
\fill[red!40] (6.5, {4*\unitheight}) rectangle +({\barwidth}, {30*\unitheight});
|
||||
\node[right, font=\scriptsize, align=left] at (8.1, {19*\unitheight}) {
|
||||
Activations\\(10-100×)\\
|
||||
\textbf{Still dominates!}
|
||||
};
|
||||
|
||||
% Total annotation
|
||||
\draw[thick, <->] (5.7, 0) -- (5.7, {34*\unitheight});
|
||||
\node[left, font=\scriptsize, align=right] at (5.6, {17*\unitheight}) {
|
||||
Total:\\34× params
|
||||
};
|
||||
|
||||
% Key insight box
|
||||
\node[draw, thick, fill=yellow!20, align=center, font=\scriptsize] at (6.5, -1.5) {
|
||||
\textbf{Key Insight:} Adam adds 2× parameter memory\\
|
||||
(3× total vs 1× for SGD), but activations\\
|
||||
still dominate overall memory usage
|
||||
};
|
||||
|
||||
% Grid lines for easier reading
|
||||
\foreach \y in {0,5,10,15,20,25,30} {
|
||||
\draw[dotted, gray] (0, {\y*\unitheight}) -- (10.5, {\y*\unitheight});
|
||||
}
|
||||
|
||||
\end{tikzpicture}
|
||||
\caption{Memory hierarchy breakdown comparing SGD and Adam optimizers. While Adam requires 3× parameter memory (parameters + gradients + momentum + variance) compared to SGD's 2× (parameters + gradients), activation memory typically dominates total memory consumption by 10-100×. This visualization clarifies that optimizer choice affects parameter memory overhead, but activation memory remains the primary concern for most models. Students learn to calculate each component from Module 01 onwards.}
|
||||
\label{fig:memory-breakdown}
|
||||
\end{figure}
|
||||
|
||||
\clearpage
|
||||
|
||||
% ============================================================
|
||||
% FIGURE C: BUILD-USE-REFLECT CYCLE
|
||||
% ============================================================
|
||||
\subsection*{Figure C: Build→Use→Reflect Cycle (HIGH PRIORITY)}
|
||||
\textbf{Location:} Section 2.3 (Module Structure), replacing or supplementing paragraph text\\
|
||||
\textbf{Pedagogical Value:} Core pedagogical pattern structuring all 20 modules. Visual makes the iterative cycle explicit.
|
||||
|
||||
\begin{figure}[h]
|
||||
\centering
|
||||
\begin{tikzpicture}[
|
||||
scale=1.0,
|
||||
every node/.style={font=\small},
|
||||
phase/.style={
|
||||
circle,
|
||||
draw,
|
||||
thick,
|
||||
minimum size=2.8cm,
|
||||
align=center,
|
||||
font=\normalsize\bfseries
|
||||
},
|
||||
example/.style={
|
||||
rectangle,
|
||||
draw,
|
||||
fill=blue!10,
|
||||
rounded corners,
|
||||
align=left,
|
||||
font=\scriptsize,
|
||||
text width=4cm
|
||||
}
|
||||
]
|
||||
|
||||
% Three main phases in circular arrangement
|
||||
\node[phase, fill=blue!30] (build) at (0, 4) {
|
||||
BUILD\\[0.3em]
|
||||
\normalfont\scriptsize Implementation
|
||||
};
|
||||
|
||||
\node[phase, fill=green!30] (use) at (4, 0) {
|
||||
USE\\[0.3em]
|
||||
\normalfont\scriptsize Integration
|
||||
};
|
||||
|
||||
\node[phase, fill=orange!30] (reflect) at (-4, 0) {
|
||||
REFLECT\\[0.3em]
|
||||
\normalfont\scriptsize Analysis
|
||||
};
|
||||
|
||||
% Arrows connecting phases
|
||||
\draw[->, ultra thick, blue!70] (build) to[bend left=20] node[midway, above right, font=\scriptsize] {Test} (use);
|
||||
\draw[->, ultra thick, green!70] (use) to[bend left=20] node[midway, below, font=\scriptsize] {Analyze} (reflect);
|
||||
\draw[->, ultra thick, orange!70] (reflect) to[bend left=20] node[midway, above left, font=\scriptsize] {Iterate} (build);
|
||||
|
||||
% Example boxes for each phase
|
||||
\node[example, below=0.8cm of build] (build-ex) {
|
||||
\textbf{Module 05 Example:}\\
|
||||
• Implement \texttt{backward()}\\
|
||||
• Build computation graph\\
|
||||
• Create gradient accumulation\\
|
||||
• Scaffold: Connection maps
|
||||
};
|
||||
|
||||
\node[example, right=0.8cm of use] (use-ex) {
|
||||
\textbf{Module 05 Example:}\\
|
||||
• Unit test: Does \texttt{.backward()} work?\\
|
||||
• Integration: Gradients through Module 03 layers?\\
|
||||
• NBGrader: Autograde results\\
|
||||
• Milestone: Train network end-to-end
|
||||
};
|
||||
|
||||
\node[example, left=0.8cm of reflect] (reflect-ex) {
|
||||
\textbf{Module 05 Example:}\\
|
||||
• Memory: Gradient storage overhead?\\
|
||||
• Complexity: $O(?)$ for backprop?\\
|
||||
• Design: Why computational graphs?\\
|
||||
• Transfer: How does PyTorch differ?
|
||||
};
|
||||
|
||||
% Connect examples to phases
|
||||
\draw[dotted] (build) -- (build-ex);
|
||||
\draw[dotted] (use) -- (use-ex);
|
||||
\draw[dotted] (reflect) -- (reflect-ex);
|
||||
|
||||
% Center annotation
|
||||
\node[align=center, font=\scriptsize\itshape] at (0, 0) {
|
||||
Repeats for\\
|
||||
all 20 modules
|
||||
};
|
||||
|
||||
% Title annotation
|
||||
\node[above=0.3cm of build, font=\normalsize\bfseries] {
|
||||
Pedagogical Cycle: Every Module
|
||||
};
|
||||
|
||||
\end{tikzpicture}
|
||||
\caption{Build→Use→Reflect pedagogical cycle structuring all TinyTorch modules. \textbf{Build:} Students implement components in Jupyter notebooks with scaffolded guidance (connection maps, TODOs). \textbf{Use:} Integration testing validates cross-module functionality via NBGrader unit tests and milestone checkpoints. \textbf{Reflect:} Systems analysis questions probe memory footprints, computational complexity, and design trade-offs. This cycle addresses cognitive apprenticeship by making expert thinking patterns explicit and assessment visible through automated feedback. Examples shown for Module 05 (Autograd).}
|
||||
\label{fig:build-use-reflect}
|
||||
\end{figure}
|
||||
|
||||
\clearpage
|
||||
|
||||
% ============================================================
|
||||
% BONUS FIGURE: MILESTONE PROGRESSION
|
||||
% ============================================================
|
||||
\subsection*{Bonus Figure D: Historical Milestone Progression (MEDIUM PRIORITY)}
|
||||
\textbf{Location:} Section 4.3 (Historical Validation), after milestone description\\
|
||||
\textbf{Pedagogical Value:} Shows 70-year capability accumulation and which modules unlock each milestone.
|
||||
|
||||
\begin{figure}[h]
|
||||
\centering
|
||||
\begin{tikzpicture}[
|
||||
scale=0.9,
|
||||
every node/.style={font=\small},
|
||||
milestone/.style={
|
||||
rectangle,
|
||||
draw,
|
||||
thick,
|
||||
rounded corners,
|
||||
minimum width=2cm,
|
||||
minimum height=1.2cm,
|
||||
align=center,
|
||||
font=\scriptsize
|
||||
}
|
||||
]
|
||||
|
||||
% Timeline axis
|
||||
\draw[thick, ->] (0, 0) -- (14, 0) node[right] {Time};
|
||||
|
||||
% Year markers
|
||||
\foreach \x/\year in {0/1957, 2.4/1969, 5.8/1986, 8.2/1998, 11.6/2017, 13/2024} {
|
||||
\draw (\x, 0.1) -- (\x, -0.1);
|
||||
\node[below, font=\tiny] at (\x, -0.3) {\year};
|
||||
}
|
||||
|
||||
% Milestones
|
||||
\node[milestone, fill=blue!20] (m1) at (0, 2) {
|
||||
\textbf{M1: Perceptron}\\
|
||||
Modules 01-04\\
|
||||
Linearly separable\\
|
||||
classification
|
||||
};
|
||||
|
||||
\node[milestone, fill=blue!30] (m2) at (2.4, 3.5) {
|
||||
\textbf{M2: XOR}\\
|
||||
Modules 01-07\\
|
||||
Multi-layer\\
|
||||
learning
|
||||
};
|
||||
|
||||
\node[milestone, fill=green!20] (m3) at (5.8, 2) {
|
||||
\textbf{M3: MNIST MLP}\\
|
||||
Modules 01-08\\
|
||||
95\%+ digit\\
|
||||
recognition
|
||||
};
|
||||
|
||||
\node[milestone, fill=green!30] (m4) at (8.2, 3.5) {
|
||||
\textbf{M4: CIFAR-10 CNN}\\
|
||||
Modules 01-09\\
|
||||
75\%+ image\\
|
||||
classification
|
||||
};
|
||||
|
||||
\node[milestone, fill=purple!20] (m5) at (11.6, 2) {
|
||||
\textbf{M5: Transformer}\\
|
||||
Modules 01-13\\
|
||||
Text generation\\
|
||||
with attention
|
||||
};
|
||||
|
||||
\node[milestone, fill=red!30] (m6) at (13, 3.5) {
|
||||
\textbf{M6: Production}\\
|
||||
All 20 modules\\
|
||||
Optimized system\\
|
||||
(Olympics)
|
||||
};
|
||||
|
||||
% Connect milestones to timeline
|
||||
\foreach \m in {m1, m2, m3, m4, m5, m6} {
|
||||
\draw[dotted] (\m) -- (\m |- 0,0);
|
||||
}
|
||||
|
||||
% Capability accumulation arrows
|
||||
\draw[->, thick, blue!50, dashed] (m1) -- (m2);
|
||||
\draw[->, thick, blue!50, dashed] (m2) -- (m3);
|
||||
\draw[->, thick, green!50, dashed] (m3) -- (m4);
|
||||
\draw[->, thick, purple!50, dashed] (m4) -- (m5);
|
||||
\draw[->, thick, red!50, dashed] (m5) -- (m6);
|
||||
|
||||
% Accuracy progression annotation
|
||||
\node[align=center, font=\scriptsize, fill=white] at (7, -2) {
|
||||
\textbf{Capability Progression:} Each milestone validates cumulative module integration.\\
|
||||
Students recreate 70 years of ML history using \emph{only} their own code.
|
||||
};
|
||||
|
||||
\end{tikzpicture}
|
||||
\caption{Historical milestone progression spanning 1957-2024. Each milestone requires progressively more modules, validating cumulative implementation correctness through historically significant achievements. Students experience ML's evolution from single-layer perceptrons (M1) through modern transformer architectures (M5) to production-optimized systems (M6). Arrows show capability accumulation - later milestones build on earlier foundations.}
|
||||
\label{fig:milestone-progression}
|
||||
\end{figure}
|
||||
|
||||
\end{document}
|
||||
Reference in New Issue
Block a user