[GH-ISSUE #723] initial ai acceleration thoughts (discussion)

GiteaMirror commented

2026-05-24 10:03:08 -05:00

Owner

Originally created by @18jeffreyma on GitHub (Feb 23, 2025).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/723

Originally assigned to: @profvjreddi on GitHub.

Really nice initial purpose section, just wanted to highlight :-)
A more general thought: do we want to introduce the hardware first then the software (as current) or do the reverse? Does it make it easier to understand how to map to the hardware if we do the reverse and first explain how workloads can be accelerated, then introduce the hardware that accelerates those primitives?
Another note: unsure which book section covers “how to accelerate a matmul, normalization, etc. or how to parallelize things intuitively?” I think these are super core info to include in a ML/DL systems textbook: my guess is that it might be good to include this in the AI acceleration section before going into the hardware: where do you think we should place it?
11.2.2: This first sentence feels weird “The evolution…” and doesn’t feel like a thesis statement. As I read it, the point that you want to emphasize is that (1) Moore’s law is holding and transistor/raw compute power is increasing (2) single-thread core performance however is tapering (denard scaling fails) (3) we need to think about parallelism as such to keep actually being able to make use of the Moore’s law scaling.
- [ ] I’ll brainstorm, but I think maybe this should be rewritten as a first statement of a section.
11.3: Could give a shout in “Complex Integration and Programming” section about NVIDIA’s CUDA moat as a good example where software has created a moat for consumers not wanting to jump to other ASICs.
11.3.6: Should this comparison come first as a “preview” for what’s to come? This table alone would be a good “table of contents”
11.4: Maybe a good exercise here would be to walk through the generations of TPUs (or GPUs) and show specialization over time?
- [ ] Software here would be models of interest, hardware would be specialized compute units or HBM sizes etc or sparse matmul cores etc.
- [ ] I think many of the sections feel like surface level themes that also were echoed in previous sections: I think a case study here has a huge chance to shine (and the opportunities and challenges can then be explained in terms of a case study)
- [ ]
11.5.1: Programming models:
- [ ] Should include Triton/Neuron Kernel Interface (tile based GPU mapping)
- [ ] Not sure if XLA/PJRT belongs here as well
No major feedback on the remaining sections (good material and clearly relevant for the AI + hardware bridge and how things reinforce each other).
I think the key bit to clarify for this section is answering the question “why can we even accelerate neural networks to begin with?” (i.e. what is the fundamental parallelism we’re exploiting in hardware later)?
- [ ] We should either answer this in a previous section or answer it in this acceleration section.

Originally created by @18jeffreyma on GitHub (Feb 23, 2025). Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/723 Originally assigned to: @profvjreddi on GitHub. - [ ] Really nice initial purpose section, just wanted to highlight :-) - [x] A more general thought: do we want to introduce the hardware first then the software (as current) or do the reverse? Does it make it easier to understand how to map to the hardware if we do the reverse and first explain how workloads can be accelerated, then introduce the hardware that accelerates those primitives? - [x] Another note: unsure which book section covers “how to accelerate a matmul, normalization, etc. or how to parallelize things intuitively?” I think these are super core info to include in a ML/DL systems textbook: my guess is that it might be good to include this in the AI acceleration section before going into the hardware: where do you think we should place it? - [x] 11.2.2: This first sentence feels weird “The evolution…” and doesn’t feel like a thesis statement. As I read it, the point that you want to emphasize is that (1) Moore’s law is holding and transistor/raw compute power is increasing (2) single-thread core performance however is tapering (denard scaling fails) (3) we need to think about parallelism as such to keep actually being able to make use of the Moore’s law scaling. - [ ] I’ll brainstorm, but I think maybe this should be rewritten as a first statement of a section. - [ ] 11.3: Could give a shout in “Complex Integration and Programming” section about NVIDIA’s CUDA moat as a good example where software has created a moat for consumers not wanting to jump to other ASICs. - [ ] 11.3.6: Should this comparison come first as a “preview” for what’s to come? This table alone would be a good “table of contents” - [x] 11.4: Maybe a good exercise here would be to walk through the generations of TPUs (or GPUs) and show specialization over time? - [ ] Software here would be models of interest, hardware would be specialized compute units or HBM sizes etc or sparse matmul cores etc. - [ ] I think many of the sections feel like surface level themes that also were echoed in previous sections: I think a case study here has a huge chance to shine (and the opportunities and challenges can then be explained in terms of a case study) - [ ] - [ ] 11.5.1: Programming models: - [ ] Should include Triton/Neuron Kernel Interface (tile based GPU mapping) - [ ] Not sure if XLA/PJRT belongs here as well - [ ] No major feedback on the remaining sections (good material and clearly relevant for the AI \+ hardware bridge and how things reinforce each other). - [ ] I think the key bit to clarify for this section is answering the question “why can we even accelerate neural networks to begin with?” (i.e. what is the fundamental parallelism we’re exploiting in hardware later)? - [ ] We should either answer this in a previous section or answer it in this acceleration section.

GiteaMirror added the area: book type: improvement labels 2026-05-24 10:03:08 -05:00

GiteaMirror closed this issue

2026-05-24 10:03:10 -05:00

GiteaMirror commented

2026-05-24 10:03:12 -05:00

Author

Owner

@18jeffreyma commented on GitHub (Feb 23, 2025):

@profvjreddi what are your thoughts on the last point (let's discuss in thread)? I realize from other passes on other chapters we're missing a section that clearly explains the parallelism inherent in alot of machine learning (and the answer to why we can even think about acceleration later)

@18jeffreyma commented on GitHub (Feb 23, 2025): @profvjreddi what are your thoughts on the last point (let's discuss in thread)? I realize from other passes on other chapters we're missing a section that clearly explains the parallelism inherent in alot of machine learning (and the answer to why we can even think about acceleration later)

GiteaMirror commented

2026-05-24 10:03:13 -05:00

Author

Owner

@18jeffreyma commented on GitHub (Mar 6, 2025):

latest (the revised chapter is incredible, almost no major notes)

https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#ai-compute-primitives, maybe provide a vector example here? was initially confused reading the Linear in the vector operations section (this is still a matmul and would be processed by matmul compute unit (i think?)
https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#mixed-precision-computing-and-hardware-evolution, maybe call out the gpu public names here (not many people know the architecture, esp book consumers)
maybe a flash attention shout out here or citation https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#transformer-architectures

@18jeffreyma commented on GitHub (Mar 6, 2025): latest (the revised chapter is incredible, almost no major notes) - [ ] https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#ai-compute-primitives, maybe provide a vector example here? was initially confused reading the Linear in the vector operations section (this is still a matmul and would be processed by matmul compute unit (i think?) - [ ] https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#mixed-precision-computing-and-hardware-evolution, maybe call out the gpu public names here (not many people know the architecture, esp book consumers) - [ ] maybe a flash attention shout out here or citation https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#transformer-architectures

GiteaMirror commented

2026-05-24 10:03:14 -05:00

Author

Owner

@RadhikaG commented on GitHub (Mar 17, 2025):

https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#fig-memory-wall is missing a proper description in the text, needs clarification on the units for the two different quantities plotted.

@RadhikaG commented on GitHub (Mar 17, 2025): - [x] [https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#fig-memory-wall](https://mlsysbook.ai/contents/core/hw_acceleration/hw_acceleration.html#fig-memory-wall) is missing a proper description in the text, needs clarification on the units for the two different quantities plotted.

GiteaMirror commented

2026-05-24 10:03:16 -05:00

Author

Owner

@profvjreddi commented on GitHub (Mar 17, 2025):

Fixed @RadhikaG feedback in 78aa69e4b0

@profvjreddi commented on GitHub (Mar 17, 2025): Fixed @RadhikaG feedback in https://github.com/harvard-edge/cs249r_book/commit/78aa69e4b0404ebe5c0515a42da13f0e8efe7a77

[GH-ISSUE #723] initial ai acceleration thoughts (discussion) #16082