Adding Clarification For RNN Parameter Storage #296

New Issue

GiteaMirror · 2026-03-22T15:35:36-05:00

GiteaMirror commented

2026-03-22 15:35:36 -05:00

Originally created by @EddieJ03 on GitHub (Jan 14, 2025).

7cc930be3d/contents/core/dnn_architectures/dnn_architectures.qmd (L685)

Forgive me if I am wrong, but I was wondering if it can be more clear that the reason RNN parameter storage is O(h^{2}) is because we are assuming in this case the input dimension N is smaller than the dimension of the hidden state h? Otherwise wouldn't O(N*h) dominate? Thanks.

Originally created by @EddieJ03 on GitHub (Jan 14, 2025). https://github.com/harvard-edge/cs249r_book/blob/7cc930be3d8ed272599f77b79197cc0f3803f9b7/contents/core/dnn_architectures/dnn_architectures.qmd#L685 Forgive me if I am wrong, but I was wondering if it can be more clear that the reason RNN parameter storage is O($h^{2}$) is because we are assuming in this case the input dimension $N$ is smaller than the dimension of the hidden state $h$? Otherwise wouldn't O($N*h$) dominate? Thanks.

GiteaMirror added the type: errata area: book type: improvement labels 2026-03-22 15:35:36 -05:00

GiteaMirror closed this issue

2026-03-22 15:35:37 -05:00

GiteaMirror commented

2026-03-22 15:35:37 -05:00

@profvjreddi commented on GitHub (Jan 16, 2025):

Hey @EddieJ03, thanks for catching that! You’re absolutely right—I was generalizing here.

The parameter storage for RNNs is ( O(h^2) ) under the assumption that the hidden state dimension ( h ) is larger than the input dimension ( N ). Otherwise, ( O(N x h) ) would dominate.

I can update the table to reflect this clarification, but if you'd like, feel free to submit a GitHub PR to address it yourself and become one of the contributors. Either way, I’m happy to help refine the fix. Let me know what works best for you!

Thanks again for pointing this out!

(Apologies for the slow reply—I’m in the thick of updating Chapter 7 before the semester starts, and things get too hectic. I'm reworking one chapter at a time from scratch because the first draft was done in a sprint, and now I am slowly cleaning it all up and making things flow. Anyway, I appreciate your question).

@profvjreddi commented on GitHub (Jan 16, 2025): Hey @EddieJ03, thanks for catching that! You’re absolutely right—I was generalizing here. The parameter storage for RNNs is $ O(h^2) $ under the assumption that the hidden state dimension $ h $ is larger than the input dimension $ N $. Otherwise, $ O(N x h) $ would dominate. I can update the table to reflect this clarification, but if you'd like, feel free to submit a GitHub PR to address it yourself and become one of the [contributors](https://mlsysbook.ai/contents/frontmatter/acknowledgements/acknowledgements.html#contributors). Either way, I’m happy to help refine the fix. Let me know what works best for you! Thanks again for pointing this out! (Apologies for the slow reply—I’m in the thick of updating Chapter 7 before the semester starts, and things get too hectic. I'm reworking one chapter at a time from scratch because the first draft was done in a sprint, and now I am slowly cleaning it all up and making things flow. Anyway, I appreciate your question).

GiteaMirror commented

2026-03-22 15:35:38 -05:00

@EddieJ03 commented on GitHub (Jan 16, 2025):

Hi @profvjreddi ! No worries and thanks for getting back to my issue. I'd be happy to raise a PR!

@EddieJ03 commented on GitHub (Jan 16, 2025): Hi @profvjreddi ! No worries and thanks for getting back to my issue. I'd be happy to raise a PR!

GiteaMirror commented

2026-03-22 15:35:39 -05:00

@profvjreddi commented on GitHub (Feb 16, 2025):

Thanks for catching this and fixing it @EddieJ03

@profvjreddi commented on GitHub (Feb 16, 2025): Thanks for catching this and fixing it @EddieJ03

GiteaMirror referenced this issue

2026-04-11 07:44:14 -05:00