Forgive me if I am wrong, but I was wondering if it can be more clear that the reason RNN parameter storage is O(h^{2}) is because we are assuming in this case the input dimension N is smaller than the dimension of the hidden state h? Otherwise wouldn't O(N*h) dominate? Thanks.
Originally created by @EddieJ03 on GitHub (Jan 14, 2025).
https://github.com/harvard-edge/cs249r_book/blob/7cc930be3d8ed272599f77b79197cc0f3803f9b7/contents/core/dnn_architectures/dnn_architectures.qmd#L685
Forgive me if I am wrong, but I was wondering if it can be more clear that the reason RNN parameter storage is O($h^{2}$) is because we are assuming in this case the input dimension $N$ is smaller than the dimension of the hidden state $h$? Otherwise wouldn't O($N*h$) dominate? Thanks.
Hey @EddieJ03, thanks for catching that! You’re absolutely right—I was generalizing here.
The parameter storage for RNNs is ( O(h^2) ) under the assumption that the hidden state dimension ( h ) is larger than the input dimension ( N ). Otherwise, ( O(N x h) ) would dominate.
I can update the table to reflect this clarification, but if you'd like, feel free to submit a GitHub PR to address it yourself and become one of the contributors. Either way, I’m happy to help refine the fix. Let me know what works best for you!
Thanks again for pointing this out!
(Apologies for the slow reply—I’m in the thick of updating Chapter 7 before the semester starts, and things get too hectic. I'm reworking one chapter at a time from scratch because the first draft was done in a sprint, and now I am slowly cleaning it all up and making things flow. Anyway, I appreciate your question).
@profvjreddi commented on GitHub (Jan 16, 2025):
Hey @EddieJ03, thanks for catching that! You’re absolutely right—I was generalizing here.
The parameter storage for RNNs is \( O(h^2) \) under the assumption that the hidden state dimension \( h \) is larger than the input dimension \( N \). Otherwise, \( O(N x h) \) would dominate.
I can update the table to reflect this clarification, but if you'd like, feel free to submit a GitHub PR to address it yourself and become one of the [contributors](https://mlsysbook.ai/contents/frontmatter/acknowledgements/acknowledgements.html#contributors). Either way, I’m happy to help refine the fix. Let me know what works best for you!
Thanks again for pointing this out!
(Apologies for the slow reply—I’m in the thick of updating Chapter 7 before the semester starts, and things get too hectic. I'm reworking one chapter at a time from scratch because the first draft was done in a sprint, and now I am slowly cleaning it all up and making things flow. Anyway, I appreciate your question).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @EddieJ03 on GitHub (Jan 14, 2025).
https://github.com/harvard-edge/cs249r_book/blob/7cc930be3d8ed272599f77b79197cc0f3803f9b7/contents/core/dnn_architectures/dnn_architectures.qmd#L685
Forgive me if I am wrong, but I was wondering if it can be more clear that the reason RNN parameter storage is O(
h^{2}) is because we are assuming in this case the input dimensionNis smaller than the dimension of the hidden stateh? Otherwise wouldn't O(N*h) dominate? Thanks.@profvjreddi commented on GitHub (Jan 16, 2025):
Hey @EddieJ03, thanks for catching that! You’re absolutely right—I was generalizing here.
The parameter storage for RNNs is ( O(h^2) ) under the assumption that the hidden state dimension ( h ) is larger than the input dimension ( N ). Otherwise, ( O(N x h) ) would dominate.
I can update the table to reflect this clarification, but if you'd like, feel free to submit a GitHub PR to address it yourself and become one of the contributors. Either way, I’m happy to help refine the fix. Let me know what works best for you!
Thanks again for pointing this out!
(Apologies for the slow reply—I’m in the thick of updating Chapter 7 before the semester starts, and things get too hectic. I'm reworking one chapter at a time from scratch because the first draft was done in a sprint, and now I am slowly cleaning it all up and making things flow. Anyway, I appreciate your question).
@EddieJ03 commented on GitHub (Jan 16, 2025):
Hi @profvjreddi ! No worries and thanks for getting back to my issue. I'd be happy to raise a PR!
@profvjreddi commented on GitHub (Feb 16, 2025):
Thanks for catching this and fixing it @EddieJ03