This hint here says "The 1.702 constant comes from √(2/π) approximation", but √(2/π) = 0.79...
I believe that the 1.702 comes from the tanh used in the GELU paper, which approximates the function to be 0.5x(1 + tanh[√(2/π)(x + 0.044715x^3)... then the next line of the paper does mention the sigmoid approximation, but the number doesn't come from this?
In other words, maybe drop the hint? Or change the module to use tanh instead?
Originally created by @oscarf189 on GitHub (Feb 4, 2026).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1154
https://github.com/harvard-edge/cs249r_book/blob/1a80e57fa07a5e551641df8db4dd2131524cb31f/tinytorch/src/02_activations/02_activations.py#L618C1-L619C1
This hint here says "The 1.702 constant comes from √(2/π) approximation", but √(2/π) = 0.79...
I believe that the 1.702 comes from the tanh used in the GELU paper, which approximates the function to be 0.5x(1 + tanh[√(2/π)(x + 0.044715x^3)... then the next line of the paper does mention the sigmoid approximation, but the number doesn't come from this?
In other words, maybe drop the hint? Or change the module to use tanh instead?
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @oscarf189 on GitHub (Feb 4, 2026).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1154
https://github.com/harvard-edge/cs249r_book/blob/1a80e57fa07a5e551641df8db4dd2131524cb31f/tinytorch/src/02_activations/02_activations.py#L618C1-L619C1
This hint here says "The 1.702 constant comes from √(2/π) approximation", but √(2/π) = 0.79...
I believe that the 1.702 comes from the tanh used in the GELU paper, which approximates the function to be 0.5x(1 + tanh[√(2/π)(x + 0.044715x^3)... then the next line of the paper does mention the sigmoid approximation, but the number doesn't come from this?
In other words, maybe drop the hint? Or change the module to use tanh instead?