github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-16 23:24:55 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

[GH-ISSUE #1692] [TinyTorch] GELU code solution issue in tinytorch/src/02_activations/02_activations.py #18065

New Issue
Closed
opened 2026-05-28 21:49:31 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-28 21:49:31 -05:00
Owner
Copy Link

Originally created by @bdub-1 on GitHub (May 6, 2026).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1692

Module

02 Activations

Type of Improvement

Code quality (refactor, style, consistency)

Description

Something to point out (relating to #1689). In this issue I referenced tinytorch/src/02_activations to populate the implementation for GELU. This fits in with how the other implementations are presented in the docs.

However, I noticed the current src code solution to be an issue. This line indicates we need to reuse the Sigmoid class we already built but then the current GELU solution uses a non-numerically stable GELU approximation approach.

Proposed Solution

Should the src code be updated in module 02 to reflect this? @profvjreddi

class GELU:
  """
  This implementation reuses our numerically stable Sigmoid implementation,
  as the instruction indicated we were supposed to.
  """
  def forward(self, x: Tensor) -> Tensor:
    result = Sigmoid()(x * 1.702) # returns newly allocated Tensor
    result *= x                   # in-place, element-wise multiplication
    return result
  • The pull request #1690 which addresses the issue #1689 would need to be updated prior to its merge or update the docs again with the source code change in a new PR
Originally created by @bdub-1 on GitHub (May 6, 2026). Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1692 ### Module 02 Activations ### Type of Improvement Code quality (refactor, style, consistency) ### Description Something to point out (relating to #1689). In this issue I referenced [tinytorch/src/02_activations](https://github.com/harvard-edge/cs249r_book/blob/ccf01ccd0808b302805bd610082910c966699b9c/tinytorch/src/02_activations/02_activations.py#L626-L633) to populate the implementation for GELU. This fits in with how the other implementations are presented in the docs. However, I noticed the current src code solution to be an issue. [This line](https://github.com/harvard-edge/cs249r_book/blob/ccf01ccd0808b302805bd610082910c966699b9c/tinytorch/src/02_activations/02_activations.py#L564) indicates we need to reuse the `Sigmoid` class we already built but then the current GELU solution uses a non-numerically stable GELU approximation approach. ### Proposed Solution Should the src code be updated in module 02 to reflect this? @profvjreddi ```py class GELU: """ This implementation reuses our numerically stable Sigmoid implementation, as the instruction indicated we were supposed to. """ def forward(self, x: Tensor) -> Tensor: result = Sigmoid()(x * 1.702) # returns newly allocated Tensor result *= x # in-place, element-wise multiplication return result ``` - The pull request #1690 which addresses the issue #1689 would need to be updated prior to its merge or update the docs again with the source code change in a new PR
GiteaMirror added the area: tinytorchtype: improvement labels 2026-05-28 21:49:31 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area: book
area: collabs
area: kits
area: labs
area: socratiq
area: staffml
area: tinytorch
area: tools
area: website
bug
dependencies
format: epub
format: pdf
javascript
link-health
link-rot
priority-high
pull-request
Mirrored from GitHub Pull Request
 staffml
type: bug
type: citation
type: code
type: errata
type: improvement
type: new
type: question
vault-sli
No Label area: tinytorch type: improvement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#18065
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?