github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-16 06:07:17 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

[PR #1870] fix(activations): numerically stable sigmoid + remove overflow warning suppressor #29148

New Issue
Open
opened 2026-06-20 13:03:49 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-06-20 13:03:49 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/harvard-edge/cs249r_book/pull/1870
Author: @Shashank-Tripathi-07
Created: 6/16/2026
Status: 🔄 Open

Base: main ← Head: fix/sigmoid-numerical-stability

📝 Commits (1)

  • 70d9988 fix sigmoid overflow and remove the warning suppressor that hid it

📊 Changes

1 file changed (+10 additions, -12 deletions)

View changed files

📝 tinytorch/src/02_activations/02_activations.py (+10 -12)

📄 Description

What breaks

Sigmoid.forward() uses the naive form:

result = 1.0 / (1.0 + np.exp(-x.data))

For very negative inputs (x << -87 in float32), np.exp(-x) overflows to inf. The result 1/(1+inf) = 0.0 is accidentally correct, but NumPy emits a RuntimeWarning. The unit test hid this with:

warnings.filterwarnings('ignore', category=RuntimeWarning)

This suppressor silences the whole RuntimeWarning category for any future code in that block -- meaning a real numerical regression would produce no signal.

Fix

Use the piece-wise stable sigmoid identity:

result = np.where(
    x_data >= 0,
    1.0 / (1.0 + np.exp(-x_data)),    # exp(-x) in [0,1], safe
    np.exp(x_data) / (1.0 + np.exp(x_data))  # exp(x) in [0,1), safe
)

Both branches compute the same mathematical function but stay in the safe exponent range. This is the same trick used by scipy.special.expit and PyTorch's torch.sigmoid.

With overflow eliminated: remove the warnings.filterwarnings calls and the now-unused import warnings.

Test plan

  • pytest tinytorch/tests/02_activations/ passes with no RuntimeWarning
  • sigmoid([-1000, 1000]) returns [~0, ~1] without any warning
  • sigmoid([0]) still returns 0.5

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/harvard-edge/cs249r_book/pull/1870 **Author:** [@Shashank-Tripathi-07](https://github.com/Shashank-Tripathi-07) **Created:** 6/16/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/sigmoid-numerical-stability` --- ### 📝 Commits (1) - [`70d9988`](https://github.com/harvard-edge/cs249r_book/commit/70d9988cefba0590e4985e28c52081325e2052c8) fix sigmoid overflow and remove the warning suppressor that hid it ### 📊 Changes **1 file changed** (+10 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `tinytorch/src/02_activations/02_activations.py` (+10 -12) </details> ### 📄 Description ## What breaks `Sigmoid.forward()` uses the naive form: ```python result = 1.0 / (1.0 + np.exp(-x.data)) ``` For very negative inputs (`x << -87` in float32), `np.exp(-x)` overflows to `inf`. The result `1/(1+inf) = 0.0` is accidentally correct, but NumPy emits a `RuntimeWarning`. The unit test hid this with: ```python warnings.filterwarnings('ignore', category=RuntimeWarning) ``` This suppressor silences the whole `RuntimeWarning` category for any future code in that block -- meaning a real numerical regression would produce no signal. ## Fix Use the piece-wise stable sigmoid identity: ```python result = np.where( x_data >= 0, 1.0 / (1.0 + np.exp(-x_data)), # exp(-x) in [0,1], safe np.exp(x_data) / (1.0 + np.exp(x_data)) # exp(x) in [0,1), safe ) ``` Both branches compute the same mathematical function but stay in the safe exponent range. This is the same trick used by `scipy.special.expit` and PyTorch's `torch.sigmoid`. With overflow eliminated: remove the `warnings.filterwarnings` calls and the now-unused `import warnings`. ## Test plan - [ ] `pytest tinytorch/tests/02_activations/` passes with no RuntimeWarning - [ ] `sigmoid([-1000, 1000])` returns `[~0, ~1]` without any warning - [ ] `sigmoid([0])` still returns `0.5` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-06-20 13:03:49 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area: book
area: collabs
area: kits
area: labs
area: socratiq
area: staffml
area: tinytorch
area: tools
area: website
bug
dependencies
format: epub
format: pdf
javascript
link-health
link-rot
priority-high
pull-request
Mirrored from GitHub Pull Request
 staffml
type: bug
type: citation
type: code
type: errata
type: improvement
type: new
type: question
vault-sli
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#29148
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?