[GH-ISSUE #1221] TinyTorch Module 02: Overflow in Sigmoid unit test, slightly confusing warning message #5696

New Issue

GiteaMirror · 2026-04-21T21:43:03-05:00

GiteaMirror commented

2026-04-21 21:43:03 -05:00

Originally created by @asgalon on GitHub (Mar 9, 2026).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1221

This showed up when running the test_unit_sigmoid cell:

That is a bit of a mixed message there. Looking at the test, the assertion holds for the Tensor([-1000,1000]), so the warning for overflow is probably expected. Perhaps it would be a good idea to add a comment saying so?

Same happens in the module integration test below, and in the Aha moment output

Originally created by @asgalon on GitHub (Mar 9, 2026). Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1221 This showed up when running the test_unit_sigmoid cell: <img width="1760" height="893" alt="Image" src="https://github.com/user-attachments/assets/b079b924-97d2-4b8a-b395-2e86bcca5ce8" /> That is a bit of a mixed message there. Looking at the test, the assertion holds for the Tensor([-1000,1000]), so the warning for overflow is probably expected. Perhaps it would be a good idea to add a comment saying so? Same happens in the module integration test below, and in the Aha moment output

GiteaMirror added the type: bug area: tinytorch labels 2026-04-21 21:43:03 -05:00

GiteaMirror closed this issue

2026-04-21 21:43:05 -05:00

GiteaMirror commented

2026-04-21 21:43:06 -05:00

@adityamulik commented on GitHub (Mar 9, 2026):

I faced this issue too, I think the np.exp() function with float32 dtype is not able to handle larger numbers. I tried fixing this.
Below was the claude response:

Based on the below, 88.72 is max else it will overflow.

float32:  np.exp(x) overflows when x > 88.72
float64:  np.exp(x) overflows when x > 709.78

Workaround for the Sigmoid forward function:

exp_neg_abs = np.exp(-np.abs(x.data))  # always safe, always tiny
result = np.where(
    x.data >= 0,
    1.0 / (1.0 + exp_neg_abs),           # positive x → standard formula
    exp_neg_abs / (1.0 + exp_neg_abs)    # negative x → rearranged formula
)
return Tensor(result)

@adityamulik commented on GitHub (Mar 9, 2026): I faced this issue too, I think the np.exp() function with float32 dtype is not able to handle larger numbers. I tried fixing this. Below was the claude response: Based on the below, 88.72 is max else it will overflow. ``` float32: np.exp(x) overflows when x > 88.72 float64: np.exp(x) overflows when x > 709.78 ``` Workaround for the Sigmoid forward function: ``` exp_neg_abs = np.exp(-np.abs(x.data)) # always safe, always tiny result = np.where( x.data >= 0, 1.0 / (1.0 + exp_neg_abs), # positive x → standard formula exp_neg_abs / (1.0 + exp_neg_abs) # negative x → rearranged formula ) return Tensor(result) ```

GiteaMirror commented

2026-04-21 21:43:07 -05:00

@asgalon commented on GitHub (Mar 10, 2026):

I think that the overflow is expected here. The problem is merely that the warning is a bit confusing because it looks like something that should be fixed, while the test explicitly checks that the result is within the expected range when exp overflows. So either this warning should be suppressed, or explicitly advertised as expected. The red color does not really help here. The assertion passes and looks useable for the expected use cases of Sigmoid, so the test result for tensor values that cause exp to overflow is correct. There is no need to complicate the algorithm, just to get rid of the confusing warning, I'd say.

@asgalon commented on GitHub (Mar 10, 2026): I think that the overflow is expected here. The problem is merely that the warning is a bit confusing because it looks like something that should be fixed, while the test explicitly checks that the result is within the expected range when exp overflows. So either this warning should be suppressed, or explicitly advertised as expected. The red color does not really help here. The assertion passes and looks useable for the expected use cases of Sigmoid, so the test result for tensor values that cause exp to overflow is correct. There is no need to complicate the algorithm, just to get rid of the confusing warning, I'd say.

GiteaMirror commented

2026-04-21 21:43:08 -05:00

@asgalon commented on GitHub (Mar 10, 2026):

I think the correct way to deal with this is to set

np.warnings.filterwarnings('ignore', category=RuntimeWarning)

just not sure where is the best place. On the other hand, this hides the fact that an overflow happens, so from an educational standpoint it may be preferable to just put a note in before it happens that this warning is expected. That is more a question of style, perhaps.

@asgalon commented on GitHub (Mar 10, 2026): I think the correct way to deal with this is to set ``` np.warnings.filterwarnings('ignore', category=RuntimeWarning) ``` just not sure where is the best place. On the other hand, this hides the fact that an overflow happens, so from an educational standpoint it may be preferable to just put a note in before it happens that this warning is expected. That is more a question of style, perhaps.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#5696