[Module 03: Layers] Inconsistency between code examples #497

New Issue

GiteaMirror · 2026-03-22T15:44:16-05:00

GiteaMirror commented

2026-03-22 15:44:16 -05:00

Originally created by @minhdang26403 on GitHub (Feb 6, 2026).

In the Module 03 - Layers, there are some inconsistencies between code examples. Specifically, in the Linear layer implementation, the book mentions requires_grad parameter, but it's not set in the code.

def __init__(self, in_features, out_features, bias=True):
    """Initialize linear layer with proper weight initialization."""
    self.in_features = in_features
    self.out_features = out_features

    # Xavier/Glorot initialization for stable gradients
    scale = np.sqrt(XAVIER_SCALE_FACTOR / in_features)
    weight_data = np.random.randn(in_features, out_features) * scale
    self.weight = Tensor(weight_data)

    # Initialize bias to zeros or None
    if bias:
        bias_data = np.zeros(out_features)
        self.bias = Tensor(bias_data)
    else:
        self.bias = None

However, in the forward pass of Dropout, the code creates Tensor and sets requires_grad parameter. This can cause some confusion while reading the book.

def forward(self, x, training=True):
    """Forward pass through dropout layer."""
    if not training or self.p == DROPOUT_MIN_PROB:
        # During inference or no dropout, pass through unchanged
        return x

    if self.p == DROPOUT_MAX_PROB:
        # Drop everything (preserve requires_grad for gradient flow)
        return Tensor(np.zeros_like(x.data), requires_grad=x.requires_grad)

    # During training, apply dropout
    keep_prob = 1.0 - self.p

    # Create random mask: True where we keep elements
    mask = np.random.random(x.data.shape) < keep_prob

    # Apply mask and scale using Tensor operations to preserve gradients
    mask_tensor = Tensor(mask.astype(np.float32), requires_grad=False)
    scale = Tensor(np.array(1.0 / keep_prob), requires_grad=False)

    # Use Tensor operations: x * mask * scale
    output = x * mask_tensor * scale
    return output

Therefore, I raise this issue to have a discussion about whether we should explicitly set requires_grad parameters here. Once we have an agreement on this, I can create PR to fix this issue.

Originally created by @minhdang26403 on GitHub (Feb 6, 2026). In the Module 03 - Layers, there are some inconsistencies between code examples. Specifically, in the Linear layer implementation, the book mentions `requires_grad` parameter, but it's not set in the code. ```python def __init__(self, in_features, out_features, bias=True): """Initialize linear layer with proper weight initialization.""" self.in_features = in_features self.out_features = out_features # Xavier/Glorot initialization for stable gradients scale = np.sqrt(XAVIER_SCALE_FACTOR / in_features) weight_data = np.random.randn(in_features, out_features) * scale self.weight = Tensor(weight_data) # Initialize bias to zeros or None if bias: bias_data = np.zeros(out_features) self.bias = Tensor(bias_data) else: self.bias = None ``` However, in the forward pass of `Dropout`, the code creates Tensor and sets `requires_grad` parameter. This can cause some confusion while reading the book. ```python def forward(self, x, training=True): """Forward pass through dropout layer.""" if not training or self.p == DROPOUT_MIN_PROB: # During inference or no dropout, pass through unchanged return x if self.p == DROPOUT_MAX_PROB: # Drop everything (preserve requires_grad for gradient flow) return Tensor(np.zeros_like(x.data), requires_grad=x.requires_grad) # During training, apply dropout keep_prob = 1.0 - self.p # Create random mask: True where we keep elements mask = np.random.random(x.data.shape) < keep_prob # Apply mask and scale using Tensor operations to preserve gradients mask_tensor = Tensor(mask.astype(np.float32), requires_grad=False) scale = Tensor(np.array(1.0 / keep_prob), requires_grad=False) # Use Tensor operations: x * mask * scale output = x * mask_tensor * scale return output ``` Therefore, I raise this issue to have a discussion about whether we should explicitly set `requires_grad` parameters here. Once we have an agreement on this, I can create PR to fix this issue.

GiteaMirror added the type: bug area: tinytorch labels 2026-03-22 15:44:16 -05:00

GiteaMirror closed this issue

2026-03-22 15:44:18 -05:00

GiteaMirror commented

2026-03-22 15:44:18 -05:00

@profvjreddi commented on GitHub (Feb 6, 2026):

Hi @minhdang26403! 👋 Thank you for the careful review of Module 03! You have a great eye for consistency issues.

The good news is that this was already fixed on in commit a363406 ("fix(layers): remove requires_grad from Linear layer Tensor calls").

The code examples you're seeing with requires_grad=x.requires_grad and requires_grad=False are from an older version. The current implementation in main has neither Linear nor Dropout using requires_grad:

Current Linear layer:

self.weight = Tensor(weight_data) # No requires_grad
self.bias = Tensor(bias_data) # No requires_grad

Current Dropout layer:

mask_tensor = Tensor(mask.astype(np.float32)) # No requires_grad
scale = Tensor(np.array(1.0 / keep_prob)) # No requires_grad

This was a bug and has been fixed, and now it is intentional, as TinyTorch follows progressive disclosure—Module 03 teaches layers, but requires_grad isn't introduced until Module 06 (Autograd). This prevents students from encountering concepts before they've been explained. Since you're looking at an older cached or forked version, I'll close this issue as already resolved. But thank you again for the thorough review 🙏

@profvjreddi commented on GitHub (Feb 6, 2026): Hi @minhdang26403! 👋 Thank you for the careful review of Module 03! You have a great eye for consistency issues. The good news is that this was already fixed on in commit [a363406](https://github.com/harvard-edge/cs249r_book/commit/a3634069026664117e28b86bbffbdebeae61e11d) ("fix(layers): remove requires_grad from Linear layer Tensor calls"). The code examples you're seeing with requires_grad=x.requires_grad and requires_grad=False are from an older version. The current implementation in main has neither Linear nor Dropout using requires_grad: Current Linear layer: self.weight = Tensor(weight_data) # No requires_grad self.bias = Tensor(bias_data) # No requires_grad Current Dropout layer: mask_tensor = Tensor(mask.astype(np.float32)) # No requires_grad scale = Tensor(np.array(1.0 / keep_prob)) # No requires_grad This was a bug and has been fixed, and now it is intentional, as TinyTorch follows progressive disclosure—Module 03 teaches layers, but requires_grad isn't introduced until Module 06 (Autograd). This prevents students from encountering concepts before they've been explained. Since you're looking at an older cached or forked version, I'll close this issue as already resolved. But thank you again for the thorough review 🙏

GiteaMirror referenced this issue

2026-04-11 07:50:08 -05:00

[GH-ISSUE #497] Unable to resolve crossref @fig-coco in benchmarking.qmd #1456

GiteaMirror referenced this issue

2026-04-19 12:05:45 -05:00

[GH-ISSUE #497] Unable to resolve crossref @fig-coco in benchmarking.qmd #4076

GiteaMirror referenced this issue

2026-04-21 21:22:58 -05:00

[GH-ISSUE #497] Unable to resolve crossref @fig-coco in benchmarking.qmd #5418

GiteaMirror referenced this issue

2026-05-12 18:30:49 -05:00

[GH-ISSUE #497] Unable to resolve crossref @fig-coco in benchmarking.qmd #10479