In the Module 03 - Layers, there are some inconsistencies between code examples. Specifically, in the Linear layer implementation, the book mentions requires_grad parameter, but it's not set in the code.
def__init__(self,in_features,out_features,bias=True):"""Initialize linear layer with proper weight initialization."""self.in_features=in_featuresself.out_features=out_features# Xavier/Glorot initialization for stable gradientsscale=np.sqrt(XAVIER_SCALE_FACTOR/in_features)weight_data=np.random.randn(in_features,out_features)*scaleself.weight=Tensor(weight_data)# Initialize bias to zeros or Noneifbias:bias_data=np.zeros(out_features)self.bias=Tensor(bias_data)else:self.bias=None
However, in the forward pass of Dropout, the code creates Tensor and sets requires_grad parameter. This can cause some confusion while reading the book.
defforward(self,x,training=True):"""Forward pass through dropout layer."""ifnottrainingorself.p==DROPOUT_MIN_PROB:# During inference or no dropout, pass through unchangedreturnxifself.p==DROPOUT_MAX_PROB:# Drop everything (preserve requires_grad for gradient flow)returnTensor(np.zeros_like(x.data),requires_grad=x.requires_grad)# During training, apply dropoutkeep_prob=1.0-self.p# Create random mask: True where we keep elementsmask=np.random.random(x.data.shape)<keep_prob# Apply mask and scale using Tensor operations to preserve gradientsmask_tensor=Tensor(mask.astype(np.float32),requires_grad=False)scale=Tensor(np.array(1.0/keep_prob),requires_grad=False)# Use Tensor operations: x * mask * scaleoutput=x*mask_tensor*scalereturnoutput
Therefore, I raise this issue to have a discussion about whether we should explicitly set requires_grad parameters here. Once we have an agreement on this, I can create PR to fix this issue.
Originally created by @minhdang26403 on GitHub (Feb 6, 2026).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1164
In the Module 03 - Layers, there are some inconsistencies between code examples. Specifically, in the Linear layer implementation, the book mentions `requires_grad` parameter, but it's not set in the code.
```python
def __init__(self, in_features, out_features, bias=True):
"""Initialize linear layer with proper weight initialization."""
self.in_features = in_features
self.out_features = out_features
# Xavier/Glorot initialization for stable gradients
scale = np.sqrt(XAVIER_SCALE_FACTOR / in_features)
weight_data = np.random.randn(in_features, out_features) * scale
self.weight = Tensor(weight_data)
# Initialize bias to zeros or None
if bias:
bias_data = np.zeros(out_features)
self.bias = Tensor(bias_data)
else:
self.bias = None
```
However, in the forward pass of `Dropout`, the code creates Tensor and sets `requires_grad` parameter. This can cause some confusion while reading the book.
```python
def forward(self, x, training=True):
"""Forward pass through dropout layer."""
if not training or self.p == DROPOUT_MIN_PROB:
# During inference or no dropout, pass through unchanged
return x
if self.p == DROPOUT_MAX_PROB:
# Drop everything (preserve requires_grad for gradient flow)
return Tensor(np.zeros_like(x.data), requires_grad=x.requires_grad)
# During training, apply dropout
keep_prob = 1.0 - self.p
# Create random mask: True where we keep elements
mask = np.random.random(x.data.shape) < keep_prob
# Apply mask and scale using Tensor operations to preserve gradients
mask_tensor = Tensor(mask.astype(np.float32), requires_grad=False)
scale = Tensor(np.array(1.0 / keep_prob), requires_grad=False)
# Use Tensor operations: x * mask * scale
output = x * mask_tensor * scale
return output
```
Therefore, I raise this issue to have a discussion about whether we should explicitly set `requires_grad` parameters here. Once we have an agreement on this, I can create PR to fix this issue.
Hi @minhdang26403! 👋 Thank you for the careful review of Module 03! You have a great eye for consistency issues.
The good news is that this was already fixed on in commit a363406 ("fix(layers): remove requires_grad from Linear layer Tensor calls").
The code examples you're seeing with requires_grad=x.requires_grad and requires_grad=False are from an older version. The current implementation in main has neither Linear nor Dropout using requires_grad:
Current Linear layer:
self.weight = Tensor(weight_data) # No requires_grad
self.bias = Tensor(bias_data) # No requires_grad
Current Dropout layer:
mask_tensor = Tensor(mask.astype(np.float32)) # No requires_grad
scale = Tensor(np.array(1.0 / keep_prob)) # No requires_grad
This was a bug and has been fixed, and now it is intentional, as TinyTorch follows progressive disclosure—Module 03 teaches layers, but requires_grad isn't introduced until Module 06 (Autograd). This prevents students from encountering concepts before they've been explained. Since you're looking at an older cached or forked version, I'll close this issue as already resolved. But thank you again for the thorough review 🙏
<!-- gh-comment-id:3858460224 -->
@profvjreddi commented on GitHub (Feb 6, 2026):
Hi @minhdang26403! 👋 Thank you for the careful review of Module 03! You have a great eye for consistency issues.
The good news is that this was already fixed on in commit [a363406](https://github.com/harvard-edge/cs249r_book/commit/a3634069026664117e28b86bbffbdebeae61e11d) ("fix(layers): remove requires_grad from Linear layer Tensor calls").
The code examples you're seeing with requires_grad=x.requires_grad and requires_grad=False are from an older version. The current implementation in main has neither Linear nor Dropout using requires_grad:
Current Linear layer:
self.weight = Tensor(weight_data) # No requires_grad
self.bias = Tensor(bias_data) # No requires_grad
Current Dropout layer:
mask_tensor = Tensor(mask.astype(np.float32)) # No requires_grad
scale = Tensor(np.array(1.0 / keep_prob)) # No requires_grad
This was a bug and has been fixed, and now it is intentional, as TinyTorch follows progressive disclosure—Module 03 teaches layers, but requires_grad isn't introduced until Module 06 (Autograd). This prevents students from encountering concepts before they've been explained. Since you're looking at an older cached or forked version, I'll close this issue as already resolved. But thank you again for the thorough review 🙏
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @minhdang26403 on GitHub (Feb 6, 2026).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1164
In the Module 03 - Layers, there are some inconsistencies between code examples. Specifically, in the Linear layer implementation, the book mentions
requires_gradparameter, but it's not set in the code.However, in the forward pass of
Dropout, the code creates Tensor and setsrequires_gradparameter. This can cause some confusion while reading the book.Therefore, I raise this issue to have a discussion about whether we should explicitly set
requires_gradparameters here. Once we have an agreement on this, I can create PR to fix this issue.@profvjreddi commented on GitHub (Feb 6, 2026):
Hi @minhdang26403! 👋 Thank you for the careful review of Module 03! You have a great eye for consistency issues.
The good news is that this was already fixed on in commit a363406 ("fix(layers): remove requires_grad from Linear layer Tensor calls").
The code examples you're seeing with requires_grad=x.requires_grad and requires_grad=False are from an older version. The current implementation in main has neither Linear nor Dropout using requires_grad:
Current Linear layer:
self.weight = Tensor(weight_data) # No requires_grad
self.bias = Tensor(bias_data) # No requires_grad
Current Dropout layer:
mask_tensor = Tensor(mask.astype(np.float32)) # No requires_grad
scale = Tensor(np.array(1.0 / keep_prob)) # No requires_grad
This was a bug and has been fixed, and now it is intentional, as TinyTorch follows progressive disclosure—Module 03 teaches layers, but requires_grad isn't introduced until Module 06 (Autograd). This prevents students from encountering concepts before they've been explained. Since you're looking at an older cached or forked version, I'll close this issue as already resolved. But thank you again for the thorough review 🙏