mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-03-12 12:23:49 -05:00
8f99655942
- Add Variable support to ReLU, Sigmoid, Tanh, and Softmax activations - Implement mathematically correct gradient functions for each activation: * ReLU: gradient = 1 if x > 0, else 0 * Sigmoid: gradient = σ(x) * (1 - σ(x)) * Tanh: gradient = 1 - tanh²(x) * Softmax: gradient with proper Jacobian computation - Maintain backward compatibility with Tensor-only usage - Add comprehensive gradient accuracy tests This enables activation functions to participate in the autograd computational graph, completing the foundation for neural network training.