{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ef7cb275",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "# Activations - Nonlinearity in Neural Networks\n",
    "\n",
    "Welcome to the Activations module! This is where neural networks get their power through nonlinearity.\n",
    "\n",
    "## Learning Goals\n",
    "- Understand why activation functions are essential for neural networks\n",
    "- Implement the four most important activation functions: ReLU, Sigmoid, Tanh, and Softmax\n",
    "- Visualize how activations transform data and enable complex learning\n",
    "- See how activations work with layers to build powerful networks\n",
    "- Master the NBGrader workflow with comprehensive testing\n",
    "\n",
    "## Build → Use → Understand\n",
    "1. **Build**: Activation functions that add nonlinearity\n",
    "2. **Use**: Transform tensors and see immediate results\n",
    "3. **Understand**: How nonlinearity enables complex pattern learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a66b7e45",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "activations-imports",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| default_exp core.activations\n",
    "\n",
    "#| export\n",
    "import math\n",
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "from typing import Union, List\n",
    "\n",
    "# Import our Tensor class - try from package first, then from local module\n",
    "try:\n",
    "    from tinytorch.core.tensor import Tensor\n",
    "except ImportError:\n",
    "    # For development, import from local tensor module\n",
    "    sys.path.append(os.path.join(os.path.dirname(__file__), '..', '01_tensor'))\n",
    "    from tensor_dev import Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70c5eb16",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "activations-welcome",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "print(\"🔥 TinyTorch Activations Module\")\n",
    "print(f\"NumPy version: {np.__version__}\")\n",
    "print(f\"Python version: {sys.version_info.major}.{sys.version_info.minor}\")\n",
    "print(\"Ready to build activation functions!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f120c05f",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 📦 Where This Code Lives in the Final Package\n",
    "\n",
    "**Learning Side:** You work in `modules/source/02_activations/activations_dev.py`  \n",
    "**Building Side:** Code exports to `tinytorch.core.activations`\n",
    "\n",
    "```python\n",
    "# Final package structure:\n",
    "from tinytorch.core.activations import ReLU, Sigmoid, Tanh, Softmax\n",
    "from tinytorch.core.tensor import Tensor  # Foundation\n",
    "from tinytorch.core.layers import Dense  # Uses activations\n",
    "```\n",
    "\n",
    "**Why this matters:**\n",
    "- **Learning:** Focused modules for deep understanding\n",
    "- **Production:** Proper organization like PyTorch's `torch.nn.ReLU`\n",
    "- **Consistency:** All activation functions live together in `core.activations`\n",
    "- **Integration:** Works seamlessly with tensors and layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e23af1",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## What Are Activation Functions?\n",
    "\n",
    "### The Problem: Linear Limitations\n",
    "Without activation functions, neural networks can only learn linear relationships:\n",
    "```\n",
    "y = W₁ · (W₂ · (W₃ · x + b₃) + b₂) + b₁\n",
    "```\n",
    "\n",
    "This simplifies to just:\n",
    "```\n",
    "y = W_combined · x + b_combined\n",
    "```\n",
    "\n",
    "**A single linear function!** No matter how many layers you add, you can't learn complex patterns like:\n",
    "- Image recognition (nonlinear pixel relationships)\n",
    "- Language understanding (nonlinear word relationships) \n",
    "- Game playing (nonlinear strategy relationships)\n",
    "\n",
    "### The Solution: Nonlinearity\n",
    "Activation functions add nonlinearity between layers:\n",
    "```\n",
    "y = W₁ · f(W₂ · f(W₃ · x + b₃) + b₂) + b₁\n",
    "```\n",
    "\n",
    "Now each layer can learn complex transformations!\n",
    "\n",
    "### Real-World Impact\n",
    "- **Before activations**: Only linear classifiers (logistic regression)\n",
    "- **After activations**: Complex pattern recognition (deep learning revolution)\n",
    "\n",
    "### What We'll Build\n",
    "1. **ReLU**: The foundation of modern deep learning\n",
    "2. **Sigmoid**: Classic activation for binary classification\n",
    "3. **Tanh**: Centered activation for better gradients\n",
    "4. **Softmax**: Probability distributions for multi-class classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8141c337",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🔧 DEVELOPMENT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f3818a",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## Step 1: ReLU - The Foundation of Deep Learning\n",
    "\n",
    "### What is ReLU?\n",
    "**ReLU (Rectified Linear Unit)** is the most important activation function in deep learning:\n",
    "\n",
    "```\n",
    "f(x) = max(0, x)\n",
    "```\n",
    "\n",
    "- **Positive inputs**: Pass through unchanged\n",
    "- **Negative inputs**: Become zero\n",
    "- **Zero**: Stays zero\n",
    "\n",
    "### Why ReLU Revolutionized Deep Learning\n",
    "1. **Computational efficiency**: Just a max operation\n",
    "2. **No vanishing gradients**: Derivative is 1 for positive values\n",
    "3. **Sparsity**: Many neurons output exactly 0\n",
    "4. **Empirical success**: Works well in practice\n",
    "\n",
    "### Visual Understanding\n",
    "```\n",
    "Input:  [-2, -1, 0, 1, 2]\n",
    "ReLU:   [ 0,  0, 0, 1, 2]\n",
    "```\n",
    "\n",
    "### Real-World Applications\n",
    "- **Image classification**: ResNet, VGG, AlexNet\n",
    "- **Object detection**: YOLO, R-CNN\n",
    "- **Language models**: Transformer feedforward layers\n",
    "- **Recommendation**: Deep collaborative filtering\n",
    "\n",
    "### Mathematical Properties\n",
    "- **Derivative**: f'(x) = 1 if x > 0, else 0\n",
    "- **Range**: [0, ∞)\n",
    "- **Sparsity**: Outputs exactly 0 for negative inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d84590de",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "relu-class",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "class ReLU:\n",
    "    \"\"\"\n",
    "    ReLU Activation Function: f(x) = max(0, x)\n",
    "    \n",
    "    The most popular activation function in deep learning.\n",
    "    Simple, fast, and effective for most applications.\n",
    "    \"\"\"\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        Apply ReLU activation: f(x) = max(0, x)\n",
    "        \n",
    "        TODO: Implement ReLU activation function.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. For each element in the input tensor, apply max(0, element)\n",
    "        2. Use NumPy's maximum function for efficient element-wise operation\n",
    "        3. Return a new tensor of the same type with the results\n",
    "        4. Preserve the input tensor's shape\n",
    "        \n",
    "        EXAMPLE USAGE:\n",
    "        ```python\n",
    "        relu = ReLU()\n",
    "        input_tensor = Tensor([[-2, -1, 0, 1, 2]])\n",
    "        output = relu(input_tensor)\n",
    "        print(output.data)  # [[0, 0, 0, 1, 2]]\n",
    "        ```\n",
    "        \n",
    "        IMPLEMENTATION HINTS:\n",
    "        - Use np.maximum(0, x.data) for element-wise max with 0\n",
    "        - Return the same type as input: return type(x)(result)\n",
    "        - The shape should remain the same as input\n",
    "        - Do not modify the input tensor (immutable operations)\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - This is like torch.nn.ReLU() in PyTorch\n",
    "        - Used in virtually every modern neural network\n",
    "        - Enables deep networks by preventing vanishing gradients\n",
    "        - Creates sparse representations (many zeros)\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        result = np.maximum(0, x.data)\n",
    "        return type(x)(result)\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def __call__(self, x):\n",
    "        \"\"\"Make the class callable: relu(x) instead of relu.forward(x)\"\"\"\n",
    "        return self.forward(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b4d900",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Test Your ReLU Implementation\n",
    "\n",
    "Once you implement the ReLU forward method above, run this cell to test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14cecf13",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": true,
     "grade_id": "test-relu-immediate",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def test_unit_relu_activation():\n",
    "    \"\"\"Unit test for the ReLU activation function.\"\"\"\n",
    "    print(\"🔬 Unit Test: ReLU Activation...\")\n",
    "\n",
    "    # Create ReLU instance\n",
    "    relu = ReLU()\n",
    "\n",
    "    # Test with mixed positive/negative values\n",
    "    test_input = Tensor([[-2, -1, 0, 1, 2]])\n",
    "    result = relu(test_input)\n",
    "    expected = np.array([[0, 0, 0, 1, 2]])\n",
    "    \n",
    "    assert np.array_equal(result.data, expected), f\"ReLU failed: expected {expected}, got {result.data}\"\n",
    "    \n",
    "    # Test that negative values become zero\n",
    "    assert np.all(result.data >= 0), \"ReLU should make all negative values zero\"\n",
    "    \n",
    "    # Test that positive values remain unchanged\n",
    "    positive_input = Tensor([[1, 2, 3, 4, 5]])\n",
    "    positive_result = relu(positive_input)\n",
    "    assert np.array_equal(positive_result.data, positive_input.data), \"ReLU should preserve positive values\"\n",
    "    \n",
    "    # Test with 2D tensor\n",
    "    matrix_input = Tensor([[-1, 2], [3, -4]])\n",
    "    matrix_result = relu(matrix_input)\n",
    "    matrix_expected = np.array([[0, 2], [3, 0]])\n",
    "    assert np.array_equal(matrix_result.data, matrix_expected), \"ReLU should work with 2D tensors\"\n",
    "    \n",
    "    # Test shape preservation\n",
    "    assert matrix_result.shape == matrix_input.shape, \"ReLU should preserve input shape\"\n",
    "    \n",
    "    print(\"✅ ReLU activation tests passed!\")\n",
    "    print(f\"✅ Negative values correctly zeroed\")\n",
    "    print(f\"✅ Positive values preserved\")\n",
    "    print(f\"✅ Shape preservation working\")\n",
    "    print(f\"✅ Works with multi-dimensional tensors\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edfce79d",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## Step 2: Sigmoid - Classic Binary Classification\n",
    "\n",
    "### What is Sigmoid?\n",
    "**Sigmoid** is the classic activation function that maps any real number to (0, 1):\n",
    "\n",
    "```\n",
    "f(x) = 1 / (1 + e^(-x))\n",
    "```\n",
    "\n",
    "### Why Sigmoid Matters\n",
    "1. **Probability interpretation**: Outputs between 0 and 1\n",
    "2. **Smooth gradients**: Differentiable everywhere\n",
    "3. **Historical importance**: Enabled early neural networks\n",
    "4. **Binary classification**: Perfect for yes/no decisions\n",
    "\n",
    "### Visual Understanding\n",
    "```\n",
    "Input:  [-∞, -2, -1, 0, 1, 2, ∞]\n",
    "Sigmoid:[0,  0.12, 0.27, 0.5, 0.73, 0.88, 1]\n",
    "```\n",
    "\n",
    "### Real-World Applications\n",
    "- **Binary classification**: Spam detection, medical diagnosis\n",
    "- **Gating mechanisms**: LSTM and GRU cells\n",
    "- **Output layers**: When you need probabilities\n",
    "- **Attention mechanisms**: Where to focus attention\n",
    "\n",
    "### Mathematical Properties\n",
    "- **Range**: (0, 1)\n",
    "- **Derivative**: f'(x) = f(x) · (1 - f(x))\n",
    "- **Centered**: f(0) = 0.5\n",
    "- **Symmetric**: f(-x) = 1 - f(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e96fbb19",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "sigmoid-class",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "class Sigmoid:\n",
    "    \"\"\"\n",
    "    Sigmoid Activation Function: f(x) = 1 / (1 + e^(-x))\n",
    "    \n",
    "    Maps any real number to the range (0, 1).\n",
    "    Useful for binary classification and probability outputs.\n",
    "    \"\"\"\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        Apply Sigmoid activation: f(x) = 1 / (1 + e^(-x))\n",
    "        \n",
    "        TODO: Implement Sigmoid activation function.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. Compute the negative of input: -x.data\n",
    "        2. Compute the exponential: np.exp(-x.data)\n",
    "        3. Add 1 to the exponential: 1 + np.exp(-x.data)\n",
    "        4. Take the reciprocal: 1 / (1 + np.exp(-x.data))\n",
    "        5. Return as new Tensor\n",
    "        \n",
    "        EXAMPLE USAGE:\n",
    "        ```python\n",
    "        sigmoid = Sigmoid()\n",
    "        input_tensor = Tensor([[-2, -1, 0, 1, 2]])\n",
    "        output = sigmoid(input_tensor)\n",
    "        print(output.data)  # [[0.119, 0.269, 0.5, 0.731, 0.881]]\n",
    "        ```\n",
    "        \n",
    "        IMPLEMENTATION HINTS:\n",
    "        - Use np.exp() for exponential function\n",
    "        - Formula: 1 / (1 + np.exp(-x.data))\n",
    "        - Handle potential overflow with np.clip(-x.data, -500, 500)\n",
    "        - Return Tensor(result)\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - This is like torch.nn.Sigmoid() in PyTorch\n",
    "        - Used in binary classification output layers\n",
    "        - Key component in LSTM and GRU gating mechanisms\n",
    "        - Historically important for early neural networks\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        # Clip to prevent overflow\n",
    "        clipped_input = np.clip(-x.data, -500, 500)\n",
    "        result = 1 / (1 + np.exp(clipped_input))\n",
    "        return type(x)(result)\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def __call__(self, x):\n",
    "        \"\"\"Make the class callable: sigmoid(x) instead of sigmoid.forward(x)\"\"\"\n",
    "        return self.forward(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b1411e8",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Test Your Sigmoid Implementation\n",
    "\n",
    "Once you implement the Sigmoid forward method above, run this cell to test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a6fc43b",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": true,
     "grade_id": "test-sigmoid-immediate",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def test_unit_sigmoid_activation():\n",
    "    \"\"\"Unit test for the Sigmoid activation function.\"\"\"\n",
    "    print(\"🔬 Unit Test: Sigmoid Activation...\")\n",
    "\n",
    "# Create Sigmoid instance\n",
    "    sigmoid = Sigmoid()\n",
    "\n",
    "    # Test with known values\n",
    "    test_input = Tensor([[0]])\n",
    "    result = sigmoid(test_input)\n",
    "    expected = 0.5\n",
    "    \n",
    "    assert abs(result.data[0][0] - expected) < 1e-6, f\"Sigmoid(0) should be 0.5, got {result.data[0][0]}\"\n",
    "    \n",
    "    # Test with positive and negative values\n",
    "    test_input = Tensor([[-2, -1, 0, 1, 2]])\n",
    "    result = sigmoid(test_input)\n",
    "    \n",
    "    # Check that all values are between 0 and 1\n",
    "    assert np.all(result.data > 0), \"Sigmoid output should be > 0\"\n",
    "    assert np.all(result.data < 1), \"Sigmoid output should be < 1\"\n",
    "    \n",
    "    # Test symmetry: sigmoid(-x) = 1 - sigmoid(x)\n",
    "    x_val = 1.0\n",
    "    pos_result = sigmoid(Tensor([[x_val]]))\n",
    "    neg_result = sigmoid(Tensor([[-x_val]]))\n",
    "    symmetry_check = abs(pos_result.data[0][0] + neg_result.data[0][0] - 1.0)\n",
    "    assert symmetry_check < 1e-6, \"Sigmoid should be symmetric around 0.5\"\n",
    "    \n",
    "    # Test with 2D tensor\n",
    "    matrix_input = Tensor([[-1, 1], [0, 2]])\n",
    "    matrix_result = sigmoid(matrix_input)\n",
    "    assert matrix_result.shape == matrix_input.shape, \"Sigmoid should preserve shape\"\n",
    "    \n",
    "    # Test extreme values (should not overflow)\n",
    "    extreme_input = Tensor([[-100, 100]])\n",
    "    extreme_result = sigmoid(extreme_input)\n",
    "    assert not np.any(np.isnan(extreme_result.data)), \"Sigmoid should handle extreme values\"\n",
    "    assert not np.any(np.isinf(extreme_result.data)), \"Sigmoid should not produce inf values\"\n",
    "    \n",
    "    print(\"✅ Sigmoid activation tests passed!\")\n",
    "    print(f\"✅ Outputs correctly bounded between 0 and 1\")\n",
    "    print(f\"✅ Symmetric property verified\")\n",
    "    print(f\"✅ Handles extreme values without overflow\")\n",
    "    print(f\"✅ Shape preservation working\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5456e34",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## Step 3: Tanh - Centered Activation\n",
    "\n",
    "### What is Tanh?\n",
    "**Tanh (Hyperbolic Tangent)** is similar to sigmoid but centered around zero:\n",
    "\n",
    "```\n",
    "f(x) = (e^x - e^(-x)) / (e^x + e^(-x))\n",
    "```\n",
    "\n",
    "### Why Tanh is Better Than Sigmoid\n",
    "1. **Zero-centered**: Outputs range from -1 to 1\n",
    "2. **Better gradients**: Helps with gradient flow in deep networks\n",
    "3. **Faster convergence**: Less bias shift during training\n",
    "4. **Stronger gradients**: Maximum gradient is 1 vs 0.25 for sigmoid\n",
    "\n",
    "### Visual Understanding\n",
    "```\n",
    "Input: [-∞, -2, -1, 0, 1, 2, ∞]\n",
    "Tanh:  [-1, -0.96, -0.76, 0, 0.76, 0.96, 1]\n",
    "```\n",
    "\n",
    "### Real-World Applications\n",
    "- **Hidden layers**: Better than sigmoid for internal activations\n",
    "- **RNN cells**: Classic RNN and LSTM use tanh\n",
    "- **Normalization**: When you need zero-centered outputs\n",
    "- **Feature scaling**: Maps inputs to [-1, 1] range\n",
    "\n",
    "### Mathematical Properties\n",
    "- **Range**: (-1, 1)\n",
    "- **Derivative**: f'(x) = 1 - f(x)²\n",
    "- **Zero-centered**: f(0) = 0\n",
    "- **Antisymmetric**: f(-x) = -f(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b4d3d36",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "tanh-class",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "class Tanh:\n",
    "    \"\"\"\n",
    "    Tanh Activation Function: f(x) = (e^x - e^(-x)) / (e^x + e^(-x))\n",
    "    \n",
    "    Zero-centered activation function with range (-1, 1).\n",
    "    Better gradient properties than sigmoid.\n",
    "    \"\"\"\n",
    "    \n",
    "    def forward(self, x: Tensor) -> Tensor:\n",
    "        \"\"\"\n",
    "        Apply Tanh activation: f(x) = (e^x - e^(-x)) / (e^x + e^(-x))\n",
    "        \n",
    "        TODO: Implement Tanh activation function.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. Use NumPy's built-in tanh function: np.tanh(x.data)\n",
    "        2. Alternatively, implement manually:\n",
    "           - Compute e^x and e^(-x)\n",
    "           - Calculate (e^x - e^(-x)) / (e^x + e^(-x))\n",
    "        3. Return as new Tensor\n",
    "        \n",
    "        EXAMPLE USAGE:\n",
    "        ```python\n",
    "        tanh = Tanh()\n",
    "        input_tensor = Tensor([[-2, -1, 0, 1, 2]])\n",
    "        output = tanh(input_tensor)\n",
    "        print(output.data)  # [[-0.964, -0.762, 0, 0.762, 0.964]]\n",
    "        ```\n",
    "        \n",
    "        IMPLEMENTATION HINTS:\n",
    "        - Use np.tanh(x.data) for simplicity\n",
    "        - Manual implementation: (np.exp(x.data) - np.exp(-x.data)) / (np.exp(x.data) + np.exp(-x.data))\n",
    "        - Handle overflow by clipping inputs: np.clip(x.data, -500, 500)\n",
    "        - Return Tensor(result)\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - This is like torch.nn.Tanh() in PyTorch\n",
    "        - Used in RNN, LSTM, and GRU cells\n",
    "        - Better than sigmoid for hidden layers\n",
    "        - Zero-centered outputs help with gradient flow\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        # Use NumPy's built-in tanh function\n",
    "        result = np.tanh(x.data)\n",
    "        return type(x)(result)\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def __call__(self, x: Tensor) -> Tensor:\n",
    "        \"\"\"Make the class callable: tanh(x) instead of tanh.forward(x)\"\"\"\n",
    "        return self.forward(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59097c9c",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Test Your Tanh Implementation\n",
    "\n",
    "Once you implement the Tanh forward method above, run this cell to test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7bb65b9",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": true,
     "grade_id": "test-tanh-immediate",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def test_unit_tanh_activation():\n",
    "    \"\"\"Unit test for the Tanh activation function.\"\"\"\n",
    "    print(\"🔬 Unit Test: Tanh Activation...\")\n",
    "\n",
    "# Create Tanh instance\n",
    "    tanh = Tanh()\n",
    "\n",
    "    # Test with zero (should be 0)\n",
    "    test_input = Tensor([[0]])\n",
    "    result = tanh(test_input)\n",
    "    expected = 0.0\n",
    "    \n",
    "    assert abs(result.data[0][0] - expected) < 1e-6, f\"Tanh(0) should be 0, got {result.data[0][0]}\"\n",
    "    \n",
    "    # Test with positive and negative values\n",
    "    test_input = Tensor([[-2, -1, 0, 1, 2]])\n",
    "    result = tanh(test_input)\n",
    "    \n",
    "    # Check that all values are between -1 and 1\n",
    "    assert np.all(result.data > -1), \"Tanh output should be > -1\"\n",
    "    assert np.all(result.data < 1), \"Tanh output should be < 1\"\n",
    "    \n",
    "    # Test antisymmetry: tanh(-x) = -tanh(x)\n",
    "    x_val = 1.5\n",
    "    pos_result = tanh(Tensor([[x_val]]))\n",
    "    neg_result = tanh(Tensor([[-x_val]]))\n",
    "    antisymmetry_check = abs(pos_result.data[0][0] + neg_result.data[0][0])\n",
    "    assert antisymmetry_check < 1e-6, \"Tanh should be antisymmetric\"\n",
    "    \n",
    "    # Test with 2D tensor\n",
    "    matrix_input = Tensor([[-1, 1], [0, 2]])\n",
    "    matrix_result = tanh(matrix_input)\n",
    "    assert matrix_result.shape == matrix_input.shape, \"Tanh should preserve shape\"\n",
    "    \n",
    "    # Test extreme values (should not overflow)\n",
    "    extreme_input = Tensor([[-100, 100]])\n",
    "    extreme_result = tanh(extreme_input)\n",
    "    assert not np.any(np.isnan(extreme_result.data)), \"Tanh should handle extreme values\"\n",
    "    assert not np.any(np.isinf(extreme_result.data)), \"Tanh should not produce inf values\"\n",
    "    \n",
    "    # Test that extreme values approach ±1\n",
    "    assert abs(extreme_result.data[0][0] - (-1)) < 1e-6, \"Tanh(-∞) should approach -1\"\n",
    "    assert abs(extreme_result.data[0][1] - 1) < 1e-6, \"Tanh(∞) should approach 1\"\n",
    "    \n",
    "    print(\"✅ Tanh activation tests passed!\")\n",
    "    print(f\"✅ Outputs correctly bounded between -1 and 1\")\n",
    "    print(f\"✅ Antisymmetric property verified\")\n",
    "    print(f\"✅ Zero-centered (tanh(0) = 0)\")\n",
    "    print(f\"✅ Handles extreme values correctly\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166814f3",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## Step 4: Softmax - Probability Distributions\n",
    "\n",
    "### What is Softmax?\n",
    "**Softmax** converts a vector of real numbers into a probability distribution:\n",
    "\n",
    "```\n",
    "f(x_i) = e^(x_i) / Σ(e^(x_j))\n",
    "```\n",
    "\n",
    "### Why Softmax is Essential\n",
    "1. **Probability distribution**: Outputs sum to 1\n",
    "2. **Multi-class classification**: Choose one class from many\n",
    "3. **Interpretable**: Each output is a probability\n",
    "4. **Differentiable**: Enables gradient-based learning\n",
    "\n",
    "### Visual Understanding\n",
    "```\n",
    "Input:  [1, 2, 3]\n",
    "Softmax:[0.09, 0.24, 0.67]  # Sums to 1.0\n",
    "```\n",
    "\n",
    "### Real-World Applications\n",
    "- **Classification**: Image classification, text classification\n",
    "- **Language models**: Next word prediction\n",
    "- **Attention mechanisms**: Where to focus attention\n",
    "- **Reinforcement learning**: Action selection probabilities\n",
    "\n",
    "### Mathematical Properties\n",
    "- **Range**: (0, 1) for each output\n",
    "- **Constraint**: Σ(f(x_i)) = 1\n",
    "- **Argmax preservation**: Doesn't change relative ordering\n",
    "- **Temperature scaling**: Can be made sharper or softer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ea27bab",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "softmax-class",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "class Softmax:\n",
    "    \"\"\"\n",
    "    Softmax Activation Function: f(x_i) = e^(x_i) / Σ(e^(x_j))\n",
    "    \n",
    "    Converts a vector of real numbers into a probability distribution.\n",
    "    Essential for multi-class classification.\n",
    "    \"\"\"\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        Apply Softmax activation: f(x_i) = e^(x_i) / Σ(e^(x_j))\n",
    "        \n",
    "        TODO: Implement Softmax activation function.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. Handle empty input case\n",
    "        2. Subtract max value for numerical stability: x - max(x)\n",
    "        3. Compute exponentials: np.exp(x - max(x))\n",
    "        4. Compute sum of exponentials: np.sum(exp_values)\n",
    "        5. Divide each exponential by the sum: exp_values / sum\n",
    "        6. Return as same tensor type as input\n",
    "        \n",
    "        EXAMPLE USAGE:\n",
    "        ```python\n",
    "        softmax = Softmax()\n",
    "        input_tensor = Tensor([[1, 2, 3]])\n",
    "        output = softmax(input_tensor)\n",
    "        print(output.data)  # [[0.09, 0.24, 0.67]]\n",
    "        print(np.sum(output.data))  # 1.0\n",
    "        ```\n",
    "        \n",
    "        IMPLEMENTATION HINTS:\n",
    "        - Handle empty case: if x.data.size == 0: return type(x)(x.data.copy())\n",
    "        - Subtract max for numerical stability: x_shifted = x.data - np.max(x.data, axis=-1, keepdims=True)\n",
    "        - Compute exponentials: exp_values = np.exp(x_shifted)\n",
    "        - Sum along last axis: sum_exp = np.sum(exp_values, axis=-1, keepdims=True)\n",
    "        - Divide: result = exp_values / sum_exp\n",
    "        - Return same type as input: return type(x)(result)\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - This is like torch.nn.Softmax() in PyTorch\n",
    "        - Used in classification output layers\n",
    "        - Key component in attention mechanisms\n",
    "        - Enables probability-based decision making\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        # Handle empty input\n",
    "        if x.data.size == 0:\n",
    "            return type(x)(x.data.copy())\n",
    "        \n",
    "        # Subtract max for numerical stability\n",
    "        x_shifted = x.data - np.max(x.data, axis=-1, keepdims=True)\n",
    "        \n",
    "        # Compute exponentials\n",
    "        exp_values = np.exp(x_shifted)\n",
    "        \n",
    "        # Sum along last axis\n",
    "        sum_exp = np.sum(exp_values, axis=-1, keepdims=True)\n",
    "        \n",
    "        # Divide to get probabilities\n",
    "        result = exp_values / sum_exp\n",
    "        \n",
    "        return type(x)(result)\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def __call__(self, x):\n",
    "        \"\"\"Make the class callable: softmax(x) instead of softmax.forward(x)\"\"\"\n",
    "        return self.forward(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877d56e6",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Test Your Softmax Implementation\n",
    "\n",
    "Once you implement the Softmax forward method above, run this cell to test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c5c426d",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": true,
     "grade_id": "test-softmax-immediate",
     "locked": true,
     "points": 15,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def test_unit_softmax_activation():\n",
    "    \"\"\"Unit test for the Softmax activation function.\"\"\"\n",
    "    print(\"🔬 Unit Test: Softmax Activation...\")\n",
    "\n",
    "# Create Softmax instance\n",
    "    softmax = Softmax()\n",
    "\n",
    "    # Test with simple input\n",
    "    test_input = Tensor([[1, 2, 3]])\n",
    "    result = softmax(test_input)\n",
    "    \n",
    "    # Check that outputs sum to 1\n",
    "    output_sum = np.sum(result.data)\n",
    "    assert abs(output_sum - 1.0) < 1e-6, f\"Softmax outputs should sum to 1, got {output_sum}\"\n",
    "    \n",
    "    # Check that all outputs are positive\n",
    "    assert np.all(result.data > 0), \"Softmax outputs should be positive\"\n",
    "    assert np.all(result.data < 1), \"Softmax outputs should be less than 1\"\n",
    "    \n",
    "    # Test with uniform input (should give equal probabilities)\n",
    "    uniform_input = Tensor([[1, 1, 1]])\n",
    "    uniform_result = softmax(uniform_input)\n",
    "    expected_prob = 1.0 / 3.0\n",
    "    \n",
    "    for prob in uniform_result.data[0]:\n",
    "        assert abs(prob - expected_prob) < 1e-6, f\"Uniform input should give equal probabilities\"\n",
    "    \n",
    "    # Test with batch input (multiple samples)\n",
    "    batch_input = Tensor([[1, 2, 3], [4, 5, 6]])\n",
    "    batch_result = softmax(batch_input)\n",
    "    \n",
    "    # Check that each row sums to 1\n",
    "    for i in range(batch_input.shape[0]):\n",
    "        row_sum = np.sum(batch_result.data[i])\n",
    "        assert abs(row_sum - 1.0) < 1e-6, f\"Each row should sum to 1, row {i} sums to {row_sum}\"\n",
    "    \n",
    "    # Test numerical stability with large values\n",
    "    large_input = Tensor([[1000, 1001, 1002]])\n",
    "    large_result = softmax(large_input)\n",
    "    \n",
    "    assert not np.any(np.isnan(large_result.data)), \"Softmax should handle large values\"\n",
    "    assert not np.any(np.isinf(large_result.data)), \"Softmax should not produce inf values\"\n",
    "    \n",
    "    large_sum = np.sum(large_result.data)\n",
    "    assert abs(large_sum - 1.0) < 1e-6, \"Large values should still sum to 1\"\n",
    "\n",
    "# Test shape preservation\n",
    "    assert batch_result.shape == batch_input.shape, \"Softmax should preserve shape\"\n",
    "    \n",
    "    print(\"✅ Softmax activation tests passed!\")\n",
    "    print(f\"✅ Outputs sum to 1 (probability distribution)\")\n",
    "    print(f\"✅ All outputs are positive\")\n",
    "    print(f\"✅ Handles uniform inputs correctly\")\n",
    "    print(f\"✅ Works with batch inputs\")\n",
    "    print(f\"✅ Numerically stable with large values\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4cbae56",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## 🎯 Comprehensive Test: All Activations Working Together\n",
    "\n",
    "### Real-World Scenario\n",
    "Let us test how all activation functions work together in a realistic neural network scenario:\n",
    "\n",
    "- **Input processing**: Raw data transformation\n",
    "- **Hidden layers**: ReLU for internal processing\n",
    "- **Output layer**: Softmax for classification\n",
    "- **Comparison**: See how different activations transform the same data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e972ac59",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": true,
     "grade_id": "test-activations-comprehensive",
     "locked": true,
     "points": 15,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def test_unit_activations_comprehensive():\n",
    "    \"\"\"Comprehensive unit test for all activation functions working together.\"\"\"\n",
    "    print(\"🔬 Unit Test: Activation Functions Comprehensive Test...\")\n",
    "    \n",
    "    # Create instances of all activation functions\n",
    "    relu = ReLU()\n",
    "    sigmoid = Sigmoid()\n",
    "    tanh = Tanh()\n",
    "    softmax = Softmax()\n",
    "    \n",
    "    # Test data: simulating neural network layer outputs\n",
    "    test_data = Tensor([[-2, -1, 0, 1, 2]])\n",
    "    \n",
    "    # Apply each activation function\n",
    "    relu_result = relu(test_data)\n",
    "    sigmoid_result = sigmoid(test_data)\n",
    "    tanh_result = tanh(test_data)\n",
    "    softmax_result = softmax(test_data)\n",
    "    \n",
    "    # Test that all functions preserve input shape\n",
    "    assert relu_result.shape == test_data.shape, \"ReLU should preserve shape\"\n",
    "    assert sigmoid_result.shape == test_data.shape, \"Sigmoid should preserve shape\"\n",
    "    assert tanh_result.shape == test_data.shape, \"Tanh should preserve shape\"\n",
    "    assert softmax_result.shape == test_data.shape, \"Softmax should preserve shape\"\n",
    "    \n",
    "    # Test that all functions return Tensor objects\n",
    "    assert isinstance(relu_result, Tensor), \"ReLU should return Tensor\"\n",
    "    assert isinstance(sigmoid_result, Tensor), \"Sigmoid should return Tensor\"\n",
    "    assert isinstance(tanh_result, Tensor), \"Tanh should return Tensor\"\n",
    "    assert isinstance(softmax_result, Tensor), \"Softmax should return Tensor\"\n",
    "    \n",
    "    # Test ReLU properties\n",
    "    assert np.all(relu_result.data >= 0), \"ReLU output should be non-negative\"\n",
    "    \n",
    "    # Test Sigmoid properties\n",
    "    assert np.all(sigmoid_result.data > 0), \"Sigmoid output should be positive\"\n",
    "    assert np.all(sigmoid_result.data < 1), \"Sigmoid output should be less than 1\"\n",
    "    \n",
    "    # Test Tanh properties\n",
    "    assert np.all(tanh_result.data > -1), \"Tanh output should be > -1\"\n",
    "    assert np.all(tanh_result.data < 1), \"Tanh output should be < 1\"\n",
    "    \n",
    "    # Test Softmax properties\n",
    "    softmax_sum = np.sum(softmax_result.data)\n",
    "    assert abs(softmax_sum - 1.0) < 1e-6, \"Softmax outputs should sum to 1\"\n",
    "    \n",
    "    # Test chaining activations (realistic neural network scenario)\n",
    "    # Hidden layer with ReLU\n",
    "    hidden_output = relu(test_data)\n",
    "    \n",
    "    # Add some weights simulation (element-wise multiplication)\n",
    "    weights = Tensor([[0.5, 0.3, 0.8, 0.2, 0.7]])\n",
    "    weighted_output = hidden_output * weights\n",
    "    \n",
    "    # Final layer with Softmax\n",
    "    final_output = softmax(weighted_output)\n",
    "    \n",
    "    # Test that chained operations work\n",
    "    assert isinstance(final_output, Tensor), \"Chained operations should return Tensor\"\n",
    "    assert abs(np.sum(final_output.data) - 1.0) < 1e-6, \"Final output should be valid probability\"\n",
    "    \n",
    "    # Test with batch data (multiple samples)\n",
    "    batch_data = Tensor([\n",
    "    [-2, -1, 0, 1, 2],\n",
    "    [1, 2, 3, 4, 5],\n",
    "    [-1, 0, 1, 2, 3]\n",
    "    ])\n",
    "    \n",
    "    batch_softmax = softmax(batch_data)\n",
    "    \n",
    "    # Each row should sum to 1\n",
    "    for i in range(batch_data.shape[0]):\n",
    "        row_sum = np.sum(batch_softmax.data[i])\n",
    "        assert abs(row_sum - 1.0) < 1e-6, f\"Batch row {i} should sum to 1\"\n",
    "    \n",
    "    print(\"✅ Activation functions comprehensive tests passed!\")\n",
    "    print(f\"✅ All functions work together seamlessly\")\n",
    "    print(f\"✅ Shape preservation across all activations\")\n",
    "    print(f\"✅ Chained operations work correctly\")\n",
    "    print(f\"✅ Batch processing works for all activations\")\n",
    "    print(f\"✅ Ready for neural network integration!\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aa04f2e",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def test_module_activation_tensor_integration():\n",
    "    \"\"\"\n",
    "    Integration test for activation functions with Tensor operations.\n",
    "    \n",
    "    Tests that activation functions properly integrate with the Tensor class\n",
    "    and maintain compatibility for neural network operations.\n",
    "    \"\"\"\n",
    "    print(\"🔬 Running Integration Test: Activation-Tensor Integration...\")\n",
    "    \n",
    "    # Test 1: Activation functions preserve Tensor types\n",
    "    input_tensor = Tensor([-2.0, -1.0, 0.0, 1.0, 2.0])\n",
    "    \n",
    "    relu_fn = ReLU()\n",
    "    sigmoid_fn = Sigmoid()\n",
    "    tanh_fn = Tanh()\n",
    "    \n",
    "    relu_result = relu_fn(input_tensor)\n",
    "    sigmoid_result = sigmoid_fn(input_tensor) \n",
    "    tanh_result = tanh_fn(input_tensor)\n",
    "    \n",
    "    assert isinstance(relu_result, Tensor), \"ReLU should return Tensor\"\n",
    "    assert isinstance(sigmoid_result, Tensor), \"Sigmoid should return Tensor\"\n",
    "    assert isinstance(tanh_result, Tensor), \"Tanh should return Tensor\"\n",
    "    \n",
    "    # Test 2: Activations work with matrix Tensors (neural network layers)\n",
    "    layer_output = Tensor([[1.0, -2.0, 3.0], \n",
    "                          [-1.0, 2.0, -3.0]])  # Simulating dense layer output\n",
    "    \n",
    "    relu_fn = ReLU()\n",
    "    activated = relu_fn(layer_output)\n",
    "    expected = np.array([[1.0, 0.0, 3.0], \n",
    "                        [0.0, 2.0, 0.0]])\n",
    "    \n",
    "    assert isinstance(activated, Tensor), \"Matrix activation should return Tensor\"\n",
    "    assert np.array_equal(activated.data, expected), \"Matrix ReLU should work correctly\"\n",
    "    \n",
    "    # Test 3: Softmax with classification scenario\n",
    "    logits = Tensor([[2.0, 1.0, 0.1],  # Batch of 2 samples\n",
    "                    [1.0, 3.0, 0.2]])   # Each with 3 classes\n",
    "    \n",
    "    softmax_fn = Softmax()\n",
    "    probabilities = softmax_fn(logits)\n",
    "    \n",
    "    assert isinstance(probabilities, Tensor), \"Softmax should return Tensor\"\n",
    "    assert probabilities.shape == logits.shape, \"Softmax should preserve shape\"\n",
    "    \n",
    "    # Each row should sum to 1 (probability distribution)\n",
    "    for i in range(logits.shape[0]):\n",
    "        row_sum = np.sum(probabilities.data[i])\n",
    "        assert abs(row_sum - 1.0) < 1e-6, f\"Probability row {i} should sum to 1\"\n",
    "    \n",
    "    # Test 4: Chaining tensor operations with activations\n",
    "    x = Tensor([1.0, 2.0, 3.0])\n",
    "    y = Tensor([4.0, 5.0, 6.0])\n",
    "    \n",
    "    # Simulate: dense layer output -> activation -> more operations\n",
    "    dense_sim = x * y  # Element-wise multiplication (simulating dense layer)\n",
    "    relu_fn = ReLU()\n",
    "    activated = relu_fn(dense_sim)  # Apply activation\n",
    "    final = activated + Tensor([1.0, 1.0, 1.0])  # More tensor operations\n",
    "    \n",
    "    expected_final = np.array([5.0, 11.0, 19.0])  # [4,10,18] -> relu -> +1 = [5,11,19]\n",
    "    \n",
    "    assert isinstance(final, Tensor), \"Chained operations should maintain Tensor type\"\n",
    "    assert np.array_equal(final.data, expected_final), \"Chained operations should work correctly\"\n",
    "    \n",
    "    print(\"✅ Integration Test Passed: Activation-Tensor integration works correctly.\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f8c25e6",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## ⚡ ML Systems: Performance Analysis & Optimization\n",
    "\n",
    "Now that you have working activation functions, let us develop **performance engineering skills**. This section teaches you to measure computational costs, understand scaling patterns, and think about production optimization.\n",
    "\n",
    "### **Learning Outcome**: *\"I understand performance trade-offs between different activation functions\"*\n",
    "\n",
    "---\n",
    "\n",
    "## Performance Profiling Tools (Light Implementation)\n",
    "\n",
    "As an ML systems engineer, you need to understand which activation functions are fast vs slow, and why. Let us build simple tools to measure and compare performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c38ee3ef",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "activation-profiler",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "import time\n",
    "\n",
    "class ActivationProfiler:\n",
    "    \"\"\"\n",
    "    Performance profiling toolkit for activation functions.\n",
    "    \n",
    "    Helps ML engineers understand computational costs and optimize\n",
    "    neural network performance for production deployment.\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.results = {}\n",
    "        \n",
    "    def time_activation(self, activation_fn, tensor, activation_name, iterations=100):\n",
    "        \"\"\"\n",
    "        Time how long an activation function takes to run.\n",
    "        \n",
    "        TODO: Implement activation timing.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. Record start time using time.time()\n",
    "        2. Run the activation function for specified iterations\n",
    "        3. Record end time\n",
    "        4. Calculate average time per iteration\n",
    "        5. Return the average time in milliseconds\n",
    "        \n",
    "        EXAMPLE:\n",
    "        profiler = ActivationProfiler()\n",
    "        relu = ReLU()\n",
    "        test_tensor = Tensor(np.random.randn(1000, 1000))\n",
    "        avg_time = profiler.time_activation(relu, test_tensor, \"ReLU\")\n",
    "        print(f\"ReLU took {avg_time:.3f} ms on average\")\n",
    "        \n",
    "        HINTS:\n",
    "        - Use time.time() for timing\n",
    "        - Run multiple iterations for better accuracy\n",
    "        - Calculate: (end_time - start_time) / iterations * 1000 for ms\n",
    "        - Return the average time per call in milliseconds\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        start_time = time.time()\n",
    "        \n",
    "        for _ in range(iterations):\n",
    "            result = activation_fn(tensor)\n",
    "        \n",
    "        end_time = time.time()\n",
    "        avg_time_ms = (end_time - start_time) / iterations * 1000\n",
    "        \n",
    "        return avg_time_ms\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def compare_activations(self, tensor_size=(1000, 1000), iterations=50):\n",
    "        \"\"\"\n",
    "        Compare performance of all activation functions.\n",
    "        \n",
    "        This function is PROVIDED to show systems analysis.\n",
    "        Students run it to understand performance differences.\n",
    "        \"\"\"\n",
    "        print(f\"⚡ ACTIVATION PERFORMANCE COMPARISON\")\n",
    "        print(f\"=\" * 50)\n",
    "        print(f\"Tensor size: {tensor_size}, Iterations: {iterations}\")\n",
    "        \n",
    "        # Create test tensor\n",
    "        test_tensor = Tensor(np.random.randn(*tensor_size))\n",
    "        tensor_mb = test_tensor.data.nbytes / (1024 * 1024)\n",
    "        print(f\"Test tensor: {tensor_mb:.2f} MB\")\n",
    "        \n",
    "        # Test all activation functions\n",
    "        activations = {\n",
    "            'ReLU': ReLU(),\n",
    "            'Sigmoid': Sigmoid(),\n",
    "            'Tanh': Tanh(),\n",
    "            'Softmax': Softmax()\n",
    "        }\n",
    "        \n",
    "        results = {}\n",
    "        for name, activation_fn in activations.items():\n",
    "            avg_time = self.time_activation(activation_fn, test_tensor, name, iterations)\n",
    "            results[name] = avg_time\n",
    "            print(f\"   {name:8}: {avg_time:.3f} ms\")\n",
    "        \n",
    "        # Calculate speed ratios relative to fastest\n",
    "        fastest_time = min(results.values())\n",
    "        fastest_name = min(results, key=results.get)\n",
    "        \n",
    "        print(f\"\\n📊 SPEED ANALYSIS:\")\n",
    "        for name, time_ms in sorted(results.items(), key=lambda x: x[1]):\n",
    "            speed_ratio = time_ms / fastest_time\n",
    "            if name == fastest_name:\n",
    "                print(f\"   {name:8}: {speed_ratio:.1f}x (fastest)\")\n",
    "            else:\n",
    "                print(f\"   {name:8}: {speed_ratio:.1f}x slower than {fastest_name}\")\n",
    "        \n",
    "        return results\n",
    "    \n",
    "    def analyze_scaling(self, activation_fn, activation_name, sizes=[100, 500, 1000]):\n",
    "        \"\"\"\n",
    "        Analyze how activation performance scales with tensor size.\n",
    "        \n",
    "        This function is PROVIDED to demonstrate scaling patterns.\n",
    "        Students use it to understand computational complexity.\n",
    "        \"\"\"\n",
    "        print(f\"\\n🔍 SCALING ANALYSIS: {activation_name}\")\n",
    "        print(f\"=\" * 40)\n",
    "        \n",
    "        scaling_results = []\n",
    "        \n",
    "        for size in sizes:\n",
    "            test_tensor = Tensor(np.random.randn(size, size))\n",
    "            avg_time = self.time_activation(activation_fn, test_tensor, activation_name, iterations=20)\n",
    "            \n",
    "            elements = size * size\n",
    "            time_per_element = avg_time / elements * 1e6  # microseconds per element\n",
    "            \n",
    "            result = {\n",
    "                'size': size,\n",
    "                'elements': elements,\n",
    "                'time_ms': avg_time,\n",
    "                'time_per_element_us': time_per_element\n",
    "            }\n",
    "            scaling_results.append(result)\n",
    "            \n",
    "            print(f\"   {size}x{size}: {avg_time:.3f}ms ({time_per_element:.3f}μs/element)\")\n",
    "        \n",
    "        # Analyze scaling pattern\n",
    "        if len(scaling_results) >= 2:\n",
    "            small = scaling_results[0]\n",
    "            large = scaling_results[-1]\n",
    "            \n",
    "            size_ratio = large['size'] / small['size']\n",
    "            time_ratio = large['time_ms'] / small['time_ms']\n",
    "            \n",
    "            print(f\"\\n📈 Scaling Pattern:\")\n",
    "            print(f\"   Size increased {size_ratio:.1f}x ({small['size']} → {large['size']})\")\n",
    "            print(f\"   Time increased {time_ratio:.1f}x\")\n",
    "            \n",
    "            if abs(time_ratio - size_ratio**2) < abs(time_ratio - size_ratio):\n",
    "                print(f\"   Pattern: O(n^2) - linear in tensor size\")\n",
    "            else:\n",
    "                print(f\"   Pattern: ~O(n) - very efficient scaling\")\n",
    "        \n",
    "        return scaling_results\n",
    "\n",
    "def benchmark_activation_suite():\n",
    "    \"\"\"\n",
    "    Comprehensive benchmark of all activation functions.\n",
    "    \n",
    "    This function is PROVIDED to show complete systems analysis.\n",
    "    Students run it to understand production performance implications.\n",
    "    \"\"\"\n",
    "    profiler = ActivationProfiler()\n",
    "    \n",
    "    print(\"🏆 COMPREHENSIVE ACTIVATION BENCHMARK\")\n",
    "    print(\"=\" * 60)\n",
    "    \n",
    "    # Test 1: Performance comparison\n",
    "    comparison_results = profiler.compare_activations(tensor_size=(800, 800), iterations=30)\n",
    "    \n",
    "    # Test 2: Scaling analysis for each activation\n",
    "    activations_to_test = [\n",
    "        (ReLU(), \"ReLU\"),\n",
    "        (Sigmoid(), \"Sigmoid\"),\n",
    "        (Tanh(), \"Tanh\")\n",
    "    ]\n",
    "    \n",
    "    for activation_fn, name in activations_to_test:\n",
    "        profiler.analyze_scaling(activation_fn, name, sizes=[200, 400, 600])\n",
    "    \n",
    "    # Test 3: Memory vs Performance trade-offs\n",
    "    print(f\"\\n💾 MEMORY vs PERFORMANCE ANALYSIS:\")\n",
    "    print(f\"=\" * 40)\n",
    "    \n",
    "    test_tensor = Tensor(np.random.randn(500, 500))\n",
    "    original_memory = test_tensor.data.nbytes / (1024 * 1024)\n",
    "    \n",
    "    for name, activation_fn in [(\"ReLU\", ReLU()), (\"Sigmoid\", Sigmoid())]:\n",
    "        start_time = time.time()\n",
    "        result = activation_fn(test_tensor)\n",
    "        end_time = time.time()\n",
    "        \n",
    "        result_memory = result.data.nbytes / (1024 * 1024)\n",
    "        time_ms = (end_time - start_time) * 1000\n",
    "        \n",
    "        print(f\"   {name}:\")\n",
    "        print(f\"     Input: {original_memory:.2f} MB\")\n",
    "        print(f\"     Output: {result_memory:.2f} MB\")\n",
    "        print(f\"     Memory overhead: {result_memory - original_memory:.2f} MB\")\n",
    "        print(f\"     Time: {time_ms:.3f} ms\")\n",
    "    \n",
    "    print(f\"\\n🎯 PRODUCTION INSIGHTS:\")\n",
    "    print(f\"   - ReLU is typically fastest (simple max operation)\")\n",
    "    print(f\"   - Sigmoid/Tanh slower due to exponential calculations\")\n",
    "    print(f\"   - All operations scale linearly with tensor size\")\n",
    "    print(f\"   - Memory usage doubles (input + output tensors)\")\n",
    "    print(f\"   - Choose activation based on accuracy vs speed trade-offs\")\n",
    "    \n",
    "    return comparison_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17904e8",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Test: Activation Performance Profiling\n",
    "\n",
    "Let us test our activation profiler with realistic performance analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25650903",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "test-activation-profiler",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def test_activation_profiler():\n",
    "    \"\"\"Test activation profiler with comprehensive scenarios.\"\"\"\n",
    "    print(\"🔬 Unit Test: Activation Performance Profiler...\")\n",
    "    \n",
    "    profiler = ActivationProfiler()\n",
    "    \n",
    "    # Create test tensor\n",
    "    test_tensor = Tensor(np.random.randn(100, 100))\n",
    "    relu = ReLU()\n",
    "    \n",
    "    # Test timing functionality\n",
    "    avg_time = profiler.time_activation(relu, test_tensor, \"ReLU\", iterations=10)\n",
    "    \n",
    "    # Verify timing results\n",
    "    assert isinstance(avg_time, (int, float)), \"Should return numeric time\"\n",
    "    assert avg_time > 0, \"Time should be positive\"\n",
    "    assert avg_time < 1000, \"Time should be reasonable (< 1000ms)\"\n",
    "    \n",
    "    print(\"✅ Basic timing functionality test passed\")\n",
    "    \n",
    "    # Test comparison functionality\n",
    "    comparison_results = profiler.compare_activations(tensor_size=(50, 50), iterations=5)\n",
    "    \n",
    "    # Verify comparison results\n",
    "    assert isinstance(comparison_results, dict), \"Should return dictionary of results\"\n",
    "    assert len(comparison_results) == 4, \"Should test all 4 activation functions\"\n",
    "    \n",
    "    expected_activations = ['ReLU', 'Sigmoid', 'Tanh', 'Softmax']\n",
    "    for activation in expected_activations:\n",
    "        assert activation in comparison_results, f\"Should include {activation}\"\n",
    "        assert comparison_results[activation] > 0, f\"{activation} time should be positive\"\n",
    "    \n",
    "    print(\"✅ Activation comparison test passed\")\n",
    "    \n",
    "    # Test scaling analysis\n",
    "    scaling_results = profiler.analyze_scaling(relu, \"ReLU\", sizes=[50, 100])\n",
    "    \n",
    "    # Verify scaling results\n",
    "    assert isinstance(scaling_results, list), \"Should return list of scaling results\"\n",
    "    assert len(scaling_results) == 2, \"Should test both sizes\"\n",
    "    \n",
    "    for result in scaling_results:\n",
    "        assert 'size' in result, \"Should include size\"\n",
    "        assert 'time_ms' in result, \"Should include timing\"\n",
    "        assert result['time_ms'] > 0, \"Time should be positive\"\n",
    "    \n",
    "    print(\"✅ Scaling analysis test passed\")\n",
    "    \n",
    "    print(\"🎯 Activation Profiler: All tests passed!\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdf9ac1c",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 🎯 Learning Activity: Activation Performance Analysis\n",
    "\n",
    "**Goal**: Learn to measure activation function performance and understand which operations are fast vs slow in production ML systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "244a1548",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "activation-performance-analysis",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Activation profiler initialization moved to main block\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    # Initialize the activation profiler\n",
    "    profiler = ActivationProfiler()\n",
    "    \n",
    "    # Run all activation tests\n",
    "    test_unit_relu_activation()\n",
    "    test_unit_sigmoid_activation()\n",
    "    test_unit_tanh_activation()\n",
    "    test_unit_softmax_activation()\n",
    "    test_unit_activations_comprehensive()\n",
    "    test_module_activation_tensor_integration()\n",
    "    test_activation_profiler()\n",
    "    \n",
    "    print(\"⚡ ACTIVATION PERFORMANCE ANALYSIS\")\n",
    "    print(\"=\" * 50)\n",
    "\n",
    "    # Create test data\n",
    "    test_tensor = Tensor(np.random.randn(500, 500))  # Medium-sized tensor for testing\n",
    "    print(f\"Test tensor size: {test_tensor.shape}\")\n",
    "    print(f\"Memory footprint: {test_tensor.data.nbytes/(1024*1024):.2f} MB\")\n",
    "\n",
    "    # Test individual activation timing\n",
    "    print(f\"\\n🎯 Individual Activation Timing:\")\n",
    "    activations_to_test = [\n",
    "        (ReLU(), \"ReLU\"),\n",
    "        (Sigmoid(), \"Sigmoid\"), \n",
    "        (Tanh(), \"Tanh\"),\n",
    "        (Softmax(), \"Softmax\")\n",
    "    ]\n",
    "\n",
    "    individual_results = {}\n",
    "    for activation_fn, name in activations_to_test:\n",
    "        # Students implement this timing call\n",
    "        avg_time = profiler.time_activation(activation_fn, test_tensor, name, iterations=50)\n",
    "        individual_results[name] = avg_time\n",
    "        print(f\"   {name:8}: {avg_time:.3f} ms average\")\n",
    "\n",
    "    # Analyze the results  \n",
    "    fastest = min(individual_results, key=individual_results.get)\n",
    "    slowest = max(individual_results, key=individual_results.get)\n",
    "    speed_ratio = individual_results[slowest] / individual_results[fastest]\n",
    "\n",
    "    print(f\"\\n📊 PERFORMANCE INSIGHTS:\")\n",
    "    print(f\"   Fastest: {fastest} ({individual_results[fastest]:.3f} ms)\")\n",
    "    print(f\"   Slowest: {slowest} ({individual_results[slowest]:.3f} ms)\")\n",
    "    print(f\"   Speed difference: {speed_ratio:.1f}x\")\n",
    "\n",
    "    print(f\"\\n💡 WHY THE DIFFERENCE?\")\n",
    "    print(f\"   - ReLU: Just max(0, x) - simple comparison\")\n",
    "    print(f\"   - Sigmoid: Requires exponential calculation\")\n",
    "    print(f\"   - Tanh: Also exponential, but often optimized\")\n",
    "    print(f\"   - Softmax: Exponentials + division\")\n",
    "\n",
    "    print(f\"\\n🏭 PRODUCTION IMPLICATIONS:\")\n",
    "    print(f\"   - ReLU dominates modern deep learning (speed + effectiveness)\")\n",
    "    print(f\"   - Sigmoid/Tanh used where probability interpretation needed\")\n",
    "    print(f\"   - Speed matters: 1000 layers × speed difference = major impact\")\n",
    "    \n",
    "    print(\"All tests passed!\")\n",
    "    print(\"Activations module complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c725a597",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🤔 ML Systems Thinking: Interactive Questions\n",
    "\n",
    "Now that you've built the nonlinear functions that enable neural network intelligence, let's connect this foundational work to broader ML systems challenges. These questions help you think critically about how activation functions scale to production ML environments.\n",
    "\n",
    "Take time to reflect thoughtfully on each question - your insights will help you understand how the activation concepts you've implemented connect to real-world ML systems engineering."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eab90f8",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### Question 1: Computational Efficiency and Numerical Stability\n",
    "\n",
    "**Context**: Your activation implementations handle basic operations like ReLU's max(0, x) and Softmax's exponential computations. In production ML systems, these operations run billions of times during training and inference, making computational efficiency and numerical stability critical for system reliability.\n",
    "\n",
    "**Reflection Question**: Design a production-grade activation function system that balances computational efficiency with numerical stability. How would you optimize ReLU for sparse computation, implement numerically stable Softmax for large vocabulary language models, and handle precision requirements across different hardware platforms? Consider scenarios where numerical instability in activation functions could cascade through deep networks and cause training failures.\n",
    "\n",
    "Think about: vectorization strategies, overflow/underflow protection, sparse computation optimization, and precision trade-offs between speed and accuracy.\n",
    "\n",
    "*Target length: 150-300 words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "677cb4c6",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "question-1-computational-efficiency",
     "locked": false,
     "points": 10,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "YOUR REFLECTION ON COMPUTATIONAL EFFICIENCY AND NUMERICAL STABILITY:\n",
    "\n",
    "TODO: Replace this text with your thoughtful response about production-grade activation function design.\n",
    "\n",
    "Consider addressing:\n",
    "- How would you optimize activation functions for both efficiency and numerical stability?\n",
    "- What strategies would you use to handle large-scale sparse computation in ReLU?\n",
    "- How would you implement numerically stable Softmax for large vocabulary models?\n",
    "- What precision trade-offs would you make across different hardware platforms?\n",
    "- How would you prevent numerical instability from cascading through deep networks?\n",
    "\n",
    "Write a technical analysis connecting your activation implementations to real production optimization challenges.\n",
    "\n",
    "GRADING RUBRIC (Instructor Use):\n",
    "- Demonstrates understanding of efficiency vs stability trade-offs (3 points)\n",
    "- Addresses numerical stability concerns in large-scale systems (3 points)\n",
    "- Shows practical knowledge of optimization strategies (2 points)\n",
    "- Demonstrates systems thinking about activation function design (2 points)\n",
    "- Clear technical reasoning and practical considerations (bonus points for innovative approaches)\n",
    "\"\"\"\n",
    "\n",
    "### BEGIN SOLUTION\n",
    "# Student response area - instructor will replace this section during grading setup\n",
    "# This is a manually graded question requiring technical analysis of activation optimization\n",
    "# Students should demonstrate understanding of efficiency and numerical stability in production systems\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3f1657b",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### Question 2: Hardware Optimization and Parallelization\n",
    "\n",
    "**Context**: Your activation functions perform element-wise operations that are ideal for parallel computation. Production ML systems deploy these functions across diverse hardware: CPUs, GPUs, TPUs, and edge devices, each with different computational characteristics and optimization opportunities.\n",
    "\n",
    "**Reflection Question**: Architect a hardware-aware activation function system that automatically optimizes for different compute platforms. How would you leverage ReLU's sparsity for GPU memory optimization, implement vectorized operations for CPU SIMD instructions, and design activation kernels for specialized AI accelerators? Consider the challenges of maintaining consistent numerical behavior across platforms while maximizing hardware-specific performance.\n",
    "\n",
    "Think about: SIMD vectorization, GPU kernel fusion, sparse computation patterns, and platform-specific optimization techniques.\n",
    "\n",
    "*Target length: 150-300 words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba879308",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "question-2-hardware-optimization",
     "locked": false,
     "points": 10,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "YOUR REFLECTION ON HARDWARE OPTIMIZATION AND PARALLELIZATION:\n",
    "\n",
    "TODO: Replace this text with your thoughtful response about hardware-aware activation function design.\n",
    "\n",
    "Consider addressing:\n",
    "- How would you design activation functions that optimize for different hardware platforms?\n",
    "- What strategies would you use to leverage GPU parallelism for activation computations?\n",
    "- How would you implement SIMD vectorization for CPU-based activation functions?\n",
    "- What role would kernel fusion play in optimizing activation performance?\n",
    "- How would you maintain numerical consistency across different hardware platforms?\n",
    "\n",
    "Write an architectural analysis connecting your activation implementations to real hardware optimization challenges.\n",
    "\n",
    "GRADING RUBRIC (Instructor Use):\n",
    "- Shows understanding of hardware-specific optimization strategies (3 points)\n",
    "- Designs practical approaches to parallel activation computation (3 points)\n",
    "- Addresses platform consistency and performance trade-offs (2 points)\n",
    "- Demonstrates systems thinking about hardware-software optimization (2 points)\n",
    "- Clear architectural reasoning with hardware insights (bonus points for comprehensive understanding)\n",
    "\"\"\"\n",
    "\n",
    "### BEGIN SOLUTION\n",
    "# Student response area - instructor will replace this section during grading setup\n",
    "# This is a manually graded question requiring understanding of hardware optimization challenges\n",
    "# Students should demonstrate knowledge of parallel computation and platform-specific optimization\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b77fa4",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### Question 3: Integration with Training Systems and Gradient Flow\n",
    "\n",
    "**Context**: Your activation functions will integrate with automatic differentiation systems for training neural networks. The choice and implementation of activation functions significantly impacts gradient flow, training stability, and convergence speed in large-scale ML training systems.\n",
    "\n",
    "**Reflection Question**: Design an activation function integration system for large-scale neural network training that optimizes gradient flow and training stability. How would you implement activation functions that support efficient gradient computation, handle the vanishing gradient problem in deep networks, and integrate with distributed training systems? Consider the challenges of maintaining training stability when activation choices affect gradient magnitude and direction across hundreds of layers.\n",
    "\n",
    "Think about: gradient flow characteristics, backpropagation efficiency, training stability, and distributed training considerations.\n",
    "\n",
    "*Target length: 150-300 words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c159289a",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "question-3-training-integration",
     "locked": false,
     "points": 10,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "YOUR REFLECTION ON INTEGRATION WITH TRAINING SYSTEMS:\n",
    "\n",
    "TODO: Replace this text with your thoughtful response about activation function integration with training systems.\n",
    "\n",
    "Consider addressing:\n",
    "- How would you design activation functions to optimize gradient flow in deep networks?\n",
    "- What strategies would you use to handle vanishing/exploding gradient problems?\n",
    "- How would you integrate activation functions with automatic differentiation systems?\n",
    "- What role would activation choices play in distributed training stability?\n",
    "- How would you balance activation complexity with training efficiency?\n",
    "\n",
    "Write a design analysis connecting your activation functions to automatic differentiation and training optimization.\n",
    "\n",
    "GRADING RUBRIC (Instructor Use):\n",
    "- Understands activation function impact on gradient flow and training (3 points)\n",
    "- Designs practical approaches to training integration and stability (3 points)\n",
    "- Addresses distributed training and efficiency considerations (2 points)\n",
    "- Shows systems thinking about training system architecture (2 points)\n",
    "- Clear design reasoning with training optimization insights (bonus points for deep understanding)\n",
    "\"\"\"\n",
    "\n",
    "### BEGIN SOLUTION\n",
    "# Student response area - instructor will replace this section during grading setup\n",
    "# This is a manually graded question requiring understanding of training system integration\n",
    "# Students should demonstrate knowledge of gradient flow and training optimization challenges\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92ba9067",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🎯 MODULE SUMMARY: Activation Functions\n",
    "\n",
    "    Congratulations! You have successfully implemented all four essential activation functions:\n",
    "\n",
    "### ✅ What You have Built\n",
    "    - **ReLU**: The foundation of modern deep learning with sparsity and efficiency\n",
    "    - **Sigmoid**: Classic activation for binary classification and probability outputs\n",
    "    - **Tanh**: Zero-centered activation with better gradient properties\n",
    "    - **Softmax**: Probability distribution for multi-class classification\n",
    "\n",
    "### ✅ Key Learning Outcomes\n",
    "    - **Understanding**: Why nonlinearity is essential for neural networks\n",
    "    - **Implementation**: Built activation functions from scratch using NumPy\n",
    "    - **Testing**: Progressive validation with immediate feedback after each function\n",
    "    - **Integration**: Saw how activations work together in neural networks\n",
    "    - **Real-world context**: Understanding where each activation is used\n",
    "\n",
    "### ✅ Mathematical Mastery\n",
    "    - **ReLU**: f(x) = max(0, x) - Simple but powerful\n",
    "    - **Sigmoid**: f(x) = 1/(1 + e^(-x)) - Maps to (0,1)\n",
    "    - **Tanh**: f(x) = tanh(x) - Zero-centered, maps to (-1,1)\n",
    "    - **Softmax**: f(x_i) = e^(x_i)/Σ(e^(x_j)) - Probability distribution\n",
    "\n",
    "### ✅ Professional Skills Developed\n",
    "    - **Numerical stability**: Handling overflow and underflow\n",
    "    - **API design**: Consistent interfaces across all functions\n",
    "    - **Testing discipline**: Immediate validation after each implementation\n",
    "    - **Integration thinking**: Understanding how components work together\n",
    "\n",
    "### ✅ Ready for Next Steps\n",
    "    Your activation functions are now ready to power:\n",
    "    - **Dense layers**: Linear transformations with nonlinear activations\n",
    "    - **Convolutional layers**: Spatial feature extraction with ReLU\n",
    "    - **Network architectures**: Complete neural networks with proper activations\n",
    "    - **Training**: Gradient computation through activation functions\n",
    "\n",
    "### 🔗 Connection to Real ML Systems\n",
    "    Your implementations mirror production systems:\n",
    "    - **PyTorch**: `torch.nn.ReLU()`, `torch.nn.Sigmoid()`, `torch.nn.Tanh()`, `torch.nn.Softmax()`\n",
    "    - **TensorFlow**: `tf.nn.relu()`, `tf.nn.sigmoid()`, `tf.nn.tanh()`, `tf.nn.softmax()`\n",
    "    - **Industry applications**: Every major deep learning model uses these functions\n",
    "\n",
    "### 🎯 The Power of Nonlinearity\n",
    "    You have unlocked the key to deep learning:\n",
    "    - **Before**: Linear models limited to simple patterns\n",
    "    - **After**: Nonlinear models can learn any pattern (universal approximation)\n",
    "\n",
    "    **Next Module**: Layers - Building blocks that combine your tensors and activations into powerful transformations!\n",
    "\n",
    "    Your activation functions are the key to neural network intelligence. Now let us build the layers that use them!"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
