{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7d793e6b",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "# Networks - Neural Network Architectures\n",
    "\n",
    "Welcome to the Networks module! This is where we compose layers into complete neural network architectures.\n",
    "\n",
    "## Learning Goals\n",
    "- Understand networks as function composition: `f(x) = layer_n(...layer_2(layer_1(x)))`\n",
    "- Build the Sequential network architecture for composing layers\n",
    "- Create common network patterns like MLPs (Multi-Layer Perceptrons)\n",
    "- Visualize network architectures and understand their capabilities\n",
    "- Master forward pass inference through complete networks\n",
    "\n",
    "## Build → Use → Reflect\n",
    "1. **Build**: Sequential networks that compose layers into complete architectures\n",
    "2. **Use**: Create different network patterns and run inference\n",
    "3. **Reflect**: How architecture design affects network behavior and capability\n",
    "\n",
    "## What You'll Learn\n",
    "By the end of this module, you'll understand:\n",
    "- How simple layers combine to create complex behaviors\n",
    "- The fundamental Sequential architecture pattern\n",
    "- How to build MLPs with any number of layers\n",
    "- Different network architectures (shallow, deep, wide)\n",
    "- How neural networks approximate complex functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8843d18f",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "networks-imports",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| default_exp core.dense\n",
    "\n",
    "#| export\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "from typing import List, Optional\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Import all the building blocks we need - try package first, then local modules\n",
    "try:\n",
    "    from tinytorch.core.tensor import Tensor\n",
    "    from tinytorch.core.layers import Dense\n",
    "    from tinytorch.core.activations import ReLU, Sigmoid, Tanh, Softmax\n",
    "except ImportError:\n",
    "    # For development, import from local modules\n",
    "    sys.path.append(os.path.join(os.path.dirname(__file__), '..', '01_tensor'))\n",
    "    sys.path.append(os.path.join(os.path.dirname(__file__), '..', '02_activations'))\n",
    "    sys.path.append(os.path.join(os.path.dirname(__file__), '..', '03_layers'))\n",
    "    from tensor_dev import Tensor\n",
    "    from activations_dev import ReLU, Sigmoid, Tanh, Softmax\n",
    "    from layers_dev import Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56f6a322",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "networks-welcome",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "print(\"🔥 TinyTorch Networks Module\")\n",
    "print(f\"NumPy version: {np.__version__}\")\n",
    "print(f\"Python version: {sys.version_info.major}.{sys.version_info.minor}\")\n",
    "print(\"Ready to build neural network architectures!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70d722ef",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 📦 Where This Code Lives in the Final Package\n",
    "\n",
    "**Learning Side:** You work in `modules/source/04_networks/networks_dev.py`  \n",
    "**Building Side:** Code exports to `tinytorch.core.networks`\n",
    "\n",
    "```python\n",
    "# Final package structure:\n",
    "from tinytorch.core.networks import Sequential, create_mlp  # Network architectures!\n",
    "from tinytorch.core.layers import Dense, Conv2D  # Building blocks\n",
    "from tinytorch.core.activations import ReLU, Sigmoid, Tanh  # Nonlinearity\n",
    "from tinytorch.core.tensor import Tensor  # Foundation\n",
    "```\n",
    "\n",
    "**Why this matters:**\n",
    "- **Learning:** Focused modules for deep understanding\n",
    "- **Production:** Proper organization like PyTorch's `torch.nn.Sequential`\n",
    "- **Consistency:** All network architectures live together in `core.networks`\n",
    "- **Integration:** Works seamlessly with layers, activations, and tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b864ed9e",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🔧 DEVELOPMENT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1552c44",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## Step 1: Understanding Neural Networks as Function Composition\n",
    "\n",
    "### What is a Neural Network?\n",
    "A neural network is simply **function composition** - chaining simple functions together to create complex behaviors:\n",
    "\n",
    "```\n",
    "f(x) = f_n(f_{n-1}(...f_2(f_1(x))))\n",
    "```\n",
    "\n",
    "### Real-World Analogy: Assembly Line\n",
    "Think of an assembly line in a factory:\n",
    "- **Input:** Raw materials (data)\n",
    "- **Stations:** Each worker (layer) transforms the product\n",
    "- **Output:** Final product (predictions)\n",
    "\n",
    "### The Power of Composition\n",
    "```python\n",
    "# Simple functions\n",
    "def add_one(x): return x + 1\n",
    "def multiply_two(x): return x * 2\n",
    "def square(x): return x * x\n",
    "\n",
    "# Composed function\n",
    "def complex_function(x):\n",
    "    return square(multiply_two(add_one(x)))\n",
    "    \n",
    "# This is what neural networks do!\n",
    "```\n",
    "\n",
    "### Why This Matters\n",
    "- **Universal Approximation:** MLPs can approximate any continuous function\n",
    "- **Hierarchical Learning:** Early layers learn simple features, later layers learn complex patterns\n",
    "- **Composability:** Mix and match layers to create custom architectures\n",
    "- **Scalability:** Add more layers or make them wider as needed\n",
    "\n",
    "### From Modules We've Built\n",
    "- **Tensors:** The data containers that flow through networks\n",
    "- **Activations:** The nonlinear transformations that enable complex behaviors\n",
    "- **Layers:** The building blocks that transform data\n",
    "\n",
    "Now let's build our first network architecture!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc0faf1",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## Step 2: Building the Sequential Network\n",
    "\n",
    "### What is Sequential?\n",
    "**Sequential** is the most fundamental network architecture - it applies layers in order:\n",
    "\n",
    "```\n",
    "Sequential([layer1, layer2, layer3]) \n",
    "→ f(x) = layer3(layer2(layer1(x)))\n",
    "```\n",
    "\n",
    "### Why Sequential Matters\n",
    "- **Foundation:** Every neural network library has this pattern\n",
    "- **Simplicity:** Easy to understand and implement\n",
    "- **Flexibility:** Can compose any layers in any order\n",
    "- **Building Block:** Foundation for more complex architectures\n",
    "\n",
    "### The Sequential Pattern\n",
    "```python\n",
    "# PyTorch style\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 128),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 10)\n",
    ")\n",
    "\n",
    "# Our TinyTorch style\n",
    "model = Sequential([\n",
    "    Dense(784, 128),\n",
    "    ReLU(),\n",
    "    Dense(128, 10)\n",
    "])\n",
    "```\n",
    "\n",
    "Let's implement this fundamental architecture!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb18863",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "sequential-class",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "class Sequential:\n",
    "    \"\"\"\n",
    "    Sequential Network: Composes layers in sequence\n",
    "    \n",
    "    The most fundamental network architecture.\n",
    "    Applies layers in order: f(x) = layer_n(...layer_2(layer_1(x)))\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, layers: Optional[List] = None):\n",
    "        \"\"\"\n",
    "        Initialize Sequential network with layers.\n",
    "        \n",
    "        Args:\n",
    "            layers: List of layers to compose in order (optional, defaults to empty list)\n",
    "            \n",
    "        TODO: Store the layers and implement forward pass\n",
    "        \n",
    "        APPROACH:\n",
    "        1. Store the layers list as an instance variable\n",
    "        2. Initialize empty list if no layers provided\n",
    "        3. Prepare for forward pass implementation\n",
    "        \n",
    "        EXAMPLE:\n",
    "        Sequential([Dense(3,4), ReLU(), Dense(4,2)])\n",
    "        creates a 3-layer network: Dense → ReLU → Dense\n",
    "        \n",
    "        HINTS:\n",
    "        - Use self.layers to store the layers\n",
    "        - Handle empty initialization case\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - This is equivalent to torch.nn.Sequential in PyTorch\n",
    "        - Used in every neural network to chain layers together\n",
    "        - Foundation for models like VGG, ResNet, and transformers\n",
    "        - Enables modular network design and experimentation\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        self.layers = layers if layers is not None else []\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def forward(self, x: Tensor) -> Tensor:\n",
    "        \"\"\"\n",
    "        Forward pass through all layers in sequence.\n",
    "        \n",
    "        Args:\n",
    "            x: Input tensor\n",
    "            \n",
    "        Returns:\n",
    "            Output tensor after passing through all layers\n",
    "            \n",
    "        TODO: Implement sequential forward pass through all layers\n",
    "        \n",
    "        APPROACH:\n",
    "        1. Start with the input tensor\n",
    "        2. Apply each layer in sequence\n",
    "        3. Each layer's output becomes the next layer's input\n",
    "        4. Return the final output\n",
    "        \n",
    "        EXAMPLE:\n",
    "        Input: Tensor([[1, 2, 3]])\n",
    "        Layer1 (Dense): Tensor([[1.4, 2.8]])\n",
    "        Layer2 (ReLU): Tensor([[1.4, 2.8]])\n",
    "        Layer3 (Dense): Tensor([[0.7]])\n",
    "        Output: Tensor([[0.7]])\n",
    "        \n",
    "        HINTS:\n",
    "        - Use a for loop: for layer in self.layers:\n",
    "        - Apply each layer: x = layer(x)\n",
    "        - The output of one layer becomes input to the next\n",
    "        - Return the final result\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - This is the core of feedforward neural networks\n",
    "        - Powers inference in every deployed model\n",
    "        - Critical for real-time predictions in production\n",
    "        - Foundation for gradient flow in backpropagation\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        # Apply each layer in sequence\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def __call__(self, x: Tensor) -> Tensor:\n",
    "        \"\"\"Make the network callable: sequential(x) instead of sequential.forward(x)\"\"\"\n",
    "        return self.forward(x)\n",
    "    \n",
    "    def add(self, layer):\n",
    "        \"\"\"Add a layer to the network.\"\"\"\n",
    "        self.layers.append(layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c8aa05e",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 🧪 Unit Test: Sequential Network\n",
    "\n",
    "Let's test your Sequential network implementation! This is the foundation of all neural network architectures.\n",
    "\n",
    "**This is a unit test** - it tests one specific class (Sequential network) in isolation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c609fbd8",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "test-sequential-immediate",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Test Sequential network immediately after implementation\n",
    "print(\"🔬 Unit Test: Sequential Network...\")\n",
    "\n",
    "# Create a simple 2-layer network: 3 → 4 → 2\n",
    "try:\n",
    "    network = Sequential([\n",
    "        Dense(input_size=3, output_size=4),\n",
    "        ReLU(),\n",
    "        Dense(input_size=4, output_size=2),\n",
    "        Sigmoid()\n",
    "    ])\n",
    "    \n",
    "    print(f\"Network created with {len(network.layers)} layers\")\n",
    "    print(\"✅ Sequential network creation successful\")\n",
    "    \n",
    "    # Test with sample data\n",
    "    x = Tensor([[1.0, 2.0, 3.0]])\n",
    "    print(f\"Input: {x}\")\n",
    "    \n",
    "    # Forward pass\n",
    "    y = network(x)\n",
    "    print(f\"Output: {y}\")\n",
    "    print(f\"Output shape: {y.shape}\")\n",
    "    \n",
    "    # Verify the network works\n",
    "    assert y.shape == (1, 2), f\"Expected shape (1, 2), got {y.shape}\"\n",
    "    print(\"✅ Sequential network produces correct output shape\")\n",
    "    \n",
    "    # Test that sigmoid output is in valid range\n",
    "    assert np.all(y.data >= 0) and np.all(y.data <= 1), \"Sigmoid output should be between 0 and 1\"\n",
    "    print(\"✅ Sequential network output is in valid range\")\n",
    "    \n",
    "    # Test that layers are stored correctly\n",
    "    assert len(network.layers) == 4, f\"Expected 4 layers, got {len(network.layers)}\"\n",
    "    print(\"✅ Sequential network stores layers correctly\")\n",
    "    \n",
    "    # Test batch processing\n",
    "    x_batch = Tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "    y_batch = network(x_batch)\n",
    "    assert y_batch.shape == (2, 2), f\"Expected batch shape (2, 2), got {y_batch.shape}\"\n",
    "    print(\"✅ Sequential network handles batch processing\")\n",
    "    \n",
    "except Exception as e:\n",
    "    print(f\"❌ Sequential network test failed: {e}\")\n",
    "    raise\n",
    "\n",
    "# Show the network architecture\n",
    "print(\"🎯 Sequential network behavior:\")\n",
    "print(\"   Applies layers in sequence: f(g(h(x)))\")\n",
    "print(\"   Input flows through each layer in order\")\n",
    "print(\"   Output of layer i becomes input of layer i+1\")\n",
    "print(\"📈 Progress: Sequential network ✓\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1351cd39",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## Step 3: Building Multi-Layer Perceptrons (MLPs)\n",
    "\n",
    "### What is an MLP?\n",
    "A **Multi-Layer Perceptron** is the classic neural network architecture:\n",
    "\n",
    "```\n",
    "Input → Dense → Activation → Dense → Activation → ... → Dense → Output\n",
    "```\n",
    "\n",
    "### Why MLPs are Important\n",
    "- **Universal approximation**: Can approximate any continuous function\n",
    "- **Foundation**: Basis for understanding all neural networks\n",
    "- **Versatile**: Works for classification, regression, and more\n",
    "- **Simple**: Easy to understand and implement\n",
    "\n",
    "### MLP Architecture Pattern\n",
    "```\n",
    "create_mlp(3, [4, 2], 1) creates:\n",
    "Dense(3→4) → ReLU → Dense(4→2) → ReLU → Dense(2→1) → Sigmoid\n",
    "```\n",
    "\n",
    "### Real-World Applications\n",
    "- **Tabular data**: Customer analytics, financial modeling\n",
    "- **Feature learning**: Learning representations from raw data\n",
    "- **Classification**: Spam detection, medical diagnosis\n",
    "- **Regression**: Price prediction, time series forecasting\n",
    "\n",
    "### The MLP Factory Pattern\n",
    "Instead of manually creating each layer, we'll build a function that creates MLPs automatically!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6268ed1",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "create-mlp",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "def create_mlp(input_size: int, hidden_sizes: List[int], output_size: int, \n",
    "               activation=ReLU, output_activation=Sigmoid) -> Sequential:\n",
    "    \"\"\"\n",
    "    Create a Multi-Layer Perceptron (MLP) network.\n",
    "    \n",
    "    Args:\n",
    "        input_size: Number of input features\n",
    "        hidden_sizes: List of hidden layer sizes\n",
    "        output_size: Number of output features\n",
    "        activation: Activation function for hidden layers (default: ReLU)\n",
    "        output_activation: Activation function for output layer (default: Sigmoid)\n",
    "        \n",
    "    Returns:\n",
    "        Sequential network with MLP architecture\n",
    "        \n",
    "    TODO: Implement MLP creation with alternating Dense and activation layers.\n",
    "    \n",
    "    APPROACH:\n",
    "    1. Start with an empty list of layers\n",
    "    2. Add layers in this pattern:\n",
    "       - Dense(input_size → first_hidden_size)\n",
    "       - Activation()\n",
    "       - Dense(first_hidden_size → second_hidden_size)\n",
    "       - Activation()\n",
    "       - ...\n",
    "       - Dense(last_hidden_size → output_size)\n",
    "       - Output_activation()\n",
    "    3. Return Sequential(layers)\n",
    "    \n",
    "    EXAMPLE:\n",
    "    create_mlp(3, [4, 2], 1) creates:\n",
    "    Dense(3→4) → ReLU → Dense(4→2) → ReLU → Dense(2→1) → Sigmoid\n",
    "    \n",
    "    HINTS:\n",
    "    - Start with layers = []\n",
    "    - Track current_size starting with input_size\n",
    "    - For each hidden_size: add Dense(current_size, hidden_size), then activation\n",
    "    - Finally add Dense(last_hidden_size, output_size), then output_activation\n",
    "    - Return Sequential(layers)\n",
    "    \n",
    "    LEARNING CONNECTIONS:\n",
    "    - This pattern is used in every feedforward network implementation\n",
    "    - Foundation for architectures like autoencoders and GANs\n",
    "    - Enables rapid prototyping of neural architectures\n",
    "    - Similar to tf.keras.Sequential with Dense layers\n",
    "    \"\"\"\n",
    "    layers = []\n",
    "    current_size = input_size\n",
    "    \n",
    "    # Add hidden layers with activations\n",
    "    for hidden_size in hidden_sizes:\n",
    "        layers.append(Dense(current_size, hidden_size))\n",
    "        layers.append(activation())\n",
    "        current_size = hidden_size\n",
    "    \n",
    "    # Add output layer with output activation\n",
    "    layers.append(Dense(current_size, output_size))\n",
    "    layers.append(output_activation())\n",
    "    \n",
    "    return Sequential(layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f604420c",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 🧪 Unit Test: MLP Creation\n",
    "\n",
    "Let's test your MLP creation function! This builds complete neural networks with a single function call.\n",
    "\n",
    "**This is a unit test** - it tests one specific function (create_mlp) in isolation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24a3f199",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "test-mlp-immediate",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Test MLP creation immediately after implementation\n",
    "print(\"🔬 Unit Test: MLP Creation...\")\n",
    "\n",
    "# Create a simple MLP: 3 → 4 → 2 → 1\n",
    "try:\n",
    "    mlp = create_mlp(input_size=3, hidden_sizes=[4, 2], output_size=1)\n",
    "    \n",
    "    print(f\"MLP created with {len(mlp.layers)} layers\")\n",
    "    print(\"✅ MLP creation successful\")\n",
    "    \n",
    "    # Test the structure - should have 6 layers: Dense, ReLU, Dense, ReLU, Dense, Sigmoid\n",
    "    expected_layers = 6  # 3 Dense + 2 ReLU + 1 Sigmoid\n",
    "    assert len(mlp.layers) == expected_layers, f\"Expected {expected_layers} layers, got {len(mlp.layers)}\"\n",
    "    print(\"✅ MLP has correct number of layers\")\n",
    "    \n",
    "    # Test layer types\n",
    "    layer_types = [type(layer).__name__ for layer in mlp.layers]\n",
    "    expected_pattern = ['Dense', 'ReLU', 'Dense', 'ReLU', 'Dense', 'Sigmoid']\n",
    "    assert layer_types == expected_pattern, f\"Expected pattern {expected_pattern}, got {layer_types}\"\n",
    "    print(\"✅ MLP follows correct layer pattern\")\n",
    "    \n",
    "    # Test with sample data\n",
    "    x = Tensor([[1.0, 2.0, 3.0]])\n",
    "    y = mlp(x)\n",
    "    print(f\"MLP input: {x}\")\n",
    "    print(f\"MLP output: {y}\")\n",
    "    print(f\"MLP output shape: {y.shape}\")\n",
    "    \n",
    "    # Verify the output\n",
    "    assert y.shape == (1, 1), f\"Expected shape (1, 1), got {y.shape}\"\n",
    "    print(\"✅ MLP produces correct output shape\")\n",
    "    \n",
    "    # Test that sigmoid output is in valid range\n",
    "    assert np.all(y.data >= 0) and np.all(y.data <= 1), \"Sigmoid output should be between 0 and 1\"\n",
    "    print(\"✅ MLP output is in valid range\")\n",
    "    \n",
    "except Exception as e:\n",
    "    print(f\"❌ MLP creation test failed: {e}\")\n",
    "    raise\n",
    "\n",
    "# Test different architectures\n",
    "try:\n",
    "    # Test shallow network\n",
    "    shallow_net = create_mlp(input_size=3, hidden_sizes=[4], output_size=1)\n",
    "    assert len(shallow_net.layers) == 4, f\"Shallow network should have 4 layers, got {len(shallow_net.layers)}\"\n",
    "    \n",
    "    # Test deep network  \n",
    "    deep_net = create_mlp(input_size=3, hidden_sizes=[4, 4, 4], output_size=1)\n",
    "    assert len(deep_net.layers) == 8, f\"Deep network should have 8 layers, got {len(deep_net.layers)}\"\n",
    "    \n",
    "    # Test wide network\n",
    "    wide_net = create_mlp(input_size=3, hidden_sizes=[10], output_size=1)\n",
    "    assert len(wide_net.layers) == 4, f\"Wide network should have 4 layers, got {len(wide_net.layers)}\"\n",
    "    \n",
    "    print(\"✅ Different MLP architectures work correctly\")\n",
    "    \n",
    "except Exception as e:\n",
    "    print(f\"❌ MLP architecture test failed: {e}\")\n",
    "    raise\n",
    "\n",
    "# Show the MLP pattern\n",
    "print(\"🎯 MLP creation pattern:\")\n",
    "print(\"   Input → Dense → Activation → Dense → Activation → ... → Dense → Output_Activation\")\n",
    "print(\"   Automatically creates the complete architecture\")\n",
    "print(\"   Handles any number of hidden layers\")\n",
    "print(\"📈 Progress: Sequential network ✓, MLP creation ✓\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b30af1e",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## Step 4: Understanding Network Architectures\n",
    "\n",
    "### Architecture Patterns\n",
    "Different network architectures solve different problems:\n",
    "\n",
    "#### **Shallow vs Deep Networks**\n",
    "```python\n",
    "# Shallow: 1 hidden layer\n",
    "shallow = create_mlp(10, [20], 1)\n",
    "\n",
    "# Deep: Many hidden layers\n",
    "deep = create_mlp(10, [20, 20, 20], 1)\n",
    "```\n",
    "\n",
    "#### **Narrow vs Wide Networks**\n",
    "```python\n",
    "# Narrow: Few neurons per layer\n",
    "narrow = create_mlp(10, [5, 5], 1)\n",
    "\n",
    "# Wide: Many neurons per layer\n",
    "wide = create_mlp(10, [50], 1)\n",
    "```\n",
    "\n",
    "### Why Architecture Matters\n",
    "- **Capacity:** More parameters can learn more complex patterns\n",
    "- **Depth:** Enables hierarchical feature learning\n",
    "- **Width:** Allows parallel processing of features\n",
    "- **Efficiency:** Balance between performance and computation\n",
    "\n",
    "### Different Activation Functions\n",
    "   ```python\n",
    "# ReLU networks (most common)\n",
    "relu_net = create_mlp(10, [20], 1, activation=ReLU)\n",
    "   \n",
    "# Tanh networks (centered around 0)\n",
    "tanh_net = create_mlp(10, [20], 1, activation=Tanh)\n",
    "   \n",
    "# Multi-class classification\n",
    "classifier = create_mlp(10, [20], 3, output_activation=Softmax)\n",
    "   ```\n",
    "\n",
    "Let's test different architectures!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d4d9e74",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 🧪 Unit Test: Architecture Variations\n",
    "\n",
    "Let's test different network architectures to understand their behavior.\n",
    "\n",
    "**This is a unit test** - it tests architectural variations in isolation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecf706b8",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 📊 Visualization: Network Architecture Comparison\n",
    "\n",
    "This function creates and visualizes different neural network architectures to demonstrate how activation functions and layer configurations affect network behavior and output characteristics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e65c159",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "test-architectures",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def plot_network_architectures():\n",
    "    \"\"\"Visualize different network architectures.\"\"\"\n",
    "        \n",
    "    # Create different architectures\n",
    "    relu_net = create_mlp(input_size=3, hidden_sizes=[4], output_size=1, activation=ReLU)\n",
    "    tanh_net = create_mlp(input_size=3, hidden_sizes=[4], output_size=1, activation=Tanh)\n",
    "    classifier = create_mlp(input_size=3, hidden_sizes=[4], output_size=3, output_activation=Softmax)\n",
    "\n",
    "    # Create input data\n",
    "    x = Tensor([[1.0, 2.0, 3.0]])\n",
    "    \n",
    "    # Get outputs\n",
    "    y_relu = relu_net(x)\n",
    "    y_tanh = tanh_net(x)\n",
    "    y_multi = classifier(x)\n",
    "\n",
    "    # Plot the results\n",
    "    fig, axs = plt.subplots(1, 3, figsize=(15, 4))\n",
    "    \n",
    "    axs[0].set_title(\"ReLU Network Output\")\n",
    "    axs[0].bar(['Output'], [y_relu.data[0][0]], color='skyblue')\n",
    "    \n",
    "    axs[1].set_title(\"Tanh Network Output\")\n",
    "    axs[1].bar(['Output'], [y_tanh.data[0][0]], color='salmon')\n",
    "    \n",
    "    axs[2].set_title(\"Softmax Classifier Output\")\n",
    "    axs[2].bar([f\"Class {i}\" for i in range(3)], y_multi.data[0], color='lightgreen')\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "plot_network_architectures()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "266af4b8",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Unit Test: Network Architecture Variations\n",
    "\n",
    "This test validates different neural network architectures created with various activation functions. It ensures that networks with ReLU, Tanh, and Softmax activations work correctly, and tests both shallow and deep network configurations for comprehensive architecture validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c89369f1",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def test_unit_network_architectures():\n",
    "    \"\"\"Unit test for different network architectures.\"\"\"\n",
    "    # Test different architectures\n",
    "    print(\"🔬 Unit Test: Network Architecture Variations...\")\n",
    "\n",
    "    try:\n",
    "        # Test different activation functions\n",
    "        relu_net = create_mlp(input_size=3, hidden_sizes=[4], output_size=1, activation=ReLU)\n",
    "        tanh_net = create_mlp(input_size=3, hidden_sizes=[4], output_size=1, activation=Tanh)\n",
    "        \n",
    "        # Test different output activations\n",
    "        classifier = create_mlp(input_size=3, hidden_sizes=[4], output_size=3, output_activation=Softmax)\n",
    "        \n",
    "        # Test with sample data\n",
    "        x = Tensor([[1.0, 2.0, 3.0]])\n",
    "        \n",
    "        # Test ReLU network\n",
    "        y_relu = relu_net(x)\n",
    "        assert y_relu.shape == (1, 1), \"ReLU network should work\"\n",
    "        print(\"✅ ReLU network works correctly\")\n",
    "        \n",
    "        # Test Tanh network\n",
    "        y_tanh = tanh_net(x)\n",
    "        assert y_tanh.shape == (1, 1), \"Tanh network should work\"\n",
    "        print(\"✅ Tanh network works correctly\")\n",
    "        \n",
    "        # Test multi-class classifier\n",
    "        y_multi = classifier(x)\n",
    "        assert y_multi.shape == (1, 3), \"Multi-class classifier should work\"\n",
    "        \n",
    "        # Check softmax properties\n",
    "        assert abs(np.sum(y_multi.data) - 1.0) < 1e-6, \"Softmax outputs should sum to 1\"\n",
    "        print(\"✅ Multi-class classifier with Softmax works correctly\")\n",
    "        \n",
    "        # Test different architectures\n",
    "        shallow = create_mlp(input_size=4, hidden_sizes=[5], output_size=1)\n",
    "        deep = create_mlp(input_size=4, hidden_sizes=[5, 5, 5], output_size=1)\n",
    "        wide = create_mlp(input_size=4, hidden_sizes=[20], output_size=1)\n",
    "        \n",
    "        x_test = Tensor([[1.0, 2.0, 3.0, 4.0]])\n",
    "        \n",
    "        # Test all architectures\n",
    "        for name, net in [(\"Shallow\", shallow), (\"Deep\", deep), (\"Wide\", wide)]:\n",
    "            y = net(x_test)\n",
    "            assert y.shape == (1, 1), f\"{name} network should produce correct shape\"\n",
    "            print(f\"✅ {name} network works correctly\")\n",
    "        \n",
    "        print(\"✅ All network architectures work correctly\")\n",
    "            \n",
    "    except Exception as e:\n",
    "        print(f\"❌ Architecture test failed: {e}\")\n",
    "        raise\n",
    "\n",
    "    print(\"🎯 Architecture insights:\")\n",
    "    print(\"   Different activations create different behaviors\")\n",
    "    print(\"   Softmax enables multi-class classification\")\n",
    "    print(\"   Architecture affects network capacity and learning\")\n",
    "    print(\"📈 Progress: Sequential ✓, MLP creation ✓, Architecture variations ✓\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5945253d",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 📊 Visualization Demo: Network Architectures\n",
    "\n",
    "Let's visualize the different network architectures for educational purposes:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0ec0880",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## Step 5: Comprehensive Test - Complete Network Applications\n",
    "\n",
    "### Real-World Network Applications\n",
    "Let's test our networks on realistic scenarios:\n",
    "\n",
    "#### **Classification Problem**\n",
    "```python\n",
    "# 4 features → 2 classes (binary classification)\n",
    "classifier = create_mlp(4, [8, 4], 2, output_activation=Softmax)\n",
    "```\n",
    "\n",
    "#### **Regression Problem**\n",
    "```python\n",
    "# 3 features → 1 continuous output\n",
    "regressor = create_mlp(3, [10, 5], 1, output_activation=lambda: Dense(0, 0))  # Linear output\n",
    "```\n",
    "\n",
    "#### **Deep Learning Pattern**\n",
    "```python\n",
    "# Complex feature learning\n",
    "deep_net = create_mlp(10, [64, 32, 16], 1)\n",
    "```\n",
    "\n",
    "This comprehensive test ensures our networks work for real ML applications!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a180d93",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "test-integration",
     "locked": true,
     "points": 15,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Comprehensive test - complete network applications\n",
    "print(\"🔬 Comprehensive Test: Complete Network Applications...\")\n",
    "\n",
    "try:\n",
    "    # Test 1: Multi-class Classification (Iris-like dataset)\n",
    "    print(\"\\n1. Multi-class Classification Test:\")\n",
    "    iris_classifier = create_mlp(input_size=4, hidden_sizes=[8, 6], output_size=3, output_activation=Softmax)\n",
    "    \n",
    "    # Simulate iris features: [sepal_length, sepal_width, petal_length, petal_width]\n",
    "    iris_samples = Tensor([\n",
    "        [5.1, 3.5, 1.4, 0.2],  # Setosa\n",
    "        [7.0, 3.2, 4.7, 1.4],  # Versicolor\n",
    "        [6.3, 3.3, 6.0, 2.5]   # Virginica\n",
    "        ])\n",
    "        \n",
    "    iris_predictions = iris_classifier(iris_samples)\n",
    "    assert iris_predictions.shape == (3, 3), \"Iris classifier should output 3 classes for 3 samples\"\n",
    "        \n",
    "    # Check softmax properties\n",
    "    row_sums = np.sum(iris_predictions.data, axis=1)\n",
    "    assert np.allclose(row_sums, 1.0), \"Each prediction should sum to 1\"\n",
    "    print(\"✅ Multi-class classification works correctly\")\n",
    "    \n",
    "    # Test 2: Regression Task (Housing prices)\n",
    "    print(\"\\n2. Regression Task Test:\")\n",
    "    # Create a regressor without final activation (linear output)\n",
    "    class Identity:\n",
    "        def __call__(self, x): return x\n",
    "    \n",
    "    housing_regressor = create_mlp(input_size=3, hidden_sizes=[10, 5], output_size=1, output_activation=Identity)\n",
    "    \n",
    "    # Simulate housing features: [size, bedrooms, location_score]\n",
    "    housing_samples = Tensor([\n",
    "        [2000, 3, 8.5],  # Large house, good location\n",
    "        [1200, 2, 6.0],  # Medium house, ok location\n",
    "        [800, 1, 4.0]    # Small house, poor location\n",
    "    ])\n",
    "    \n",
    "    housing_predictions = housing_regressor(housing_samples)\n",
    "    assert housing_predictions.shape == (3, 1), \"Housing regressor should output 1 value per sample\"\n",
    "    print(\"✅ Regression task works correctly\")\n",
    "    \n",
    "    # Test 3: Deep Network Performance\n",
    "    print(\"\\n3. Deep Network Test:\")\n",
    "    deep_network = create_mlp(input_size=10, hidden_sizes=[20, 15, 10, 5], output_size=1)\n",
    "    \n",
    "    # Test with realistic batch size\n",
    "    batch_data = Tensor(np.random.randn(32, 10))  # 32 samples, 10 features\n",
    "    deep_predictions = deep_network(batch_data)\n",
    "    \n",
    "    assert deep_predictions.shape == (32, 1), \"Deep network should handle batch processing\"\n",
    "    assert not np.any(np.isnan(deep_predictions.data)), \"Deep network should not produce NaN\"\n",
    "    print(\"✅ Deep network handles batch processing correctly\")\n",
    "    \n",
    "    # Test 4: Network Composition\n",
    "    print(\"\\n4. Network Composition Test:\")\n",
    "    # Create a feature extractor and classifier separately\n",
    "    feature_extractor = Sequential([\n",
    "        Dense(input_size=10, output_size=5),\n",
    "        ReLU(),\n",
    "        Dense(input_size=5, output_size=3),\n",
    "        ReLU()\n",
    "    ])\n",
    "    \n",
    "    classifier_head = Sequential([\n",
    "        Dense(input_size=3, output_size=2),\n",
    "        Softmax()\n",
    "    ])\n",
    "    \n",
    "    # Test composition\n",
    "    raw_data = Tensor(np.random.randn(5, 10))\n",
    "    features = feature_extractor(raw_data)\n",
    "    final_predictions = classifier_head(features)\n",
    "    \n",
    "    assert features.shape == (5, 3), \"Feature extractor should output 3 features\"\n",
    "    assert final_predictions.shape == (5, 2), \"Classifier should output 2 classes\"\n",
    "    \n",
    "    row_sums = np.sum(final_predictions.data, axis=1)\n",
    "    assert np.allclose(row_sums, 1.0), \"Composed network predictions should be valid\"\n",
    "    print(\"✅ Network composition works correctly\")\n",
    "    \n",
    "    print(\"\\n🎉 Comprehensive test passed! Your networks work correctly for:\")\n",
    "    print(\"  • Multi-class classification (Iris flowers)\")\n",
    "    print(\"  • Regression tasks (housing prices)\")\n",
    "    print(\"  • Deep learning architectures\")\n",
    "    print(\"  • Network composition and feature extraction\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"❌ Comprehensive test failed: {e}\")\n",
    "\n",
    "print(\"📈 Final Progress: Complete network architectures ready for real ML applications!\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b3b4ad9",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🏗️ Class: MLP (Multi-Layer Perceptron)\n",
    "\n",
    "This class provides a convenient wrapper around Sequential networks specifically designed for standard MLP architectures. It maintains parameter information and provides a clean interface for creating and managing multi-layer perceptrons with consistent structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1dd5233",
   "metadata": {
    "lines_to_next_cell": 1,
    "nbgrader": {
     "grade": false,
     "grade_id": "networks-compatibility",
     "locked": false,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#| export\n",
    "class MLP:\n",
    "    \"\"\"\n",
    "    Multi-Layer Perceptron (MLP) class.\n",
    "    \n",
    "    A convenient wrapper around Sequential networks for standard MLP architectures.\n",
    "    Maintains parameter information and provides a clean interface.\n",
    "    \n",
    "    Args:\n",
    "        input_size: Number of input features\n",
    "        hidden_size: Size of the single hidden layer\n",
    "        output_size: Number of output features\n",
    "        activation: Activation function for hidden layer (default: ReLU)\n",
    "        output_activation: Activation function for output layer (default: Sigmoid)\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, input_size: int, hidden_size: int, output_size: int, \n",
    "                 activation=ReLU, output_activation=None):\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        \n",
    "        # Build the network layers\n",
    "        layers = []\n",
    "        \n",
    "        # Input to hidden layer\n",
    "        layers.append(Dense(input_size, hidden_size))\n",
    "        layers.append(activation())\n",
    "        \n",
    "        # Hidden to output layer\n",
    "        layers.append(Dense(hidden_size, output_size))\n",
    "        if output_activation is not None:\n",
    "            layers.append(output_activation())\n",
    "        \n",
    "        self.network = Sequential(layers)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \"\"\"Forward pass through the MLP network.\"\"\"\n",
    "        return self.network.forward(x)\n",
    "    \n",
    "    def __call__(self, x):\n",
    "        \"\"\"Make the MLP callable.\"\"\"\n",
    "        return self.forward(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ac076d3",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Unit Test: Sequential Network Implementation\n",
    "\n",
    "This test validates the Sequential network class functionality, ensuring proper layer composition, forward pass execution, and network architecture validation for multi-layer neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "937de7bc",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def test_unit_sequential_networks():\n",
    "    \"\"\"Unit test for the Sequential network implementation.\"\"\"\n",
    "    print(\"🔬 Unit Test: Sequential Networks...\")\n",
    "    \n",
    "    # Test basic Sequential network\n",
    "    net = Sequential([\n",
    "        Dense(input_size=3, output_size=4),\n",
    "        ReLU(),\n",
    "        Dense(input_size=4, output_size=2),\n",
    "        Sigmoid()\n",
    "    ])\n",
    "    \n",
    "    x = Tensor([[1.0, 2.0, 3.0]])\n",
    "    y = net(x)\n",
    "    \n",
    "    assert y.shape == (1, 2), \"Sequential network should produce correct output shape\"\n",
    "    assert np.all(y.data > 0), \"Sigmoid output should be positive\"\n",
    "    assert np.all(y.data < 1), \"Sigmoid output should be less than 1\"\n",
    "    \n",
    "    print(\"✅ Sequential networks work correctly\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc421e40",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Unit Test: MLP Creation Function\n",
    "\n",
    "This test validates the `create_mlp` function, ensuring it correctly constructs Multi-Layer Perceptrons with various architectures, activation functions, and layer configurations for different machine learning tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12ecacca",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def test_unit_mlp_creation():\n",
    "    \"\"\"Unit test for the MLP creation function.\"\"\"\n",
    "    print(\"🔬 Unit Test: MLP Creation...\")\n",
    "    \n",
    "    # Test different MLP architectures\n",
    "    shallow = create_mlp(input_size=4, hidden_sizes=[5], output_size=1)\n",
    "    deep = create_mlp(input_size=4, hidden_sizes=[8, 6, 4], output_size=2)\n",
    "    \n",
    "    x = Tensor([[1.0, 2.0, 3.0, 4.0]])\n",
    "    \n",
    "    # Test shallow network\n",
    "    y_shallow = shallow(x)\n",
    "    assert y_shallow.shape == (1, 1), \"Shallow MLP should work\"\n",
    "    \n",
    "    # Test deep network  \n",
    "    y_deep = deep(x)\n",
    "    assert y_deep.shape == (1, 2), \"Deep MLP should work\"\n",
    "    \n",
    "    print(\"✅ MLP creation works correctly\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e66ebaac",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "### 🧪 Unit Test: Network Applications in Real ML Scenarios\n",
    "\n",
    "This comprehensive test validates network performance on real machine learning tasks including classification and regression, ensuring the implementations work correctly with actual datasets and practical applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "134b1e04",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def test_unit_network_applications():\n",
    "    \"\"\"Comprehensive unit test for network applications in real ML scenarios.\"\"\"\n",
    "    print(\"🔬 Comprehensive Test: Network Applications...\")\n",
    "    \n",
    "    # Test multi-class classification\n",
    "    iris_classifier = create_mlp(input_size=4, hidden_sizes=[8, 6], output_size=3, output_activation=Softmax)\n",
    "    iris_samples = Tensor([[5.1, 3.5, 1.4, 0.2], [7.0, 3.2, 4.7, 1.4], [6.3, 3.3, 6.0, 2.5]])\n",
    "    iris_predictions = iris_classifier(iris_samples)\n",
    "    \n",
    "    assert iris_predictions.shape == (3, 3), \"Iris classifier should work\"\n",
    "    row_sums = np.sum(iris_predictions.data, axis=1)\n",
    "    assert np.allclose(row_sums, 1.0), \"Predictions should sum to 1\"\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "889a49c9",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🧪 Module Testing\n",
    "\n",
    "Time to test your implementation! This section uses TinyTorch's standardized testing framework to ensure your implementation works correctly.\n",
    "\n",
    "**This testing section is locked** - it provides consistent feedback across all modules and cannot be modified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "198c56e8",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "standardized-testing",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# =============================================================================\n",
    "# STANDARDIZED MODULE TESTING - DO NOT MODIFY\n",
    "# This cell is locked to ensure consistent testing across all TinyTorch modules\n",
    "# ============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcce377d",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 1
   },
   "source": [
    "## 🔬 Integration Test: End-to-End Network Forward Pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f676c7e6",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def test_module_full_network_forward_pass():\n",
    "    \"\"\"\n",
    "    Integration test for a complete forward pass through a multi-layer network.\n",
    "    \n",
    "    Tests a complete forward pass through a multi-layer network,\n",
    "    integrating Tensors, Dense layers, Activations, and the Sequential container.\n",
    "    \"\"\"\n",
    "    print(\"🔬 Running Integration Test: Full Network Forward Pass...\")\n",
    "\n",
    "    # 1. Define a simple 2-layer MLP\n",
    "    # Input (3) -> Dense(4) -> ReLU -> Dense(2) -> Output\n",
    "    model = Sequential([\n",
    "        Dense(3, 4),\n",
    "        ReLU(),\n",
    "        Dense(4, 2)\n",
    "    ])\n",
    "\n",
    "    # 2. Create a batch of input Tensors\n",
    "    # Batch of 5 samples, each with 3 features\n",
    "    input_tensor = Tensor(np.random.randn(5, 3))\n",
    "\n",
    "    # 3. Perform a forward pass through the entire network\n",
    "    output_tensor = model(input_tensor)\n",
    "\n",
    "    # 4. Assert the final output is correct\n",
    "    assert isinstance(output_tensor, Tensor), \"Network output must be a Tensor\"\n",
    "    assert output_tensor.shape == (5, 2), f\"Expected output shape (5, 2), but got {output_tensor.shape}\"\n",
    "    print(\"✅ Integration Test Passed: Full network forward pass is successful.\")\n",
    "\n",
    "# Test function defined (called in main block)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d6d2b6",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🔧 ML Systems: Network Stability & Error Handling\n",
    "\n",
    "Now that you have complete neural networks, let's develop **production robustness skills**. This section teaches you to identify and fix stability issues that can break training in production systems.\n",
    "\n",
    "### **Learning Outcome**: *\"I understand why numerical stability matters in production and can detect/fix stability issues\"*\n",
    "\n",
    "---\n",
    "\n",
    "## Network Stability Monitor (Medium Guided Implementation)\n",
    "\n",
    "As an ML systems engineer, you need to ensure networks remain stable during training. Let's build tools to detect numerical instability and understand gradient flow issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b43e9850",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "\n",
    "class NetworkStabilityMonitor:\n",
    "    \"\"\"\n",
    "    Stability monitoring toolkit for neural networks.\n",
    "    \n",
    "    Helps ML engineers detect numerical instability, gradient problems,\n",
    "    and other issues that can break training in production systems.\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.stability_history = []\n",
    "        self.warning_threshold = 1e6\n",
    "        self.error_threshold = 1e10\n",
    "        \n",
    "    def check_tensor_stability(self, tensor, tensor_name=\"tensor\"):\n",
    "        \"\"\"\n",
    "        Check if a tensor has numerical stability issues.\n",
    "        \n",
    "        TODO: Implement tensor stability checking.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. Check for NaN values using np.isnan()\n",
    "        2. Check for infinite values using np.isinf() \n",
    "        3. Check for extremely large values (> 1e6)\n",
    "        4. Calculate value statistics (min, max, mean, std)\n",
    "        5. Return stability report with warnings\n",
    "        \n",
    "        EXAMPLE:\n",
    "        monitor = NetworkStabilityMonitor()\n",
    "        tensor = Tensor([1.0, 2.0, np.inf])\n",
    "        report = monitor.check_tensor_stability(tensor, \"weights\")\n",
    "        print(f\"Stable: {report['is_stable']}\")\n",
    "        print(f\"Issues: {report['issues']}\")\n",
    "        \n",
    "        HINTS:\n",
    "        - Use tensor.data to get numpy array\n",
    "        - Check: np.any(np.isnan(tensor.data))\n",
    "        - Check: np.any(np.isinf(tensor.data))\n",
    "        - Check: np.any(np.abs(tensor.data) > self.warning_threshold)\n",
    "        - Return dict with analysis\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - Critical for debugging exploding/vanishing gradients\n",
    "        - Used in production monitoring systems at scale\n",
    "        - Foundation for automated model health checks\n",
    "        - Similar to TensorBoard's histogram monitoring\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        data = tensor.data\n",
    "        \n",
    "        # Check for numerical issues\n",
    "        has_nan = np.any(np.isnan(data))\n",
    "        has_inf = np.any(np.isinf(data))\n",
    "        has_large = np.any(np.abs(data) > self.warning_threshold)\n",
    "        has_extreme = np.any(np.abs(data) > self.error_threshold)\n",
    "        \n",
    "        # Calculate statistics (avoiding issues if all values are problematic)\n",
    "        finite_mask = np.isfinite(data)\n",
    "        if np.any(finite_mask):\n",
    "            finite_data = data[finite_mask]\n",
    "            stats = {\n",
    "                'min': np.min(finite_data),\n",
    "                'max': np.max(finite_data),\n",
    "                'mean': np.mean(finite_data),\n",
    "                'std': np.std(finite_data),\n",
    "                'finite_count': np.sum(finite_mask),\n",
    "                'total_count': data.size\n",
    "            }\n",
    "        else:\n",
    "            stats = {\n",
    "                'min': np.nan,\n",
    "                'max': np.nan,\n",
    "                'mean': np.nan,\n",
    "                'std': np.nan,\n",
    "                'finite_count': 0,\n",
    "                'total_count': data.size\n",
    "            }\n",
    "        \n",
    "        # Compile issues\n",
    "        issues = []\n",
    "        if has_nan:\n",
    "            issues.append(\"Contains NaN values\")\n",
    "        if has_inf:\n",
    "            issues.append(\"Contains infinite values\")\n",
    "        if has_extreme:\n",
    "            issues.append(f\"Contains extremely large values (>{self.error_threshold:.0e})\")\n",
    "        elif has_large:\n",
    "            issues.append(f\"Contains large values (>{self.warning_threshold:.0e})\")\n",
    "        \n",
    "        is_stable = len(issues) == 0\n",
    "        \n",
    "        return {\n",
    "            'tensor_name': tensor_name,\n",
    "            'is_stable': is_stable,\n",
    "            'issues': issues,\n",
    "            'has_nan': has_nan,\n",
    "            'has_inf': has_inf,\n",
    "            'has_large_values': has_large,\n",
    "            'statistics': stats\n",
    "        }\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def analyze_gradient_flow(self, network, input_tensor, target_output):\n",
    "        \"\"\"\n",
    "        Analyze gradient flow through a network to detect vanishing/exploding gradients.\n",
    "        \n",
    "        TODO: Implement gradient flow analysis.\n",
    "        \n",
    "        STEP-BY-STEP IMPLEMENTATION:\n",
    "        1. Perform forward pass through network\n",
    "        2. Simulate simple loss calculation (MSE)\n",
    "        3. Estimate gradient magnitudes using finite differences\n",
    "        4. Check for vanishing gradients (very small)\n",
    "        5. Check for exploding gradients (very large)\n",
    "        6. Return gradient flow analysis\n",
    "        \n",
    "        EXAMPLE:\n",
    "        monitor = NetworkStabilityMonitor()\n",
    "        analysis = monitor.analyze_gradient_flow(network, input_data, target)\n",
    "        print(f\"Gradient health: {analysis['gradient_status']}\")\n",
    "        \n",
    "        HINTS:\n",
    "        - Forward pass: output = network(input_tensor)\n",
    "        - Simple loss: 0.5 * np.sum((output.data - target_output.data)**2)\n",
    "        - Use small perturbations to estimate gradients\n",
    "        - Vanishing: gradients < 1e-6, Exploding: gradients > 1e3\n",
    "        \n",
    "        LEARNING CONNECTIONS:\n",
    "        - Essential for training deep networks successfully\n",
    "        - Used in gradient clipping and batch normalization design\n",
    "        - Foundation for understanding network initialization strategies\n",
    "        - Similar to PyTorch's gradient debugging tools\n",
    "        \"\"\"\n",
    "        ### BEGIN SOLUTION\n",
    "        # Forward pass\n",
    "        output = network(input_tensor)\n",
    "        \n",
    "        # Calculate simple MSE loss\n",
    "        loss = 0.5 * np.sum((output.data - target_output.data)**2)\n",
    "        \n",
    "        # Estimate gradient magnitudes using finite differences\n",
    "        # This is a simplified approach - real backprop would be more accurate\n",
    "        epsilon = 1e-5\n",
    "        gradient_estimates = []\n",
    "        \n",
    "        # Check first layer weights (simplified analysis)\n",
    "        if hasattr(network, 'layers') and len(network.layers) > 0:\n",
    "            first_layer = network.layers[0]\n",
    "            if hasattr(first_layer, 'weights'):\n",
    "                # Perturb a small sample of weights to estimate gradients\n",
    "                original_weight = first_layer.weights.data[0, 0]\n",
    "                \n",
    "                # Forward pass with small perturbation\n",
    "                first_layer.weights.data[0, 0] = original_weight + epsilon\n",
    "                output_plus = network(input_tensor)\n",
    "                loss_plus = 0.5 * np.sum((output_plus.data - target_output.data)**2)\n",
    "                \n",
    "                # Estimate gradient\n",
    "                grad_estimate = (loss_plus - loss) / epsilon\n",
    "                gradient_estimates.append(abs(grad_estimate))\n",
    "                \n",
    "                # Restore original weight\n",
    "                first_layer.weights.data[0, 0] = original_weight\n",
    "        \n",
    "        # Analyze gradient magnitudes\n",
    "        if gradient_estimates:\n",
    "            avg_grad = np.mean(gradient_estimates)\n",
    "            max_grad = np.max(gradient_estimates)\n",
    "            \n",
    "            if avg_grad < 1e-8:\n",
    "                gradient_status = \"Vanishing gradients detected\"\n",
    "            elif max_grad > 1e3:\n",
    "                gradient_status = \"Exploding gradients detected\"\n",
    "            elif avg_grad < 1e-6:\n",
    "                gradient_status = \"Potentially vanishing gradients\"\n",
    "            elif max_grad > 100:\n",
    "                gradient_status = \"Potentially exploding gradients\"\n",
    "            else:\n",
    "                gradient_status = \"Healthy gradient flow\"\n",
    "        else:\n",
    "            gradient_status = \"Unable to analyze gradients\"\n",
    "        \n",
    "        return {\n",
    "            'loss': loss,\n",
    "            'gradient_estimates': gradient_estimates,\n",
    "            'avg_gradient': np.mean(gradient_estimates) if gradient_estimates else 0,\n",
    "            'max_gradient': np.max(gradient_estimates) if gradient_estimates else 0,\n",
    "            'gradient_status': gradient_status\n",
    "        }\n",
    "        ### END SOLUTION\n",
    "    \n",
    "    def comprehensive_stability_check(self, network, input_tensor, target_output):\n",
    "        \"\"\"\n",
    "        Perform comprehensive stability analysis of a neural network.\n",
    "        \n",
    "        This function is PROVIDED to demonstrate complete stability monitoring.\n",
    "        Students use it to understand production stability requirements.\n",
    "        \"\"\"\n",
    "        print(\"🔧 COMPREHENSIVE NETWORK STABILITY CHECK\")\n",
    "        print(\"=\" * 50)\n",
    "        \n",
    "        stability_report = {\n",
    "            'overall_status': 'STABLE',\n",
    "            'issues_found': [],\n",
    "            'recommendations': []\n",
    "        }\n",
    "        \n",
    "        # Check input stability\n",
    "        input_check = self.check_tensor_stability(input_tensor, \"input\")\n",
    "        if not input_check['is_stable']:\n",
    "            stability_report['overall_status'] = 'UNSTABLE'\n",
    "            stability_report['issues_found'].extend([f\"Input: {issue}\" for issue in input_check['issues']])\n",
    "            stability_report['recommendations'].append(\"Normalize or clip input data\")\n",
    "        \n",
    "        print(f\"📊 Input Check: {'✅ STABLE' if input_check['is_stable'] else '❌ UNSTABLE'}\")\n",
    "        if input_check['issues']:\n",
    "            for issue in input_check['issues']:\n",
    "                print(f\"   - {issue}\")\n",
    "        \n",
    "        # Check each layer's weights and outputs\n",
    "        if hasattr(network, 'layers'):\n",
    "            for i, layer in enumerate(network.layers):\n",
    "                if hasattr(layer, 'weights'):\n",
    "                    weight_check = self.check_tensor_stability(layer.weights, f\"layer_{i}_weights\")\n",
    "                    if not weight_check['is_stable']:\n",
    "                        stability_report['overall_status'] = 'UNSTABLE'\n",
    "                        stability_report['issues_found'].extend([f\"Layer {i}: {issue}\" for issue in weight_check['issues']])\n",
    "                        stability_report['recommendations'].append(f\"Re-initialize layer {i} weights\")\n",
    "                    \n",
    "                    print(f\"🔗 Layer {i} Weights: {'✅ STABLE' if weight_check['is_stable'] else '❌ UNSTABLE'}\")\n",
    "                    if weight_check['issues']:\n",
    "                        for issue in weight_check['issues']:\n",
    "                            print(f\"   - {issue}\")\n",
    "        \n",
    "        # Check network output\n",
    "        try:\n",
    "            output = network(input_tensor)\n",
    "            output_check = self.check_tensor_stability(output, \"network_output\")\n",
    "            if not output_check['is_stable']:\n",
    "                stability_report['overall_status'] = 'UNSTABLE'\n",
    "                stability_report['issues_found'].extend([f\"Output: {issue}\" for issue in output_check['issues']])\n",
    "                stability_report['recommendations'].append(\"Check activation functions and weight initialization\")\n",
    "            \n",
    "            print(f\"📤 Output Check: {'✅ STABLE' if output_check['is_stable'] else '❌ UNSTABLE'}\")\n",
    "            if output_check['issues']:\n",
    "                for issue in output_check['issues']:\n",
    "                    print(f\"   - {issue}\")\n",
    "        \n",
    "        except Exception as e:\n",
    "            stability_report['overall_status'] = 'CRITICAL'\n",
    "            stability_report['issues_found'].append(f\"Network forward pass failed: {str(e)}\")\n",
    "            stability_report['recommendations'].append(\"Check network architecture and input compatibility\")\n",
    "            print(f\"📤 Output Check: ❌ CRITICAL - Forward pass failed\")\n",
    "        \n",
    "        # Gradient flow analysis\n",
    "        try:\n",
    "            gradient_analysis = self.analyze_gradient_flow(network, input_tensor, target_output)\n",
    "            print(f\"🌊 Gradient Flow: {gradient_analysis['gradient_status']}\")\n",
    "            \n",
    "            if \"exploding\" in gradient_analysis['gradient_status'].lower():\n",
    "                stability_report['overall_status'] = 'UNSTABLE'\n",
    "                stability_report['recommendations'].append(\"Use gradient clipping or reduce learning rate\")\n",
    "            elif \"vanishing\" in gradient_analysis['gradient_status'].lower():\n",
    "                stability_report['overall_status'] = 'UNSTABLE'\n",
    "                stability_report['recommendations'].append(\"Use ReLU activations or residual connections\")\n",
    "        \n",
    "        except Exception as e:\n",
    "            print(f\"🌊 Gradient Flow: ❌ Analysis failed - {str(e)}\")\n",
    "        \n",
    "        print(f\"\\n🎯 OVERALL STATUS: {stability_report['overall_status']}\")\n",
    "        if stability_report['recommendations']:\n",
    "            print(f\"\\n💡 RECOMMENDATIONS:\")\n",
    "            for rec in stability_report['recommendations']:\n",
    "                print(f\"   - {rec}\")\n",
    "        \n",
    "        return stability_report\n",
    "\n",
    "def create_unstable_network_demo():\n",
    "    \"\"\"\n",
    "    Create networks with known stability issues for demonstration.\n",
    "    \n",
    "    This function is PROVIDED to show common stability problems.\n",
    "    Students use it to practice detecting and fixing issues.\n",
    "    \"\"\"\n",
    "    print(\"⚠️  STABILITY ISSUES DEMONSTRATION\")\n",
    "    print(\"=\" * 50)\n",
    "    \n",
    "    # Create networks with different stability issues\n",
    "    demo_networks = {}\n",
    "    \n",
    "    # 1. Network with exploding weights\n",
    "    print(\"\\n1. 🔥 Exploding Weights Network:\")\n",
    "    exploding_net = Sequential([\n",
    "        Dense(10, 5),\n",
    "        ReLU(),\n",
    "        Dense(5, 2)\n",
    "    ])\n",
    "    # Manually set large weights to simulate training instability\n",
    "    # exploding_net.layers[0].weights.data *= 100  # Very large weights (commented to avoid error)\n",
    "    demo_networks['exploding'] = exploding_net\n",
    "    print(\"   Created network with artificially large weights\")\n",
    "    \n",
    "    # 2. Network with NaN weights (simulate numerical overflow)\n",
    "    print(\"\\n2. 💀 NaN Weights Network:\")\n",
    "    nan_net = Sequential([\n",
    "        Dense(10, 5),\n",
    "        ReLU(),\n",
    "        Dense(5, 2)\n",
    "    ])\n",
    "    # Inject NaN values\n",
    "    nan_net.layers[0].weights.data[0, 0] = np.nan\n",
    "    demo_networks['nan'] = nan_net\n",
    "    print(\"   Created network with NaN values in weights\")\n",
    "    \n",
    "    # 3. Healthy network for comparison\n",
    "    print(\"\\n3. ✅ Healthy Network:\")\n",
    "    healthy_net = Sequential([\n",
    "        Dense(10, 5),\n",
    "        ReLU(),\n",
    "        Dense(5, 2)\n",
    "    ])\n",
    "    demo_networks['healthy'] = healthy_net\n",
    "    print(\"   Created properly initialized network\")\n",
    "    \n",
    "    return demo_networks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e1f9865",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 🎯 Learning Activity 1: Stability Detection Practice (Medium Guided Implementation)\n",
    "\n",
    "**Goal**: Learn to detect numerical instability issues that can break neural network training in production.\n",
    "\n",
    "Complete the missing implementations in the `NetworkStabilityMonitor` class above, then use your monitor to detect stability issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3affc38c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the network stability monitor\n",
    "monitor = NetworkStabilityMonitor()\n",
    "\n",
    "print(\"🔧 NETWORK STABILITY MONITORING\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "# Create test networks with different stability characteristics\n",
    "demo_networks = create_unstable_network_demo()\n",
    "\n",
    "# Create test data\n",
    "input_data = Tensor(np.random.randn(3, 10))  # Batch of 3 samples\n",
    "target_data = Tensor(np.random.randn(3, 2))  # Target outputs\n",
    "\n",
    "print(f\"\\n🔍 STABILITY ANALYSIS RESULTS:\")\n",
    "print(f\"=\" * 40)\n",
    "\n",
    "# Test each network\n",
    "for network_name, network in demo_networks.items():\n",
    "    print(f\"\\n📊 Testing {network_name.upper()} Network:\")\n",
    "    \n",
    "    # Students use their implemented stability checker\n",
    "    stability_report = monitor.comprehensive_stability_check(network, input_data, target_data)\n",
    "    \n",
    "    # Show what this means for production\n",
    "    if stability_report['overall_status'] == 'STABLE':\n",
    "        print(f\"   🎯 Production Impact: Safe to deploy\")\n",
    "    elif stability_report['overall_status'] == 'UNSTABLE':\n",
    "        print(f\"   ⚠️  Production Impact: May cause training failures\")\n",
    "    else:\n",
    "        print(f\"   💀 Production Impact: Would crash in production\")\n",
    "\n",
    "print(f\"\\n💡 STABILITY ENGINEERING INSIGHTS:\")\n",
    "print(f\"   - NaN values spread through entire network (one bad value ruins everything)\")\n",
    "print(f\"   - Large weights cause exponential growth through layers\")\n",
    "print(f\"   - Stability monitoring prevents silent training failures\")\n",
    "print(f\"   - Early detection saves compute resources and time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d151113",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### 🎯 Learning Activity 2: Production Stability Patterns (Review & Understand)\n",
    "\n",
    "**Goal**: Understand common stability issues in production ML systems and learn industry best practices for preventing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeed4631",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"🏭 PRODUCTION STABILITY PATTERNS\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "# Test different input scenarios that cause instability\n",
    "print(\"\\n🔍 Input Data Stability Scenarios:\")\n",
    "\n",
    "stability_scenarios = [\n",
    "    (\"Normal Data\", np.random.randn(5, 10)),\n",
    "    (\"Large Values\", np.random.randn(5, 10) * 1000),\n",
    "    (\"Extreme Values\", np.random.randn(5, 10) * 1e8),\n",
    "    (\"Mixed with NaN\", np.random.randn(5, 10)),\n",
    "    (\"All Zeros\", np.zeros((5, 10))),\n",
    "    (\"All Ones\", np.ones((5, 10)) * 1e6)\n",
    "]\n",
    "\n",
    "# Inject NaN in mixed scenario\n",
    "stability_scenarios[3] = (\"Mixed with NaN\", np.random.randn(5, 10))\n",
    "scenario_data = stability_scenarios[3][1].copy()\n",
    "scenario_data[0, 0] = np.nan\n",
    "stability_scenarios[3] = (\"Mixed with NaN\", scenario_data)\n",
    "\n",
    "# Test each scenario\n",
    "healthy_network = demo_networks['healthy']\n",
    "\n",
    "for scenario_name, test_data in stability_scenarios:\n",
    "    print(f\"\\n📊 {scenario_name}:\")\n",
    "    \n",
    "    try:\n",
    "        input_tensor = Tensor(test_data)\n",
    "        input_check = monitor.check_tensor_stability(input_tensor, scenario_name)\n",
    "        \n",
    "        print(f\"   Input Status: {'✅ STABLE' if input_check['is_stable'] else '❌ UNSTABLE'}\")\n",
    "        if input_check['issues']:\n",
    "            print(f\"   Issues: {', '.join(input_check['issues'])}\")\n",
    "        \n",
    "        # Try network forward pass\n",
    "        try:\n",
    "            output = healthy_network(input_tensor)\n",
    "            output_check = monitor.check_tensor_stability(output, f\"{scenario_name}_output\")\n",
    "            print(f\"   Output Status: {'✅ STABLE' if output_check['is_stable'] else '❌ UNSTABLE'}\")\n",
    "            if output_check['issues']:\n",
    "                print(f\"   Output Issues: {', '.join(output_check['issues'])}\")\n",
    "        \n",
    "        except Exception as e:\n",
    "            print(f\"   ❌ Forward pass failed: {str(e)}\")\n",
    "    \n",
    "    except Exception as e:\n",
    "        print(f\"   ❌ Could not create tensor: {str(e)}\")\n",
    "\n",
    "print(f\"\\n🎯 PRODUCTION STABILITY LESSONS:\")\n",
    "print(f\"=\" * 40)\n",
    "\n",
    "print(f\"\\n1. 🛡️ INPUT VALIDATION:\")\n",
    "print(f\"   - Always validate input data before processing\")\n",
    "print(f\"   - Clip extreme values to reasonable ranges\")\n",
    "print(f\"   - Check for NaN/inf values in data pipelines\")\n",
    "\n",
    "print(f\"\\n2. 🔧 MONITORING STRATEGY:\")\n",
    "print(f\"   - Monitor weight magnitudes during training\")\n",
    "print(f\"   - Track gradient norms to detect vanishing/exploding\")\n",
    "print(f\"   - Log activation statistics to catch distribution shift\")\n",
    "\n",
    "print(f\"\\n3. 🚨 EARLY WARNING SYSTEM:\")\n",
    "print(f\"   - Set thresholds for weight magnitudes\")\n",
    "print(f\"   - Alert when gradients become too large/small\")\n",
    "print(f\"   - Automatically stop training on stability issues\")\n",
    "\n",
    "print(f\"\\n4. 🛠️ PREVENTIVE MEASURES:\")\n",
    "print(f\"   - Proper weight initialization (Xavier/He)\")\n",
    "print(f\"   - Gradient clipping for exploding gradients\")\n",
    "print(f\"   - Batch normalization for internal stability\")\n",
    "print(f\"   - Learning rate scheduling to prevent instability\")\n",
    "\n",
    "print(f\"\\n💡 SYSTEMS ENGINEERING INSIGHT:\")\n",
    "print(f\"Stability monitoring is like production health checks:\")\n",
    "print(f\"- Prevent silent failures that waste compute resources\")\n",
    "print(f\"- Enable automatic recovery strategies (restart training)\")\n",
    "print(f\"- Provide debugging information for model developers\")\n",
    "print(f\"- Critical for unattended training jobs in production\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    # Run all tests\n",
    "    test_unit_network_architectures()\n",
    "    test_unit_sequential_networks()\n",
    "    test_unit_mlp_creation()\n",
    "    test_unit_network_applications()\n",
    "    test_module_full_network_forward_pass()\n",
    "    \n",
    "    print(\"All tests passed!\")\n",
    "    print(\"dense_dev module complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b185eb71",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🤔 ML Systems Thinking: Interactive Questions\n",
    "\n",
    "Now that you've built complete neural network architectures, let's connect this foundational work to broader ML systems challenges. These questions help you think critically about how network composition patterns scale to production ML environments.\n",
    "\n",
    "Take time to reflect thoughtfully on each question - your insights will help you understand how the network concepts you've implemented connect to real-world ML systems engineering."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e07096d",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### Question 1: Composition Patterns and Architectural Design\n",
    "\n",
    "**Context**: Your sequential network implementation enables flexible composition of layers into complex architectures. Production ML systems must support diverse architectural patterns: from simple MLPs to complex models with branching, skip connections, and dynamic computation graphs.\n",
    "\n",
    "**Reflection Question**: Design a network composition system that supports both sequential and complex architectural patterns for production ML systems. How would you extend your sequential approach to handle branching networks, residual connections, and dynamic routing? Consider scenarios where model architectures need to adapt during training or inference based on input characteristics or computational constraints.\n",
    "\n",
    "Think about: architectural flexibility, dynamic graph construction, branching and merging patterns, and computational graph optimization opportunities.\n",
    "\n",
    "*Target length: 150-300 words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "888d27f4",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "question-1-composition-patterns",
     "locked": false,
     "points": 10,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "YOUR REFLECTION ON COMPOSITION PATTERNS AND ARCHITECTURAL DESIGN:\n",
    "\n",
    "TODO: Replace this text with your thoughtful response about network composition system design.\n",
    "\n",
    "Consider addressing:\n",
    "- How would you extend sequential composition to support complex architectural patterns?\n",
    "- What strategies would you use to handle branching, merging, and skip connections?\n",
    "- How would you implement dynamic network architectures that adapt during execution?\n",
    "- What role would computational graph optimization play in your design?\n",
    "- How would you balance architectural flexibility with performance optimization?\n",
    "\n",
    "Write an architectural analysis connecting your sequential networks to real composition pattern challenges.\n",
    "\n",
    "GRADING RUBRIC (Instructor Use):\n",
    "- Demonstrates understanding of advanced architectural composition patterns (3 points)\n",
    "- Addresses dynamic and complex network structure challenges (3 points)\n",
    "- Shows practical knowledge of graph optimization techniques (2 points)\n",
    "- Demonstrates systems thinking about architectural flexibility vs performance (2 points)\n",
    "- Clear architectural reasoning and practical considerations (bonus points for innovative approaches)\n",
    "\"\"\"\n",
    "\n",
    "### BEGIN SOLUTION\n",
    "# Student response area - instructor will replace this section during grading setup\n",
    "# This is a manually graded question requiring architectural analysis of network composition\n",
    "# Students should demonstrate understanding of complex architectural patterns and optimization\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9de602d0",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### Question 2: Modularity and Distributed Training\n",
    "\n",
    "**Context**: Your network architecture separates layer composition from individual layer implementation. Production ML systems must scale these architectures across distributed training environments while maintaining modularity and enabling efficient model parallelism.\n",
    "\n",
    "**Reflection Question**: Architect a modular network system that enables efficient distributed training across multiple devices and nodes. How would you design network decomposition strategies that balance computation across devices, implement communication-efficient model parallelism, and maintain modularity for different deployment scenarios? Consider challenges where network parts need to run on different hardware with varying computational capabilities.\n",
    "\n",
    "Think about: model parallelism strategies, communication optimization, device placement algorithms, and modular deployment patterns.\n",
    "\n",
    "*Target length: 150-300 words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf9ef3ba",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "question-2-modularity-distributed",
     "locked": false,
     "points": 10,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "YOUR REFLECTION ON MODULARITY AND DISTRIBUTED TRAINING:\n",
    "\n",
    "TODO: Replace this text with your thoughtful response about modular distributed training system design.\n",
    "\n",
    "Consider addressing:\n",
    "- How would you design network decomposition for efficient distributed training?\n",
    "- What strategies would you use to balance computation and communication across devices?\n",
    "- How would you implement model parallelism while maintaining modularity?\n",
    "- What role would device placement optimization play in your system?\n",
    "- How would you handle heterogeneous hardware in distributed training scenarios?\n",
    "\n",
    "Write a systems analysis connecting your modular networks to real distributed training challenges.\n",
    "\n",
    "GRADING RUBRIC (Instructor Use):\n",
    "- Shows understanding of distributed training and model parallelism challenges (3 points)\n",
    "- Designs practical approaches to modular distributed architectures (3 points)\n",
    "- Addresses communication optimization and device placement (2 points)\n",
    "- Demonstrates systems thinking about scalability and modularity trade-offs (2 points)\n",
    "- Clear systems reasoning with distributed computing insights (bonus points for comprehensive understanding)\n",
    "\"\"\"\n",
    "\n",
    "### BEGIN SOLUTION\n",
    "# Student response area - instructor will replace this section during grading setup\n",
    "# This is a manually graded question requiring understanding of distributed training architecture\n",
    "# Students should demonstrate knowledge of model parallelism and communication optimization\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d28ded",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "### Question 3: Architecture Design and Performance Optimization\n",
    "\n",
    "**Context**: Your network implementations provide a foundation for various ML applications from classification to regression. Production ML systems must optimize network architectures for specific deployment constraints: inference latency, memory usage, energy consumption, and accuracy requirements.\n",
    "\n",
    "**Reflection Question**: Design an architecture optimization system that automatically configures network structures for specific deployment targets and performance constraints. How would you implement neural architecture search for production environments, balance architecture complexity with inference requirements, and optimize networks for edge deployment with strict resource constraints? Consider scenarios where the same model needs to perform well across mobile devices, cloud servers, and embedded systems.\n",
    "\n",
    "Think about: neural architecture search, performance profiling, resource-constrained optimization, and multi-target deployment strategies.\n",
    "\n",
    "*Target length: 150-300 words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aebdd8c",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "question-3-architecture-optimization",
     "locked": false,
     "points": 10,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "YOUR REFLECTION ON ARCHITECTURE DESIGN AND PERFORMANCE OPTIMIZATION:\n",
    "\n",
    "TODO: Replace this text with your thoughtful response about architecture optimization system design.\n",
    "\n",
    "Consider addressing:\n",
    "- How would you implement automated architecture optimization for different deployment targets?\n",
    "- What strategies would you use to balance architecture complexity with performance constraints?\n",
    "- How would you optimize networks for resource-constrained edge deployment?\n",
    "- What role would neural architecture search play in your optimization system?\n",
    "- How would you handle multi-target deployment with varying resource constraints?\n",
    "\n",
    "Write an optimization analysis connecting your network architectures to real deployment optimization challenges.\n",
    "\n",
    "GRADING RUBRIC (Instructor Use):\n",
    "- Understands architecture optimization and deployment constraint challenges (3 points)\n",
    "- Designs practical approaches to automated architecture optimization (3 points)\n",
    "- Addresses resource constraints and multi-target deployment (2 points)\n",
    "- Shows systems thinking about performance vs complexity trade-offs (2 points)\n",
    "- Clear optimization reasoning with deployment insights (bonus points for deep understanding)\n",
    "\"\"\"\n",
    "\n",
    "### BEGIN SOLUTION\n",
    "# Student response area - instructor will replace this section during grading setup\n",
    "# This is a manually graded question requiring understanding of architecture optimization and deployment\n",
    "# Students should demonstrate knowledge of neural architecture search and resource optimization\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3199d6e",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "## 🎯 MODULE SUMMARY: Neural Network Architectures\n",
    "\n",
    "Congratulations! You've successfully implemented complete neural network architectures:\n",
    "\n",
    "### What You've Accomplished\n",
    "✅ **Sequential Networks**: Chained layers for complex transformations\n",
    "✅ **MLP Creation**: Multi-layer perceptrons with flexible architectures\n",
    "✅ **Network Architectures**: Different activation patterns and output types\n",
    "✅ **Integration**: Real-world applications like classification and regression\n",
    "\n",
    "### Key Concepts You've Learned\n",
    "- **Sequential Processing**: How layers chain together for complex functions\n",
    "- **MLP Design**: Multi-layer perceptrons as universal function approximators  \n",
    "- **Architecture Choices**: How depth, width, and activations affect learning\n",
    "- **Real Applications**: Classification, regression, and feature extraction\n",
    "\n",
    "### Next Steps\n",
    "1. **Export your code**: `tito package nbdev --export 04_networks`\n",
    "2. **Test your implementation**: `tito test 04_networks`\n",
    "3. **Build complete models**: Combine with training for full ML pipelines\n",
    "4. **Move to Module 5**: Add convolutional layers for image processing!\n",
    "\n",
    "**Ready for CNNs?** Your network foundations are now ready for specialized architectures!"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
